Extract cameras from E

Reconstruct model by intersection

Refine with bundle adjustment

Separate exterior orientation of two images

Solve two resections, with GCP

Combined, single stage orientation (bundle block adjustment with block=2)

Use GCP, but can work without (arbitrarily fix the gauge freedom)

Two-steps combined orientation

relative orientation

absolute orientation

Use GCP but can work without

Stereo-models are correct up to a similarity

Separate exterior orientation of two images

Combined single stage orientation

Relative + absolute orientation

Global modeling

Merge of partial models

Camera (frame) space

Points space

Global motion-first

Independent models

block adjustment

Resection/intersection

Hierachical variation

Multiple views (block)

Use points

and

cameras

and all of them simultaneously

Alignment via points correspondences

Points are used to compute orientations:

absolute in IBA

exterior in resection/intersection

both in the hierarchical method

Only some points/image are used at intermediate steps. The final model results from a merge.

Use two-view tools iteratively.

Alignment of camera frames (w/o points)

Intersection only after cameras are aligned

This is partially global: the first stage is local (many relative orientations, but the alignment of camera frames is global (w/o points though).

Computer Vision

Photogrammetry

Two views (stereo)

Epipolar geometry

**Let's hybridize**

This was my home page

And this was the state of the art in modeling from images (CVPR'97)

I was challenged by my PhD supervisor with the problem of reconstructing an object's geometry from casual, uncalibrated images.

Of Course, I did not succeeded, but (as a partial apology):

"It is good to have an end to journey toward; but

it is the journey that matters

, in the end." [E. Hemingway]

Andrea Fusiello

DIEG - University of Udine

It's image-based modelling

It's structure and/from motion

It's shape reconstruction

It's stereo/block processing

It's block adjustment

It's aerial/photogrammetric triangulation

Structure from motion: state of the art, examples and open issues - the Computer Vision perspective

Piazza Erbe, Udine

1996

1999

2013

Back in 1996...

Nowadays

What was a challenge in 1997 has become a commodity.

There are now software applications that solves the problem off-the-shelf.

**Let's agree on names**

**Start**

**Let's put it in perspective**

**Some background**

**A possible taxonomy**

References

Specific CV references are in the relevant slides. These are general refs on the CV-PH connection

R. I. Hartley and J. L. Mundy, "

The relationship between photogrammetry and computer vision

," SPIE93-photogrammetry, 1993, pp. 92--105.

Förstner. "

Computer Vision and Photogrammetry Mutual Questions: Geometry, Statistics and Cognition.

" Int. Symp. Photogrammetry meets Geoinformatics, 2002

Förstner.

Computer Vision and Remote Sensing – Lessons Learned

. Photogrammetrische Woche 2009

Remondino, Fraser.

Digital camera calibration methods: Considerations and comparisons

. ISPRS Comm. V Symp. 2006.

Credits

Fabio Crosilla for his guidance into the Photogrammetry world

Fabio Remondino for his suggestions that helped to improve this presentations

Roberto Toldo is the main programmer of "Samantha" and produced the results that have been shown

Seitz, Dyer. Photorealistic Scene Reconstruction by Voxel Coloring

Pollefeys, Van Gool. A Stratified Approach to Metric Self-Calibration

Stein, Shashua. Model-based Brightness Constraints: on Direct Estimation of Structure and Motion

Bundle Block

Adjustment

Multifocal constraints

Structure and motion

Compute 3D points (structure) and camera exterior parameters (motion) from image point correspondences (tie-points)

Factorization

The gold-standard method is bundle adjustment, used in CV to refine solutions provided by other methods.

Focusing on automation, BA is not seen in CV as a stand-alone method.

CV methods concentrate on a free-network solution, as GCP are considered an optional piece of information. If GCPs are available, solve an absolute orientation in the end.

CV methods are devised for irregular, unknown blocks;

a lot of effort is put on recovering the

epipolar graph

(= block structure)

Sequential SfM (Bundler) and its hierarchical variation (Samantha) proved the most effective in practical applications. They are cousins of IMBA.

Global motion-first methods are very promising (use frames vs points).

For each pair of overlapping photographs a stereo-model is built (by relative orientation)

Then, all these independent models are simultaneously transformed into a common reference frame by similarity transformations.

The common reference system is usually the ground ref. system, but if no GCP are available it can be any model coordinates system.

**Closure**

Comments

CV tools: a quick overview

All points must be seen in all views (in principle)

Due to the low-rank constraint, matrix completion can be used

Projective reconstruction (Euclidean upgrade is required)

Initial guess needed

Refs:

Moulon, Monasse, Marlet. Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. ICCV 2013

Robust global motion estimation with matrix completion - F. Arrigoni, B. Rossi, F. Malapelle, P. Fragneto and A. Fusiello.

WG1 poster

(Tue PM)

Equivalent to collinearity equations

Like in the case of the bifocal tensor (E), camera matrices can be extracted from the trifocal and quadrifocal tensors.

This can be seen as generalization to 3 and 4 views of the relative orientation

The first step step of the global-motion first methods is linked to the global registration of multiple 3D point sets in

frame space

The second step of the IMBA is linked to the global registration of multiple 3D point sets in

point space

note: rigid vs similarity

Global registration

of multiple 3D point sets

Refs:

Sharp, Lee, Wehe, 2001. Toward multiview registration in frame space

Benjemaa, Schmit, 1998. A solution for the registration of multiple 3D point sets using unit quaternions

Beinat, A., Crosilla, F., 2001. Generalized procrustes analysis for size and shape 3d object reconstruction

Refs:

R. Gherardi, M. Farenzena, A. Fusiello. Improving the Efficiency of Hierarchical Structure-and-Motion. CVPR 2010 (Samantha)

Ref:

Snavely, Seitz, Szeliski, 2008. Modeling the world from internet photo collections (Bundler)

Ref:

Bill Triggs, Philip F. McLauchlan, Richard I. Hartley, and Andrew W. Fitzgibbon. Bundle adjustment - a modern synthesis. International Workshop on Vision Algorithms, 2000.

[Diagram from Triggs et al]

It's SLAM

On the two-views methods there is a good agreement.

Relative orientation is central, even if the Essential matrix formulation is peculiar of CV, it is equivalent to the coplanarity constraint

Exterior orientation (resection) is not used in the context of two-views processing, but it is known in CV (also as PnP or :camera pose")

BA is common background. In CV used as a refinement step (discussed ahead)

Photographs courtesy of F. Remondino

Results obtained with 3Dflow's

Zephyr

software (

www.3dflow.net

)

Demo available at the exposition booth

Results obtained with "Samantha" (code available from

samantha.3dflow.net

)

Comments

Summary

Refs:

P. Sturm and B. Triggs. A factorization based algorithm for multi-image projective structure and motion. EECV 1996

John Oliensis and Richard Hartley. Iterative extensions of the sturm/triggs algorithm: Convergence and nonconvergence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(12), 2007.

A relative orientation variant

This presentation on Prezi web site:

https://prezi.com/t4kibkvjekh6/

A unifying note: The Gauss-Helmert method can be seen as a Gauss-Newton iteration with a special approximation of the Hessian [Kanatani, Niitsuma 2011]

Modulo different notation, in the end there is agreement on the problems to be solved, usually non-linear LS.

Also the solution methods are largely overlapping, after a closer look

Thanks to the projective geometry approach, CV also developed fast, linear sub-optimal methods (DLT-like) that can be used for initialization.

Assessment of accuracy of results, sensitivity/reliability analysis is less a concern in CV

Ref: Börlin, Grussenmeyer. BUNDLE ADJUSTMENT WITH AND WITHOUT DAMPING. The Photogrammetric Record 28(144): 2013

The numerical implementations of BA can differ, but all of them (from CV and PH) stem from the Gauss-Newton method.

Indeed, if the cost function is weighted by the true measurement covariances, there is no difference between the Gauss-Newton method and the so-called Gauss-Markov adjustment (from PH).

L-M (from CV) is basically the Gauss-Newton method, to which the gradient descent principle is combined to improve convergence.

This can be seen as an instance of a more general class of damped Gauss-Newton methods (see ref.)

Trifocal constraint

To travel is to discover that everyone is wrong about other countries

[W. Förstner citing A. Huxley]