图像理解样式编辑母版文本样式

1、Image StitchingGap Closing Estimate a series of rotation matrices and focal lengths to create large panoramas Unfortunately, accumulated errors Rarely produce a closed 360 pararomaRichard SzeliskiImage Stitching2 Wrong focal length: f=510; correct focal length f=468Gap Closing Estimate a series of r

2、otation matrices and focal lengths to create large panoramas Unfortunately, accumulated errors Hard to estimate accurate focal length Rarely produce a closed 360 pararomaRichard SzeliskiImage Stitching3Gap Closing Matching the first image and the last one Compute the gap angle Distribute the gap ang

3、le evenly across the whole sequence Modify rotations by Update focal length Only works for 1D panorama where the camera is continuously turning in the same direction Richard SzeliskiImage Stitching4g/gimageN(1/360 )gffRichard SzeliskiImage Stitching5Assembling the panorama Stitch pairs together, ble

4、nd, then cropRichard SzeliskiImage Stitching6Problem: Drift Error accumulation small (vertical) errors accumulate over time apply correction so that sum = 0 (for 360pan.)Richard SzeliskiImage Stitching7Problem: Drift Solution add another copy of first image at the end this gives a constraint: yn = y

5、1 there are a bunch of ways to solve this problem add displacement of (y1 yn)/(n -1) to each image after the first compute a global warp: y = y + ax run a big optimization problem, incorporating this constraint best solution, but more complicated known as “bundle adjustment”(x1,y1)copy of first imag

6、e(xn,yn)Richard SzeliskiImage Stitching8Global alignment Pairwise matching global energy involves all of the per-image pose parameters Local adjustments to reduce blurring due to local mis-registration Parallax removal Bundle Adjustment Developed in photogrammetry in 50s , for general structure from

7、 motion (Szeliski and Kang 1994) Pairwise alignmentix ixix)h(px,Bundle Adjustment N features, appears in M images3D Point2D Point on image I_jCamera jBundle Adjustment-1 Given initial set of Rj, Kj, refine them together Disadvantages xik depends on xij, error-in-variable Overweighted for feature obs

8、erved many times Derivatives of xik wrt Rj, fj are cumbersome Observed location of feature i in image k Predicted location of feature i in image k ikx),(ijfRBundle Adjustment-2 True BA: estimate camera pose and 3d points Disadvantages A lot of variables to solve Slow convergence ix),(ijfRixCamera j

9、),(ijfRijx ijxlinearized GaussNewton step can be reduced using sparse matrix techniques Bundle Adjustment-3 Minimize the error in 3D projected ray directions Shum & Szeliski 2000ixCamera j ),(ijfRij-1j-1jixKRxijijjijiijDBAfc23|),;(|ExRxxixijx Bundle Adjustment-4 Minimize a pairwise energy in 3D

10、projected ray directions Shum & Szeliski 2000ijijjijiijDBAfc23|),;(|ExRxxCamera j ),(ijfRixijx ijkkikijjijiikijDffcck23pairsall|),;(),;(|ERxxRxxCamera k ),(kkfRBundle Adjustment Developed in photogrammetry in 50s Non-linear least squares Sparse matrix techniques Levenberg-Marquardt Gradient desc

11、ent Move fast to the minimal when far from it Move slowly to the minimal when close from it Gauss-Newton Move fast to the minimal when close to it Sensitive to the starting point Convergence is not guaranteed Levenberg-MarquardtNon-Linear Least Squares Problem: Minimize Gradient descent )(1Stttt)(2)

12、(fyJTtSJacobian Matrix Gradient NxM)(tttfyResidual error xfJ),(Non-Linear Least Squares Problem: Minimize Solution: Taylor Expansion Minimize Iteratively Minimize tttJ Non-Linear Least Squares Gauss-Newton Minimize Taking the derivative with respect to Setting the result to zero 0JfyJ)(2T)(TTfyJJJLe

13、venberg-Marquardt Levenberg: damped version is adjusted at each iteration If reduction of is rapid, a smaller value can be used, bringing the algorithm closer to the GaussNewton algorithm if an iteration gives insufficient reduction in the residual, can be increased, giving a step closer to the grad

14、ient descent direction )(TTfyJJJ)(2fyJTtLevenberg-Marquardt Levenberg: When is large, inverting JJ+I is not used at all Marguardt: Scale each component of the gradient according to the curvature so that there is larger movement along the directions where the gradient is smaller Avoid slow convergenc

15、e in the direction of small gradient 22Non-linear least squares Linear approximation of residual allows quadratic approximation of sum- of-squaresJ0e0T1 -T0TTJJJ0J2J2Jee JJ00eeTJ)diag(JJJNTTN(extra term = descent term)Minimization corresponds to finding zeros of derivativeLevenberg-Marquardt: extra

16、term to deal with singular N(decrease/increase l if success/failure to descent)23Bundle AdjustmentU1U2U3WTWVP1P2P3M Jacobian of has sparse block structure cameras independent of other cameras, points independent of other pointsJJJNT12xm3xn(in general much larger)im.pts. view 1minjjiijD112MP,mNeeded

17、for non-linear minimizationK CamerasM 3D Points24Bundle AdjustmentEliminate dependence of camera/motion parameters on structure parametersNote in general 3n 11mWTVU-WV-1WTNI0WVI111xm3xnAllows much more efficient computations e.g. 100 views,10000 points, solve 1000 x1000, not 30000 x30000Often still

18、band diagonaluse sparse linear algebra algorithms Up Vector Selection Keep the upright direction at the final stitched image Multiply each rotation by a global rotation RgUp Vector Selection rg1 is the smallest eigenvector of the scatter or moment matrix spanned by the individual camera rotation x-v

19、ectors Full rotation matrix RgBundle Adjustment New images initialised with rotation, focal length of best matching imageBundle Adjustment New images initialised with rotation, focal length of best matching imageParallax Removal Blurry or ghosting Unmodeled radial distortion 3D parallax: failure to

20、rotate the camera around its optical center Small scene motion, large scale scene motion Different approaches for themParallax Removal Full-3D BA compute 3D point location reproject it to images Interpolate a dense correction field u(x)Deghosting a mosaic with motion parallax (Shum and Szeliski 2000

21、) c 2000 IEEE: (a) composite with parallax; (b) after a single deghosting step (patch size 32); (c) after multiple steps (sizes 32, 16 and 8)Parallax Removal By Image blending Seamless image composition Content-preserving image warping Example: Recognising PanoramasM. Brown and D. Lowe, University o

22、f British ColumbiaWhy “Recognising Panoramas”?Why “Recognising Panoramas”? 1D Rotations (q) Ordering matching imagesWhy “Recognising Panoramas”? 1D Rotations (q) Ordering matching imagesWhy “Recognising Panoramas”? 1D Rotations (q) Ordering matching imagesWhy “Recognising Panoramas”? 2D Rotations (q

23、, f) Ordering matching images 1D Rotations (q) Ordering matching imagesWhy “Recognising Panoramas”? 1D Rotations (q) Ordering matching images 2D Rotations (q, f) Ordering matching imagesWhy “Recognising Panoramas”? 1D Rotations (q) Ordering matching images 2D Rotations (q, f) Ordering matching image

24、sWhy “Recognising Panoramas”?Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results ConclusionsRANSAC for HomographyRANSAC for HomographyRANSAC for HomographyFinding the panoramasFinding the panoramasFinding the panoramasFinding the panoramasMulti-band Blending Burt & Adelson 1983 Blend frequency bands over range lResultsRichard SzeliskiImage Stitching51Get you own free copy Richard SzeliskiImage