Xử lý ảnh trong cơ điện tử machine vision chapter 9 camera calibration and 3d reconstruction

Camera calibration ❖ Pinhole Camera Model ▪ The calibration algorithm calculates the camera matrix using ➢ The intrinsic parameters ➢ The extrinsic parameters Scale factor Image points I

XỬ LÝ ẢNH TRONG CƠ ĐIỆN TỬ

Machine Vision

1

TRƯỜNG ĐẠI HỌC BÁCH KHOA HÀ NỘI

Giảng viên: TS Nguyễn Thành Hùng Đơn vị: Bộ môn Cơ điện tử, Viện Cơ khí

Hà Nội, 2021

Camera calibration

❖ Geometric camera calibration, also referred to as camera resectioning, estimates the

parameters of a lens and image sensor of an image or video camera.

❖ You can use these parameters to correct for lens distortion, measure the size of an

object in world units, or determine the location of the camera in the scene.

❖ These tasks are used in applications such as machine vision to detect and measure

objects They are also used in robotics, for navigation systems, and 3-D scene reconstruction.

Camera calibration

Camera calibration

❖ Camera parameters include intrinsics, extrinsics, and distortion coefficients.

❖ To estimate the camera parameters, you need to have 3-D world points and their

corresponding 2-D image points.

❖ You can get these correspondences using multiple images of a calibration pattern,

such as a checkerboard Using the correspondences, you can solve for the camera parameters.

Camera calibration

❖ After you calibrate a camera, to evaluate the accuracy of the estimated parameters,

you can:

➢ Plot the relative locations of the camera and the calibration pattern

➢ Calculate the reprojection errors

➢ Calculate the parameter estimation errors.

Camera calibration

❖ Pinhole Camera Model

➢ A pinhole camera is a simple camera without a lens and with a single small aperture

Camera calibration

❖ Pinhole Camera Model

Camera calibration

❖ Pinhole Camera Model

▪ The calibration algorithm calculates the camera matrix using

➢ The intrinsic parameters

➢ The extrinsic parameters

Scale factor Image points Intrinsic

matrix

Extrinsics Rotation and translation

World points

Camera calibration

❖ Pinhole Camera Model

➢ The world points are transformed to camera coordinates using the extrinsics parameters The

camera coordinates are mapped into the image plane using the intrinsics parameters

Camera calibration

❖ Camera Calibration Parameters

Camera calibration

❖ Extrinsic Parameters

➢ The extrinsic parameters consist of a rotation, R, and a translation, t The origin of the

camera’s coordinate system is at its optical center and its x- and y-axis define the image plane.

Camera calibration

❖ Intrinsic Parameters

➢ The intrinsic parameters include the focal length, the optical center, also known as the

principal point, and the skew coefficient The camera intrinsic matrix, K, is defined as:

➢ The pixel skew is defined as:

Camera calibration

❖ Intrinsic Parameters

Camera calibration

❖ Distortion in Camera Calibration

➢ The camera matrix does not account for lens distortion because an ideal pinhole camera does

not have a lens To accurately represent a real camera, the camera model includes the radialand tangential lens distortion

Camera calibration

❖ Radial Distortion

➢ Radial distortion occurs when light rays bend more near the edges of a lens than they do at its

optical center The smaller the lens, the greater the distortion

Camera calibration

❖ Radial Distortion

➢ The radial distortion coefficients model this type of distortion The distorted points are

denoted as (xdistorted, ydistorted):

•x, y — Undistorted pixel locations x and y are in normalized image coordinates Normalized

image coordinates are calculated from pixel coordinates by translating to the optical center and

dividing by the focal length in pixels Thus, x and y are dimensionless.

•k1, k2, and k3 — Radial distortion coefficients of the lens

•r2: x2 + y2

Camera calibration

❖ Tangential Distortion

➢ Tangential distortion occurs when the lens and the image plane are not parallel The

tangential distortion coefficients model this type of distortion

Camera calibration

❖ Tangential Distortion

➢ The distorted points are denoted as (xdistorted, ydistorted):

•x, y — Undistorted pixel locations x and y are in normalized image coordinates Normalized

image coordinates are calculated from pixel coordinates by translating to the optical center and

dividing by the focal length in pixels Thus, x and y are dimensionless.

•p1 and p2 — Tangential distortion coefficients of the lens

•r2: x2 + y2

Camera calibration

❖ Camera model Camera coordinate system

Image plane

Camera Calibration with OpenCV

❖ Function calibrateCamera use the above model to do the following:

➢ Project 3D points to the image plane given intrinsic and extrinsic parameters

➢ Compute extrinsic parameters given intrinsic parameters, a few 3D points, and their

projections

➢ Estimate intrinsic and extrinsic camera parameters from several views of a known calibration

pattern (every view is described by several 3D-2D point correspondences)

➢ Estimate the relative position and orientation of the stereo camera “heads” and compute the

rectification transformation that makes the camera optical axes parallel

Robot Camera Calibration

Camera fixed to an independent structure Camera mounted on the robot

Ai: transformation from Base → Gripper

Bi: transformation from Cam → Target

X: transformation from Gripper → Cam???

Ai: transformation from Base → Gripper

Bi: transformation from Target → Cam

X: transformation from Gripper → Target???

Ai: transformation from Base → Gripper

Bi: transformation from Target → Cam

X: transformation from Gripper → Target???

Ai: transformation from Base → Gripper

Bi: transformation from Cam → Target

X: transformation from Gripper → Cam???

Camera Pose Estimation

❖ 3D pose estimation is the problem of determining the transformation of an object in

a 2D image which gives the 3D object.

❖ 3D pose estimation from a calibrated 2D camera

➢ Given a 2D image of an object, and the camera that is calibrated with respect to a world

coordinate system, it is also possible to find the pose which gives the 3D object in its objectcoordinate system

➢ Starting with a 2D image, image points are extracted which correspond to corners in an

image The projection rays from the image points are reconstructed from the 2D points sothat the 3D points, which must be incident with the reconstructed rays, can be determined

Camera Pose Estimation

❖ Algorithm

➢ The algorithm for determining pose estimation is based on the Iterative Closest Point

algorithm The main idea is to determine the correspondences between 2D image features andpoints on the 3D model curve

(a) Reconstruct projection rays from the image points

(b) Estimate the nearest point of each projection ray to a point on the 3D contour

(c) Estimate the pose of the contour with the use of this correspondence set

(d) goto (b)

Camera Pose Estimation with OpenCV

Stereo Vision

❖ What is stereo correspondence?

➢ only have 2D information when capture images

➢ the depth information is lost

➢ our brain takes two images and builds a 3D map using stereo vision

➢ capture two photos of the same scene using different viewpoints, and then match the

corresponding points to obtain the depth map of the scene → stereo vision algorithms

Stereo Vision

❖ What is stereo correspondence?

➢ The absolute difference between d1 and d2 is greater than the absolute difference between d3

and d4

➢ The camera moved by the same amount, there is a big difference between the apparent

distances between the initial and fnal positions → This happens because we can bring theobject closer to the camera; the apparent movement decreases when you capture two imagesfrom different angles

➢ This is the concept behind stereo correspondence: we capture two images and use this

knowledge to extract the depth information from a given scene

Stereo Vision

❖ What is epipolar geometry?

Stereo Vision

❖ What is epipolar geometry?

➢ Our goal is to match the keypoints in these two images to extract the scene information

➢ The way we do this is by extracting a matrix that can associate the corresponding points

between two stereo images → the fundamental matrix.

➢ The point at which the epipolar lines converge is called epipole

➢ If you match the keypoints using SIFT, and draw the lines towards the meeting point on the

left and right images, they will look like this:

Stereo Vision

❖ What is epipolar geometry?

Stereo Vision

❖ What is epipolar geometry?

➢ If two frames are positioned in 3D, then each epipolar line between the two frames must

intersect the corresponding feature in each frame and each of the camera origins

➢ This can be used to estimate the pose of the cameras with respect to the 3D environment

➢ We will use this information later on, to extract 3D information from the scene

Stereo Vision

❖ Building the 3D map

Stereo Vision

❖ Building the 3D map

➢ The frst step is to extract the disparity map between the two images

➢ Once we fnd the matching points between the two images, we can fnd the disparity by using

epipolar lines to impose epipolar constraints

Stereo Vision

❖ Building the 3D map

➢ x and x′ are the distance between points in image plane corresponding to the scene point 3D

and their camera center

➢ B is the distance between two cameras (which we know) and f is the focal length of camera

(already known)

➢ So in short, above equation says that the depth of a point in a scene is inversely proportional

to the difference in distance of corresponding image points and their camera centers

➢ So with this information, we can derive the depth of all pixels in an image

Stereo Vision

❖ Building the 3D map