OpenCV.js Example

# Camera Calibration Source: https://opencv-opencv.mintlify.app/api/calib3d/calibration Functions for calibrating monocular cameras, detecting calibration patterns, and estimating camera intrinsic parameters ## Overview Camera calibration estimates intrinsic parameters (focal length, principal point, distortion coefficients) from multiple views of a calibration pattern. OpenCV supports chessboard and circular grid patterns. ## Core Functions ### calibrateCamera Finds the camera intrinsic and extrinsic parameters from several views of a calibration pattern. ```cpp theme={null} double cv::calibrateCamera( InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints, Size imageSize, InputOutputArray cameraMatrix, InputOutputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, int flags = 0, TermCriteria criteria = TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 30, DBL_EPSILON) ) ``` Vector of vectors of calibration pattern points in the calibration pattern coordinate space (e.g. `std::vector>`). The outer vector contains as many elements as pattern views. For planar patterns, Z-coordinate is 0. Vector of vectors of the projections of calibration pattern points (e.g. `std::vector>`). Must match the size of objectPoints. Size of the image used only to initialize the camera intrinsic matrix. Input/output 3x3 floating-point camera intrinsic matrix. If `CALIB_USE_INTRINSIC_GUESS` is specified, some or all of fx, fy, cx, cy must be initialized before calling. Input/output vector of distortion coefficients `(k1, k2, p1, p2[, k3[, k4, k5, k6[, s1, s2, s3, s4[, τx, τy]]]])`. Output vector of rotation vectors (Rodrigues) estimated for each pattern view. Each rotation vector brings the calibration pattern from object coordinate space to camera coordinate space. Output vector of translation vectors estimated for each pattern view. Different flags for calibration behavior (see Calibration Flags below). Termination criteria for the iterative optimization algorithm. **Returns:** The overall RMS re-projection error. The algorithm is based on Zhang2000. It performs: 1. Compute initial intrinsic parameters (for planar patterns) or read from input 2. Estimate initial camera pose using solvePnP 3. Run global Levenberg-Marquardt optimization to minimize reprojection error ### Extended Version ```cpp theme={null} double cv::calibrateCamera( InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints, Size imageSize, InputOutputArray cameraMatrix, InputOutputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, OutputArray stdDeviationsIntrinsics, OutputArray stdDeviationsExtrinsics, OutputArray perViewErrors, int flags = 0, TermCriteria criteria = TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 30, DBL_EPSILON) ) ``` Extended version provides additional outputs: Output vector of standard deviations estimated for intrinsic parameters. Order: `(fx, fy, cx, cy, k1, k2, p1, p2, k3, k4, k5, k6, s1, s2, s3, s4, τx, τy)` Output vector of standard deviations estimated for extrinsic parameters. Order: `(R0, T0, ..., R_{M-1}, T_{M-1})` where M is the number of pattern views. Output vector of the RMS re-projection error estimated for each pattern view. ## Calibration Flags Flags control which parameters are estimated or fixed during calibration: | Flag | Value | Description | | --------------------------- | --------- | --------------------------------------------------------------------- | | `CALIB_USE_INTRINSIC_GUESS` | 0x00001 | cameraMatrix contains valid initial values that are optimized further | | `CALIB_FIX_PRINCIPAL_POINT` | 0x00004 | Principal point is not changed during optimization | | `CALIB_FIX_ASPECT_RATIO` | 0x00002 | Only fy is estimated, ratio fx/fy stays same as input | | `CALIB_ZERO_TANGENT_DIST` | 0x00008 | Tangential distortion coefficients (p1, p2) are set to zero | | `CALIB_FIX_FOCAL_LENGTH` | 0x00010 | Focal length is not changed (requires `CALIB_USE_INTRINSIC_GUESS`) | | `CALIB_FIX_K1` | 0x00020 | k1 distortion coefficient is not changed | | `CALIB_FIX_K2` | 0x00040 | k2 distortion coefficient is not changed | | `CALIB_FIX_K3` | 0x00080 | k3 distortion coefficient is not changed | | `CALIB_FIX_K4` | 0x00800 | k4 distortion coefficient is not changed | | `CALIB_FIX_K5` | 0x01000 | k5 distortion coefficient is not changed | | `CALIB_FIX_K6` | 0x02000 | k6 distortion coefficient is not changed | | `CALIB_RATIONAL_MODEL` | 0x04000 | Enable k4, k5, k6 coefficients (8+ coefficients) | | `CALIB_THIN_PRISM_MODEL` | 0x08000 | Enable s1, s2, s3, s4 coefficients (12+ coefficients) | | `CALIB_FIX_S1_S2_S3_S4` | 0x10000 | Thin prism distortion coefficients are not changed | | `CALIB_TILTED_MODEL` | 0x40000 | Enable tauX and tauY coefficients (14 coefficients) | | `CALIB_FIX_TAUX_TAUY` | 0x80000 | Tilted sensor model coefficients are not changed | | `CALIB_USE_QR` | 0x100000 | Use QR instead of SVD decomposition (faster but less precise) | | `CALIB_FIX_TANGENT_DIST` | 0x200000 | Fix tangential distortion coefficients | | `CALIB_USE_LU` | 1 \<\< 17 | Use LU instead of SVD decomposition (much faster but less precise) | ## Pattern Detection ### findChessboardCorners Finds the positions of internal corners of the chessboard. ```cpp theme={null} bool cv::findChessboardCorners( InputArray image, Size patternSize, OutputArray corners, int flags = CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_NORMALIZE_IMAGE ) ``` Source chessboard view. Must be an 8-bit grayscale or color image. Number of inner corners per chessboard row and column: `Size(points_per_row, points_per_column)` = `Size(columns, rows)`. For an 8x8 chessboard, use `Size(7, 7)`. Output array of detected corners. Operation flags: * `CALIB_CB_ADAPTIVE_THRESH` (1): Use adaptive thresholding * `CALIB_CB_NORMALIZE_IMAGE` (2): Normalize image gamma with equalizeHist * `CALIB_CB_FILTER_QUADS` (4): Use additional criteria to filter false quads * `CALIB_CB_FAST_CHECK` (8): Run fast check for chessboard corners * `CALIB_CB_PLAIN` (256): Take image as-is without processing **Returns:** Non-zero if all corners are found and placed in order (row by row, left to right), otherwise 0. The function requires white space (like a square-thick border) around the board to make detection more robust. Without borders, the outer black squares cannot be segmented properly. **Example:** ```cpp theme={null} Size patternsize(8, 6); // interior number of corners Mat gray = ...; // source image vector corners; bool patternfound = findChessboardCorners(gray, patternsize, corners, CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_NORMALIZE_IMAGE + CALIB_CB_FAST_CHECK); if(patternfound) cornerSubPix(gray, corners, Size(11, 11), Size(-1, -1), TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 30, 0.1)); drawChessboardCorners(img, patternsize, Mat(corners), patternfound); ``` ### findChessboardCornersSB Finds chessboard corners using a sector-based approach (more accurate and robust). ```cpp theme={null} bool cv::findChessboardCornersSB( InputArray image, Size patternSize, OutputArray corners, int flags = 0 ) ``` Operation flags: * `CALIB_CB_NORMALIZE_IMAGE` (2): Normalize image gamma * `CALIB_CB_EXHAUSTIVE` (16): Run exhaustive search to improve detection rate * `CALIB_CB_ACCURACY` (32): Upsample input image for better sub-pixel accuracy * `CALIB_CB_LARGER` (64): Allow detected pattern to be larger than patternSize * `CALIB_CB_MARKER` (128): Pattern must have a marker (for consistent coordinate system) This method uses a localized Radon transformation and is: * More robust to noise * Faster on larger images * Returns more accurate sub-pixel positions than cornerSubPix Based on the paper "Accurate Detection and Localization of Checkerboard Corners for Calibration" (Duda 2018). ### drawChessboardCorners Renders the detected chessboard corners. ```cpp theme={null} void cv::drawChessboardCorners( InputOutputArray image, Size patternSize, InputArray corners, bool patternWasFound ) ``` Destination image. Must be an 8-bit color image. Number of inner corners per chessboard row and column. Array of detected corners from findChessboardCorners. Parameter indicating whether the complete board was found. Pass the return value of findChessboardCorners. Draws individual corners as red circles (if board not found) or as colored corners connected with lines (if board found). ## Helper Functions ### initCameraMatrix2D Finds an initial camera intrinsic matrix from 3D-2D point correspondences. ```cpp theme={null} Mat cv::initCameraMatrix2D( InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints, Size imageSize, double aspectRatio = 1.0 ) ``` If zero or negative, both fx and fy are estimated independently. Otherwise, fx = fy \* aspectRatio. **Returns:** Initial camera intrinsic matrix for calibration process. Currently only supports planar calibration patterns (Z-coordinate = 0). ### calibrationMatrixValues Computes useful camera characteristics from the camera intrinsic matrix. ```cpp theme={null} void cv::calibrationMatrixValues( InputArray cameraMatrix, Size imageSize, double apertureWidth, double apertureHeight, CV_OUT double& fovx, CV_OUT double& fovy, CV_OUT double& focalLength, CV_OUT Point2d& principalPoint, CV_OUT double& aspectRatio ) ``` Physical width of the sensor in mm. Physical height of the sensor in mm. Output field of view in degrees along horizontal sensor axis. Output field of view in degrees along vertical sensor axis. Focal length of the lens in mm. Principal point in mm. fy/fx ratio. ## Advanced Calibration ### calibrateCameraRO Calibrates camera using the releasing object method for improved precision. ```cpp theme={null} double cv::calibrateCameraRO( InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints, Size imageSize, int iFixedPoint, InputOutputArray cameraMatrix, InputOutputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, OutputArray newObjPoints, int flags = 0, TermCriteria criteria = TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 30, DBL_EPSILON) ) ``` Index of the 3D object point in objectPoints\[0] to be fixed. Range \[1, objectPoints\[0].size()-2] enables object-releasing method. Values outside this range use standard calibration. Updated output vector of calibration pattern points with potentially scaled coordinates. This method can dramatically improve precision for inaccurate, unmeasured, roughly planar targets. Requires identical calibration board fully visible in all views. Calibration time may be much longer with this method. Use CALIB\_USE\_QR or CALIB\_USE\_LU for faster calibration. ## See Also * [Pose Estimation](/api/calib3d/pose-estimation) - solvePnP, solvePnPRansac for estimating camera pose * [Stereo Calibration](/api/calib3d/stereo) - stereoCalibrate for calibrating stereo camera systems * OpenCV samples: `calibration.cpp`, `3calibration.cpp` # Pose Estimation Source: https://opencv-opencv.mintlify.app/api/calib3d/pose-estimation Functions for estimating camera pose from 3D-2D point correspondences using PnP (Perspective-n-Point) algorithms ## Overview Pose estimation determines the transformation (rotation and translation) from the object coordinate system to the camera coordinate system. This is essential for: * Augmented reality applications * Robot navigation and manipulation * 3D scene reconstruction * Object tracking and localization ## Core Functions ### solvePnP Finds an object pose from 3D-2D point correspondences. ```cpp theme={null} bool cv::solvePnP( InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess = false, int flags = SOLVEPNP_ITERATIVE ) ``` Array of object points in the object coordinate space, Nx3 1-channel or 1xN/Nx1 3-channel, where N is the number of points. `vector` can also be passed. Array of corresponding image points in pixel coordinates, Nx2 1-channel or 1xN/Nx1 2-channel. `vector` can also be passed. Input camera intrinsic matrix (3x3). Input vector of distortion coefficients. If the vector is NULL/empty, zero distortion coefficients are assumed. Output rotation vector (see Rodrigues) that, together with tvec, brings points from the model coordinate system to the camera coordinate system. Output translation vector. If true, the function uses the provided rvec and tvec values as initial approximations and further optimizes them (used with SOLVEPNP\_ITERATIVE). Method for solving the PnP problem (see SolvePnP Methods below). **Returns:** True if a solution is found, false otherwise. **Coordinate Systems:** * Input objectPoints: 3D points in **world coordinate frame** * Output rvec/tvec: Transformation from world to **camera coordinate frame** * The transformation `Xc = R * Xw + t` brings world points to camera coordinates ### SolvePnP Methods | Method | Value | Description | Requirements | | ---------------------- | ----- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | `SOLVEPNP_ITERATIVE` | 0 | Levenberg-Marquardt optimization. DLT for non-planar (≥6 pts), homography for planar (≥4 pts) | ≥4 points | | `SOLVEPNP_EPNP` | 1 | Efficient PnP based on Lepetit et al. 2009 | ≥4 points | | `SOLVEPNP_P3P` | 2 | P3P algorithm based on Ding et al. 2023 | Exactly 4 points (3 for estimation, 1 for validation) | | `SOLVEPNP_DLS` | 3 | **Broken - fallback to EPnP** | Not recommended | | `SOLVEPNP_UPNP` | 4 | **Broken - fallback to EPnP** | Not recommended | | `SOLVEPNP_AP3P` | 5 | Efficient algebraic solution by Ke & Roumeliotis 2017 | Exactly 4 points | | `SOLVEPNP_IPPE` | 6 | Infinitesimal Plane-Based Pose Estimation | ≥4 coplanar points | | `SOLVEPNP_IPPE_SQUARE` | 7 | IPPE for square markers (returns 2 solutions) | Exactly 4 coplanar points in specific order | | `SOLVEPNP_SQPNP` | 8 | SQPnP: Fast and globally optimal solution | ≥3 points | For SOLVEPNP\_IPPE\_SQUARE, object points must be defined in this exact order: * point 0: `[-squareLength/2, squareLength/2, 0]` * point 1: `[ squareLength/2, squareLength/2, 0]` * point 2: `[ squareLength/2, -squareLength/2, 0]` * point 3: `[-squareLength/2, -squareLength/2, 0]` **Example:** ```cpp theme={null} // Define object points (e.g., 3D corners of a marker) vector objectPoints = { {-0.05f, 0.05f, 0.0f}, { 0.05f, 0.05f, 0.0f}, { 0.05f, -0.05f, 0.0f}, {-0.05f, -0.05f, 0.0f} }; // Corresponding 2D image points (detected in image) vector imagePoints = { {234.5f, 156.3f}, {456.2f, 167.8f}, {443.1f, 389.4f}, {221.7f, 378.9f} }; Mat cameraMatrix = ...; // 3x3 camera intrinsic matrix Mat distCoeffs = ...; // Distortion coefficients Mat rvec, tvec; bool success = solvePnP(objectPoints, imagePoints, cameraMatrix, distCoeffs, rvec, tvec, false, SOLVEPNP_IPPE_SQUARE); ``` ### solvePnPRansac Finds object pose from 3D-2D point correspondences using RANSAC to handle outliers. ```cpp theme={null} bool cv::solvePnPRansac( InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess = false, int iterationsCount = 100, float reprojectionError = 8.0, double confidence = 0.99, OutputArray inliers = noArray(), int flags = SOLVEPNP_ITERATIVE ) ``` Number of RANSAC iterations. Inlier threshold value in pixels. The maximum allowed distance between observed and computed point projections to consider it an inlier. The probability that the algorithm produces a useful result (typically 0.99). Output vector that contains indices of inliers in objectPoints and imagePoints. **Returns:** True if a solution is found. This function estimates an object pose and is resistant to outliers using RANSAC. The algorithm: 1. Randomly selects minimal subsets of points 2. Estimates pose for each subset 3. Counts inliers (points within reprojectionError threshold) 4. Refines final pose using all inliers **Minimal Sample Sets:** * Default method uses SOLVEPNP\_EPNP for minimal sample estimation * If you choose SOLVEPNP\_P3P or SOLVEPNP\_AP3P, these methods are used * If exactly 4 input points, SOLVEPNP\_P3P is automatically used * Final pose is refined using all inliers with the method specified in flags (unless P3P/AP3P, then EPNP is used) ### USAC-based solvePnPRansac Advanced robust estimation using USAC (Universal RANSAC) framework. ```cpp theme={null} bool cv::solvePnPRansac( InputArray objectPoints, InputArray imagePoints, InputOutputArray cameraMatrix, InputArray distCoeffs, OutputArray rvec, OutputArray tvec, OutputArray inliers, const UsacParams& params = UsacParams() ) ``` USAC provides several advanced RANSAC variants with configurable parameters for better performance and accuracy. ### solvePnPGeneric Returns all possible solutions for pose estimation (multiple solutions from P3P methods). ```cpp theme={null} int cv::solvePnPGeneric( InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, bool useExtrinsicGuess = false, SolvePnPMethod flags = SOLVEPNP_ITERATIVE, InputArray rvec = noArray(), InputArray tvec = noArray(), OutputArray reprojectionError = noArray() ) ``` Vector of output rotation vectors. P3P methods return 0-4 solutions, SOLVEPNP\_IPPE returns 2 solutions, others return 1 solution. Vector of output translation vectors corresponding to rvecs. Optional output array of reprojection error (RMSE) for each solution. **Returns:** Number of solutions found. P3P solutions are sorted by reprojection errors (lowest to highest). ### solveP3P Finds an object pose from 3 3D-2D point correspondences. ```cpp theme={null} int cv::solveP3P( InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, int flags ) ``` Array of object points, 3x3 1-channel or 1x3/3x1 3-channel. Exactly 3 points required. Array of corresponding image points, 3x2 1-channel or 1x3/3x1 2-channel. Exactly 3 points required. Method for solving P3P: * `SOLVEPNP_P3P`: Based on Ding et al. 2023 * `SOLVEPNP_AP3P`: Based on Ke & Roumeliotis 2017 **Returns:** Number of solutions (0-4). Solutions are sorted by reprojection errors. ## Pose Refinement ### solvePnPRefineLM Refines a pose using Levenberg-Marquardt optimization. ```cpp theme={null} void cv::solvePnPRefineLM( InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, InputOutputArray rvec, InputOutputArray tvec, TermCriteria criteria = TermCriteria(TermCriteria::EPS + TermCriteria::COUNT, 20, FLT_EPSILON) ) ``` Input/Output rotation vector. Input values used as initial solution. Input/Output translation vector. Input values used as initial solution. Termination criteria for the iterative optimization algorithm. Minimizes projection error using Levenberg-Marquardt iterative minimization. Requires at least 3 object points and an initial pose estimate. ### solvePnPRefineVVS Refines a pose using Virtual Visual Servoing (VVS). ```cpp theme={null} void cv::solvePnPRefineVVS( InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray distCoeffs, InputOutputArray rvec, InputOutputArray tvec, TermCriteria criteria = TermCriteria(TermCriteria::EPS + TermCriteria::COUNT, 20, FLT_EPSILON), double VVSlambda = 1 ) ``` Gain for the virtual visual servoing control law, equivalent to the α gain in the Damped Gauss-Newton formulation. Minimizes projection error using Virtual Visual Servoing scheme (Chaumette 2006, Marchand 2016). ## Homography-Based Methods ### findHomography Finds a perspective transformation between two planes. ```cpp theme={null} Mat cv::findHomography( InputArray srcPoints, InputArray dstPoints, int method = 0, double ransacReprojThreshold = 3, OutputArray mask = noArray(), const int maxIters = 2000, const double confidence = 0.995 ) ``` Coordinates of points in the original plane, CV\_32FC2 or `vector`. Coordinates of points in the target plane, CV\_32FC2 or `vector`. Method for computing homography: * 0: Regular method using all points (least squares) * `RANSAC` (8): RANSAC-based robust method * `LMEDS` (4): Least-Median robust method * `RHO` (16): PROSAC-based robust method Maximum allowed reprojection error to treat a point pair as an inlier (pixels). Used in RANSAC and RHO methods. Optional output mask set by robust methods. Input mask values are ignored. Maximum number of RANSAC iterations. Confidence level, between 0 and 1. **Returns:** The 3x3 homography matrix H such that: ``` s_i [x'_i, y'_i, 1]^T ≈ H [x_i, y_i, 1]^T ``` The function finds the perspective transformation between source and destination planes. Useful for: * Finding initial intrinsic and extrinsic matrices * Planar object tracking * Image rectification If H cannot be estimated, an empty matrix is returned. ### USAC-based findHomography ```cpp theme={null} Mat cv::findHomography( InputArray srcPoints, InputArray dstPoints, OutputArray mask, const UsacParams& params ) ``` Uses USAC framework for robust homography estimation with configurable parameters. ## Decomposition Methods ### decomposeProjectionMatrix Decomposes a projection matrix into rotation matrix and camera intrinsic matrix. ```cpp theme={null} void cv::decomposeProjectionMatrix( InputArray projMatrix, OutputArray cameraMatrix, OutputArray rotMatrix, OutputArray transVect, OutputArray rotMatrixX = noArray(), OutputArray rotMatrixY = noArray(), OutputArray rotMatrixZ = noArray(), OutputArray eulerAngles = noArray() ) ``` 3x4 input projection matrix P. Output 3x3 camera intrinsic matrix. Output 3x3 external rotation matrix R. Output 4x1 translation vector T. Optional 3x3 rotation matrix around x-axis. Optional 3x3 rotation matrix around y-axis. Optional 3x3 rotation matrix around z-axis. Optional three-element vector containing three Euler angles of rotation in degrees. Decomposes a projection matrix into calibration and rotation matrix and the position of a camera. Based on RQDecomp3x3. ### RQDecomp3x3 Computes an RQ decomposition of 3x3 matrices. ```cpp theme={null} Vec3d cv::RQDecomp3x3( InputArray src, OutputArray mtxR, OutputArray mtxQ, OutputArray Qx = noArray(), OutputArray Qy = noArray(), OutputArray Qz = noArray() ) ``` 3x3 input matrix. Output 3x3 upper-triangular matrix. Output 3x3 orthogonal matrix. **Returns:** Three Euler angles in degrees. Used in decomposeProjectionMatrix to decompose the left 3x3 submatrix of a projection matrix. ## Helper Functions ### projectPoints Projects 3D points to an image plane. ```cpp theme={null} void cv::projectPoints( InputArray objectPoints, InputArray rvec, InputArray tvec, InputArray cameraMatrix, InputArray distCoeffs, OutputArray imagePoints, OutputArray jacobian = noArray(), double aspectRatio = 0 ) ``` Array of object points in world coordinate frame, 3xN/Nx3 1-channel or 1xN/Nx1 3-channel. Rotation vector (Rodrigues) that performs change of basis from world to camera coordinate system. Translation vector. Output array of image points in **pixel coordinates**, 1xN/Nx1 2-channel, or `vector`. Optional output 2Nx(10+numDistCoeffs) Jacobian matrix of derivatives of image points with respect to rotation, translation, focal lengths, principal point, and distortion coefficients. Optional fixed aspect ratio parameter. If not 0, the function assumes aspect ratio (fx/fy) is fixed. Computes 2D projections of 3D points given intrinsic and extrinsic camera parameters. Used during optimization in calibrateCamera, solvePnP, and stereoCalibrate. ### drawFrameAxes Draws axes of the world/object coordinate system from pose estimation. ```cpp theme={null} void cv::drawFrameAxes( InputOutputArray image, InputArray cameraMatrix, InputArray distCoeffs, InputArray rvec, InputArray tvec, float length, int thickness = 3 ) ``` Length of the painted axes in the same unit as tvec (usually meters). Line thickness of the painted axes. Draws the world/object coordinate system axes w\.r.t. the camera frame: * OX is drawn in red * OY is drawn in green * OZ is drawn in blue ## See Also * [Camera Calibration](/api/calib3d/calibration) - calibrateCamera for obtaining camera intrinsics * [Stereo Vision](/api/calib3d/stereo) - Stereo calibration and rectification * OpenCV samples: `plane_ar.py` (planar augmented reality) # Stereo Vision Source: https://opencv-opencv.mintlify.app/api/calib3d/stereo Functions for stereo camera calibration, rectification, and depth computation from stereo image pairs ## Overview Stereo vision uses two cameras to compute depth information by triangulation. OpenCV provides functions for: * Calibrating stereo camera systems * Computing rectification transformations * Stereo correspondence algorithms (StereoBM, StereoSGBM) * 3D reconstruction from disparity maps ## Stereo Calibration ### stereoCalibrate Calibrates a stereo camera setup by finding intrinsic parameters for each camera and extrinsic parameters between them. ```cpp theme={null} double cv::stereoCalibrate( InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints1, InputArrayOfArrays imagePoints2, InputOutputArray cameraMatrix1, InputOutputArray distCoeffs1, InputOutputArray cameraMatrix2, InputOutputArray distCoeffs2, Size imageSize, OutputArray R, OutputArray T, OutputArray E, OutputArray F, int flags = CALIB_FIX_INTRINSIC, TermCriteria criteria = TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 30, 1e-6) ) ``` Vector of vectors of calibration pattern points. Both cameras need to see the same object points. Structure same as calibrateCamera. Vector of vectors of projections of calibration pattern points observed by the first camera. Vector of vectors of projections of calibration pattern points observed by the second camera. Input/output camera intrinsic matrix for the first camera. Input/output vector of distortion coefficients for the first camera. Input/output camera intrinsic matrix for the second camera. Input/output vector of distortion coefficients for the second camera. Size of the image used only to initialize camera intrinsic matrices. Output rotation matrix between the first and second camera coordinate systems. This matrix brings points from the first camera's coordinate system to the second camera's coordinate system. Output translation vector between the coordinate systems of the cameras. Equivalent to the position of the first camera with respect to the second camera. Output essential matrix. Output fundamental matrix. Different flags for stereo calibration (see Stereo Calibration Flags below). Termination criteria for the iterative optimization algorithm. **Returns:** The overall RMS re-projection error. The function estimates the transformation between two cameras: ``` R2 = R * R1 T2 = R * T1 + T ``` Optionally computes the essential matrix E: ``` E = [T]_x * R ``` where `[T]_x` is the skew-symmetric matrix of T. And the fundamental matrix F: ``` F = cameraMatrix2^(-T) * E * cameraMatrix1^(-1) ``` Due to high dimensionality and noise, the function can diverge. If intrinsic parameters can be estimated with high accuracy for each camera individually (using calibrateCamera), it's recommended to pass CALIB\_FIX\_INTRINSIC flag with the computed intrinsic parameters. ### Extended Version ```cpp theme={null} double cv::stereoCalibrate( InputArrayOfArrays objectPoints, InputArrayOfArrays imagePoints1, InputArrayOfArrays imagePoints2, InputOutputArray cameraMatrix1, InputOutputArray distCoeffs1, InputOutputArray cameraMatrix2, InputOutputArray distCoeffs2, Size imageSize, InputOutputArray R, InputOutputArray T, OutputArray E, OutputArray F, OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs, OutputArray perViewErrors, int flags = CALIB_FIX_INTRINSIC, TermCriteria criteria = TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 30, 1e-6) ) ``` Output vector of rotation vectors (Rodrigues) estimated for each pattern view in the coordinate system of the first camera. Output vector of translation vectors estimated for each pattern view. Output vector of the RMS re-projection error estimated for each pattern view. ### Stereo Calibration Flags Flags control the calibration behavior (in addition to single camera flags): | Flag | Value | Description | | --------------------------------- | --------- | --------------------------------------------------------------------------- | | `CALIB_FIX_INTRINSIC` | 0x00100 | Fix cameraMatrix1/2 and distCoeffs1/2 so that only R, T, E, F are estimated | | `CALIB_USE_INTRINSIC_GUESS` | 0x00001 | Optimize some or all intrinsic parameters according to specified flags | | `CALIB_USE_EXTRINSIC_GUESS` | 1 \<\< 22 | R and T contain valid initial values that are optimized further | | `CALIB_FIX_PRINCIPAL_POINT` | 0x00004 | Fix principal points during optimization | | `CALIB_FIX_FOCAL_LENGTH` | 0x00010 | Fix fx and fy for both cameras | | `CALIB_FIX_ASPECT_RATIO` | 0x00002 | Optimize fy, fix ratio fx/fy | | `CALIB_SAME_FOCAL_LENGTH` | 0x00200 | Enforce fx^(0) = fx^(1) and fy^(0) = fy^(1) | | `CALIB_ZERO_TANGENT_DIST` | 0x00008 | Set tangential distortion coefficients to zero for each camera | | `CALIB_FIX_K1` ... `CALIB_FIX_K6` | Various | Do not change corresponding radial distortion coefficient | | `CALIB_RATIONAL_MODEL` | 0x04000 | Enable k4, k5, k6 coefficients (8 coefficients total) | | `CALIB_THIN_PRISM_MODEL` | 0x08000 | Enable s1, s2, s3, s4 coefficients (12 coefficients total) | | `CALIB_FIX_S1_S2_S3_S4` | 0x10000 | Thin prism distortion coefficients are not changed | | `CALIB_TILTED_MODEL` | 0x40000 | Enable tauX and tauY coefficients (14 coefficients) | | `CALIB_FIX_TAUX_TAUY` | 0x80000 | Tilted sensor model coefficients are not changed | It's usually reasonable to restrict some parameters, e.g., pass CALIB\_SAME\_FOCAL\_LENGTH and CALIB\_ZERO\_TANGENT\_DIST flags. ## Stereo Rectification ### stereoRectify Computes rectification transforms for each head of a calibrated stereo camera. ```cpp theme={null} void cv::stereoRectify( InputArray cameraMatrix1, InputArray distCoeffs1, InputArray cameraMatrix2, InputArray distCoeffs2, Size imageSize, InputArray R, InputArray T, OutputArray R1, OutputArray R2, OutputArray P1, OutputArray P2, OutputArray Q, int flags = CALIB_ZERO_DISPARITY, double alpha = -1, Size newImageSize = Size(), CV_OUT Rect* validPixROI1 = 0, CV_OUT Rect* validPixROI2 = 0 ) ``` First camera intrinsic matrix. First camera distortion parameters. Second camera intrinsic matrix. Second camera distortion parameters. Size of the image used for stereo calibration. Rotation matrix from the coordinate system of the first camera to the second camera (from stereoCalibrate). Translation vector from the coordinate system of the first camera to the second camera (from stereoCalibrate). Output 3x3 rectification transform (rotation matrix) for the first camera. Performs change of basis from unrectified to rectified first camera's coordinate system. Output 3x3 rectification transform (rotation matrix) for the second camera. Output 3x4 projection matrix in the new (rectified) coordinate systems for the first camera. Projects points given in the rectified first camera coordinate system into the rectified first camera's image. Output 3x4 projection matrix in the new (rectified) coordinate systems for the second camera. Output 4x4 disparity-to-depth mapping matrix (see reprojectImageTo3D). Operation flags: * `CALIB_ZERO_DISPARITY` (0x00400): Makes principal points of each camera have the same pixel coordinates in rectified views Free scaling parameter between 0 and 1: * `alpha=0`: Rectified images are zoomed and shifted so only valid pixels are visible (no black areas) * `alpha=1`: Rectified images are decimated and shifted so all pixels from original images are retained * `-1`: Default scaling New image resolution after rectification. When (0,0), it's set to the original imageSize. Setting to larger value helps preserve details. Optional output rectangle inside the rectified first image where all pixels are valid. Optional output rectangle inside the rectified second image where all pixels are valid. The function computes rotation matrices for each camera that make both camera image planes the same plane. This makes all epipolar lines parallel, simplifying dense stereo correspondence. **Horizontal Stereo:** For cameras shifted mainly along x-axis, the projection matrices are: ``` P1 = [f 0 cx1 0 ] [0 f cy 0 ] [0 0 1 0 ] P2 = [f 0 cx2 Tx*f] [0 f cy 0 ] [0 0 1 0 ] Q = [1 0 0 -cx1 ] [0 1 0 -cy ] [0 0 0 f ] [0 0 -1/Tx (cx1-cx2)/Tx] ``` where Tx is horizontal shift between cameras and cx1=cx2 if CALIB\_ZERO\_DISPARITY is set. **Vertical Stereo:** For cameras shifted mainly along y-axis: ``` P1 = [f 0 cx 0 ] [0 f cy1 0 ] [0 0 1 0 ] P2 = [f 0 cx 0 ] [0 f cy2 Ty*f] [0 0 1 0 ] Q = [1 0 0 -cx ] [0 1 0 -cy1 ] [0 0 0 f ] [0 0 -1/Ty (cy1-cy2)/Ty] ``` The first three columns of P1 and P2 are the new "rectified" camera matrices. Pass these with R1 and R2 to initUndistortRectifyMap to initialize rectification maps. ### stereoRectifyUncalibrated Computes a rectification transform for an uncalibrated stereo camera. ```cpp theme={null} bool cv::stereoRectifyUncalibrated( InputArray points1, InputArray points2, InputArray F, Size imgSize, OutputArray H1, OutputArray H2, double threshold = 5 ) ``` Array of feature points in the first image. Corresponding points in the second image. Input fundamental matrix. Can be computed from the same point pairs using findFundamentalMat. Size of the image. Output rectification homography matrix for the first image. Output rectification homography matrix for the second image. Optional threshold to filter outliers. If >0, point pairs not complying with epipolar geometry are rejected. Otherwise all points are considered inliers. Computes rectification transformations without knowing intrinsic parameters. Implements the algorithm from Hartley99. Algorithm heavily depends on epipolar geometry. If camera lenses have significant distortion, correct it before computing fundamental matrix and calling this function. ## Utility Functions ### getOptimalNewCameraMatrix Returns the new camera intrinsic matrix based on the free scaling parameter. ```cpp theme={null} Mat cv::getOptimalNewCameraMatrix( InputArray cameraMatrix, InputArray distCoeffs, Size imageSize, double alpha, Size newImgSize = Size(), CV_OUT Rect* validPixROI = 0, bool centerPrincipalPoint = false ) ``` Input camera intrinsic matrix. Input vector of distortion coefficients. If NULL/empty, zero distortion is assumed. Original image size. Free scaling parameter between 0 (only valid pixels) and 1 (retain all source pixels). See stereoRectify for details. Image size after rectification. By default, set to imageSize. Optional output rectangle outlining all-good-pixels region in undistorted image. Optional flag indicating whether the principal point should be at image center or chosen to best fit source image (determined by alpha). **Returns:** New camera intrinsic matrix. By varying alpha parameter, you can retrieve only sensible pixels (alpha=0), keep all original pixels (alpha=1), or get something in between. When alpha>0, undistorted result likely has black pixels corresponding to "virtual" pixels outside captured distorted image. ### rectify3Collinear Computes rectification transforms for 3-head camera where all heads are on the same line. ```cpp theme={null} float cv::rectify3Collinear( InputArray cameraMatrix1, InputArray distCoeffs1, InputArray cameraMatrix2, InputArray distCoeffs2, InputArray cameraMatrix3, InputArray distCoeffs3, InputArrayOfArrays imgpt1, InputArrayOfArrays imgpt3, Size imageSize, InputArray R12, InputArray T12, InputArray R13, InputArray T13, OutputArray R1, OutputArray R2, OutputArray R3, OutputArray P1, OutputArray P2, OutputArray P3, OutputArray Q, double alpha, Size newImgSize, CV_OUT Rect* roi1, CV_OUT Rect* roi2, int flags ) ``` Computes rectification transformations for tri-focal stereo camera systems with collinear arrangement. ## Stereo Matching Classes ### StereoBM Class for computing stereo correspondence using the block matching algorithm. ```cpp theme={null} class CV_EXPORTS_W StereoBM : public StereoMatcher { public: static Ptr create(int numDisparities = 0, int blockSize = 21); // Parameters CV_WRAP virtual int getPreFilterType() const = 0; CV_WRAP virtual void setPreFilterType(int preFilterType) = 0; CV_WRAP virtual int getPreFilterSize() const = 0; CV_WRAP virtual void setPreFilterSize(int preFilterSize) = 0; CV_WRAP virtual int getPreFilterCap() const = 0; CV_WRAP virtual void setPreFilterCap(int preFilterCap) = 0; CV_WRAP virtual int getTextureThreshold() const = 0; CV_WRAP virtual void setTextureThreshold(int textureThreshold) = 0; CV_WRAP virtual int getUniquenessRatio() const = 0; CV_WRAP virtual void setUniquenessRatio(int uniquenessRatio) = 0; CV_WRAP virtual int getSmallerBlockSize() const = 0; CV_WRAP virtual void setSmallerBlockSize(int blockSize) = 0; CV_WRAP virtual Rect getROI1() const = 0; CV_WRAP virtual void setROI1(Rect roi1) = 0; CV_WRAP virtual Rect getROI2() const = 0; CV_WRAP virtual void setROI2(Rect roi2) = 0; }; ``` **Key Parameters:** Maximum disparity minus minimum disparity. Must be divisible by 16. Typical value: 16, 32, 48, 64, etc. Matched block size. Must be odd number ≥1. Typical values: 5-21. Larger blocks produce smoother but less detailed disparity maps. Type of the prefilter: * `PREFILTER_NORMALIZED_RESPONSE`: Normalized response * `PREFILTER_XSOBEL`: Sobel prefilter Prefilter window size (5-255, must be odd). Truncation value for prefiltered image pixels (1-63). Minimum texture for disparity computation. Areas with low texture are filtered out. Margin in percentage by which best computed cost function value should "win" second best value. Typically 5-15. **Example:** ```cpp theme={null} Ptr stereo = StereoBM::create(64, 15); stereo->setPreFilterCap(31); stereo->setUniquenessRatio(10); Mat disparity; stereo->compute(leftImage, rightImage, disparity); ``` ### StereoSGBM Class for computing stereo correspondence using Semi-Global Block Matching algorithm. ```cpp theme={null} class CV_EXPORTS_W StereoSGBM : public StereoMatcher { public: enum { MODE_SGBM = 0, MODE_HH = 1, MODE_SGBM_3WAY = 2, MODE_HH4 = 3 }; static Ptr create( int minDisparity = 0, int numDisparities = 16, int blockSize = 3, int P1 = 0, int P2 = 0, int disp12MaxDiff = 0, int preFilterCap = 0, int uniquenessRatio = 0, int speckleWindowSize = 0, int speckleRange = 0, int mode = MODE_SGBM ); CV_WRAP virtual int getPreFilterCap() const = 0; CV_WRAP virtual void setPreFilterCap(int preFilterCap) = 0; CV_WRAP virtual int getUniquenessRatio() const = 0; CV_WRAP virtual void setUniquenessRatio(int uniquenessRatio) = 0; CV_WRAP virtual int getP1() const = 0; CV_WRAP virtual void setP1(int P1) = 0; CV_WRAP virtual int getP2() const = 0; CV_WRAP virtual void setP2(int P2) = 0; CV_WRAP virtual int getMode() const = 0; CV_WRAP virtual void setMode(int mode) = 0; }; ``` **Key Parameters:** Minimum possible disparity value. Typically 0, but can be adjusted. Maximum disparity minus minimum disparity. Must be divisible by 16. Values: 16, 32, 48, 64, 96, 128, etc. Matched block size. Must be odd number ≥1. Values: 3, 5, 7, etc. SGBM works well with smaller blocks than BM. First parameter controlling disparity smoothness. Penalty for disparity change by ±1. If 0, default is `8 * channels * blockSize^2`. Second parameter controlling disparity smoothness. Penalty for disparity change by more than 1. If 0, default is `32 * channels * blockSize^2`. P2 > P1. Maximum allowed difference in left-right disparity check. Set to negative value to disable check. Truncation value for prefiltered image pixels. Default: 63. Margin by which best cost function value should "win" second best. Typically 5-15. Maximum size of smooth disparity regions to consider noise speckles and invalidate. Set to 0 to disable. Typical: 50-200. Maximum disparity variation within connected component. Typical: 1-2. Algorithm mode: * `MODE_SGBM`: Standard Semi-Global Block Matching * `MODE_HH`: Hirschmuller algorithm * `MODE_SGBM_3WAY`: Modified SGBM * `MODE_HH4`: Full-scale two-pass algorithm SGBM produces smoother and more accurate disparity maps than BM but is computationally more expensive. **Example:** ```cpp theme={null} Ptr stereo = StereoSGBM::create( 0, // minDisparity 96, // numDisparities 5, // blockSize 600, // P1 2400, // P2 1, // disp12MaxDiff 63, // preFilterCap 10, // uniquenessRatio 100, // speckleWindowSize 32, // speckleRange StereoSGBM::MODE_SGBM_3WAY ); Mat disparity; stereo->compute(leftImage, rightImage, disparity); // Convert to float disparity disparity.convertTo(disparity, CV_32F, 1.0/16.0); ``` SGBM is more suitable for real-time applications and produces better results than BM, especially in textured regions. Consider using MODE\_SGBM\_3WAY or MODE\_HH4 for best quality. ## See Also * [Camera Calibration](/api/calib3d/calibration) - calibrateCamera for obtaining camera intrinsics * [Pose Estimation](/api/calib3d/pose-estimation) - solvePnP for 3D-2D correspondences * OpenCV samples: `stereo_calib.cpp`, `stereo_match.cpp` # Mat Class Source: https://opencv-opencv.mintlify.app/api/core/mat N-dimensional dense array class for storing images, matrices, and multi-dimensional data ## Overview The `Mat` class is the primary data structure in OpenCV for representing n-dimensional dense arrays. It can store real or complex-valued vectors, matrices, grayscale or color images, voxel volumes, vector fields, point clouds, tensors, and histograms. ## Constructors ### Default Constructor ```cpp theme={null} Mat() ``` Creates an empty matrix with no allocated data. ### Size and Type Constructor ```cpp theme={null} Mat(int rows, int cols, int type) Mat(Size size, int type) ``` Number of rows in a 2D array Number of columns in a 2D array 2D array size: Size(cols, rows) Array type. Use CV\_8UC1, ..., CV\_64FC4 to create 1-4 channel matrices, or CV\_8UC(n), ..., CV\_64FC(n) to create multi-channel matrices (up to CV\_CN\_MAX channels) **Example:** ```cpp theme={null} // Create a 100x100 8-bit unsigned single-channel matrix Mat img(100, 100, CV_8UC1); // Create a 640x480 8-bit 3-channel color image Mat colorImage(Size(640, 480), CV_8UC3); ``` ### Constructor with Initialization ```cpp theme={null} Mat(int rows, int cols, int type, const Scalar& s) Mat(Size size, int type, const Scalar& s) ``` Optional value to initialize each matrix element with **Example:** ```cpp theme={null} // Create a 7x7 complex matrix filled with 1+3j Mat M(7, 7, CV_32FC2, Scalar(1, 3)); // Create a 100x100 matrix filled with zeros Mat zeros(100, 100, CV_64F, Scalar(0)); ``` ### Multi-dimensional Constructor ```cpp theme={null} Mat(int ndims, const int* sizes, int type) Mat(const std::vector& sizes, int type) Mat(int ndims, const int* sizes, int type, const Scalar& s) ``` Array dimensionality Array of integers specifying an n-dimensional array shape **Example:** ```cpp theme={null} // Create a 100x100x100 8-bit 3D array int sz[] = {100, 100, 100}; Mat bigCube(3, sz, CV_8U, Scalar::all(0)); ``` ### User Data Constructor ```cpp theme={null} Mat(int rows, int cols, int type, void* data, size_t step=AUTO_STEP) Mat(Size size, int type, void* data, size_t step=AUTO_STEP) ``` Pointer to user data. No data is copied; the matrix header points to the specified data Number of bytes each matrix row occupies. If AUTO\_STEP, no padding is assumed The external data is not automatically deallocated, so you should manage it manually. ### Copy Constructor ```cpp theme={null} Mat(const Mat& m) ``` Creates a matrix header for the same data. This is an O(1) operation that does not copy data but increments the reference counter. ### ROI Constructors ```cpp theme={null} Mat(const Mat& m, const Range& rowRange, const Range& colRange=Range::all()) Mat(const Mat& m, const Rect& roi) Mat(const Mat& m, const Range* ranges) ``` Range of rows to take from the source matrix Range of columns to take from the source matrix Region of interest rectangle **Example:** ```cpp theme={null} Mat img(320, 240, CV_8UC3); // Select a ROI Mat roi(img, Rect(10, 10, 100, 100)); // Fill the ROI with green color roi = Scalar(0, 255, 0); ``` ## Key Methods ### create ```cpp theme={null} void create(int rows, int cols, int type) void create(Size size, int type) void create(int ndims, const int* sizes, int type) ``` Allocates new array data if needed. If the array already has the specified size and type, the method does nothing. ### clone ```cpp theme={null} Mat clone() const ``` Creates a full copy of the array and underlying data. **Example:** ```cpp theme={null} Mat A = Mat::eye(3, 3, CV_32F); Mat B = A.clone(); // B is an independent copy ``` ### copyTo ```cpp theme={null} void copyTo(OutputArray m) const void copyTo(OutputArray m, InputArray mask) const ``` Destination matrix. Reallocated if needed Operation mask (8-bit single channel). Non-zero elements indicate which matrix elements to copy ### convertTo ```cpp theme={null} void convertTo(OutputArray m, int rtype, double alpha=1, double beta=0) const ``` Output matrix Desired output matrix type or depth Optional scale factor Optional delta added to scaled values Converts array to another data type with optional scaling: `m(x,y) = saturate_cast(alpha*(*this)(x,y) + beta)` ### row / col ```cpp theme={null} Mat row(int y) const Mat col(int x) const ``` Creates a matrix header for the specified matrix row or column. This is an O(1) operation. 0-based row index 0-based column index **Example:** ```cpp theme={null} // Add the 5-th row, multiplied by 3 to the 3rd row M.row(3) = M.row(3) + M.row(5) * 3; ``` ### rowRange / colRange ```cpp theme={null} Mat rowRange(int startrow, int endrow) const Mat rowRange(const Range& r) const Mat colRange(int startcol, int endcol) const Mat colRange(const Range& r) const ``` Creates a matrix header for the specified row or column span. ### diag ```cpp theme={null} Mat diag(int d=0) const static Mat diag(const Mat& d) ``` Index of the diagonal. d=0 is the main diagonal, d\<0 is below, d>0 is above Extracts a diagonal from a matrix or creates a diagonal matrix. ### reshape ```cpp theme={null} Mat reshape(int cn, int rows=0) const Mat reshape(int cn, int newndims, const int* newsz) const ``` New number of channels. If 0, the number of channels remains unchanged New number of rows. If 0, the number of rows remains unchanged Changes the shape and/or number of channels without copying data. ### at ```cpp theme={null} template _Tp& at(int i0, int i1) template const _Tp& at(int i0, int i1) const template _Tp& at(Point pt) template _Tp& at(int i0, int i1, int i2) ``` Returns a reference to the specified array element. **Example:** ```cpp theme={null} Mat M(100, 100, CV_64F); M.at(i, j) += 1.0; Mat colorImage(480, 640, CV_8UC3); Vec3b& pixel = colorImage.at(y, x); pixel[0] = 255; // Blue channel ``` No bounds checking is performed in Release builds. Use with caution. ### ptr ```cpp theme={null} template _Tp* ptr(int i0=0) template const _Tp* ptr(int i0=0) const ``` Returns a pointer to the specified matrix row. **Example:** ```cpp theme={null} // Efficient row-wise processing for(int i = 0; i < M.rows; i++) { const double* Mi = M.ptr(i); for(int j = 0; j < M.cols; j++) sum += std::max(Mi[j], 0.); } ``` ## Static Initialization Methods ### zeros ```cpp theme={null} static Mat zeros(int rows, int cols, int type) static Mat zeros(Size size, int type) static Mat zeros(int ndims, const int* sz, int type) ``` Returns a zero array of the specified size and type. **Example:** ```cpp theme={null} Mat Z = Mat::zeros(3, 3, CV_32F); ``` ### ones ```cpp theme={null} static Mat ones(int rows, int cols, int type) static Mat ones(Size size, int type) static Mat ones(int ndims, const int* sz, int type) ``` Returns an array of all 1's of the specified size and type. ### eye ```cpp theme={null} static Mat eye(int rows, int cols, int type) static Mat eye(Size size, int type) ``` Returns an identity matrix of the specified size and type. **Example:** ```cpp theme={null} // Add identity matrix to M M += Mat::eye(M.rows, M.cols, CV_64F); ``` ## Properties ### Data Layout Number of rows (for 2D arrays) Number of columns (for 2D arrays) Number of matrix dimensions (≥ 2) Pointer to the data Array of strides (step\[0] contains the full row length in bytes) ### Type Information Returns the matrix element type (CV\_8UC1, CV\_32FC3, etc.) Returns the depth of the matrix elements (CV\_8U, CV\_32F, etc.) Returns the number of channels Returns element size in bytes Returns size of each element channel in bytes ### Size Information Returns the matrix size (for 2D matrices) Returns the total number of array elements Returns true if the array has no elements Returns true if the matrix is continuous (no gaps at the end of rows) Returns true if the matrix is a submatrix of another matrix ## Operators ### Assignment ```cpp theme={null} Mat& operator=(const Mat& m) Mat& operator=(const MatExpr& expr) ``` Matrix assignment is an O(1) operation that copies the header and increments the reference counter. ### Element Access ```cpp theme={null} template _Tp& operator()(int row, int col) template _Tp& operator()(Point pt) template _Tp& operator()(const int* idx) ``` Provides element access similar to at() method. ### Arithmetic Operators Matrix arithmetic operators (+, -, \*, /) are supported through matrix expressions: ```cpp theme={null} Mat A, B, C; C = A + B; // Addition C = A - B; // Subtraction C = A * B; // Matrix multiplication C = A / 2.0; // Scalar division ``` ## Memory Management Mat uses reference counting for automatic memory management. When no one references the data, it's automatically deallocated. **Example:** ```cpp theme={null} Mat A(100, 100, CV_32F); Mat B = A; // B references the same data (O(1) operation) Mat C = A.clone(); // C is an independent copy // Manually release data before destructor A.release(); ``` ## Data Layout For a 2D array, element (i,j) address is computed as: ``` addr(M[i,j]) = M.data + M.step[0]*i + M.step[1]*j ``` Matrices are stored row-by-row, with `step[0]` being the row length in bytes. The data is stored continuously if `isContinuous()` returns true, meaning there are no gaps between rows. This allows for more efficient processing. # Array Operations Source: https://opencv-opencv.mintlify.app/api/core/operations Per-element operations, matrix arithmetic, and array manipulation functions ## Arithmetic Operations ### add ```cpp theme={null} void add(InputArray src1, InputArray src2, OutputArray dst, InputArray mask = noArray(), int dtype = -1) ``` Calculates the per-element sum of two arrays or an array and a scalar. First input array or scalar Second input array or scalar Output array with the same size and number of channels as input arrays Optional 8-bit single channel mask that specifies elements to be changed Optional depth of the output array. Default is -1 (same as input) **Formula:** ``` dst(I) = saturate(src1(I) + src2(I)) if mask(I) ≠ 0 ``` **Example:** ```cpp theme={null} Mat A(100, 100, CV_8UC1, Scalar(50)); Mat B(100, 100, CV_8UC1, Scalar(100)); Mat C; add(A, B, C); // C = A + B // Equivalent matrix expression C = A + B; ``` Saturation is not applied when the output array has depth CV\_32S. You may get incorrect sign in case of overflow. ### subtract ```cpp theme={null} void subtract(InputArray src1, InputArray src2, OutputArray dst, InputArray mask = noArray(), int dtype = -1) ``` Calculates the per-element difference between two arrays or array and a scalar. First input array or scalar Second input array or scalar Output array Optional operation mask Optional depth of the output array **Formula:** ``` dst(I) = saturate(src1(I) - src2(I)) if mask(I) ≠ 0 ``` **Example:** ```cpp theme={null} Mat result; subtract(imageA, imageB, result); // Equivalent: result = imageA - imageB; ``` ### multiply ```cpp theme={null} void multiply(InputArray src1, InputArray src2, OutputArray dst, double scale = 1, int dtype = -1) ``` Calculates the per-element scaled product of two arrays. First input array Second input array of the same size and type as src1 Output array Optional scale factor Optional depth of the output array **Formula:** ``` dst(I) = saturate(scale · src1(I) · src2(I)) ``` **Example:** ```cpp theme={null} Mat A, B, C; multiply(A, B, C, 2.5); // Element-wise multiplication with scaling ``` For matrix multiplication (not element-wise), use gemm() function. ### divide ```cpp theme={null} void divide(InputArray src1, InputArray src2, OutputArray dst, double scale = 1, int dtype = -1) void divide(double scale, InputArray src2, OutputArray dst, int dtype = -1) ``` Performs per-element division of two arrays or a scalar by an array. First input array (numerator) Second input array (denominator) Scalar factor Output array **Formula:** ``` dst(I) = saturate(src1(I) * scale / src2(I)) dst(I) = saturate(scale / src2(I)) // Second overload ``` **Example:** ```cpp theme={null} Mat A, B, C; divide(A, B, C); // Element-wise division divide(255.0, B, C); // Scalar divided by array ``` For integer types, when src2(I) is zero, dst(I) will also be zero. For floating-point data, expect IEEE-754 behavior (NaN, Inf values). ### scaleAdd ```cpp theme={null} void scaleAdd(InputArray src1, double alpha, InputArray src2, OutputArray dst) ``` Calculates the sum of a scaled array and another array (SAXPY/DAXPY operation). First input array Scale factor for the first array Second input array (same size and type as src1) Output array **Formula:** ``` dst(I) = scale · src1(I) + src2(I) ``` **Example:** ```cpp theme={null} Mat A, B, C; scaleAdd(A, 2.0, B, C); // C = 2*A + B ``` ### addWeighted ```cpp theme={null} void addWeighted(InputArray src1, double alpha, InputArray src2, double beta, double gamma, OutputArray dst, int dtype = -1) ``` Calculates the weighted sum of two arrays. First input array Weight of the first array elements Second input array Weight of the second array elements Scalar added to each sum Output array **Formula:** ``` dst(I) = saturate(src1(I) · alpha + src2(I) · beta + gamma) ``` **Example:** ```cpp theme={null} // Image blending Mat img1, img2, blended; addWeighted(img1, 0.7, img2, 0.3, 0.0, blended); ``` ## Bitwise Operations ### bitwise\_and ```cpp theme={null} void bitwise_and(InputArray src1, InputArray src2, OutputArray dst, InputArray mask = noArray()) ``` Calculates per-element bit-wise conjunction of two arrays or array and a scalar. First input array or scalar Second input array or scalar Output array Optional operation mask **Formula:** ``` dst(I) = src1(I) & src2(I) if mask(I) ≠ 0 ``` ### bitwise\_or ```cpp theme={null} void bitwise_or(InputArray src1, InputArray src2, OutputArray dst, InputArray mask = noArray()) ``` Calculates per-element bit-wise disjunction of two arrays or array and a scalar. **Formula:** ``` dst(I) = src1(I) | src2(I) if mask(I) ≠ 0 ``` ### bitwise\_xor ```cpp theme={null} void bitwise_xor(InputArray src1, InputArray src2, OutputArray dst, InputArray mask = noArray()) ``` Calculates per-element bit-wise exclusive OR of two arrays or array and a scalar. **Formula:** ``` dst(I) = src1(I) ^ src2(I) if mask(I) ≠ 0 ``` ### bitwise\_not ```cpp theme={null} void bitwise_not(InputArray src, OutputArray dst, InputArray mask = noArray()) ``` Inverts every bit of an array. Input array Output array **Formula:** ``` dst(I) = ~src(I) if mask(I) ≠ 0 ``` **Example:** ```cpp theme={null} Mat mask, inverted_mask; bitwise_not(mask, inverted_mask); ``` ## Comparison Operations ### compare ```cpp theme={null} void compare(InputArray src1, InputArray src2, OutputArray dst, int cmpop) ``` Performs per-element comparison of two arrays or array and scalar. First input array or scalar Second input array or scalar Output array of type CV\_8U (0 or 255 values) Comparison operation: CMP\_EQ, CMP\_GT, CMP\_GE, CMP\_LT, CMP\_LE, CMP\_NE **Example:** ```cpp theme={null} Mat A, B, mask; compare(A, B, mask, CMP_GT); // mask = (A > B) ``` ### min ```cpp theme={null} void min(InputArray src1, InputArray src2, OutputArray dst) ``` Calculates per-element minimum of two arrays or array and scalar. **Example:** ```cpp theme={null} Mat A, B, C; min(A, B, C); // C = min(A, B) element-wise ``` ### max ```cpp theme={null} void max(InputArray src1, InputArray src2, OutputArray dst) ``` Calculates per-element maximum of two arrays or array and scalar. **Example:** ```cpp theme={null} Mat A, B, C; max(A, B, C); // C = max(A, B) element-wise ``` ### inRange ```cpp theme={null} void inRange(InputArray src, InputArray lowerb, InputArray upperb, OutputArray dst) ``` Checks if array elements lie between two bounds. Input array Inclusive lower boundary array or scalar Inclusive upper boundary array or scalar Output array of type CV\_8U (0 or 255) **Formula:** ``` dst(I) = 255 if lowerb(I) ≤ src(I) ≤ upperb(I), else 0 ``` **Example:** ```cpp theme={null} // Color segmentation Mat hsv, mask; cvtColor(image, hsv, COLOR_BGR2HSV); inRange(hsv, Scalar(100, 50, 50), Scalar(130, 255, 255), mask); ``` ## Mathematical Operations ### sqrt ```cpp theme={null} void sqrt(InputArray src, OutputArray dst) ``` Calculates square root of array elements. **Formula:** ``` dst(I) = √src(I) ``` ### pow ```cpp theme={null} void pow(InputArray src, double power, OutputArray dst) ``` Raises every array element to a power. Input array Exponent of power Output array **Formula:** ``` dst(I) = src(I)^power ``` **Example:** ```cpp theme={null} Mat A, B; pow(A, 2.0, B); // Square each element ``` ### exp ```cpp theme={null} void exp(InputArray src, OutputArray dst) ``` Calculates the exponent of every array element. **Formula:** ``` dst(I) = e^src(I) ``` ### log ```cpp theme={null} void log(InputArray src, OutputArray dst) ``` Calculates the natural logarithm of every array element. **Formula:** ``` dst(I) = log(|src(I)|) ``` **Example:** ```cpp theme={null} Mat A, logA; log(A, logA); ``` ### absdiff ```cpp theme={null} void absdiff(InputArray src1, InputArray src2, OutputArray dst) ``` Calculates per-element absolute difference between two arrays or array and scalar. **Formula:** ``` dst(I) = |src1(I) - src2(I)| ``` **Example:** ```cpp theme={null} // Frame difference for motion detection Mat frame1, frame2, diff; absdiff(frame1, frame2, diff); ``` ### magnitude ```cpp theme={null} void magnitude(InputArray x, InputArray y, OutputArray magnitude) ``` Calculates magnitude of 2D vectors. Floating-point array of x-coordinates Floating-point array of y-coordinates (same size as x) Output array of magnitudes **Formula:** ``` magnitude(I) = √(x(I)² + y(I)²) ``` ### phase ```cpp theme={null} void phase(InputArray x, InputArray y, OutputArray angle, bool angleInDegrees = false) ``` Calculates the rotation angle of 2D vectors. Array of x-coordinates (floating-point) Array of y-coordinates (same size and type as x) Output array of angles When true, angles are in degrees (0-360), otherwise radians (0-2π) **Formula:** ``` angle(I) = atan2(y(I), x(I)) ``` ### cartToPolar ```cpp theme={null} void cartToPolar(InputArray x, InputArray y, OutputArray magnitude, OutputArray angle, bool angleInDegrees = false) ``` Calculates the magnitude and angle of 2D vectors. **Example:** ```cpp theme={null} Mat dx, dy, magnitude, angle; // ... compute gradients dx, dy ... cartToPolar(dx, dy, magnitude, angle, true); ``` ### polarToCart ```cpp theme={null} void polarToCart(InputArray magnitude, InputArray angle, OutputArray x, OutputArray y, bool angleInDegrees = false) ``` Converts polar coordinates to Cartesian. Array of vector magnitudes Array of vector angles Output array of x-coordinates Output array of y-coordinates ## Matrix Operations ### gemm ```cpp theme={null} void gemm(InputArray src1, InputArray src2, double alpha, InputArray src3, double beta, OutputArray dst, int flags = 0) ``` Performs generalized matrix multiplication (GEMM). First input matrix Second input matrix Weight for src1 \* src2 Third input matrix (added to product) Weight for src3 Output matrix Operation flags: GEMM\_1\_T (transpose src1), GEMM\_2\_T (transpose src2), GEMM\_3\_T (transpose src3) **Formula:** ``` dst = alpha · src1 · src2 + beta · src3 ``` **Example:** ```cpp theme={null} Mat A, B, C, D; gemm(A, B, 1.0, C, 1.0, D); // D = A*B + C gemm(A, B, 2.0, C, 0.5, D, GEMM_1_T); // D = 2*A^T*B + 0.5*C ``` ### transpose ```cpp theme={null} void transpose(InputArray src, OutputArray dst) ``` Transposes a matrix. **Formula:** ``` dst(i,j) = src(j,i) ``` **Example:** ```cpp theme={null} Mat A, AT; transpose(A, AT); ``` ### transform ```cpp theme={null} void transform(InputArray src, OutputArray dst, InputArray m) ``` Performs matrix transformation of every array element. Input array (must be continuous) Output array Transformation matrix (2x2, 2x3, 3x3, or 3x4 for 2D, 3x3 or 3x4 for 3D) **Example:** ```cpp theme={null} // Rotate all 2D points by 45 degrees std::vector points, rotated; Mat R = getRotationMatrix2D(Point2f(0,0), 45, 1.0); transform(points, rotated, R); ``` ### mulTransposed ```cpp theme={null} void mulTransposed(InputArray src, OutputArray dst, bool aTa, InputArray delta = noArray(), double scale = 1, int dtype = -1) ``` Calculates the product of a matrix and its transposition. Input matrix Output square matrix If true, computes src^T \* src; if false, computes src \* src^T Optional delta matrix subtracted from src before multiplication Optional scale factor for the matrix product **Formula:** ``` dst = scale · (src - delta)^T · (src - delta) // if aTa=true dst = scale · (src - delta) · (src - delta)^T // if aTa=false ``` ## Array Manipulation ### flip ```cpp theme={null} void flip(InputArray src, OutputArray dst, int flipCode) ``` Flips a 2D array around vertical, horizontal, or both axes. Input array Output array Flip type: 0 (vertical flip), positive (horizontal flip), negative (both axes) **Example:** ```cpp theme={null} Mat img, flipped; flip(img, flipped, 1); // Horizontal flip flip(img, flipped, 0); // Vertical flip flip(img, flipped, -1); // Both axes ``` ### rotate ```cpp theme={null} void rotate(InputArray src, OutputArray dst, int rotateCode) ``` Rotates a 2D array in multiples of 90 degrees. Rotation type: ROTATE\_90\_CLOCKWISE, ROTATE\_180, ROTATE\_90\_COUNTERCLOCKWISE **Example:** ```cpp theme={null} Mat img, rotated; rotate(img, rotated, ROTATE_90_CLOCKWISE); ``` ### repeat ```cpp theme={null} void repeat(InputArray src, int ny, int nx, OutputArray dst) Mat repeat(const Mat& src, int ny, int nx) ``` Fills the output array with repeated copies of the input array. Number of times to repeat along the vertical axis Number of times to repeat along the horizontal axis **Example:** ```cpp theme={null} Mat pattern(2, 2, CV_8U, Scalar(255)); Mat tiled = repeat(pattern, 5, 5); // Create 10x10 tiled pattern ``` ### hconcat ```cpp theme={null} void hconcat(InputArray src1, InputArray src2, OutputArray dst) void hconcat(InputArrayOfArrays src, OutputArray dst) ``` Concatenates arrays horizontally (along columns). **Example:** ```cpp theme={null} Mat A, B, C; hconcat(A, B, C); // Concatenate side by side ``` ### vconcat ```cpp theme={null} void vconcat(InputArray src1, InputArray src2, OutputArray dst) void vconcat(InputArrayOfArrays src, OutputArray dst) ``` Concatenates arrays vertically (along rows). **Example:** ```cpp theme={null} Mat A, B, C; vconcat(A, B, C); // Stack vertically ``` ## Channel Operations ### split ```cpp theme={null} void split(InputArray src, OutputArrayOfArrays mv) ``` Divides a multi-channel array into several single-channel arrays. Input multi-channel array Output vector of arrays **Example:** ```cpp theme={null} Mat bgr, channels[3]; split(bgr, channels); // channels[0] = blue, channels[1] = green, channels[2] = red ``` ### merge ```cpp theme={null} void merge(InputArrayOfArrays mv, OutputArray dst) ``` Merges several arrays to make a multi-channel array. **Example:** ```cpp theme={null} std::vector channels; Mat bgr; merge(channels, bgr); ``` ### mixChannels ```cpp theme={null} void mixChannels(InputArrayOfArrays src, InputOutputArrayOfArrays dst, const int* fromTo, size_t npairs) void mixChannels(InputArrayOfArrays src, InputOutputArrayOfArrays dst, const std::vector& fromTo) ``` Copies specified channels from input arrays to specified channels of output arrays. Array of index pairs: fromTo\[k*2] is source channel, fromTo\[k*2+1] is destination channel **Example:** ```cpp theme={null} Mat bgr, bgra; // Copy BGR to BGRA and set alpha to 255 int from_to[] = {0,0, 1,1, 2,2, -1,3}; mixChannels(&bgr, 1, &bgra, 1, from_to, 4); ``` Channel indexing starts from 0. Use -1 to set a channel to zero. # Utility Functions Source: https://opencv-opencv.mintlify.app/api/core/utilities Statistical functions, reduction operations, and utility helpers for array processing ## Statistical Functions ### sum ```cpp theme={null} Scalar sum(InputArray src) ``` Calculates the sum of array elements. Input array (1 to 4 channels) Sum of all array elements for each channel **Example:** ```cpp theme={null} Mat img(100, 100, CV_8UC3); Scalar total = sum(img); // Returns sum for each channel double blueSum = total[0]; double greenSum = total[1]; double redSum = total[2]; ``` ### mean ```cpp theme={null} Scalar mean(InputArray src, InputArray mask = noArray()) ``` Calculates the mean value of array elements. Input array (1 to 4 channels) Optional operation mask (8-bit single channel) Mean value for each channel **Example:** ```cpp theme={null} Mat img; Scalar avgColor = mean(img); // Average color Mat mask; Scalar avgInRegion = mean(img, mask); // Average within masked region ``` ### meanStdDev ```cpp theme={null} void meanStdDev(InputArray src, OutputArray mean, OutputArray stddev, InputArray mask = noArray()) ``` Calculates mean and standard deviation of array elements. Input array (1 to 4 channels) Output parameter: calculated mean value Output parameter: calculated standard deviation Optional operation mask **Example:** ```cpp theme={null} Mat img; Mat mean, stddev; meanStdDev(img, mean, stddev); std::cout << "Mean: " << mean << std::endl; std::cout << "Std Dev: " << stddev << std::endl; ``` ### minMaxLoc ```cpp theme={null} void minMaxLoc(InputArray src, double* minVal, double* maxVal = 0, Point* minLoc = 0, Point* maxLoc = 0, InputArray mask = noArray()) ``` Finds the global minimum and maximum in an array. Input single-channel array Pointer to returned minimum value (can be NULL) Pointer to returned maximum value (can be NULL) Pointer to returned minimum location (can be NULL) Pointer to returned maximum location (can be NULL) Optional mask to select a sub-array **Example:** ```cpp theme={null} Mat img; double minVal, maxVal; Point minLoc, maxLoc; minMaxLoc(img, &minVal, &maxVal, &minLoc, &maxLoc); std::cout << "Min: " << minVal << " at " << minLoc << std::endl; std::cout << "Max: " << maxVal << " at " << maxLoc << std::endl; ``` ### norm ```cpp theme={null} double norm(InputArray src1, int normType = NORM_L2, InputArray mask = noArray()) double norm(InputArray src1, InputArray src2, int normType = NORM_L2, InputArray mask = noArray()) ``` Calculates an absolute array norm, absolute difference norm, or relative difference norm. First input array Second input array (for difference norms) Type of norm: NORM\_INF, NORM\_L1, NORM\_L2, NORM\_L2SQR, NORM\_HAMMING, NORM\_HAMMING2 Optional operation mask Calculated norm value **Norm Types:** * `NORM_INF`: max(|x\_i|) * `NORM_L1`: Σ|x\_i| * `NORM_L2`: √(Σx\_i²) * `NORM_L2SQR`: Σx\_i² **Example:** ```cpp theme={null} Mat vec1, vec2; double l2norm = norm(vec1, NORM_L2); double diff = norm(vec1, vec2, NORM_L2); // Euclidean distance ``` ### normalize ```cpp theme={null} void normalize(InputArray src, InputOutputArray dst, double alpha = 1, double beta = 0, int norm_type = NORM_L2, int dtype = -1, InputArray mask = noArray()) ``` Normalizes the norm or value range of an array. Input array Output array (same size as src) Norm value to normalize to or lower range boundary in range normalization Upper range boundary in range normalization (not used for norm normalization) Normalization type: NORM\_INF, NORM\_L1, NORM\_L2, or NORM\_MINMAX Optional depth of output array Optional operation mask **Example:** ```cpp theme={null} Mat src, dst; // Normalize to range [0, 255] normalize(src, dst, 0, 255, NORM_MINMAX, CV_8U); // Normalize to unit norm normalize(src, dst, 1.0, 0, NORM_L2); ``` ### countNonZero ```cpp theme={null} int countNonZero(InputArray src) ``` Counts non-zero array elements. Single-channel array Number of non-zero elements **Example:** ```cpp theme={null} Mat binary; threshold(img, binary, 128, 255, THRESH_BINARY); int whitePixels = countNonZero(binary); ``` ### hasNonZero ```cpp theme={null} bool hasNonZero(InputArray src) ``` Checks if there are any non-zero elements in array. True if at least one non-zero element exists **Example:** ```cpp theme={null} Mat mask; if (hasNonZero(mask)) { // Process masked region } ``` ### findNonZero ```cpp theme={null} void findNonZero(InputArray src, OutputArray idx) ``` Returns the list of locations of non-zero pixels. Single-channel array (8-bit or floating-point) Output array of Point locations (N×1 or 1×N) **Example:** ```cpp theme={null} Mat binary; threshold(img, binary, 128, 255, THRESH_BINARY); std::vector locations; findNonZero(binary, locations); for(const Point& pt : locations) { // Process each non-zero pixel location } ``` ## Reduction Operations ### reduce ```cpp theme={null} void reduce(InputArray src, OutputArray dst, int dim, int rtype, int dtype = -1) ``` Reduces a matrix to a vector by applying an operation along a specified dimension. Input array Output vector Dimension to reduce: 0 (reduce to single row), 1 (reduce to single column) Reduction operation: REDUCE\_SUM, REDUCE\_AVG, REDUCE\_MAX, REDUCE\_MIN, REDUCE\_SUM2 Optional depth of output array **Operations:** * `REDUCE_SUM`: Sum of all rows/columns * `REDUCE_AVG`: Mean of all rows/columns * `REDUCE_MAX`: Maximum of all rows/columns * `REDUCE_MIN`: Minimum of all rows/columns * `REDUCE_SUM2`: Sum of squared values **Example:** ```cpp theme={null} Mat matrix(100, 100, CV_32F); Mat columnSums, rowMeans; reduce(matrix, columnSums, 0, REDUCE_SUM); // Sum each column reduce(matrix, rowMeans, 1, REDUCE_AVG); // Mean of each row ``` ### reduceArgMin ```cpp theme={null} void reduceArgMin(InputArray src, OutputArray dst, int axis, bool lastIndex = false) ``` Finds indices of minimum elements along specified axis. Input array Output array of indices Dimension to reduce along Whether to return last index in case of multiple minimum values **Example:** ```cpp theme={null} Mat data, minIndices; reduceArgMin(data, minIndices, 1); // Index of min in each row ``` ### reduceArgMax ```cpp theme={null} void reduceArgMax(InputArray src, OutputArray dst, int axis, bool lastIndex = false) ``` Finds indices of maximum elements along specified axis. **Example:** ```cpp theme={null} Mat data, maxIndices; reduceArgMax(data, maxIndices, 0); // Index of max in each column ``` ## Sorting ### sort ```cpp theme={null} void sort(InputArray src, OutputArray dst, int flags) ``` Sorts each matrix row or column in ascending or descending order. Input single-channel array Output array (same size and type as src) Operation flags: SORT\_EVERY\_ROW, SORT\_EVERY\_COLUMN, SORT\_ASCENDING, SORT\_DESCENDING **Example:** ```cpp theme={null} Mat data, sorted; sort(data, sorted, SORT_EVERY_ROW | SORT_ASCENDING); ``` ### sortIdx ```cpp theme={null} void sortIdx(InputArray src, OutputArray dst, int flags) ``` Sorts each matrix row or column and returns sorted indices instead of values. Output integer array of sorted indices **Example:** ```cpp theme={null} Mat values, indices; sortIdx(values, indices, SORT_EVERY_ROW | SORT_DESCENDING); ``` ## Linear Algebra ### determinant ```cpp theme={null} double determinant(InputArray mtx) ``` Returns the determinant of a square matrix. Input matrix (must be square) Determinant value **Example:** ```cpp theme={null} Mat A(3, 3, CV_64F); double det = determinant(A); ``` ### trace ```cpp theme={null} Scalar trace(InputArray mtx) ``` Returns the trace (sum of diagonal elements) of a matrix. Input matrix Trace of the matrix **Formula:** ``` trace(A) = Σ A[i,i] ``` **Example:** ```cpp theme={null} Mat A; Scalar tr = trace(A); ``` ### invert ```cpp theme={null} double invert(InputArray src, OutputArray dst, int flags = DECOMP_LU) ``` Finds the inverse or pseudo-inverse of a matrix. Input floating-point matrix Output matrix of the same size and type as src Inversion method: DECOMP\_LU, DECOMP\_SVD, DECOMP\_CHOLESKY Reciprocal condition number (for SVD) or 0 if singular **Decomposition Methods:** * `DECOMP_LU`: LU decomposition (fastest for well-conditioned matrices) * `DECOMP_SVD`: Singular value decomposition (works for singular matrices) * `DECOMP_CHOLESKY`: Cholesky decomposition (for symmetric positive-definite matrices) **Example:** ```cpp theme={null} Mat A, invA; double rcond = invert(A, invA, DECOMP_SVD); if (rcond > 1e-6) { // Matrix is well-conditioned } ``` ### solve ```cpp theme={null} bool solve(InputArray src1, InputArray src2, OutputArray dst, int flags = DECOMP_LU) ``` Solves one or more linear systems or least-squares problems. Coefficient matrix (A in Ax=b) Right-hand side matrix (b in Ax=b) Output solution (x in Ax=b) Solution method: DECOMP\_LU, DECOMP\_SVD, DECOMP\_CHOLESKY, DECOMP\_QR, DECOMP\_NORMAL True if solution exists **Example:** ```cpp theme={null} Mat A, b, x; // Solve Ax = b if (solve(A, b, x, DECOMP_LU)) { std::cout << "Solution: " << x << std::endl; } ``` ### eigen ```cpp theme={null} bool eigen(InputArray src, OutputArray eigenvalues, OutputArray eigenvectors = noArray()) ``` Calculates eigenvalues and eigenvectors of a symmetric matrix. Input symmetric square matrix Output vector of eigenvalues (in descending order) Output matrix of eigenvectors (one per row) True if successful **Example:** ```cpp theme={null} Mat covar, eigenvalues, eigenvectors; calcCovarMatrix(samples, covar, mean, COVAR_NORMAL | COVAR_ROWS); eigen(covar, eigenvalues, eigenvectors); // First eigenvector (principal component) Mat pc1 = eigenvectors.row(0); ``` This function is optimized for symmetric matrices. For general matrices, use eigenNonSymmetric(). ### calcCovarMatrix ```cpp theme={null} void calcCovarMatrix(InputArray samples, OutputArray covar, InputOutputArray mean, int flags, int ctype = CV_64F) ``` Calculates covariance matrix of a set of vectors. Input samples (each row or column is a sample) Output covariance matrix Input or output mean vector Operation flags: COVAR\_SCRAMBLED, COVAR\_NORMAL, COVAR\_USE\_AVG, COVAR\_SCALE, COVAR\_ROWS, COVAR\_COLS Type of output matrices (CV\_32F or CV\_64F) **Example:** ```cpp theme={null} Mat samples(1000, 5, CV_32F); // 1000 samples, 5 dimensions Mat covar, mean; calcCovarMatrix(samples, covar, mean, COVAR_NORMAL | COVAR_ROWS | COVAR_SCALE, CV_32F); ``` ## Timing and Profiling ### getTickCount ```cpp theme={null} int64 getTickCount() ``` Returns the number of ticks since a certain event (e.g., machine startup). Current tick count **Example:** ```cpp theme={null} int64 t1 = getTickCount(); // ... perform operation ... int64 t2 = getTickCount(); double time = (t2 - t1) / getTickFrequency(); std::cout << "Time: " << time << " seconds" << std::endl; ``` ### getTickFrequency ```cpp theme={null} double getTickFrequency() ``` Returns the number of ticks per second. Tick frequency in Hz ### TickMeter ```cpp theme={null} class TickMeter { public: void start(); void stop(); void reset(); double getTimeSec() const; double getTimeMilli() const; double getTimeMicro() const; int64 getCounter() const; double getFPS() const; }; ``` A class to measure passing time and calculate performance metrics. **Example:** ```cpp theme={null} cv::TickMeter tm; tm.start(); // ... perform operation ... tm.stop(); std::cout << "Time: " << tm.getTimeMilli() << " ms" << std::endl; std::cout << "FPS: " << tm.getFPS() << std::endl; ``` ## System Information ### getNumberOfCPUs ```cpp theme={null} int getNumberOfCPUs() ``` Returns the number of logical CPUs available for the process. ### setNumThreads ```cpp theme={null} void setNumThreads(int nthreads) ``` Sets the number of threads used by OpenCV for parallel regions. Number of threads. Use 0 or negative values to reset to default ### getNumThreads ```cpp theme={null} int getNumThreads() ``` Returns the number of threads used by OpenCV for parallel regions. ### getBuildInformation ```cpp theme={null} String getBuildInformation() ``` Returns full configuration time cmake output including version, compiler, enabled modules, etc. **Example:** ```cpp theme={null} std::cout << getBuildInformation() << std::endl; ``` ### getVersionString ```cpp theme={null} String getVersionString() ``` Returns library version string (e.g., "4.8.0"). ### getCPUFeaturesLine ```cpp theme={null} std::string getCPUFeaturesLine() ``` Returns a string containing CPU features enabled during compilation. **Example output:** ``` SSE SSE2 SSE3 *SSE4.1 *SSE4.2 *AVX *AVX2 ``` * No marker: baseline features * `*`: features enabled in dispatcher * `?`: features enabled but not available in hardware ## Utility Functions ### setUseOptimized ```cpp theme={null} void setUseOptimized(bool onoff) ``` Enables or disables optimized code (SSE, AVX, etc.). True to enable optimizations, false to disable ### useOptimized ```cpp theme={null} bool useOptimized() ``` Returns the status of optimized code usage. ### checkRange ```cpp theme={null} bool checkRange(InputArray a, bool quiet = true, Point* pos = 0, double minVal = -DBL_MAX, double maxVal = DBL_MAX) ``` Checks every element of an input array for invalid values. Input array If true, doesn't throw exceptions on invalid values Optional output parameter for position of first invalid value Minimum valid value (inclusive) Maximum valid value (inclusive) True if all elements are within range and not NaN/Inf **Example:** ```cpp theme={null} Mat result; // ... computation ... if (!checkRange(result)) { std::cerr << "Invalid values detected (NaN or Inf)" << std::endl; } ``` ### patchNaNs ```cpp theme={null} void patchNaNs(InputOutputArray a, double val = 0) ``` Replaces all NaN values in an array with specified value. Input/output floating-point array Value to replace NaNs with **Example:** ```cpp theme={null} Mat data; // ... computation that might produce NaNs ... patchNaNs(data, 0.0); // Replace NaNs with zeros ``` ### LUT ```cpp theme={null} void LUT(InputArray src, InputArray lut, OutputArray dst) ``` Performs a look-up table transform of an array. Input array (8-bit elements) Look-up table (256 elements) Output array (same size as src) **Example:** ```cpp theme={null} // Create a gamma correction LUT Mat lut(1, 256, CV_8U); for(int i = 0; i < 256; i++) lut.at(i) = saturate_cast(pow(i / 255.0, 0.5) * 255.0); Mat img, corrected; LUT(img, lut, corrected); // Apply gamma correction ``` ### convertScaleAbs ```cpp theme={null} void convertScaleAbs(InputArray src, OutputArray dst, double alpha = 1, double beta = 0) ``` Scales, calculates absolute values, and converts to 8-bit unsigned type. Input array Output array (CV\_8U type) Scale factor Delta added to scaled values **Formula:** ``` dst(I) = saturate_cast(|src(I) · alpha + beta|) ``` **Example:** ```cpp theme={null} Mat gradient_x, abs_gradient; Sobel(img, gradient_x, CV_16S, 1, 0); convertScaleAbs(gradient_x, abs_gradient); ``` ### PSNR ```cpp theme={null} double PSNR(InputArray src1, InputArray src2, double R = 255.0) ``` Computes Peak Signal-to-Noise Ratio (PSNR) between two images. First input array Second input array (same size and type as src1) Maximum pixel value (255.0 for 8-bit images) PSNR value in decibels (dB) **Example:** ```cpp theme={null} Mat original, compressed; double psnr = PSNR(original, compressed); std::cout << "PSNR: " << psnr << " dB" << std::endl; ``` Higher PSNR values indicate better quality. Typical values range from 20 to 50 dB, with 30-50 dB being good quality. # DNN Inference Utilities Source: https://opencv-opencv.mintlify.app/api/dnn/inference Helper functions and utilities for neural network inference ## Overview OpenCV DNN provides utility functions for preparing inputs, processing outputs, and managing inference workflows. ## Blob Creation ### blobFromImage Convert single image to 4D blob: ```cpp theme={null} Mat blobFromImage( InputArray image, double scalefactor = 1.0, const Size& size = Size(), const Scalar& mean = Scalar(), bool swapRB = false, bool crop = false, int ddepth = CV_32F ); ``` **Parameters**: * `image`: Input image (any size, any channels) * `scalefactor`: Multiplier for pixel values * `size`: Target spatial dimensions * `mean`: Values to subtract from channels * `swapRB`: Swap red and blue channels (BGR to RGB) * `crop`: Crop image after resize * `ddepth`: Output depth (typically CV\_32F) **Example**: ```cpp theme={null} Mat img = imread("image.jpg"); // Basic usage Mat blob = blobFromImage(img, 1.0/255, Size(224, 224)); // With mean subtraction Mat blob = blobFromImage( img, 1.0, Size(224, 224), Scalar(104, 117, 123), // ImageNet mean true, // BGR to RGB false // No crop ); // Output shape: [1, 3, 224, 224] ``` ### blobFromImages Convert multiple images to single blob (batching): ```cpp theme={null} Mat blobFromImages( InputArrayOfArrays images, double scalefactor = 1.0, Size size = Size(), const Scalar& mean = Scalar(), bool swapRB = false, bool crop = false, int ddepth = CV_32F ); ``` **Example**: ```cpp theme={null} std::vector images; images.push_back(imread("img1.jpg")); images.push_back(imread("img2.jpg")); images.push_back(imread("img3.jpg")); Mat blob = blobFromImages( images, 1.0/255, Size(224, 224), Scalar(), true ); // Output shape: [3, 3, 224, 224] // batch_size=3, channels=3, height=224, width=224 ``` ### imagesFromBlob Convert blob back to images: ```cpp theme={null} void imagesFromBlob( const Mat& blob, OutputArrayOfArrays images ); ``` **Example**: ```cpp theme={null} std::vector images; imagesFromBlob(blob, images); for(const Mat& img : images) { imshow("Image", img); waitKey(0); } ``` ## Blob Utilities ### getPlane Extract single plane from blob: ```cpp theme={null} Mat getPlane(const Mat& m, int n, int cn); ``` **Parameters**: * `m`: 4D blob \[N, C, H, W] * `n`: Batch index * `cn`: Channel index **Example**: ```cpp theme={null} // Extract first channel of first image Mat plane = getPlane(blob, 0, 0); ``` ### getMatFromNet Get layer activations: ```cpp theme={null} Mat net.getParam(const String& layer, int paramId); ``` ## NMS (Non-Maximum Suppression) ### NMSBoxes Filter overlapping bounding boxes: ```cpp theme={null} void NMSBoxes( const std::vector& bboxes, const std::vector& scores, const float score_threshold, const float nms_threshold, std::vector& indices, const float eta = 1.f, const int top_k = 0 ); ``` **Parameters**: * `bboxes`: Bounding boxes * `scores`: Confidence scores * `score_threshold`: Minimum score to keep * `nms_threshold`: IoU threshold (typically 0.4-0.5) * `indices`: Output indices of kept boxes * `eta`: Adaptive NMS parameter * `top_k`: Keep top K boxes (0 = all) **Example**: ```cpp theme={null} std::vector boxes = {/* detected boxes */}; std::vector confidences = {/* scores */}; std::vector indices; NMSBoxes( boxes, confidences, 0.5, // score_threshold 0.4, // nms_threshold indices ); // Draw kept boxes for(int idx : indices) { rectangle(img, boxes[idx], Scalar(0, 255, 0), 2); } ``` ### NMSBoxesBatched NMS for batched detections: ```cpp theme={null} void NMSBoxesBatched( const std::vector& bboxes, const std::vector& scores, const std::vector& class_ids, const std::vector& batch_ids, const float score_threshold, const float nms_threshold, std::vector& indices, const float eta = 1.f, const int top_k = 0 ); ``` ## Softmax ### softmax Apply softmax activation: ```cpp theme={null} Mat softmax(const Mat& src); void softmax( InputArray src, OutputArray dst, int axis = 1 ); ``` **Example**: ```cpp theme={null} Mat logits = net.forward(); Mat probs; softmax(logits, probs, 1); // Get top class Point classIdPoint; minMaxLoc(probs.reshape(1, 1), 0, 0, 0, &classIdPoint); int classId = classIdPoint.x; ``` ## Backend Queries ### getAvailableBackends Query available backends: ```cpp theme={null} std::vector> getAvailableBackends(); ``` **Example**: ```cpp theme={null} auto backends = getAvailableBackends(); for(auto& pair : backends) { std::cout << "Backend: " << pair.first << ", Target: " << pair.second << std::endl; } ``` ### getAvailableTargets Query targets for backend: ```cpp theme={null} std::vector getAvailableTargets(Backend be); ``` **Example**: ```cpp theme={null} auto targets = getAvailableTargets(DNN_BACKEND_CUDA); for(Target t : targets) { std::cout << "Target: " << t << std::endl; } ``` ## Model Diagnostics ### enableModelDiagnostics Enable verbose model loading: ```cpp theme={null} void enableModelDiagnostics(bool isDiagnosticsMode); ``` **Example**: ```cpp theme={null} enableModelDiagnostics(true); Net net = readNet("model.onnx"); // Prints detailed info ``` ## Complete Examples ### Image Classification ```cpp theme={null} #include #include #include using namespace cv; using namespace cv::dnn; int main() { // Load model Net net = readNet("model.onnx"); net.setPreferableBackend(DNN_BACKEND_CUDA); net.setPreferableTarget(DNN_TARGET_CUDA); // Load image Mat img = imread("image.jpg"); // Create blob Mat blob = blobFromImage( img, 1.0/255.0, Size(224, 224), Scalar(0.485, 0.456, 0.406) * 255, // ImageNet mean true, // swapRB false // crop ); // Inference net.setInput(blob); Mat output = net.forward(); // Softmax Mat probs; softmax(output, probs, 1); // Get top-5 predictions Mat flat = probs.reshape(1, 1); Mat sorted; sortIdx(flat, sorted, SORT_EVERY_ROW | SORT_DESCENDING); std::cout << "Top 5 predictions:\n"; for(int i = 0; i < 5; i++) { int classId = sorted.at(i); float prob = flat.at(classId); std::cout << i+1 << ". Class " << classId << ": " << prob*100 << "%\n"; } return 0; } ``` ### Object Detection (YOLO) ```cpp theme={null} #include #include #include using namespace cv; using namespace cv::dnn; int main() { // Load YOLO Net net = readNetFromDarknet("yolov4.cfg", "yolov4.weights"); // Load image Mat img = imread("image.jpg"); // Create blob Mat blob = blobFromImage(img, 1/255.0, Size(416, 416), Scalar(), true, false); // Inference net.setInput(blob); std::vector outputs; net.forward(outputs, net.getUnconnectedOutLayersNames()); // Process detections std::vector boxes; std::vector confidences; std::vector classIds; for(const Mat& output : outputs) { for(int i = 0; i < output.rows; i++) { const float* data = output.ptr(i); float confidence = data[4]; if(confidence > 0.5) { Mat scores = output.row(i).colRange(5, output.cols); Point classIdPoint; double maxScore; minMaxLoc(scores, 0, &maxScore, 0, &classIdPoint); if(maxScore > 0.5) { int centerX = data[0] * img.cols; int centerY = data[1] * img.rows; int width = data[2] * img.cols; int height = data[3] * img.rows; boxes.push_back(Rect( centerX - width/2, centerY - height/2, width, height )); confidences.push_back(confidence); classIds.push_back(classIdPoint.x); } } } } // NMS std::vector indices; NMSBoxes(boxes, confidences, 0.5, 0.4, indices); // Draw detections for(int idx : indices) { Rect box = boxes[idx]; rectangle(img, box, Scalar(0, 255, 0), 2); String label = format("Class %d: %.2f", classIds[idx], confidences[idx]); putText(img, label, Point(box.x, box.y-5), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0,255,0), 2); } imshow("Detections", img); waitKey(0); return 0; } ``` ## Best Practices Match preprocessing used during training Use blobFromImages for multiple images Remove overlapping detections Query available backends for optimization ## See Also * [Net Class](/api/dnn/network) - Network operations * [DNN Module](/modules/dnn) - Module overview * [DNN Layers](/api/dnn/layers) - Layer types # DNN Layer Types Source: https://opencv-opencv.mintlify.app/api/dnn/layers Built-in layer types and custom layer implementation ## Overview OpenCV DNN module supports numerous layer types from different frameworks. This document covers common layers and how to implement custom ones. ## Base Layer Class ### Layer Interface ```cpp theme={null} class Layer : public Algorithm { public: // Layer parameters std::vector blobs; String name; String type; // Initialize layer virtual void finalize( InputArrayOfArrays inputs, OutputArrayOfArrays outputs ); // Forward pass virtual void forward( InputArrayOfArrays inputs, OutputArrayOfArrays outputs, OutputArrayOfArrays internals ); // Query methods virtual int inputNameToIndex(String inputName); virtual int outputNameToIndex(const String& outputName); }; ``` ## Convolution Layers ### Convolution **Type**: `Convolution` **Parameters**: * `num_output`: Number of output channels * `kernel_size`: Kernel dimensions * `stride`: Stride * `pad`: Padding * `dilation`: Dilation * `group`: Number of groups **Example network**: ``` Convolution: num_output: 64 kernel_size: 3 stride: 1 pad: 1 ``` ### Deconvolution **Type**: `Deconvolution` / `ConvolutionTranspose` Transposed convolution for upsampling. ### Depthwise Convolution Implemented as regular convolution with `group = num_input`. ## Pooling Layers ### MaxPooling **Type**: `Pooling` with `pool: MAX` **Parameters**: * `kernel_size`: Pool window size * `stride`: Stride * `pad`: Padding **Example**: ``` Pooling: pool: MAX kernel_size: 2 stride: 2 ``` ### AveragePooling **Type**: `Pooling` with `pool: AVE` ### GlobalPooling **Type**: `Pooling` with `global_pooling: true` Reduces spatial dimensions to 1x1. ## Activation Layers ### ReLU **Type**: `ReLU` ```cpp theme={null} class ActivationLayer : public Layer { public: virtual void forwardSlice( const float* src, float* dst, int len, size_t planeSize, int cn ) const = 0; }; ``` ### LeakyReLU **Type**: `ReLU` with `negative_slope` **Parameters**: * `negative_slope`: Slope for negative values (e.g., 0.1) ### PReLU **Type**: `PReLU` Parametric ReLU with learned slopes. ### ELU **Type**: `ELU` **Parameters**: * `alpha`: Scale factor ### Sigmoid **Type**: `Sigmoid` ### TanH **Type**: `TanH` ### Swish / SiLU **Type**: `Swish` ### Mish **Type**: `Mish` ## Normalization Layers ### BatchNormalization **Type**: `BatchNorm` **Parameters**: * `eps`: Epsilon for numerical stability * Learned parameters: gamma, beta, mean, variance ```cpp theme={null} // Batch norm stores 4 parameters: // blobs[0] - mean // blobs[1] - variance // blobs[2] - scale (gamma) // blobs[3] - shift (beta) ``` ### LayerNormalization **Type**: `LayerNorm` Normalizes across channel dimension. ### InstanceNormalization **Type**: `InstanceNorm` Normalizes each sample independently. ### GroupNormalization **Type**: `GroupNorm` **Parameters**: * `num_groups`: Number of groups ## Fully Connected Layers ### InnerProduct / Dense **Type**: `InnerProduct` **Parameters**: * `num_output`: Output dimension ```cpp theme={null} // Parameters stored in blobs: // blobs[0] - weights [num_output x num_input] // blobs[1] - biases [num_output] (optional) ``` ## Reshape Layers ### Reshape **Type**: `Reshape` **Parameters**: * `dim`: New dimensions (can use -1 for auto) ### Flatten **Type**: `Flatten` Reshapes to 2D (batch\_size, features). ### Permute **Type**: `Permute` Transposes dimensions. **Parameters**: * `order`: New axis order (e.g., \[0, 2, 3, 1]) ### Slice **Type**: `Slice` Slices along an axis. **Parameters**: * `axis`: Axis to slice * `slice_point`: Split points ### Concat **Type**: `Concat` Concatenates along an axis. **Parameters**: * `axis`: Concatenation axis ## Attention Layers ### Attention (Generic) **Type**: `Attention` Multi-head self-attention mechanism. ### ScaledDotProductAttention Implemented for transformer models. ## Dropout **Type**: `Dropout` **Parameters**: * `dropout_ratio`: Probability of dropping (0-1) Dropout is typically disabled during inference (automatically handled). ## Element-wise Operations ### Eltwise **Type**: `Eltwise` **Operations**: * `SUM`: Element-wise addition * `PROD`: Element-wise multiplication * `MAX`: Element-wise maximum **Parameters**: * `operation`: Operation type * `coeff`: Optional coefficients for SUM ### Scale **Type**: `Scale` Scales and shifts: `output = scale * input + bias` ### Shift **Type**: `Shift` Adds a bias term. ## Upsampling Layers ### Resize **Type**: `Resize` / `Upsample` **Parameters**: * `zoom_factor`: Scale factor * `interpolation`: NEAREST, BILINEAR ### UpsamplingNearest **Type**: `ResizeNearest` Nearest neighbor upsampling. ### UpsamplingBilinear **Type**: `ResizeBilinear` Bilinear upsampling. ## Utility Layers ### Split **Type**: `Split` Duplicates input to multiple outputs. ### Crop **Type**: `Crop` Crops spatial dimensions. ### Padding **Type**: `Padding` Adds padding to input. ### Exp **Type**: `Exp` Element-wise exponential. ### Log **Type**: `Log` Element-wise logarithm. ### Power **Type**: `Power` Raises to power: `output = (shift + scale * input) ^ power` ### Abs **Type**: `AbsVal` Absolute value. ### BNLL **Type**: `BNLL` Binomial normal log likelihood. ## Custom Layers ### Implementing Custom Layer ```cpp theme={null} class MyCustomLayer : public Layer { public: MyCustomLayer(const LayerParams& params) : Layer(params) { // Initialize from params myParam = params.get("my_param", 0); } virtual bool getMemoryShapes( const std::vector& inputs, const int requiredOutputs, std::vector& outputs, std::vector& internals ) const override { // Define output shapes outputs.resize(1); outputs[0] = inputs[0]; // Same as input return false; } virtual void forward( InputArrayOfArrays inputs_arr, OutputArrayOfArrays outputs_arr, OutputArrayOfArrays internals_arr ) override { std::vector inputs, outputs; inputs_arr.getMatVector(inputs); outputs_arr.getMatVector(outputs); const Mat& input = inputs[0]; Mat& output = outputs[0]; // Implement forward pass output = input * 2; // Example: multiply by 2 } private: int myParam; }; ``` ### Registering Custom Layer ```cpp theme={null} CV_DNN_REGISTER_LAYER_CLASS(MyCustom, MyCustomLayer); ``` ### Using Custom Layer The layer will be automatically used when loading models containing that layer type. ## Layer Parameters ### LayerParams Class ```cpp theme={null} class LayerParams : public Dict { public: std::vector blobs; // Learned parameters String name; // Layer name String type; // Layer type // Get parameter template T get(const String& key, const T& defaultValue) const; }; ``` ### Accessing Parameters ```cpp theme={null} // In custom layer constructor int numOutput = params.get("num_output"); float scale = params.get("scale", 1.0f); // With default // Access learned weights if(!params.blobs.empty()) { Mat weights = params.blobs[0]; Mat biases = params.blobs[1]; } ``` ## Backend Support ### CPU Implementation Default implementation for all layers. ### OpenCL Implementation Many layers have optimized OpenCL kernels. ### CUDA Implementation ```cpp theme={null} class Layer { public: virtual Ptr initCUDA( void* context, const std::vector>& inputs, const std::vector>& outputs ); }; ``` ### Adding Backend Support ```cpp theme={null} virtual bool supportBackend(int backendId) override { return backendId == DNN_BACKEND_OPENCV || backendId == DNN_BACKEND_CUDA; } virtual Ptr initCUDA(...) override { // Implement CUDA version return Ptr(); } ``` ## Best Practices Validate parameter blob dimensions in constructor Define output shapes for memory allocation Provide CUDA/OpenCL implementations when possible Compare outputs with reference implementation ## See Also * [Net Class](/api/dnn/network) - Network loading and inference * [DNN Module](/modules/dnn) - DNN module overview * [DNN Inference](/api/dnn/inference) - Inference utilities # Net Class Source: https://opencv-opencv.mintlify.app/api/dnn/network The main class for loading and running deep neural network models ## Overview The `Net` class is the core of OpenCV's DNN module. It represents a neural network loaded from various frameworks and provides methods for inference. ## Loading Models ### readNet (Auto-detect) ```cpp theme={null} Net readNet(const String& model, const String& config = "", const String& framework = ""); ``` Automatically detects model format: ```cpp theme={null} // ONNX Net net = readNet("model.onnx"); // TensorFlow Net net = readNet("model.pb", "config.pbtxt"); // Caffe Net net = readNet("model.caffemodel", "deploy.prototxt"); ``` ### Framework-Specific Loaders #### ONNX ```cpp theme={null} Net readNetFromONNX(const String& onnxFile); // From buffer Net readNetFromONNX(const std::vector& buffer); ``` #### TensorFlow ```cpp theme={null} Net readNetFromTensorflow( const String& model, const String& config = String() ); // From buffers Net readNetFromTensorflow( const std::vector& bufferModel, const std::vector& bufferConfig = std::vector() ); ``` #### Caffe ```cpp theme={null} Net readNetFromCaffe( const String& prototxt, const String& caffeModel = String() ); // From buffers Net readNetFromCaffe( const std::vector& bufferProto, const std::vector& bufferModel = std::vector() ); ``` #### Darknet (YOLO) ```cpp theme={null} Net readNetFromDarknet( const String& cfgFile, const String& darknetModel = String() ); // From buffers Net readNetFromDarknet( const std::vector& bufferCfg, const std::vector& bufferModel = std::vector() ); ``` ## Backend and Target ### setPreferableBackend ```cpp theme={null} void Net::setPreferableBackend(int backendId); ``` **Available backends**: * `DNN_BACKEND_DEFAULT`: Default OpenCV implementation * `DNN_BACKEND_OPENCV`: Pure OpenCV implementation * `DNN_BACKEND_CUDA`: NVIDIA CUDA * `DNN_BACKEND_INFERENCE_ENGINE`: Intel OpenVINO * `DNN_BACKEND_VKCOM`: Vulkan ```cpp theme={null} net.setPreferableBackend(DNN_BACKEND_CUDA); ``` ### setPreferableTarget ```cpp theme={null} void Net::setPreferableTarget(int targetId); ``` **Available targets**: * `DNN_TARGET_CPU`: CPU * `DNN_TARGET_OPENCL`: OpenCL (GPU) * `DNN_TARGET_OPENCL_FP16`: OpenCL with FP16 * `DNN_TARGET_CUDA`: CUDA * `DNN_TARGET_CUDA_FP16`: CUDA with FP16 * `DNN_TARGET_MYRIAD`: Intel Myriad * `DNN_TARGET_FPGA`: FPGA ```cpp theme={null} net.setPreferableTarget(DNN_TARGET_CUDA); ``` ### Backend/Target Compatibility ```cpp theme={null} // CPU net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // OpenCL GPU net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_OPENCL); // CUDA GPU net.setPreferableBackend(DNN_BACKEND_CUDA); net.setPreferableTarget(DNN_TARGET_CUDA); // Intel OpenVINO net.setPreferableBackend(DNN_BACKEND_INFERENCE_ENGINE); net.setPreferableTarget(DNN_TARGET_CPU); ``` ## Setting Inputs ### setInput ```cpp theme={null} void Net::setInput( InputArray blob, const String& name = "", double scalefactor = 1.0, const Scalar& mean = Scalar() ); ``` **Parameters**: * `blob`: 4D blob (NCHW format) * `name`: Input layer name (optional if single input) * `scalefactor`: Multiplicative scaling factor * `mean`: Mean values to subtract ```cpp theme={null} Mat blob = blobFromImage(img, 1.0/255, Size(224, 224)); net.setInput(blob); // With preprocessing net.setInput(blob, "data", 1.0/255, Scalar(104, 117, 123)); ``` ## Forward Pass ### forward (Single Output) ```cpp theme={null} Mat Net::forward(const String& outputName = String()); ``` ```cpp theme={null} // Get output from last layer Mat output = net.forward(); // Get specific output layer Mat output = net.forward("conv5"); ``` ### forward (Multiple Outputs) ```cpp theme={null} void Net::forward( OutputArrayOfArrays outputBlobs, const String& outputName = String() ); void Net::forward( OutputArrayOfArrays outputBlobs, const std::vector& outBlobNames ); ``` ```cpp theme={null} // Multiple outputs std::vector outputs; net.forward(outputs, {"output1", "output2", "output3"}); // All unconnected outputs std::vector outNames = net.getUnconnectedOutLayersNames(); std::vector outputs; net.forward(outputs, outNames); ``` ### forwardAsync (Asynchronous) ```cpp theme={null} AsyncArray Net::forwardAsync(const String& outputName = String()); ``` ```cpp theme={null} net.setInput(blob); AsyncArray async = net.forwardAsync(); // Do other work... Mat output = async.get(); // Wait for result ``` ## Network Information ### empty ```cpp theme={null} bool Net::empty() const; ``` Check if network is loaded: ```cpp theme={null} if(net.empty()) { std::cerr << "Failed to load model\n"; return -1; } ``` ### getLayerNames ```cpp theme={null} std::vector Net::getLayerNames() const; ``` Get all layer names: ```cpp theme={null} std::vector layerNames = net.getLayerNames(); for(const String& name : layerNames) { std::cout << name << std::endl; } ``` ### getLayerId ```cpp theme={null} int Net::getLayerId(const String& layer) const; ``` Get layer ID by name: ```cpp theme={null} int layerId = net.getLayerId("conv1"); ``` ### getLayer ```cpp theme={null} Ptr Net::getLayer(int layerId) const; Ptr Net::getLayer(const String& layerName) const; ``` Get layer object: ```cpp theme={null} Ptr layer = net.getLayer("conv1"); String type = layer->type; ``` ### getUnconnectedOutLayersNames ```cpp theme={null} std::vector Net::getUnconnectedOutLayersNames() const; ``` Get output layer names: ```cpp theme={null} std::vector outNames = net.getUnconnectedOutLayersNames(); for(const String& name : outNames) { std::cout << "Output: " << name << std::endl; } ``` ## Network Modification ### setInputsNames ```cpp theme={null} void Net::setInputsNames(const std::vector& inputBlobNames); ``` ### setInputShape ```cpp theme={null} void Net::setInputShape(const String& inputName, const MatShape& shape); ``` ### getParam ```cpp theme={null} Mat Net::getParam(int layer, int numParam = 0) const; Mat Net::getParam(const String& layerName, int numParam = 0) const; ``` Get layer parameters (weights): ```cpp theme={null} Mat weights = net.getParam("conv1", 0); // Weights Mat biases = net.getParam("conv1", 1); // Biases ``` ### setParam ```cpp theme={null} void Net::setParam(int layer, int numParam, const Mat& blob); void Net::setParam(const String& layerName, int numParam, const Mat& blob); ``` Set layer parameters: ```cpp theme={null} Mat newWeights = /* ... */; net.setParam("conv1", 0, newWeights); ``` ## Performance Analysis ### getPerfProfile ```cpp theme={null} int64 Net::getPerfProfile(std::vector& timings); ``` Get layer-wise timing: ```cpp theme={null} std::vector timings; int64 overall = net.getPerfProfile(timings); std::vector layerNames = net.getLayerNames(); for(size_t i = 0; i < timings.size(); i++) { std::cout << layerNames[i] << ": " << timings[i] << " ms\n"; } std::cout << "Total: " << overall / 1000.0 << " ms\n"; ``` ### getFLOPS ```cpp theme={null} int64 Net::getFLOPS(const MatShape& netInputShape) const; int64 Net::getFLOPS(const std::vector& netInputShapes) const; ``` Compute FLOPs: ```cpp theme={null} MatShape inputShape = {1, 3, 224, 224}; int64 flops = net.getFLOPS(inputShape); std::cout << "Model FLOPs: " << flops / 1e9 << " G\n"; ``` ### getMemoryConsumption ```cpp theme={null} void Net::getMemoryConsumption( const MatShape& netInputShape, size_t& weights, size_t& blobs ) const; ``` Get memory usage: ```cpp theme={null} size_t weights, blobs; net.getMemoryConsumption({1, 3, 224, 224}, weights, blobs); std::cout << "Weights: " << weights / 1e6 << " MB\n"; std::cout << "Blobs: " << blobs / 1e6 << " MB\n"; ``` ## Network Optimization ### enableFusion ```cpp theme={null} void Net::enableFusion(bool fusion); ``` Enable/disable layer fusion: ```cpp theme={null} net.enableFusion(true); // Default ``` ### enableWinograd ```cpp theme={null} void Net::enableWinograd(bool useWinograd); ``` Enable Winograd convolution optimization: ```cpp theme={null} net.enableWinograd(true); // Default ``` ## Debugging ### dump ```cpp theme={null} String Net::dump(); ``` Get network structure: ```cpp theme={null} String structure = net.dump(); std::cout << structure; ``` ### dumpToFile ```cpp theme={null} void Net::dumpToFile(const String& path); ``` Save structure to file: ```cpp theme={null} net.dumpToFile("network_structure.txt"); ``` ### dumpToPbtxt ```cpp theme={null} void Net::dumpToPbtxt(const String& path); ``` Save as protobuf text (viewable in Netron): ```cpp theme={null} net.dumpToPbtxt("network.pbtxt"); // View at https://netron.app ``` ## Complete Example ```cpp theme={null} #include #include #include using namespace cv; using namespace cv::dnn; int main() { // Load model Net net = readNet("model.onnx"); if(net.empty()) { std::cerr << "Failed to load model\n"; return -1; } // Configure backend/target net.setPreferableBackend(DNN_BACKEND_CUDA); net.setPreferableTarget(DNN_TARGET_CUDA); // Load and preprocess image Mat img = imread("image.jpg"); Mat blob = blobFromImage(img, 1.0/255, Size(224, 224), Scalar(), true, false); // Set input net.setInput(blob); // Forward pass Mat output = net.forward(); // Get performance info std::vector timings; int64 t = net.getPerfProfile(timings); std::cout << "Inference time: " << t / 1000.0 << " ms\n"; // Process output Point classIdPoint; minMaxLoc(output.reshape(1, 1), 0, 0, 0, &classIdPoint); int classId = classIdPoint.x; std::cout << "Predicted class: " << classId << std::endl; return 0; } ``` ## See Also * [DNN Module](/modules/dnn) - DNN module overview * [DNN Layers](/api/dnn/layers) - Layer types * [DNN Inference](/api/dnn/inference) - Inference utilities # Color Space Conversions Source: https://opencv-opencv.mintlify.app/api/imgproc/color-conversion Functions for converting images between different color spaces OpenCV provides extensive support for color space conversions, allowing you to convert images between various color representations. ## Main Function ### cvtColor Converts an image from one color space to another. ```cpp theme={null} void cvtColor(InputArray src, OutputArray dst, int code, int dstCn = 0, AlgorithmHint hint = cv::ALGO_HINT_DEFAULT); ``` Input image: 8-bit unsigned, 16-bit unsigned (CV\_16UC...), or single-precision floating-point. Output image of the same size and depth as src. Color space conversion code. See ColorConversionCodes. Number of channels in the destination image; if the parameter is 0, the number of the channels is derived automatically from src and code. Implementation modification flags. The function converts an input image from one color space to another. The function ignores the colorModel indicator in IplImage header, and determines the source color space from the number of channels. ```cpp theme={null} Mat src = imread("image.jpg"); Mat gray, hsv; // Convert BGR to grayscale cvtColor(src, gray, COLOR_BGR2GRAY); // Convert BGR to HSV cvtColor(src, hsv, COLOR_BGR2HSV); ``` ```python theme={null} src = cv2.imread('image.jpg') # Convert BGR to grayscale gray = cv2.cvtColor(src, cv2.COLOR_BGR2GRAY) # Convert BGR to HSV hsv = cv2.cvtColor(src, cv2.COLOR_BGR2HSV) ``` The source image (src) must be of an appropriate type for the desired color conversion. Different conversion codes support different input types. *** ### cvtColorTwoPlane Converts an image from one color space to another where the source image is stored in two planes. ```cpp theme={null} void cvtColorTwoPlane(InputArray src1, InputArray src2, OutputArray dst, int code, AlgorithmHint hint = cv::ALGO_HINT_DEFAULT); ``` 8-bit image (CV\_8U) of the Y plane. Image containing interleaved U/V plane. Output image. Specifies the type of conversion. Supported codes: COLOR\_YUV2BGR\_NV12, COLOR\_YUV2RGB\_NV12, COLOR\_YUV2BGRA\_NV12, COLOR\_YUV2RGBA\_NV12, COLOR\_YUV2BGR\_NV21, COLOR\_YUV2RGB\_NV21, COLOR\_YUV2BGRA\_NV21, COLOR\_YUV2RGBA\_NV21. This function only supports YUV420 to RGB conversion as of now. *** ## ColorConversionCodes Enum The color conversion codes specify the type of conversion to perform. Here are the most commonly used ones: ### RGB/BGR Conversions * `COLOR_BGR2BGRA` - Add alpha channel to BGR image \[8U/16U/32F] * `COLOR_RGB2RGBA` - Add alpha channel to RGB image \[8U/16U/32F] * `COLOR_BGRA2BGR` - Remove alpha channel from BGR image \[8U/16U/32F] * `COLOR_RGBA2RGB` - Remove alpha channel from RGB image \[8U/16U/32F] * `COLOR_BGR2RGB` - Convert between RGB and BGR \[8U/16U/32F] * `COLOR_BGRA2RGBA` - Convert between RGBA and BGRA \[8U/16U/32F] ### Grayscale Conversions * `COLOR_BGR2GRAY` - Convert BGR to grayscale \[8U/16U/32F] * `COLOR_RGB2GRAY` - Convert RGB to grayscale \[8U/16U/32F] * `COLOR_GRAY2BGR` - Convert grayscale to BGR \[8U/16U/32F] * `COLOR_GRAY2BGRA` - Convert grayscale to BGRA \[8U/16U/32F] * `COLOR_BGRA2GRAY` - Convert BGRA to grayscale \[8U/16U/32F] * `COLOR_RGBA2GRAY` - Convert RGBA to grayscale \[8U/16U/32F] ### HSV Conversions * `COLOR_BGR2HSV` - Convert BGR to HSV with H range 0..180 if 8 bit image \[8U/32F] * `COLOR_RGB2HSV` - Convert RGB to HSV \[8U/32F] * `COLOR_HSV2BGR` - Convert HSV to BGR \[8U/32F] * `COLOR_HSV2RGB` - Convert HSV to RGB \[8U/32F] * `COLOR_BGR2HSV_FULL` - Convert BGR to HSV with H range 0..255 if 8 bit image \[8U/32F] * `COLOR_HSV2BGR_FULL` - Convert HSV to BGR with full H range \[8U/32F] ### HLS Conversions * `COLOR_BGR2HLS` - Convert BGR to HLS (hue lightness saturation) \[8U/32F] * `COLOR_RGB2HLS` - Convert RGB to HLS \[8U/32F] * `COLOR_HLS2BGR` - Convert HLS to BGR \[8U/32F] * `COLOR_HLS2RGB` - Convert HLS to RGB \[8U/32F] * `COLOR_BGR2HLS_FULL` - Convert BGR to HLS with H range 0..255 \[8U/32F] * `COLOR_HLS2BGR_FULL` - Convert HLS to BGR with full H range \[8U/32F] ### CIE Lab Conversions * `COLOR_BGR2Lab` - Convert BGR to CIE Lab \[8U/32F] * `COLOR_RGB2Lab` - Convert RGB to CIE Lab \[8U/32F] * `COLOR_Lab2BGR` - Convert CIE Lab to BGR \[8U/32F] * `COLOR_Lab2RGB` - Convert CIE Lab to RGB \[8U/32F] ### CIE Luv Conversions * `COLOR_BGR2Luv` - Convert BGR to CIE Luv \[8U/32F] * `COLOR_RGB2Luv` - Convert RGB to CIE Luv \[8U/32F] * `COLOR_Luv2BGR` - Convert CIE Luv to BGR \[8U/32F] * `COLOR_Luv2RGB` - Convert CIE Luv to RGB \[8U/32F] ### CIE XYZ Conversions * `COLOR_BGR2XYZ` - Convert BGR to CIE XYZ \[8U/16U/32F] * `COLOR_RGB2XYZ` - Convert RGB to CIE XYZ \[8U/16U/32F] * `COLOR_XYZ2BGR` - Convert CIE XYZ to BGR \[8U/16U/32F] * `COLOR_XYZ2RGB` - Convert CIE XYZ to RGB \[8U/16U/32F] ### YCrCb Conversions * `COLOR_BGR2YCrCb` - Convert BGR to YCrCb (luma-chroma) \[8U/16U/32F] * `COLOR_RGB2YCrCb` - Convert RGB to YCrCb \[8U/16U/32F] * `COLOR_YCrCb2BGR` - Convert YCrCb to BGR \[8U/16U/32F] * `COLOR_YCrCb2RGB` - Convert YCrCb to RGB \[8U/16U/32F] ### YUV Conversions * `COLOR_BGR2YUV` - Convert between RGB/BGR and YUV \[8U/16U/32F] * `COLOR_RGB2YUV` - Convert RGB to YUV \[8U/16U/32F] * `COLOR_YUV2BGR` - Convert YUV to BGR \[8U/16U/32F] * `COLOR_YUV2RGB` - Convert YUV to RGB \[8U/16U/32F] ### YUV 4:2:0 Conversions (NV12/NV21) * `COLOR_YUV2RGB_NV12` - YUV NV12 to RGB \[8U] * `COLOR_YUV2BGR_NV12` - YUV NV12 to BGR \[8U] * `COLOR_YUV2RGB_NV21` - YUV NV21 to RGB \[8U] * `COLOR_YUV2BGR_NV21` - YUV NV21 to BGR \[8U] * `COLOR_YUV2RGBA_NV12` - YUV NV12 to RGBA \[8U] * `COLOR_YUV2BGRA_NV12` - YUV NV12 to BGRA \[8U] ### YUV 4:2:0 Conversions (YV12/IYUV) * `COLOR_YUV2RGB_YV12` - YUV YV12 to RGB \[8U] * `COLOR_YUV2BGR_YV12` - YUV YV12 to BGR \[8U] * `COLOR_YUV2RGB_IYUV` - YUV IYUV/I420 to RGB \[8U] * `COLOR_YUV2BGR_IYUV` - YUV IYUV/I420 to BGR \[8U] ### Bayer Pattern Conversions * `COLOR_BayerBG2BGR` - Bayer BG/RGGB pattern to BGR \[8U/16U] * `COLOR_BayerGB2BGR` - Bayer GB/GRBG pattern to BGR \[8U/16U] * `COLOR_BayerRG2BGR` - Bayer RG/BGGR pattern to BGR \[8U/16U] * `COLOR_BayerGR2BGR` - Bayer GR/GBRG pattern to BGR \[8U/16U] * `COLOR_BayerBG2GRAY` - Bayer BG/RGGB pattern to grayscale \[8U/16U] * `COLOR_BayerGB2GRAY` - Bayer GB/GRBG pattern to grayscale \[8U/16U] ### Bayer VNG (Variable Number of Gradients) * `COLOR_BayerBG2BGR_VNG` - Bayer BG to BGR using VNG \[8U] * `COLOR_BayerGB2BGR_VNG` - Bayer GB to BGR using VNG \[8U] * `COLOR_BayerRG2BGR_VNG` - Bayer RG to BGR using VNG \[8U] * `COLOR_BayerGR2BGR_VNG` - Bayer GR to BGR using VNG \[8U] ### Bayer Edge-Aware * `COLOR_BayerBG2BGR_EA` - Bayer BG to BGR using edge-aware \[8U/16U] * `COLOR_BayerGB2BGR_EA` - Bayer GB to BGR using edge-aware \[8U/16U] * `COLOR_BayerRG2BGR_EA` - Bayer RG to BGR using edge-aware \[8U/16U] * `COLOR_BayerGR2BGR_EA` - Bayer GR to BGR using edge-aware \[8U/16U] ## Color Space Information ### RGB/BGR The default color space in OpenCV is BGR (Blue-Green-Red), not RGB. When you read an image using imread(), it returns a BGR image. Many other libraries (like matplotlib) expect RGB format. ### Grayscale Grayscale images have a single channel. The conversion from RGB/BGR to grayscale uses: $$ \text{Gray} = 0.299 \cdot R + 0.587 \cdot G + 0.114 \cdot B $$ ### HSV (Hue, Saturation, Value) HSV separates image intensity (value) from color information (hue and saturation). The hue range is: * 0..180 for 8-bit images (default) * 0..255 for 8-bit images (FULL variant) * 0..360 for 32-bit images ### HLS (Hue, Lightness, Saturation) Similar to HSV but uses lightness instead of value. The hue range is the same as HSV. ### Lab/Luv CIE Lab and Luv are perceptually uniform color spaces, useful for color-based segmentation and comparison. ### YCrCb/YUV Luma-chroma color spaces commonly used in video encoding. Y represents luminance (brightness), while Cr/Cb (or U/V) represent chrominance (color information). # Drawing Functions Source: https://opencv-opencv.mintlify.app/api/imgproc/drawing Functions for drawing geometric shapes and text on images Drawing functions work with matrices/images of arbitrary depth. The boundaries of shapes can be rendered with antialiasing (implemented only for 8-bit images for now). ## Color Convention For color images, the channel ordering is normally Blue, Green, Red (BGR). This is what imshow, imread, and imwrite expect. If you form a color using the Scalar constructor, it should look like: ```cpp theme={null} Scalar(blue_component, green_component, red_component[, alpha_component]) ``` ## Line Drawing ### line Draws a line segment connecting two points. ```cpp theme={null} void line(InputOutputArray img, Point pt1, Point pt2, const Scalar& color, int thickness = 1, int lineType = LINE_8, int shift = 0); ``` Image. First point of the line segment. Second point of the line segment. Line color. Line thickness. Type of the line. See LineTypes. Number of fractional bits in the point coordinates. The function line draws the line segment between pt1 and pt2 points in the image. The line is clipped by the image boundaries. ```cpp theme={null} Mat img = Mat::zeros(400, 400, CV_8UC3); line(img, Point(50, 50), Point(350, 350), Scalar(0, 255, 0), 2); ``` ```python theme={null} img = np.zeros((400, 400, 3), dtype=np.uint8) cv2.line(img, (50, 50), (350, 350), (0, 255, 0), 2) ``` *** ### arrowedLine Draws an arrow segment pointing from the first point to the second one. ```cpp theme={null} void arrowedLine(InputOutputArray img, Point pt1, Point pt2, const Scalar& color, int thickness=1, int line_type=8, int shift=0, double tipLength=0.1); ``` Image. The point the arrow starts from. The point the arrow points to. Line color. Line thickness. Type of the line. See LineTypes. Number of fractional bits in the point coordinates. The length of the arrow tip in relation to the arrow length. *** ## Shape Drawing ### rectangle Draws a simple, thick, or filled up-right rectangle. ```cpp theme={null} void rectangle(InputOutputArray img, Point pt1, Point pt2, const Scalar& color, int thickness = 1, int lineType = LINE_8, int shift = 0); void rectangle(InputOutputArray img, Rect rec, const Scalar& color, int thickness = 1, int lineType = LINE_8, int shift = 0); ``` Image. Vertex of the rectangle. Vertex of the rectangle opposite to pt1. Alternative rectangle specification. Rectangle color or brightness (grayscale image). Thickness of lines that make up the rectangle. Negative values, like FILLED, mean that the function has to draw a filled rectangle. Type of the line. See LineTypes. Number of fractional bits in the point coordinates. ```cpp theme={null} // Draw rectangle outline rectangle(img, Point(100, 100), Point(300, 200), Scalar(255, 0, 0), 2); // Draw filled rectangle rectangle(img, Rect(50, 250, 200, 100), Scalar(0, 0, 255), FILLED); ``` ```python theme={null} # Draw rectangle outline cv2.rectangle(img, (100, 100), (300, 200), (255, 0, 0), 2) # Draw filled rectangle cv2.rectangle(img, (50, 250, 200, 100), (0, 0, 255), cv2.FILLED) ``` *** ### circle Draws a circle. ```cpp theme={null} void circle(InputOutputArray img, Point center, int radius, const Scalar& color, int thickness = 1, int lineType = LINE_8, int shift = 0); ``` Image where the circle is drawn. Center of the circle. Radius of the circle. Circle color. Thickness of the circle outline, if positive. Negative values, like FILLED, mean that a filled circle is to be drawn. Type of the circle boundary. See LineTypes. Number of fractional bits in the coordinates of the center and in the radius value. ```cpp theme={null} circle(img, Point(200, 200), 50, Scalar(0, 255, 255), 3); circle(img, Point(300, 300), 75, Scalar(255, 255, 0), FILLED); ``` ```python theme={null} cv2.circle(img, (200, 200), 50, (0, 255, 255), 3) cv2.circle(img, (300, 300), 75, (255, 255, 0), cv2.FILLED) ``` *** ### ellipse Draws a simple or thick elliptic arc or fills an ellipse sector. ```cpp theme={null} void ellipse(InputOutputArray img, Point center, Size axes, double angle, double startAngle, double endAngle, const Scalar& color, int thickness = 1, int lineType = LINE_8, int shift = 0); void ellipse(InputOutputArray img, const RotatedRect& box, const Scalar& color, int thickness = 1, int lineType = LINE_8); ``` Image. Center of the ellipse. Half of the size of the ellipse main axes. Ellipse rotation angle in degrees. Starting angle of the elliptic arc in degrees. Ending angle of the elliptic arc in degrees. Alternative ellipse representation via RotatedRect. This means that the function draws an ellipse inscribed in the rotated rectangle. Ellipse color. Thickness of the ellipse arc outline, if positive. Otherwise, this indicates that a filled ellipse sector is to be drawn. Type of the ellipse boundary. See LineTypes. Number of fractional bits in the coordinates of the center and values of axes. To draw the whole ellipse, not an arc, pass startAngle=0 and endAngle=360. If startAngle is greater than endAngle, they are swapped. *** ## Polygon Drawing ### polylines Draws several polygonal curves. ```cpp theme={null} void polylines(InputOutputArray img, InputArrayOfArrays pts, bool isClosed, const Scalar& color, int thickness = 1, int lineType = LINE_8, int shift = 0); ``` Image. Array of polygonal curves. Flag indicating whether the drawn polylines are closed or not. If they are closed, the function draws a line from the last vertex of each curve to its first vertex. Polyline color. Thickness of the polyline edges. Type of the line segments. See LineTypes. Number of fractional bits in the vertex coordinates. *** ### fillPoly Fills the area bounded by one or more polygons. ```cpp theme={null} void fillPoly(InputOutputArray img, InputArrayOfArrays pts, const Scalar& color, int lineType = LINE_8, int shift = 0, Point offset = Point()); ``` Image. Array of polygons where each polygon is represented as an array of points. Polygon color. Type of the polygon boundaries. See LineTypes. Number of fractional bits in the vertex coordinates. Optional offset of all points of the contours. The function fillPoly fills an area bounded by several polygonal contours. The function can fill complex areas, for example, areas with holes, contours with self-intersections, and so forth. *** ### fillConvexPoly Fills a convex polygon. ```cpp theme={null} void fillConvexPoly(InputOutputArray img, InputArray points, const Scalar& color, int lineType = LINE_8, int shift = 0); ``` Image. Polygon vertices. Polygon color. Type of the polygon boundaries. See LineTypes. Number of fractional bits in the vertex coordinates. This function is much faster than fillPoly. It can fill not only convex polygons but any monotonic polygon without self-intersections. *** ## Text Rendering ### putText Draws a text string. ```cpp theme={null} void putText(InputOutputArray img, const String& text, Point org, int fontFace, double fontScale, Scalar color, int thickness = 1, int lineType = LINE_8, bool bottomLeftOrigin = false); ``` Image. Text string to be drawn. Bottom-left corner of the text string in the image. Font type. See HersheyFonts. Font scale factor that is multiplied by the font-specific base size. Text color. Thickness of the lines used to draw a text. Line type. See LineTypes. When true, the image data origin is at the bottom-left corner. Otherwise, it is at the top-left corner. The function putText renders the specified text string in the image. Symbols that cannot be rendered using the specified font are replaced by question marks. ```cpp theme={null} putText(img, "OpenCV", Point(50, 100), FONT_HERSHEY_SIMPLEX, 1.5, Scalar(255, 255, 255), 2); ``` ```python theme={null} cv2.putText(img, 'OpenCV', (50, 100), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (255, 255, 255), 2) ``` *** ### getTextSize Calculates the width and height of a text string. ```cpp theme={null} Size getTextSize(const String& text, int fontFace, double fontScale, int thickness, int* baseLine); ``` Input text string. Font to use. See HersheyFonts. Font scale factor that is multiplied by the font-specific base size. Thickness of lines used to render the text. y-coordinate of the baseline relative to the bottom-most text point. The function calculates and returns the size of a box that contains the specified text. *** ## Enumerations ### LineTypes Types of line: * `FILLED` (-1) - Filled shape * `LINE_4` (4) - 4-connected line * `LINE_8` (8) - 8-connected line * `LINE_AA` (16) - Antialiased line ### HersheyFonts Hershey font types: * `FONT_HERSHEY_SIMPLEX` (0) - Normal size sans-serif font * `FONT_HERSHEY_PLAIN` (1) - Small size sans-serif font * `FONT_HERSHEY_DUPLEX` (2) - Normal size sans-serif font (more complex than SIMPLEX) * `FONT_HERSHEY_COMPLEX` (3) - Normal size serif font * `FONT_HERSHEY_TRIPLEX` (4) - Normal size serif font (more complex than COMPLEX) * `FONT_HERSHEY_COMPLEX_SMALL` (5) - Smaller version of COMPLEX * `FONT_HERSHEY_SCRIPT_SIMPLEX` (6) - Hand-writing style font * `FONT_HERSHEY_SCRIPT_COMPLEX` (7) - More complex variant of SCRIPT\_SIMPLEX * `FONT_ITALIC` (16) - Flag for italic font ### MarkerTypes Marker types used for the drawMarker function: * `MARKER_CROSS` (0) - A crosshair marker shape * `MARKER_TILTED_CROSS` (1) - A 45 degree tilted crosshair marker shape * `MARKER_STAR` (2) - A star marker shape * `MARKER_DIAMOND` (3) - A diamond marker shape * `MARKER_SQUARE` (4) - A square marker shape * `MARKER_TRIANGLE_UP` (5) - An upwards pointing triangle marker shape * `MARKER_TRIANGLE_DOWN` (6) - A downwards pointing triangle marker shape # Feature Detection Source: https://opencv-opencv.mintlify.app/api/imgproc/feature-detection Functions for detecting edges, corners, lines, and circles in images This module provides functions for detecting various features in images, including edges, corners, lines, and circles. ## Edge Detection ### Canny Finds edges in an image using the Canny algorithm. ```cpp theme={null} void Canny(InputArray image, OutputArray edges, double threshold1, double threshold2, int apertureSize = 3, bool L2gradient = false); void Canny(InputArray dx, InputArray dy, OutputArray edges, double threshold1, double threshold2, bool L2gradient = false); ``` 8-bit input image. 16-bit x derivative of input image (CV\_16SC1 or CV\_16SC3). 16-bit y derivative of input image (same type as dx). Output edge map; single channels 8-bit image, which has the same size as image. First threshold for the hysteresis procedure. Second threshold for the hysteresis procedure. Aperture size for the Sobel operator. A flag, indicating whether a more accurate L2 norm should be used to calculate the image gradient magnitude (L2gradient=true), or whether the default L1 norm is enough (L2gradient=false). The function finds edges in the input image and marks them in the output map edges using the Canny algorithm. The smallest value between threshold1 and threshold2 is used for edge linking. The largest value is used to find initial segments of strong edges. ```cpp theme={null} Mat src = imread("image.jpg", IMREAD_GRAYSCALE); Mat edges; Canny(src, edges, 50, 150); ``` ```python theme={null} src = cv2.imread('image.jpg', cv2.IMREAD_GRAYSCALE) edges = cv2.Canny(src, 50, 150) ``` *** ## Derivatives and Gradients ### Sobel Calculates the first, second, third, or mixed image derivatives using an extended Sobel operator. ```cpp theme={null} void Sobel(InputArray src, OutputArray dst, int ddepth, int dx, int dy, int ksize = 3, double scale = 1, double delta = 0, int borderType = BORDER_DEFAULT); ``` Input image. Output image of the same size and the same number of channels as src. Output image depth. In the case of 8-bit input images it will result in truncated derivatives. Order of the derivative x. Order of the derivative y. Size of the extended Sobel kernel; it must be 1, 3, 5, or 7. Optional scale factor for the computed derivative values; by default, no scaling is applied. Optional delta value that is added to the results prior to storing them in dst. Pixel extrapolation method. BORDER\_WRAP is not supported. The Sobel operators combine Gaussian smoothing and differentiation. Most often, the function is called with (xorder = 1, yorder = 0, ksize = 3) or (xorder = 0, yorder = 1, ksize = 3) to calculate the first x- or y- image derivative. *** ### Scharr Calculates the first x- or y- image derivative using Scharr operator. ```cpp theme={null} void Scharr(InputArray src, OutputArray dst, int ddepth, int dx, int dy, double scale = 1, double delta = 0, int borderType = BORDER_DEFAULT); ``` Input image. Output image of the same size and the same number of channels as src. Output image depth. Order of the derivative x. Order of the derivative y. Optional scale factor for the computed derivative values. Optional delta value that is added to the results prior to storing them in dst. Pixel extrapolation method. BORDER\_WRAP is not supported. The Scharr operator may give more accurate results than the 3×3 Sobel. The Scharr aperture is: $$ \begin{bmatrix} -3 & 0 & 3 \\ -10 & 0 & 10 \\ -3 & 0 & 3 \end{bmatrix} $$ for the x-derivative, or transposed for the y-derivative. *** ### Laplacian Calculates the Laplacian of an image. ```cpp theme={null} void Laplacian(InputArray src, OutputArray dst, int ddepth, int ksize = 1, double scale = 1, double delta = 0, int borderType = BORDER_DEFAULT); ``` Source image. Destination image of the same size and the same number of channels as src. Desired depth of the destination image. Aperture size used to compute the second-derivative filters. The size must be positive and odd. Optional scale factor for the computed Laplacian values. Optional delta value that is added to the results prior to storing them in dst. Pixel extrapolation method. BORDER\_WRAP is not supported. The function calculates the Laplacian of the source image by adding up the second x and y derivatives calculated using the Sobel operator: $$ \Delta \text{src} = \frac{\partial^2 \text{src}}{\partial x^2} + \frac{\partial^2 \text{src}}{\partial y^2} $$ *** ## Corner Detection ### cornerHarris Harris corner detector. ```cpp theme={null} void cornerHarris(InputArray src, OutputArray dst, int blockSize, int ksize, double k, int borderType = BORDER_DEFAULT); ``` Input single-channel 8-bit or floating-point image. Image to store the Harris detector responses. It has the type CV\_32FC1 and the same size as src. Neighborhood size. Aperture parameter for the Sobel operator. Harris detector free parameter. Pixel extrapolation method. BORDER\_WRAP is not supported. The function runs the Harris corner detector on the image. Corners in the image can be found as the local maxima of this response map. *** ### cornerMinEigenVal Calculates the minimal eigenvalue of gradient matrices for corner detection. ```cpp theme={null} void cornerMinEigenVal(InputArray src, OutputArray dst, int blockSize, int ksize = 3, int borderType = BORDER_DEFAULT); ``` Input single-channel 8-bit or floating-point image. Image to store the minimal eigenvalues. It has the type CV\_32FC1 and the same size as src. Neighborhood size. Aperture parameter for the Sobel operator. Pixel extrapolation method. BORDER\_WRAP is not supported. The function is similar to cornerEigenValsAndVecs but it calculates and stores only the minimal eigenvalue of the covariance matrix of derivatives. *** ### goodFeaturesToTrack Determines strong corners on an image. ```cpp theme={null} void goodFeaturesToTrack(InputArray image, OutputArray corners, int maxCorners, double qualityLevel, double minDistance, InputArray mask = noArray(), int blockSize = 3, bool useHarrisDetector = false, double k = 0.04); ``` Input 8-bit or floating-point 32-bit, single-channel image. Output vector of detected corners. Maximum number of corners to return. If there are more corners than are found, the strongest of them is returned. maxCorners \<= 0 implies that no limit on the maximum is set. Parameter characterizing the minimal accepted quality of image corners. The parameter value is multiplied by the best corner quality measure. Minimum possible Euclidean distance between the returned corners. Optional region of interest. Size of an average block for computing a derivative covariation matrix over each pixel neighborhood. Parameter indicating whether to use a Harris detector or cornerMinEigenVal. Free parameter of the Harris detector. The function finds the most prominent corners in the image or in the specified image region. *** ## Hough Transform ### HoughLines Finds lines in a binary image using the standard Hough transform. ```cpp theme={null} void HoughLines(InputArray image, OutputArray lines, double rho, double theta, int threshold, double srn = 0, double stn = 0, double min_theta = 0, double max_theta = CV_PI, bool use_edgeval = false); ``` 8-bit, single-channel binary source image. The image may be modified by the function. Output vector of lines. Each line is represented by a 2 or 3 element vector (ρ, θ) or (ρ, θ, votes). Distance resolution of the accumulator in pixels. Angle resolution of the accumulator in radians. Accumulator threshold parameter. Only those lines are returned that get enough votes (>threshold). For the multi-scale Hough transform, it is a divisor for the distance resolution rho. For the multi-scale Hough transform, it is a divisor for the distance resolution theta. Minimum angle to check for lines. Must fall between 0 and max\_theta. Upper bound for the angle. Must fall between min\_theta and CV\_PI. The function implements the standard or standard multi-scale Hough transform algorithm for line detection. *** ### HoughLinesP Finds line segments in a binary image using the probabilistic Hough transform. ```cpp theme={null} void HoughLinesP(InputArray image, OutputArray lines, double rho, double theta, int threshold, double minLineLength = 0, double maxLineGap = 0); ``` 8-bit, single-channel binary source image. The image may be modified by the function. Output vector of lines. Each line is represented by a 4-element vector (x₁, y₁, x₂, y₂), where (x₁,y₁) and (x₂, y₂) are the ending points of each detected line segment. Distance resolution of the accumulator in pixels. Angle resolution of the accumulator in radians. Accumulator threshold parameter. Only those lines are returned that get enough votes (>threshold). Minimum line length. Line segments shorter than that are rejected. Maximum allowed gap between points on the same line to link them. The function implements the probabilistic Hough transform algorithm for line detection. *** ### HoughCircles Finds circles in a grayscale image using the Hough transform. ```cpp theme={null} void HoughCircles(InputArray image, OutputArray circles, int method, double dp, double minDist, double param1 = 100, double param2 = 100, int minRadius = 0, int maxRadius = 0); ``` 8-bit, single-channel, grayscale input image. Output vector of found circles. Each vector is encoded as 3 or 4 element floating-point vector (x, y, radius) or (x, y, radius, votes). Detection method. The available methods are HOUGH\_GRADIENT and HOUGH\_GRADIENT\_ALT. Inverse ratio of the accumulator resolution to the image resolution. For example, if dp=1, the accumulator has the same resolution as the input image. Minimum distance between the centers of the detected circles. First method-specific parameter. In case of HOUGH\_GRADIENT and HOUGH\_GRADIENT\_ALT, it is the higher threshold of the two passed to the Canny edge detector. Second method-specific parameter. In case of HOUGH\_GRADIENT, it is the accumulator threshold for the circle centers at the detection stage. Minimum circle radius. Maximum circle radius. If \<= 0, uses the maximum image dimension. The function finds circles in a grayscale image using a modification of the Hough transform. ```cpp theme={null} Mat src = imread("image.jpg", IMREAD_GRAYSCALE); Mat blurred; GaussianBlur(src, blurred, Size(9, 9), 2, 2); vector circles; HoughCircles(blurred, circles, HOUGH_GRADIENT, 1, src.rows/8, 200, 100); ``` ```python theme={null} src = cv2.imread('image.jpg', cv2.IMREAD_GRAYSCALE) blurred = cv2.GaussianBlur(src, (9, 9), 2) circles = cv2.HoughCircles(blurred, cv2.HOUGH_GRADIENT, 1, src.shape[0]/8, param1=200, param2=100) ``` Usually the function detects the centers of circles well. However, it may fail to find correct radii. You can assist to the function by specifying the radius range (minRadius and maxRadius) if you know it. *** ## Enumerations ### HoughModes Variants of Hough transform: * `HOUGH_STANDARD` - Classical or standard Hough transform * `HOUGH_PROBABILISTIC` - Probabilistic Hough transform (more efficient) * `HOUGH_MULTI_SCALE` - Multi-scale variant of the classical Hough transform * `HOUGH_GRADIENT` - 21HT for circles * `HOUGH_GRADIENT_ALT` - Variation of HOUGH\_GRADIENT to get better accuracy # Image Filtering Source: https://opencv-opencv.mintlify.app/api/imgproc/filtering Functions for performing linear and non-linear filtering operations on 2D images This module provides functions to perform various linear or non-linear filtering operations on 2D images. For each pixel location in the source image, its neighborhood is considered and used to compute the response. ## Border Extrapolation Many filtering functions need to extrapolate values of non-existing pixels (e.g., when processing pixels near image borders). OpenCV provides several border extrapolation methods via the `BorderTypes` enum. ## Smoothing Filters ### blur Blurs an image using the normalized box filter. ```cpp theme={null} void blur(InputArray src, OutputArray dst, Size ksize, Point anchor = Point(-1,-1), int borderType = BORDER_DEFAULT); ``` Input image; it can have any number of channels, which are processed independently, but the depth should be CV\_8U, CV\_16U, CV\_16S, CV\_32F or CV\_64F. Output image of the same size and type as src. Blurring kernel size. Anchor point; default value Point(-1,-1) means that the anchor is at the kernel center. Border mode used to extrapolate pixels outside of the image. BORDER\_WRAP is not supported. The function smooths an image using the kernel: $$ \texttt{K} = \frac{1}{\texttt{ksize.width*ksize.height}} \begin{bmatrix} 1 & 1 & 1 & \cdots & 1 & 1 \\ 1 & 1 & 1 & \cdots & 1 & 1 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 1 & 1 & 1 & \cdots & 1 & 1 \end{bmatrix} $$ *** ### GaussianBlur Blurs an image using a Gaussian filter. ```cpp theme={null} void GaussianBlur(InputArray src, OutputArray dst, Size ksize, double sigmaX, double sigmaY = 0, int borderType = BORDER_DEFAULT, AlgorithmHint hint = cv::ALGO_HINT_DEFAULT); ``` Input image; the image can have any number of channels, which are processed independently, but the depth should be CV\_8U, CV\_16U, CV\_16S, CV\_32F or CV\_64F. Output image of the same size and type as src. Gaussian kernel size. ksize.width and ksize.height can differ but they both must be positive and odd. Or, they can be zeros and then they are computed from sigma. Gaussian kernel standard deviation in X direction. Gaussian kernel standard deviation in Y direction; if sigmaY is zero, it is set to be equal to sigmaX. Pixel extrapolation method. BORDER\_WRAP is not supported. The function convolves the source image with the specified Gaussian kernel. In-place filtering is supported. *** ### medianBlur Blurs an image using the median filter. ```cpp theme={null} void medianBlur(InputArray src, OutputArray dst, int ksize); ``` Input 1-, 3-, or 4-channel image; when ksize is 3 or 5, the image depth should be CV\_8U, CV\_16U, or CV\_32F, for larger aperture sizes, it can only be CV\_8U. Destination array of the same size and type as src. Aperture linear size; it must be odd and greater than 1, for example: 3, 5, 7 ... The function smoothes an image using the median filter with the ksize × ksize aperture. Each channel of a multi-channel image is processed independently. In-place operation is supported. The median filter uses BORDER\_REPLICATE internally to cope with border pixels. *** ### bilateralFilter Applies the bilateral filter to an image. ```cpp theme={null} void bilateralFilter(InputArray src, OutputArray dst, int d, double sigmaColor, double sigmaSpace, int borderType = BORDER_DEFAULT); ``` Source 8-bit or floating-point, 1-channel or 3-channel image. Destination image of the same size and type as src. Diameter of each pixel neighborhood that is used during filtering. If it is non-positive, it is computed from sigmaSpace. Filter sigma in the color space. A larger value means that farther colors within the pixel neighborhood will be mixed together. Filter sigma in the coordinate space. A larger value means that farther pixels will influence each other as long as their colors are close enough. Border mode used to extrapolate pixels outside of the image. The bilateral filter can reduce unwanted noise very well while keeping edges fairly sharp. However, it is very slow compared to most filters. **Sigma values**: For simplicity, you can set the 2 sigma values to be the same. If they are small (\< 10), the filter will not have much effect, whereas if they are large (> 150), they will have a very strong effect. **Filter size**: Large filters (d > 5) are very slow, so it is recommended to use d=5 for real-time applications, and perhaps d=9 for offline applications. This filter does not work inplace. *** ### boxFilter Blurs an image using the box filter. ```cpp theme={null} void boxFilter(InputArray src, OutputArray dst, int ddepth, Size ksize, Point anchor = Point(-1,-1), bool normalize = true, int borderType = BORDER_DEFAULT); ``` Input image. Output image of the same size and type as src. The output image depth (-1 to use src.depth()). Blurring kernel size. Anchor point; default value Point(-1,-1) means that the anchor is at the kernel center. Flag, specifying whether the kernel is normalized by its area or not. Border mode used to extrapolate pixels outside of the image. BORDER\_WRAP is not supported. Unnormalized box filter is useful for computing various integral characteristics over each pixel neighborhood, such as covariance matrices of image derivatives. *** ## Custom Filters ### filter2D Convolves an image with the kernel. ```cpp theme={null} void filter2D(InputArray src, OutputArray dst, int ddepth, InputArray kernel, Point anchor = Point(-1,-1), double delta = 0, int borderType = BORDER_DEFAULT); ``` Input image. Output image of the same size and the same number of channels as src. Desired depth of the destination image. See combinations in the documentation. Convolution kernel (or rather a correlation kernel), a single-channel floating point matrix. Anchor of the kernel that indicates the relative position of a filtered point within the kernel; default value (-1,-1) means that the anchor is at the kernel center. Optional value added to the filtered pixels before storing them in dst. Pixel extrapolation method. BORDER\_WRAP is not supported. The function applies an arbitrary linear filter to an image. In-place operation is supported. The function uses the DFT-based algorithm in case of sufficiently large kernels (\~11 x 11 or larger) and the direct algorithm for small kernels. The function actually computes correlation, not convolution. If you need a real convolution, flip the kernel using flip() and set the new anchor. *** ### sepFilter2D Applies a separable linear filter to an image. ```cpp theme={null} void sepFilter2D(InputArray src, OutputArray dst, int ddepth, InputArray kernelX, InputArray kernelY, Point anchor = Point(-1,-1), double delta = 0, int borderType = BORDER_DEFAULT); ``` Source image. Destination image of the same size and the same number of channels as src. Destination image depth. Coefficients for filtering each row. Coefficients for filtering each column. Anchor position within the kernel. The default value (-1,-1) means that the anchor is at the kernel center. Value added to the filtered results before storing them. Pixel extrapolation method. BORDER\_WRAP is not supported. The function applies a separable linear filter to the image. First, every row of src is filtered with the 1D kernel kernelX. Then, every column of the result is filtered with the 1D kernel kernelY. *** ## Helper Functions ### getGaussianKernel Returns Gaussian filter coefficients. ```cpp theme={null} Mat getGaussianKernel(int ksize, double sigma, int ktype = CV_64F); ``` Aperture size. It should be odd and positive. Gaussian standard deviation. If it is non-positive, it is computed from ksize as sigma = 0.3\*((ksize-1)\*0.5 - 1) + 0.8. Type of filter coefficients. It can be CV\_32F or CV\_64F. The function computes and returns the ksize × 1 matrix of Gaussian filter coefficients. Two of such generated kernels can be passed to sepFilter2D or used with GaussianBlur. *** ### getDerivKernels Returns filter coefficients for computing spatial image derivatives. ```cpp theme={null} void getDerivKernels(OutputArray kx, OutputArray ky, int dx, int dy, int ksize, bool normalize = false, int ktype = CV_32F); ``` Output matrix of row filter coefficients. Output matrix of column filter coefficients. Derivative order in respect of x. Derivative order in respect of y. Aperture size. It can be FILTER\_SCHARR, 1, 3, 5, or 7. Flag indicating whether to normalize (scale down) the filter coefficients or not. Type of filter coefficients. It can be CV\_32F or CV\_64F. The function computes and returns the filter coefficients for spatial image derivatives. When ksize=FILTER\_SCHARR, the Scharr 3 × 3 kernels are generated. Otherwise, Sobel kernels are generated. *** ### getStructuringElement Returns a structuring element of the specified size and shape for morphological operations. ```cpp theme={null} Mat getStructuringElement(int shape, Size ksize, Point anchor = Point(-1,-1)); ``` Element shape that could be one of MorphShapes: MORPH\_RECT, MORPH\_CROSS, MORPH\_ELLIPSE, MORPH\_DIAMOND. Size of the structuring element. Anchor position within the element. The default value (-1, -1) means that the anchor is at the center. The function constructs and returns the structuring element that can be further passed to erode, dilate or morphologyEx. *** ## Enumerations ### MorphShapes Shape of the structuring element: * `MORPH_RECT` - A rectangular structuring element * `MORPH_CROSS` - A cross-shaped structuring element * `MORPH_ELLIPSE` - An elliptic structuring element * `MORPH_DIAMOND` - A diamond structuring element defined by Manhattan distance ### SpecialFilter * `FILTER_SCHARR` - Scharr filter (-1) # Geometric Image Transformations Source: https://opencv-opencv.mintlify.app/api/imgproc/geometric Functions for performing geometric transformations of 2D images The functions in this module perform various geometrical transformations of 2D images. They do not change the image content but deform the pixel grid and map this deformed grid to the destination image. ## Image Resizing ### resize Resizes an image. ```cpp theme={null} void resize(InputArray src, OutputArray dst, Size dsize, double fx = 0, double fy = 0, int interpolation = INTER_LINEAR); ``` Input image. Output image; it has the size dsize (when it is non-zero) or the size computed from src.size(), fx, and fy; the type of dst is the same as of src. Output image size; if it equals zero, it is computed as dsize = Size(round(fx*src.cols), round(fy*src.rows)). Either dsize or both fx and fy must be non-zero. Scale factor along the horizontal axis; when it equals 0, it is computed as (double)dsize.width/src.cols. Scale factor along the vertical axis; when it equals 0, it is computed as (double)dsize.height/src.rows. Interpolation method. See InterpolationFlags. The function resize resizes the image src down to or up to the specified size. ```cpp theme={null} // Explicitly specify dsize resize(src, dst, Size(640, 480), 0, 0, INTER_LINEAR); // Specify fx and fy resize(src, dst, Size(), 0.5, 0.5, INTER_AREA); ``` ```python theme={null} # Explicitly specify dsize dst = cv2.resize(src, (640, 480), interpolation=cv2.INTER_LINEAR) # Specify fx and fy dst = cv2.resize(src, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA) ``` To shrink an image, it will generally look best with INTER\_AREA interpolation, whereas to enlarge an image, it will generally look best with INTER\_CUBIC (slow) or INTER\_LINEAR (faster but still looks OK). *** ## Affine Transformations ### warpAffine Applies an affine transformation to an image. ```cpp theme={null} void warpAffine(InputArray src, OutputArray dst, InputArray M, Size dsize, int flags = INTER_LINEAR, int borderMode = BORDER_CONSTANT, const Scalar& borderValue = Scalar()); ``` Input image. Output image that has the size dsize and the same type as src. 2×3 transformation matrix. Size of the output image. Combination of interpolation methods (see InterpolationFlags) and the optional flag WARP\_INVERSE\_MAP that means that M is the inverse transformation. Pixel extrapolation method; when borderMode=BORDER\_TRANSPARENT, it means that the pixels in the destination image corresponding to the "outliers" in the source image are not modified by the function. Value used in case of a constant border; by default, it is 0. The function warpAffine transforms the source image using the specified matrix: $$ \texttt{dst}(x,y) = \texttt{src}(M_{11} x + M_{12} y + M_{13}, M_{21} x + M_{22} y + M_{23}) $$ when the flag WARP\_INVERSE\_MAP is set. Otherwise, the transformation is first inverted with invertAffineTransform. The function cannot operate in-place. *** ### getRotationMatrix2D Calculates an affine matrix of 2D rotation. ```cpp theme={null} Mat getRotationMatrix2D(Point2f center, double angle, double scale); ``` Center of the rotation in the source image. Rotation angle in degrees. Positive values mean counter-clockwise rotation (the coordinate origin is assumed to be the top-left corner). Isotropic scale factor. The function calculates the 2×3 rotation matrix. The transformation maps the rotation center to itself. If this is not the target, adjust the shift. *** ### getAffineTransform Calculates an affine transform from three pairs of the corresponding points. ```cpp theme={null} Mat getAffineTransform(const Point2f src[], const Point2f dst[]); Mat getAffineTransform(InputArray src, InputArray dst); ``` Coordinates of triangle vertices in the source image. Coordinates of the corresponding triangle vertices in the destination image. The function calculates the 2 × 3 matrix of an affine transform so that the three source points are mapped to the three destination points. *** ### invertAffineTransform Inverts an affine transformation. ```cpp theme={null} void invertAffineTransform(InputArray M, OutputArray iM); ``` Original affine transformation. Output reverse affine transformation. The function computes an inverse affine transformation represented by 2 × 3 matrix M. *** ## Perspective Transformations ### warpPerspective Applies a perspective transformation to an image. ```cpp theme={null} void warpPerspective(InputArray src, OutputArray dst, InputArray M, Size dsize, int flags = INTER_LINEAR, int borderMode = BORDER_CONSTANT, const Scalar& borderValue = Scalar()); ``` Input image. Output image that has the size dsize and the same type as src. 3×3 transformation matrix. Size of the output image. Combination of interpolation methods (INTER\_LINEAR or INTER\_NEAREST) and the optional flag WARP\_INVERSE\_MAP. Pixel extrapolation method (BORDER\_CONSTANT or BORDER\_REPLICATE). Value used in case of a constant border; by default, it equals 0. The function warpPerspective transforms the source image using the specified matrix: $$ \texttt{dst}(x,y) = \texttt{src}\left(\frac{M_{11} x + M_{12} y + M_{13}}{M_{31} x + M_{32} y + M_{33}}, \frac{M_{21} x + M_{22} y + M_{23}}{M_{31} x + M_{32} y + M_{33}}\right) $$ when the flag WARP\_INVERSE\_MAP is set. The function cannot operate in-place. *** ### getPerspectiveTransform Calculates a perspective transform from four pairs of the corresponding points. ```cpp theme={null} Mat getPerspectiveTransform(InputArray src, InputArray dst, int solveMethod = DECOMP_LU); Mat getPerspectiveTransform(const Point2f src[], const Point2f dst[], int solveMethod = DECOMP_LU); ``` Coordinates of quadrangle vertices in the source image. Coordinates of the corresponding quadrangle vertices in the destination image. Method passed to cv::solve. The function calculates the 3 × 3 matrix of a perspective transform so that the four source points are mapped to the four destination points. *** ## Generic Remapping ### remap Applies a generic geometrical transformation to an image. ```cpp theme={null} void remap(InputArray src, OutputArray dst, InputArray map1, InputArray map2, int interpolation, int borderMode = BORDER_CONSTANT, const Scalar& borderValue = Scalar()); ``` Source image. Destination image. It has the same size as map1 and the same type as src. The first map of either (x,y) points or just x values having the type CV\_16SC2, CV\_32FC1, or CV\_32FC2. The second map of y values having the type CV\_16UC1, CV\_32FC1, or none (empty map if map1 is (x,y) points), respectively. Interpolation method. The methods INTER\_AREA, INTER\_LINEAR\_EXACT and INTER\_NEAREST\_EXACT are not supported by this function. Pixel extrapolation method. When borderMode=BORDER\_TRANSPARENT, it means that the pixels in the destination image that corresponds to the "outliers" in the source image are not modified by the function. Value used in case of a constant border. By default, it is 0. The function remap transforms the source image using the specified map: $$ \texttt{dst}(x,y) = \texttt{src}(\texttt{map}_x(x,y), \texttt{map}_y(x,y)) $$ This function cannot operate in-place. Due to current implementation limitations the size of an input and output images should be less than 32767x32767. *** ### getRectSubPix Retrieves a pixel rectangle from an image with sub-pixel accuracy. ```cpp theme={null} void getRectSubPix(InputArray image, Size patchSize, Point2f center, OutputArray patch, int patchType = -1); ``` Source image. Size of the extracted patch. Floating point coordinates of the center of the extracted rectangle within the source image. The center must be inside the image. Extracted patch that has the size patchSize and the same number of channels as src. Depth of the extracted pixels. By default, they have the same depth as src. The function getRectSubPix extracts pixels from src using bilinear interpolation. While the center of the rectangle must be inside the image, parts of the rectangle may be outside. *** ## Enumerations ### InterpolationFlags Interpolation algorithm: * `INTER_NEAREST` - Nearest neighbor interpolation * `INTER_LINEAR` - Bilinear interpolation * `INTER_CUBIC` - Bicubic interpolation * `INTER_AREA` - Resampling using pixel area relation (preferred for image decimation) * `INTER_LANCZOS4` - Lanczos interpolation over 8x8 neighborhood * `INTER_LINEAR_EXACT` - Bit exact bilinear interpolation * `INTER_NEAREST_EXACT` - Bit exact nearest neighbor interpolation * `INTER_MAX` - Mask for interpolation codes * `WARP_FILL_OUTLIERS` - Flag, fills all of the destination image pixels * `WARP_INVERSE_MAP` - Flag, inverse transformation ### WarpPolarMode Polar mapping mode: * `WARP_POLAR_LINEAR` - Remaps an image to/from polar space * `WARP_POLAR_LOG` - Remaps an image to/from semilog-polar space # Classification Algorithms Source: https://opencv-opencv.mintlify.app/api/ml/classification Machine learning classification algorithms including SVM, K-Nearest Neighbors, Decision Trees, Boosting, Random Trees, and Naive Bayes classifier The OpenCV Machine Learning module provides several powerful classification algorithms for supervised learning tasks. ## SVM - Support Vector Machines Support Vector Machines are powerful classifiers that work well for both linear and non-linear classification problems. ### Creating an SVM Model ```cpp theme={null} Ptr svm = SVM::create(); ``` ### Key Methods Sets the type of SVM formulation **Parameters:** * `val` (int): SVM type. See `SVM::Types` **Types:** * `SVM::C_SVC` (100): C-Support Vector Classification for n-class classification * `SVM::NU_SVC` (101): ν-Support Vector Classification with parameter ν * `SVM::ONE_CLASS` (102): Distribution Estimation (One-class SVM) * `SVM::EPS_SVR` (103): ε-Support Vector Regression * `SVM::NU_SVR` (104): ν-Support Vector Regression Initialize with one of predefined kernels **Parameters:** * `kernelType` (int): Kernel type. See `SVM::KernelTypes` **Kernel Types:** * `SVM::LINEAR` (0): Linear kernel, no mapping done * `SVM::POLY` (1): Polynomial kernel * `SVM::RBF` (2): Radial basis function, good choice in most cases * `SVM::SIGMOID` (3): Sigmoid kernel * `SVM::CHI2` (4): Exponential Chi2 kernel * `SVM::INTER` (5): Histogram intersection kernel Sets parameter C of a SVM optimization problem **Parameters:** * `val` (double): Parameter C for C\_SVC, EPS\_SVR or NU\_SVR Sets parameter γ of a kernel function **Parameters:** * `val` (double): Parameter gamma for POLY, RBF, SIGMOID or CHI2 kernels Trains the SVM model **Parameters:** * `trainData` (Ptr\): Training data * `flags` (int): Optional flags **Returns:** bool - true if training succeeded Predicts response for input samples **Parameters:** * `samples` (InputArray): Input samples, floating-point matrix * `results` (OutputArray): Optional output matrix of results * `flags` (int): Optional flags **Returns:** float - predicted response for single sample Trains an SVM with optimal parameters using cross-validation **Parameters:** * `data` (Ptr\): Training data * `kFold` (int): Cross-validation parameter (default: 10) * `Cgrid` (ParamGrid): Grid for C parameter * `gammaGrid` (ParamGrid): Grid for gamma parameter * `pGrid` (ParamGrid): Grid for p parameter * `nuGrid` (ParamGrid): Grid for nu parameter * `coeffGrid` (ParamGrid): Grid for coeff parameter * `degreeGrid` (ParamGrid): Grid for degree parameter * `balanced` (bool): Create balanced cross-validation subsets **Returns:** bool - true if training succeeded Retrieves all the support vectors **Returns:** Mat - matrix where support vectors are stored as rows ### Example Usage ```cpp theme={null} #include using namespace cv::ml; // Create and configure SVM Ptr svm = SVM::create(); svm->setType(SVM::C_SVC); svm->setKernel(SVM::RBF); svm->setGamma(0.5); svm->setC(1.0); svm->setTermCriteria(TermCriteria(TermCriteria::MAX_ITER, 100, 1e-6)); // Train the SVM Ptr trainData = TrainData::create(samples, ROW_SAMPLE, labels); svm->train(trainData); // Predict Mat results; svm->predict(testSamples, results); ``` ```python theme={null} import cv2 as cv import numpy as np # Create and configure SVM svm = cv.ml.SVM_create() svm.setType(cv.ml.SVM_C_SVC) svm.setKernel(cv.ml.SVM_RBF) svm.setGamma(0.5) svm.setC(1.0) svm.setTermCriteria((cv.TERM_CRITERIA_MAX_ITER, 100, 1e-6)) # Train the SVM svm.train(samples, cv.ml.ROW_SAMPLE, labels) # Predict result = svm.predict(test_samples) ``` *** ## KNearest - K-Nearest Neighbors The K-Nearest Neighbors algorithm finds the k nearest neighbors and predicts the response based on their values. ### Creating a KNearest Model ```cpp theme={null} Ptr knn = KNearest::create(); ``` ### Key Methods Sets the default number of neighbors to use in predict method **Parameters:** * `val` (int): Number of neighbors (must be greater than 1) Sets whether classification or regression model should be trained **Parameters:** * `val` (bool): true for classification, false for regression Sets the algorithm type **Parameters:** * `val` (int): Algorithm type **Types:** * `KNearest::BRUTE_FORCE` (1): Brute force search * `KNearest::KDTREE` (2): KD-tree based search Finds the neighbors and predicts responses for input vectors **Parameters:** * `samples` (InputArray): Input samples (rows are samples) * `k` (int): Number of nearest neighbors * `results` (OutputArray): Vector with prediction results * `neighborResponses` (OutputArray): Optional output for neighbor responses * `dist` (OutputArray): Optional output distances to neighbors **Returns:** float - predicted value for single input vector ### Example Usage ```cpp theme={null} #include using namespace cv::ml; // Create KNN classifier Ptr knn = KNearest::create(); knn->setDefaultK(5); knn->setIsClassifier(true); knn->setAlgorithmType(KNearest::BRUTE_FORCE); // Train knn->train(trainSamples, ROW_SAMPLE, trainLabels); // Predict Mat results, neighbors, distances; knn->findNearest(testSamples, 5, results, neighbors, distances); ``` *** ## DTrees - Decision Trees Decision trees are tree-based classifiers that split data based on feature values. ### Creating a Decision Tree Model ```cpp theme={null} Ptr dtree = DTrees::create(); ``` ### Key Methods Sets the maximum possible depth of the tree **Parameters:** * `val` (int): Maximum depth (root node has depth 0) Sets the minimum number of samples required to split a node **Parameters:** * `val` (int): Minimum sample count (default: 10) Sets the maximum number of categories for clustering **Parameters:** * `val` (int): Maximum categories (default: 10) Sets the number of folds for cross-validation pruning **Parameters:** * `val` (int): Number of folds (default: 10) Sets a priori class probabilities **Parameters:** * `val` (Mat): Array of class probabilities sorted by label ### Example Usage ```cpp theme={null} // Create and configure decision tree Ptr dtree = DTrees::create(); dtree->setMaxDepth(10); dtree->setMinSampleCount(10); dtree->setCVFolds(10); // Train dtree->train(trainData); // Predict float response = dtree->predict(testSample); ``` *** ## Boost - Boosted Trees Boosted tree classifier that combines multiple weak classifiers into a strong one. ### Creating a Boost Model ```cpp theme={null} Ptr boost = Boost::create(); ``` ### Key Methods Sets the type of boosting algorithm **Parameters:** * `val` (int): Boost type **Types:** * `Boost::DISCRETE` (0): Discrete AdaBoost * `Boost::REAL` (1): Real AdaBoost (default, works well with categorical data) * `Boost::LOGIT` (2): LogitBoost (good for regression) * `Boost::GENTLE` (3): Gentle AdaBoost (good with regression data) Sets the number of weak classifiers **Parameters:** * `val` (int): Number of weak classifiers (default: 100) Sets the threshold for computational time savings **Parameters:** * `val` (double): Weight trim rate between 0 and 1 (default: 0.95) ### Example Usage ```cpp theme={null} // Create and configure Boost Ptr boost = Boost::create(); boost->setBoostType(Boost::REAL); boost->setWeakCount(100); boost->setWeightTrimRate(0.95); boost->setMaxDepth(1); // Train boost->train(trainData); // Predict float result = boost->predict(testSample); ``` *** ## RTrees - Random Trees (Random Forest) Random Forest is an ensemble learning method that constructs multiple decision trees. ### Creating a Random Trees Model ```cpp theme={null} Ptr rtrees = RTrees::create(); ``` ### Key Methods Enable/disable variable importance calculation **Parameters:** * `val` (bool): true to calculate variable importance Sets the size of randomly selected subset of features at each tree node **Parameters:** * `val` (int): Number of active variables (0 = sqrt of total features) Sets termination criteria for training **Parameters:** * `val` (TermCriteria): Criteria specifying max iterations or accuracy Returns the variable importance array **Returns:** Mat - variable importance vector (if enabled during training) Returns the result of each individual tree in the forest **Parameters:** * `samples` (InputArray): Samples for which votes will be calculated * `results` (OutputArray): Matrix where results will be written * `flags` (int): Flags for defining the type of RTrees ### Example Usage ```cpp theme={null} // Create and configure Random Forest Ptr rtrees = RTrees::create(); rtrees->setMaxDepth(10); rtrees->setMinSampleCount(10); rtrees->setCalculateVarImportance(true); rtrees->setActiveVarCount(0); // sqrt of total features rtrees->setTermCriteria(TermCriteria(TermCriteria::MAX_ITER, 100, 0)); // Train rtrees->train(trainData); // Get variable importance Mat varImportance = rtrees->getVarImportance(); // Predict float result = rtrees->predict(testSample); ``` *** ## NormalBayesClassifier - Naive Bayes Bayes classifier for normally distributed data using Bayesian statistics. ### Creating a Naive Bayes Model ```cpp theme={null} Ptr bayes = NormalBayesClassifier::create(); ``` ### Key Methods Trains the Bayes classifier **Parameters:** * `trainData` (Ptr\): Training data * `flags` (int): Optional flags **Returns:** bool - true if training succeeded Predicts response for input samples **Parameters:** * `samples` (InputArray): Input samples * `results` (OutputArray): Output predictions * `flags` (int): Optional flags **Returns:** float - predicted class for single sample Predicts the response and returns probabilities **Parameters:** * `inputs` (InputArray): Input vectors (one or more) * `outputs` (OutputArray): Predicted classes * `outputProbs` (OutputArray): Output probabilities for each class * `flags` (int): Optional flags **Returns:** float - predicted class for single input ### Example Usage ```cpp theme={null} // Create Naive Bayes classifier Ptr bayes = NormalBayesClassifier::create(); // Train bayes->train(trainData); // Predict with probabilities Mat results, probs; bayes->predictProb(testSamples, results, probs); ``` The Naive Bayes classifier assumes that features are normally distributed and independent. It works best when these assumptions hold true. ## See Also * [Regression Algorithms](/api/ml/regression) - Linear and logistic regression methods * [Clustering Algorithms](/api/ml/clustering) - K-means and EM clustering * [StatModel Base Class](https://docs.opencv.org/4.x/dd/ded/classcv_1_1ml_1_1StatModel.html) - Base class for all ML models # Clustering Algorithms Source: https://opencv-opencv.mintlify.app/api/ml/clustering Unsupervised machine learning clustering algorithms including k-means and Expectation Maximization (EM) for Gaussian mixture models Clustering algorithms group data points into clusters based on their similarity without requiring labeled training data. ## kmeans - K-Means Clustering The k-means algorithm finds centers of clusters and groups input samples around the clusters. It's one of the most popular clustering algorithms. ### Function Signature ```cpp theme={null} double cv::kmeans( InputArray data, int K, InputOutputArray bestLabels, TermCriteria criteria, int attempts, int flags, OutputArray centers = noArray() ); ``` ### Parameters Data for clustering. An array of N-dimensional points with float coordinates. Examples: * `Mat points(count, 2, CV_32F)` - 2D points as rows * `Mat points(count, 1, CV_32FC2)` - 2D points as single channel * `Mat points(1, count, CV_32FC2)` - 2D points as columns * `std::vector points(sampleCount)` - vector of points Number of clusters to split the set by. Must be at least 2. Input/output integer array that stores the cluster indices for every sample. Each element is in the range \[0, K-1] indicating which cluster the sample belongs to. The algorithm termination criteria: maximum number of iterations and/or desired accuracy. The accuracy is specified as `criteria.epsilon`. The algorithm stops when each cluster center moves by less than epsilon. Example: `TermCriteria(TermCriteria::MAX_ITER + TermCriteria::EPS, 100, 0.01)` Number of times the algorithm is executed using different initial labellings. The algorithm returns the labels that yield the best compactness. Use at least 3 attempts for better results. Flag specifying the method for center initialization. **Flags:** * `KMEANS_RANDOM_CENTERS` (0): Select random initial centers in each attempt * `KMEANS_PP_CENTERS` (2): Use kmeans++ center initialization (recommended) * `KMEANS_USE_INITIAL_LABELS` (1): Use user-supplied labels for first attempt Output matrix of the cluster centers, one row per each cluster center. Size: K × dimensions ### Returns **Type:** `double` The function returns the compactness measure computed as: $\sum_i \|\text{samples}_i - \text{centers}_{\text{labels}_i}\|^2$ The best (minimum) compactness value is chosen among all attempts, and the corresponding labels and cluster centers are returned. ### Example Usage ```cpp theme={null} #include #include #include using namespace cv; using namespace std; int main() { // Generate random 2D points int sampleCount = 100; Mat points(sampleCount, 2, CV_32F); randu(points, Scalar(0, 0), Scalar(100, 100)); // K-means parameters int K = 3; Mat labels; Mat centers; TermCriteria criteria(TermCriteria::EPS + TermCriteria::MAX_ITER, 100, 0.01); // Run k-means double compactness = kmeans( points, K, labels, criteria, 3, // attempts KMEANS_PP_CENTERS, centers ); cout << "Compactness: " << compactness << endl; cout << "Centers:\n" << centers << endl; // Access cluster labels for (int i = 0; i < 10; i++) { cout << "Point " << i << " belongs to cluster " << labels.at(i) << endl; } return 0; } ``` ```python theme={null} import cv2 as cv import numpy as np import matplotlib.pyplot as plt # Generate random 2D points points = np.random.randint(0, 100, (100, 2)).astype(np.float32) # K-means parameters K = 3 criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 100, 0.01) # Run k-means compactness, labels, centers = cv.kmeans( points, K, None, criteria, attempts=3, flags=cv.KMEANS_PP_CENTERS ) print(f"Compactness: {compactness}") print(f"Centers:\n{centers}") # Visualize results colors = ['red', 'blue', 'green'] for i in range(K): cluster_points = points[labels.flatten() == i] plt.scatter(cluster_points[:, 0], cluster_points[:, 1], c=colors[i], label=f'Cluster {i}') plt.scatter(centers[:, 0], centers[:, 1], marker='x', s=200, c='black', label='Centers') plt.legend() plt.show() ``` For best results, use `KMEANS_PP_CENTERS` flag which implements the kmeans++ initialization algorithm by Arthur and Vassilvitskii. This provides better initial centers than random selection. *** ## EM - Expectation Maximization The Expectation Maximization algorithm implements Gaussian Mixture Models (GMM) for clustering. It models data as a mixture of multiple Gaussian distributions. ### Creating an EM Model ```cpp theme={null} Ptr em = EM::create(); ``` ### Key Methods Sets the number of mixture components **Parameters:** * `val` (int): Number of clusters/mixtures (default: 5) Sets the constraint on covariance matrices **Parameters:** * `val` (int): Type of covariance matrices **Types:** * `EM::COV_MAT_SPHERICAL` (0): Scaled identity matrix μ\_k \* I * `EM::COV_MAT_DIAGONAL` (1): Diagonal matrix with positive diagonal elements (recommended) * `EM::COV_MAT_GENERIC` (2): Symmetric positive definite matrix Sets the termination criteria of the EM algorithm **Parameters:** * `val` (TermCriteria): Criteria for max iterations or likelihood change The EM algorithm terminates when: * Maximum number of iterations (M-steps) is reached, OR * Relative change of likelihood logarithm is less than epsilon Default: `TermCriteria(TermCriteria::MAX_ITER + TermCriteria::EPS, 100, epsilon)` Estimates Gaussian mixture parameters from a sample set (Expectation step start) **Parameters:** * `samples` (InputArray): Samples from which GMM will be estimated (CV\_64F or will be converted) * `logLikelihoods` (OutputArray): Optional output matrix of likelihood logarithm values * `labels` (OutputArray): Optional output "class label" for each sample * `probs` (OutputArray): Optional posterior probabilities matrix (nsamples × nclusters) **Returns:** bool - true if training succeeded This variation starts with Expectation step. Initial values are estimated by k-means algorithm. Estimates Gaussian mixture parameters with initial means provided **Parameters:** * `samples` (InputArray): Training samples matrix * `means0` (InputArray): Initial means of mixture components (nclusters × dims) * `covs0` (InputArray): Optional initial covariance matrices * `weights0` (InputArray): Optional initial weights of mixture components * `logLikelihoods` (OutputArray): Optional likelihood logarithm output * `labels` (OutputArray): Optional cluster labels output * `probs` (OutputArray): Optional posterior probabilities output **Returns:** bool - true if training succeeded Estimates Gaussian mixture parameters starting with Maximization step **Parameters:** * `samples` (InputArray): Training samples * `probs0` (InputArray): Initial probabilities * `logLikelihoods` (OutputArray): Optional likelihood output * `labels` (OutputArray): Optional labels output * `probs` (OutputArray): Optional probabilities output **Returns:** bool - true if training succeeded Returns posterior probabilities for the provided samples **Parameters:** * `samples` (InputArray): Input samples matrix * `results` (OutputArray): Optional output matrix of results (nsamples × nclusters) * `flags` (int): Optional flags (ignored) **Returns:** float - predicted class for single sample Returns likelihood logarithm value and index of most probable mixture component **Parameters:** * `sample` (InputArray): A sample for classification (1 × dims or dims × 1) * `probs` (OutputArray): Optional posterior probabilities (1 × nclusters, CV\_64FC1) **Returns:** Vec2d * Element \[0]: Likelihood logarithm value * Element \[1]: Index of most probable mixture component Returns weights of the mixtures **Returns:** Mat - vector with number of elements equal to number of mixtures Returns the cluster centers (means of the Gaussian mixture) **Returns:** Mat - matrix with rows = number of mixtures, cols = space dimensionality Returns covariance matrices **Parameters:** * `covs` (std::vector\&): Output vector of covariance matrices Returns vector of covariation matrices (one per mixture, each is NxN where N is dimensionality) ### Example Usage ```cpp theme={null} #include #include using namespace cv; using namespace cv::ml; using namespace std; int main() { // Generate sample data Mat samples(300, 2, CV_32F); randn(samples.rowRange(0, 100), Scalar(0, 0), Scalar(10, 10)); randn(samples.rowRange(100, 200), Scalar(50, 50), Scalar(10, 10)); randn(samples.rowRange(200, 300), Scalar(25, 75), Scalar(10, 10)); // Create and configure EM Ptr em = EM::create(); em->setClustersNumber(3); em->setCovarianceMatrixType(EM::COV_MAT_DIAGONAL); em->setTermCriteria(TermCriteria( TermCriteria::MAX_ITER + TermCriteria::EPS, 100, 0.1 )); // Train EM model Mat labels, probs, logLikelihoods; em->trainEM(samples, logLikelihoods, labels, probs); // Get model parameters Mat means = em->getMeans(); Mat weights = em->getWeights(); vector covs; em->getCovs(covs); cout << "Means:\n" << means << endl; cout << "Weights:\n" << weights << endl; // Predict for new sample Mat testSample = (Mat_(1, 2) << 5.0, 5.0); Mat outputProbs; Vec2d prediction = em->predict2(testSample, outputProbs); cout << "Log likelihood: " << prediction[0] << endl; cout << "Most probable cluster: " << prediction[1] << endl; cout << "Probabilities: " << outputProbs << endl; return 0; } ``` ```python theme={null} import cv2 as cv import numpy as np import matplotlib.pyplot as plt # Generate sample data (3 Gaussian clusters) samples1 = np.random.randn(100, 2) * 10 + [0, 0] samples2 = np.random.randn(100, 2) * 10 + [50, 50] samples3 = np.random.randn(100, 2) * 10 + [25, 75] samples = np.vstack([samples1, samples2, samples3]).astype(np.float32) # Create and configure EM em = cv.ml.EM_create() em.setClustersNumber(3) em.setCovarianceMatrixType(cv.ml.EM_COV_MAT_DIAGONAL) em.setTermCriteria((cv.TERM_CRITERIA_MAX_ITER + cv.TERM_CRITERIA_EPS, 100, 0.1)) # Train EM model em.trainEM(samples) # Get model parameters means = em.getMeans() weights = em.getWeights() print(f"Means:\n{means}") print(f"Weights:\n{weights}") # Predict for new samples retval, probs = em.predict(samples) labels = probs.argmax(axis=1) # Visualize colors = ['red', 'blue', 'green'] for i in range(3): cluster_points = samples[labels == i] plt.scatter(cluster_points[:, 0], cluster_points[:, 1], c=colors[i], label=f'Cluster {i}') plt.scatter(means[:, 0], means[:, 1], marker='x', s=200, c='black', label='Means') plt.legend() plt.title('EM Clustering') plt.show() ``` ### When to Use EM vs K-Means **Use EM when:** * Clusters have different shapes and sizes * You need probabilistic cluster assignments * Data follows Gaussian distributions * You want to model uncertainty in cluster membership **Use K-Means when:** * Clusters are roughly spherical and similar in size * You need hard cluster assignments * Speed is critical (K-means is faster) * You have very large datasets The EM algorithm is more flexible than k-means as it can model elliptical clusters with different orientations and sizes. However, it's more computationally expensive and requires more samples for reliable estimation. ## See Also * [Classification Algorithms](/api/ml/classification) - SVM, KNN, Decision Trees, and more * [Regression Algorithms](/api/ml/regression) - Linear and logistic regression methods * [TrainData Class](https://docs.opencv.org/4.x/dc/d32/classcv_1_1ml_1_1TrainData.html) - Managing training data # Regression Algorithms Source: https://opencv-opencv.mintlify.app/api/ml/regression Machine learning regression algorithms including Logistic Regression and linear regression methods for predicting continuous and categorical outputs Regression algorithms predict continuous or categorical output values based on input features. ## LogisticRegression - Logistic Regression Classifier Logistic Regression is a statistical method for binary and multi-class classification problems. Despite its name, it's primarily used for classification rather than regression. ### Creating a Logistic Regression Model ```cpp theme={null} Ptr lr = LogisticRegression::create(); ``` ### Key Methods Sets the learning rate for gradient descent **Parameters:** * `val` (double): Learning rate (step size for parameter updates) Typical values: 0.001 to 0.1. Higher values train faster but may overshoot the optimal solution. Sets the number of training iterations **Parameters:** * `val` (int): Maximum number of iterations More iterations can lead to better convergence but increase training time. Sets the kind of regularization to be applied **Parameters:** * `val` (int): Regularization type **Types:** * `LogisticRegression::REG_DISABLE` (-1): No regularization * `LogisticRegression::REG_L1` (0): L1 norm regularization (promotes sparsity) * `LogisticRegression::REG_L2` (1): L2 norm regularization (prevents overfitting) Sets the training method used **Parameters:** * `val` (int): Training method **Methods:** * `LogisticRegression::BATCH` (0): Batch gradient descent * `LogisticRegression::MINI_BATCH` (1): Mini-batch gradient descent Sets the number of training samples taken in each Mini-Batch Gradient Descent step **Parameters:** * `val` (int): Mini-batch size (must be less than total training samples) Only used when training method is `MINI_BATCH`. Smaller batches provide more frequent updates but noisier gradients. Sets the termination criteria of the algorithm **Parameters:** * `val` (TermCriteria): Criteria for stopping training Example: `TermCriteria(TermCriteria::MAX_ITER + TermCriteria::EPS, 1000, 0.001)` Trains the logistic regression model **Parameters:** * `samples` (InputArray): Training samples (CV\_32F type) * `layout` (int): Sample layout (ROW\_SAMPLE or COL\_SAMPLE) * `responses` (InputArray): Training labels/responses **Returns:** bool - true if training succeeded Predicts responses for input samples **Parameters:** * `samples` (InputArray): Input data for prediction (CV\_32F type, m × n matrix) * `results` (OutputArray): Predicted labels as column matrix (CV\_32S type) * `flags` (int): Optional flags (not used) **Returns:** float - predicted value for single sample Returns the trained parameters **Returns:** Mat - learnt parameters of Logistic Regression (CV\_32F type) For two-class classification, returns a row matrix. These are the weights/coefficients learned during training. ### Example Usage ```cpp theme={null} #include #include using namespace cv; using namespace cv::ml; using namespace std; int main() { // Prepare training data (binary classification) Mat trainData = (Mat_(6, 2) << 1.0, 1.0, // Class 0 2.0, 1.0, 1.0, 2.0, 6.0, 6.0, // Class 1 5.0, 6.0, 6.0, 5.0 ); Mat labels = (Mat_(6, 1) << 0, 0, 0, 1, 1, 1); // Create and configure Logistic Regression Ptr lr = LogisticRegression::create(); lr->setLearningRate(0.001); lr->setIterations(10000); lr->setRegularization(LogisticRegression::REG_L2); lr->setTrainMethod(LogisticRegression::BATCH); lr->setMiniBatchSize(1); // Train the model Ptr tData = TrainData::create( trainData, ROW_SAMPLE, labels ); lr->train(tData); // Get learned parameters Mat theta = lr->get_learnt_thetas(); cout << "Learned parameters (theta):\n" << theta << endl; // Predict on test data Mat testData = (Mat_(2, 2) << 1.5, 1.5, // Should predict class 0 5.5, 5.5 // Should predict class 1 ); Mat predictions; lr->predict(testData, predictions); cout << "Predictions:\n" << predictions << endl; // Evaluate accuracy Mat trainPredictions; lr->predict(trainData, trainPredictions); int correct = 0; for (int i = 0; i < labels.rows; i++) { if (trainPredictions.at(i) == labels.at(i)) correct++; } cout << "Training accuracy: " << (100.0 * correct / labels.rows) << "%" << endl; return 0; } ``` ```python theme={null} import cv2 as cv import numpy as np # Prepare training data (binary classification) train_data = np.array([ [1.0, 1.0], # Class 0 [2.0, 1.0], [1.0, 2.0], [6.0, 6.0], # Class 1 [5.0, 6.0], [6.0, 5.0] ], dtype=np.float32) labels = np.array([[0], [0], [0], [1], [1], [1]], dtype=np.int32) # Create and configure Logistic Regression lr = cv.ml.LogisticRegression_create() lr.setLearningRate(0.001) lr.setIterations(10000) lr.setRegularization(cv.ml.LogisticRegression_REG_L2) lr.setTrainMethod(cv.ml.LogisticRegression_BATCH) lr.setMiniBatchSize(1) # Train the model lr.train(train_data, cv.ml.ROW_SAMPLE, labels) # Get learned parameters theta = lr.get_learnt_thetas() print(f"Learned parameters (theta):\n{theta}") # Predict on test data test_data = np.array([ [1.5, 1.5], # Should predict class 0 [5.5, 5.5] # Should predict class 1 ], dtype=np.float32) retval, predictions = lr.predict(test_data) print(f"Predictions:\n{predictions}") # Evaluate accuracy retval, train_predictions = lr.predict(train_data) accuracy = np.mean(train_predictions.flatten() == labels.flatten()) * 100 print(f"Training accuracy: {accuracy}%") ``` ### Multi-Class Classification For multi-class problems (more than 2 classes), Logistic Regression uses a one-vs-rest approach: ```cpp theme={null} // Multi-class example (3 classes) Mat trainData = (Mat_(9, 2) << 1.0, 1.0, // Class 0 2.0, 1.0, 1.0, 2.0, 6.0, 6.0, // Class 1 5.0, 6.0, 6.0, 5.0, 3.0, 8.0, // Class 2 4.0, 9.0, 3.5, 8.5 ); Mat labels = (Mat_(9, 1) << 0, 0, 0, 1, 1, 1, 2, 2, 2); // Train and predict as before Ptr lr = LogisticRegression::create(); lr->setLearningRate(0.01); lr->setIterations(5000); lr->train(trainData, ROW_SAMPLE, labels); Mat predictions; lr->predict(testData, predictions); ``` For binary classification, ensure your labels are 0 and 1. For multi-class classification, use consecutive integers starting from 0 (e.g., 0, 1, 2, 3...). *** ## Linear Regression with Normal Equations While OpenCV doesn't have a dedicated linear regression class, you can perform linear regression using the `solve()` function with normal equations or use the `SVM` class with `SVM::EPS_SVR` type. ### Method 1: Using Normal Equations Linear regression can be solved directly using the normal equation: θ = (X^T X)^(-1) X^T y ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { // Training data: y = 2x + 3 Mat X = (Mat_(5, 2) << 1.0, 1.0, 1.0, 2.0, 1.0, 3.0, 1.0, 4.0, 1.0, 5.0 ); // First column is bias term (1s) Mat y = (Mat_(5, 1) << 5.0, 7.0, 9.0, 11.0, 13.0); // Solve normal equation: theta = (X^T * X)^(-1) * X^T * y Mat theta; solve(X, y, theta, DECOMP_SVD); cout << "Learned parameters:\n" << theta << endl; // Should be approximately [3.0, 2.0] (intercept, slope) // Make predictions Mat testX = (Mat_(3, 2) << 1.0, 6.0, 1.0, 7.0, 1.0, 8.0 ); Mat predictions = testX * theta; cout << "Predictions:\n" << predictions << endl; return 0; } ``` ### Method 2: Using SVM for Regression For more robust regression with regularization, use `SVM::EPS_SVR`: ```cpp theme={null} #include using namespace cv; using namespace cv::ml; // Prepare training data Mat trainData = (Mat_(5, 1) << 1.0, 2.0, 3.0, 4.0, 5.0); Mat responses = (Mat_(5, 1) << 5.0, 7.0, 9.0, 11.0, 13.0); // Create SVM for regression Ptr svm = SVM::create(); svm->setType(SVM::EPS_SVR); svm->setKernel(SVM::LINEAR); svm->setC(1.0); svm->setP(0.1); // epsilon parameter // Train Ptr tData = TrainData::create( trainData, ROW_SAMPLE, responses ); svm->train(tData); // Predict Mat testData = (Mat_(3, 1) << 6.0, 7.0, 8.0); Mat predictions; svm->predict(testData, predictions); ``` ### Method 3: Using Decision Trees for Regression `DTrees` can also be used for regression by setting appropriate parameters: ```cpp theme={null} #include using namespace cv::ml; Ptr dtree = DTrees::create(); dtree->setMaxDepth(10); dtree->setMinSampleCount(2); dtree->setRegressionAccuracy(0.01f); // Train with continuous response values Ptr trainData = TrainData::create( samples, ROW_SAMPLE, continuousResponses // Regression targets ); dtree->train(trainData); // Predict float prediction = dtree->predict(testSample); ``` *** ## Polynomial Regression For polynomial regression, transform your input features to include polynomial terms: ```cpp theme={null} #include using namespace cv; // Function to create polynomial features Mat createPolynomialFeatures(const Mat& X, int degree) { int rows = X.rows; int cols = X.cols; // Calculate number of output features int outCols = 1; // bias term for (int d = 1; d <= degree; d++) { outCols += cols; // Add linear terms, squared terms, etc. } Mat polyFeatures(rows, outCols, CV_32F); for (int i = 0; i < rows; i++) { int colIdx = 0; polyFeatures.at(i, colIdx++) = 1.0f; // bias for (int d = 1; d <= degree; d++) { for (int j = 0; j < cols; j++) { float val = X.at(i, j); polyFeatures.at(i, colIdx++) = pow(val, d); } } } return polyFeatures; } // Example usage Mat X = (Mat_(5, 1) << 1.0, 2.0, 3.0, 4.0, 5.0); Mat y = (Mat_(5, 1) << 1.0, 4.0, 9.0, 16.0, 25.0); // y = x^2 // Create polynomial features (degree 2) Mat X_poly = createPolynomialFeatures(X, 2); // Solve using normal equations Mat theta; solve(X_poly, y, theta, DECOMP_SVD); ``` When using polynomial features, consider normalizing your data first to prevent numerical instability. Higher degree polynomials can lead to overfitting. *** ## Regularized Regression For Ridge Regression (L2 regularization), modify the normal equation: θ = (X^T X + λI)^(-1) X^T y ```cpp theme={null} Mat XtX = X.t() * X; Mat identity = Mat::eye(XtX.rows, XtX.cols, XtX.type()); float lambda = 0.1; // Regularization parameter Mat regularized = XtX + lambda * identity; Mat Xty = X.t() * y; Mat theta; solve(regularized, Xty, theta, DECOMP_CHOLESKY); ``` *** ## Best Practices ### Feature Scaling Always normalize features when using gradient-based methods: ```cpp theme={null} // Normalize features to [0, 1] or standardize to mean=0, std=1 Mat mean, stddev; cv::meanStdDev(trainData, mean, stddev); Mat normalizedData = (trainData - mean) / stddev; ``` ### Cross-Validation Use cross-validation to evaluate model performance: ```cpp theme={null} Ptr data = TrainData::create( samples, ROW_SAMPLE, responses ); // Split into train and test data->setTrainTestSplitRatio(0.8, true); Ptr lr = LogisticRegression::create(); lr->train(data->getTrainSamples(), ROW_SAMPLE, data->getTrainResponses()); float trainError = lr->calcError(data, false, noArray()); float testError = lr->calcError(data, true, noArray()); ``` ### Hyperparameter Tuning Try different learning rates and regularization parameters: ```cpp theme={null} vector learningRates = {0.001, 0.01, 0.1}; vector regularizations = { LogisticRegression::REG_DISABLE, LogisticRegression::REG_L1, LogisticRegression::REG_L2 }; float bestError = FLT_MAX; Ptr bestModel; for (double lr : learningRates) { for (int reg : regularizations) { Ptr model = LogisticRegression::create(); model->setLearningRate(lr); model->setRegularization(reg); model->train(trainData); float error = model->calcError(testData, true, noArray()); if (error < bestError) { bestError = error; bestModel = model; } } } ``` ## See Also * [Classification Algorithms](/api/ml/classification) - SVM, Decision Trees, and classifiers * [Clustering Algorithms](/api/ml/clustering) - K-means and EM clustering * [StatModel Base Class](https://docs.opencv.org/4.x/dd/ded/classcv_1_1ml_1_1StatModel.html) - Base class for all ML models # ArUco Marker Detection Source: https://opencv-opencv.mintlify.app/api/objdetect/aruco API reference for ArUco marker detection, CharUco board detection, and dictionary management # ArUco Marker Detection API reference for detecting ArUco markers and CharUco boards for robust camera pose estimation. ## ArucoDetector Main class for detecting ArUco markers in images. ### Constructor ```cpp theme={null} cv::aruco::ArucoDetector::ArucoDetector( const Dictionary& dictionary = getPredefinedDictionary(cv::aruco::DICT_4X4_50), const DetectorParameters& detectorParams = DetectorParameters(), const RefineParameters& refineParams = RefineParameters() ) ``` Dictionary indicating the type of markers that will be searched Marker detection parameters Marker refine detection parameters #### Multi-Dictionary Constructor ```cpp theme={null} cv::aruco::ArucoDetector::ArucoDetector( const std::vector& dictionaries, const DetectorParameters& detectorParams = DetectorParameters(), const RefineParameters& refineParams = RefineParameters() ) ``` Multiple dictionaries for marker detection. Cannot be empty. ### Methods #### detectMarkers Basic marker detection in an image. ```cpp theme={null} void detectMarkers( InputArray image, OutputArrayOfArrays corners, OutputArray ids, OutputArrayOfArrays rejectedImgPoints = noArray() ) const ``` Input image where markers will be detected Vector of detected marker corners. For each marker, its four corners are provided (clockwise order). For N detected markers, dimensions are Nx4. Vector of identifiers of the detected markers. For N detected markers, the size is N. Contains the corners of squares whose inner code has incorrect codification. Useful for debugging. The function does not correct lens distortion. It's recommended to undistort the input image if camera parameters are known. #### detectMarkersWithConfidence Marker detection with confidence computation. ```cpp theme={null} void detectMarkersWithConfidence( InputArray image, OutputArrayOfArrays corners, OutputArray ids, OutputArray markersConfidence, OutputArrayOfArrays rejectedImgPoints = noArray() ) const ``` Contains the normalized confidence \[0;1] of the markers' detection, defined as 1 minus the normalized uncertainty (percentage of incorrect pixel detections). #### refineDetectedMarkers Refine undetected markers based on already detected markers and board layout. ```cpp theme={null} void refineDetectedMarkers( InputArray image, const Board& board, InputOutputArrayOfArrays detectedCorners, InputOutputArray detectedIds, InputOutputArrayOfArrays rejectedCorners, InputArray cameraMatrix = noArray(), InputArray distCoeffs = noArray(), OutputArray recoveredIdxs = noArray() ) const ``` Input image Layout of markers in the board Vector of already detected marker corners Vector of already detected marker identifiers Vector of rejected candidates during the marker detection process Optional 3x3 floating-point camera matrix Optional vector of distortion coefficients Optional array to return the indexes of recovered candidates in the original rejectedCorners array #### getDictionary / setDictionary ```cpp theme={null} const Dictionary& getDictionary() const void setDictionary(const Dictionary& dictionary) ``` Gets or sets the first dictionary used for marker detection. #### getDetectorParameters / setDetectorParameters ```cpp theme={null} const DetectorParameters& getDetectorParameters() const void setDetectorParameters(const DetectorParameters& detectorParameters) ``` Gets or sets the detector parameters. #### getRefineParameters / setRefineParameters ```cpp theme={null} const RefineParameters& getRefineParameters() const void setRefineParameters(const RefineParameters& refineParameters) ``` Gets or sets the refine parameters. ### Example Usage ```cpp theme={null} #include #include // Create ArUco detector with 6x6 dictionary cv::aruco::Dictionary dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_250); cv::aruco::DetectorParameters detectorParams = cv::aruco::DetectorParameters(); cv::aruco::ArucoDetector detector(dictionary, detectorParams); // Detect markers std::vector> corners; std::vector ids; std::vector> rejected; detector.detectMarkers(image, corners, ids, rejected); // Draw detected markers if (!ids.empty()) { cv::aruco::drawDetectedMarkers(image, corners, ids); } std::cout << "Detected " << ids.size() << " markers" << std::endl; ``` ```python theme={null} import cv2 import numpy as np # Create ArUco detector with 6x6 dictionary dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250) parameters = cv2.aruco.DetectorParameters() detector = cv2.aruco.ArucoDetector(dictionary, parameters) # Detect markers corners, ids, rejected = detector.detectMarkers(image) # Draw detected markers if ids is not None: cv2.aruco.drawDetectedMarkers(image, corners, ids) print(f"Detected {len(ids) if ids is not None else 0} markers") ``` *** ## Dictionary A dictionary is a set of unique ArUco markers of the same size. ### Constructor ```cpp theme={null} cv::aruco::Dictionary::Dictionary() cv::aruco::Dictionary::Dictionary( const Mat& bytesList, int markerSize, int maxCorrectionBits = 0 ) ``` Bits for all ArUco markers in dictionary (CV\_8UC4 type) ArUco marker size in units (number of bits per dimension) Maximum number of bits that can be corrected ### Properties * `bytesList` (Mat): Marker code information stored as 2D matrix with 4 channels * `markerSize` (int): Number of bits per dimension * `maxCorrectionBits` (int): Maximum number of bits that can be corrected ### Methods #### identify Given a matrix of bits, returns whether the marker is identified. ```cpp theme={null} bool identify( const Mat& onlyBits, int& idx, int& rotation, double maxCorrectionRate ) const ``` Input matrix of bits Output marker ID in the dictionary (if any) Output marker rotation (0-3) Maximum error correction rate **Returns:** `true` if marker is identified #### generateImageMarker Generates a canonical marker image. ```cpp theme={null} void generateImageMarker( int id, int sidePixels, OutputArray img, int borderBits = 1 ) const ``` Marker ID to generate Size of the output image in pixels Output marker image Width of the marker border ### Predefined Dictionaries #### getPredefinedDictionary ```cpp theme={null} Dictionary getPredefinedDictionary(PredefinedDictionaryType name) Dictionary getPredefinedDictionary(int dict) ``` Available predefined dictionaries: * `DICT_4X4_50` - 4x4 bits, 50 markers, hamming distance 4 * `DICT_4X4_100` - 4x4 bits, 100 markers, hamming distance 3 * `DICT_4X4_250` - 4x4 bits, 250 markers, hamming distance 3 * `DICT_4X4_1000` - 4x4 bits, 1000 markers, hamming distance 2 * `DICT_5X5_50` - 5x5 bits, 50 markers, hamming distance 8 * `DICT_5X5_100` - 5x5 bits, 100 markers, hamming distance 7 * `DICT_5X5_250` - 5x5 bits, 250 markers, hamming distance 6 * `DICT_5X5_1000` - 5x5 bits, 1000 markers, hamming distance 5 * `DICT_6X6_50` - 6x6 bits, 50 markers, hamming distance 13 * `DICT_6X6_100` - 6x6 bits, 100 markers, hamming distance 12 * `DICT_6X6_250` - 6x6 bits, 250 markers, hamming distance 11 * `DICT_6X6_1000` - 6x6 bits, 1000 markers, hamming distance 9 * `DICT_7X7_50` - 7x7 bits, 50 markers, hamming distance 19 * `DICT_7X7_100` - 7x7 bits, 100 markers, hamming distance 18 * `DICT_7X7_250` - 7x7 bits, 250 markers, hamming distance 17 * `DICT_7X7_1000` - 7x7 bits, 1000 markers, hamming distance 14 * `DICT_ARUCO_ORIGINAL` - 6x6 bits, 1024 markers (standard ArUco Library) * `DICT_APRILTAG_16h5` - 4x4 bits, 30 markers, hamming distance 5 * `DICT_APRILTAG_25h9` - 5x5 bits, 35 markers, hamming distance 9 * `DICT_APRILTAG_36h10` - 6x6 bits, 2320 markers, hamming distance 10 * `DICT_APRILTAG_36h11` - 6x6 bits, 587 markers, hamming distance 11 * `DICT_ARUCO_MIP_36h12` - 6x6 bits, 250 markers, hamming distance 12 *** ## CharucoDetector Detector for ChArUco boards (chessboard + ArUco markers). ### Constructor ```cpp theme={null} cv::aruco::CharucoDetector::CharucoDetector( const CharucoBoard& board, const CharucoParameters& charucoParams = CharucoParameters(), const DetectorParameters& detectorParams = DetectorParameters(), const RefineParameters& refineParams = RefineParameters() ) ``` ChArUco board configuration ChArUco detection parameters Marker detection parameters Marker refine detection parameters ### Methods #### detectBoard Detects ArUco markers and interpolates ChArUco board corners. ```cpp theme={null} void detectBoard( InputArray image, OutputArray charucoCorners, OutputArray charucoIds, InputOutputArrayOfArrays markerCorners = noArray(), InputOutputArray markerIds = noArray() ) const ``` Input image necessary for corner refinement Interpolated chessboard corners Interpolated chessboard corner identifiers Vector of already detected marker corners. If empty, the function will detect markers. List of identifiers for each marker in corners. If empty, the function will detect markers. After OpenCV 4.6.0, there was an incompatible change in the ChArUco pattern generation algorithm for even row counts. Use `CharucoBoard::setLegacyPattern()` to ensure compatibility with patterns created before 4.6.0. #### detectDiamonds Detects ChArUco Diamond markers. ```cpp theme={null} void detectDiamonds( InputArray image, OutputArrayOfArrays diamondCorners, OutputArray diamondIds, InputOutputArrayOfArrays markerCorners = noArray(), InputOutputArray markerIds = noArray() ) const ``` Input image necessary for corner subpixel accuracy Output list of detected diamond corners (4 corners per diamond) in clockwise order IDs of the diamonds. Each diamond has 4 IDs corresponding to the ArUco markers composing it. List of detected marker corners. If empty, the function will detect markers. List of marker IDs. If empty, the function will detect markers. ### Example Usage ```cpp theme={null} #include #include // Create ChArUco board cv::aruco::Dictionary dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_250); cv::aruco::CharucoBoard board(cv::Size(5, 7), 0.04f, 0.02f, dictionary); // Create detector cv::aruco::CharucoDetector detector(board); // Detect ChArUco corners std::vector charucoCorners; std::vector charucoIds; std::vector> markerCorners; std::vector markerIds; detector.detectBoard(image, charucoCorners, charucoIds, markerCorners, markerIds); // Draw detected corners if (!charucoIds.empty()) { cv::aruco::drawDetectedCornersCharuco(image, charucoCorners, charucoIds, cv::Scalar(255, 0, 0)); } ``` ```python theme={null} import cv2 import numpy as np # Create ChArUco board dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250) board = cv2.aruco.CharucoBoard((5, 7), 0.04, 0.02, dictionary) # Create detector detector = cv2.aruco.CharucoDetector(board) # Detect ChArUco corners charuco_corners, charuco_ids, marker_corners, marker_ids = \ detector.detectBoard(image) # Draw detected corners if charuco_ids is not None and len(charuco_ids) > 0: cv2.aruco.drawDetectedCornersCharuco(image, charuco_corners, charuco_ids, (255, 0, 0)) ``` ## DetectorParameters Parameters for ArUco marker detection. ### Key Parameters ```cpp theme={null} struct DetectorParameters { int adaptiveThreshWinSizeMin; // default: 3 int adaptiveThreshWinSizeMax; // default: 23 int adaptiveThreshWinSizeStep; // default: 10 double adaptiveThreshConstant; // default: 7 double minMarkerPerimeterRate; // default: 0.03 double maxMarkerPerimeterRate; // default: 4.0 double polygonalApproxAccuracyRate; // default: 0.03 double minCornerDistanceRate; // default: 0.05 int minDistanceToBorder; // default: 3 double minMarkerDistanceRate; // default: 0.125 int cornerRefinementMethod; // default: CORNER_REFINE_NONE int cornerRefinementWinSize; // default: 5 int cornerRefinementMaxIterations; // default: 30 double cornerRefinementMinAccuracy; // default: 0.1 int markerBorderBits; // default: 1 int perspectiveRemovePixelPerCell; // default: 4 double errorCorrectionRate; // default: 0.6 bool detectInvertedMarker; // default: false bool useAruco3Detection; // default: false } ``` Corner refinement method: * `CORNER_REFINE_NONE` - No refinement * `CORNER_REFINE_SUBPIX` - Subpixel corner refinement * `CORNER_REFINE_CONTOUR` - Contour-based refinement * `CORNER_REFINE_APRILTAG` - AprilTag approach Enable the new and faster ArUco 3 detection strategy (from Romero-Ramirez et al. 2018) ## Utility Functions ### drawDetectedMarkers Draws detected markers in an image. ```cpp theme={null} void drawDetectedMarkers( InputOutputArray image, InputArrayOfArrays corners, InputArray ids = noArray(), Scalar borderColor = Scalar(0, 255, 0) ) ``` ### generateImageMarker Generates a canonical marker image. ```cpp theme={null} void generateImageMarker( const Dictionary& dictionary, int id, int sidePixels, OutputArray img, int borderBits = 1 ) ``` ## See Also * [Cascade Classifier](/api/objdetect/cascade) * [Face Detection](/api/objdetect/face) * [QR Code Detection](/api/objdetect/qrcode) # Cascade Classifier Source: https://opencv-opencv.mintlify.app/api/objdetect/cascade API reference for CascadeClassifier and HOGDescriptor classes for object detection # Cascade Classifier API reference for cascade-based object detection and HOG descriptor computation. ## CascadeClassifier Cascade classifier class for object detection using Haar or LBP features. ### Constructor ```cpp theme={null} cv::CascadeClassifier::CascadeClassifier() cv::CascadeClassifier::CascadeClassifier(const String& filename) ``` Path to the classifier file (e.g., haarcascade\_frontalface\_default.xml) ### Methods #### load Loads a classifier from a file. ```cpp theme={null} bool load(const String& filename) ``` Name of the file from which the classifier is loaded. The file may contain an old HAAR classifier trained by the haartraining application or a new cascade classifier trained by the traincascade application. **Returns:** `true` if the classifier was loaded successfully #### detectMultiScale Detects objects of different sizes in the input image. ```cpp theme={null} void detectMultiScale( InputArray image, std::vector& objects, double scaleFactor = 1.1, int minNeighbors = 3, int flags = 0, Size minSize = Size(), Size maxSize = Size() ) ``` Matrix of type CV\_8U containing an image where objects are detected Output vector of rectangles where each rectangle contains a detected object Parameter specifying how much the image size is reduced at each image scale Parameter specifying how many neighbors each candidate rectangle should have to retain it Parameter with the same meaning for an old cascade as in cvHaarDetectObjects. Not used for new cascades. Minimum possible object size. Objects smaller than this are ignored. Maximum possible object size. Objects larger than this are ignored. If maxSize == minSize, model is evaluated on single scale. The function does not correct lens distortion. If camera parameters are known, it's recommended to undistort the input image first. #### detectMultiScale (with detection counts) ```cpp theme={null} void detectMultiScale( InputArray image, std::vector& objects, std::vector& numDetections, double scaleFactor = 1.1, int minNeighbors = 3, int flags = 0, Size minSize = Size(), Size maxSize = Size() ) ``` Vector of detection numbers for the corresponding objects. An object's number of detections is the number of neighboring positively classified rectangles that were joined together. #### detectMultiScale (with confidence levels) ```cpp theme={null} void detectMultiScale( InputArray image, std::vector& objects, std::vector& rejectLevels, std::vector& levelWeights, double scaleFactor = 1.1, int minNeighbors = 3, int flags = 0, Size minSize = Size(), Size maxSize = Size(), bool outputRejectLevels = false ) ``` Output vector of reject levels for each detection Output vector containing the certainty of classification at the final stage for each detection Set to true to retrieve the final stage decision certainty of classification #### empty Checks whether the classifier has been loaded. ```cpp theme={null} bool empty() const ``` **Returns:** `true` if the classifier is empty ### Example Usage ```cpp theme={null} #include #include // Load the cascade classifier cv::CascadeClassifier face_cascade; face_cascade.load("haarcascade_frontalface_default.xml"); // Detect faces cv::Mat gray; cv::cvtColor(image, gray, cv::COLOR_BGR2GRAY); std::vector faces; face_cascade.detectMultiScale(gray, faces, 1.1, 3); // Draw rectangles around detected faces for (const auto& face : faces) { cv::rectangle(image, face, cv::Scalar(255, 0, 0), 2); } ``` ```python theme={null} import cv2 # Load the cascade classifier face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml') # Convert to grayscale gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) # Detect faces faces = face_cascade.detectMultiScale(gray, 1.1, 3) # Draw rectangles around detected faces for (x, y, w, h) in faces: cv2.rectangle(image, (x, y), (x+w, y+h), (255, 0, 0), 2) ``` *** ## HOGDescriptor Implementation of HOG (Histogram of Oriented Gradients) descriptor and object detector. Based on the algorithm introduced by Navneet Dalal and Bill Triggs. ### Constructor ```cpp theme={null} cv::HOGDescriptor::HOGDescriptor() cv::HOGDescriptor::HOGDescriptor( Size winSize, Size blockSize, Size blockStride, Size cellSize, int nbins, int derivAperture = 1, double winSigma = -1, HOGDescriptor::HistogramNormType histogramNormType = HOGDescriptor::L2Hys, double L2HysThreshold = 0.2, bool gammaCorrection = false, int nlevels = HOGDescriptor::DEFAULT_NLEVELS, bool signedGradient = false ) cv::HOGDescriptor::HOGDescriptor(const String& filename) ``` Detection window size. Default is Size(64, 128). Must align to block size and block stride. Block size in pixels. Default is Size(16, 16). Must align to cell size. Block stride. Default is Size(8, 8). Must be a multiple of cell size. Cell size. Default is Size(8, 8). Number of bins used in the calculation of histogram of gradients. Default is 9. File name containing HOGDescriptor properties and coefficients for the linear SVM classifier ### Methods #### compute Computes HOG descriptors of given image. ```cpp theme={null} void compute( InputArray img, std::vector& descriptors, Size winStride = Size(), Size padding = Size(), const std::vector& locations = std::vector() ) const ``` Matrix of type CV\_8U containing an image where HOG features will be calculated Output matrix of type CV\_32F containing computed descriptors Window stride. Must be a multiple of block stride. Padding around the image Vector of specific locations to compute descriptors at #### detect Performs object detection without a multi-scale window. ```cpp theme={null} void detect( InputArray img, std::vector& foundLocations, std::vector& weights, double hitThreshold = 0, Size winStride = Size(), Size padding = Size(), const std::vector& searchLocations = std::vector() ) const ``` Matrix of type CV\_8U or CV\_8UC3 containing an image where objects are detected Vector of points where each point contains left-top corner of detected object boundaries Vector that will contain confidence values for each detected object Threshold for the distance between features and SVM classifying plane Window stride. Must be a multiple of block stride. Padding around the image Vector of specific locations to search #### detectMultiScale Detects objects of different sizes in the input image. ```cpp theme={null} void detectMultiScale( InputArray img, std::vector& foundLocations, std::vector& foundWeights, double hitThreshold = 0, Size winStride = Size(), Size padding = Size(), double scale = 1.05, double groupThreshold = 2.0, bool useMeanshiftGrouping = false ) const ``` Matrix of type CV\_8U or CV\_8UC3 containing an image where objects are detected Vector of rectangles where each rectangle contains the detected object Vector that will contain confidence values for each detected object Coefficient of the detection window increase Coefficient to regulate the similarity threshold. When detected, some objects can be covered by many rectangles. 0 means not to perform grouping. Indicates whether to use meanshift grouping algorithm #### setSVMDetector Sets coefficients for the linear SVM classifier. ```cpp theme={null} void setSVMDetector(InputArray svmdetector) ``` Coefficients for the linear SVM classifier #### getDefaultPeopleDetector Returns coefficients of the classifier trained for people detection (for 64x128 windows). ```cpp theme={null} static std::vector getDefaultPeopleDetector() ``` **Returns:** Vector of SVM coefficients for people detection #### getDaimlerPeopleDetector Returns coefficients of the classifier trained for people detection (for 48x96 windows). ```cpp theme={null} static std::vector getDaimlerPeopleDetector() ``` **Returns:** Vector of SVM coefficients for Daimler people detection ### Properties * `winSize` (Size): Detection window size. Default Size(64, 128). * `blockSize` (Size): Block size in pixels. Default Size(16, 16). * `blockStride` (Size): Block stride. Default Size(8, 8). * `cellSize` (Size): Cell size. Default Size(8, 8). * `nbins` (int): Number of bins. Default 9. * `derivAperture` (int): Derivative aperture. * `winSigma` (double): Gaussian smoothing window parameter. * `histogramNormType` (HistogramNormType): Histogram normalization type. * `L2HysThreshold` (double): L2-Hys normalization method shrinkage. * `gammaCorrection` (bool): Flag to specify gamma correction preprocessing. * `svmDetector` (std::vector\): Coefficients for the linear SVM classifier. * `nlevels` (int): Maximum number of detection window increases. Default 64. * `signedGradient` (bool): Indicates whether signed gradient will be used. ### Example Usage ```cpp theme={null} #include #include // Create HOG descriptor with default people detector cv::HOGDescriptor hog; hog.setSVMDetector(cv::HOGDescriptor::getDefaultPeopleDetector()); // Detect people in the image std::vector found; std::vector weights; hog.detectMultiScale(image, found, weights, 0, cv::Size(8, 8), cv::Size(32, 32), 1.05, 2, false); // Draw rectangles around detected people for (const auto& rect : found) { cv::rectangle(image, rect, cv::Scalar(0, 255, 0), 2); } ``` ```python theme={null} import cv2 # Create HOG descriptor with default people detector hog = cv2.HOGDescriptor() hog.setSVMDetector(cv2.HOGDescriptor_getDefaultPeopleDetector()) # Detect people in the image found, weights = hog.detectMultiScale(image, winStride=(8, 8), padding=(32, 32), scale=1.05) # Draw rectangles around detected people for (x, y, w, h) in found: cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2) ``` ## See Also * [Face Detection](/api/objdetect/face) * [ArUco Detection](/api/objdetect/aruco) * [QR Code Detection](/api/objdetect/qrcode) # Face Detection and Recognition Source: https://opencv-opencv.mintlify.app/api/objdetect/face API reference for DNN-based face detection and recognition classes # Face Detection and Recognition DNN-based face detection and recognition using the FaceDetectorYN and FaceRecognizerSF classes. ## FaceDetectorYN DNN-based face detector class. Model download link: [Face Detection YuNet](https://github.com/opencv/opencv_zoo/tree/master/models/face_detection_yunet) ### Constructor Use the static `create()` method to create an instance. #### create (from file) ```cpp theme={null} static Ptr create( const String& model, const String& config, const Size& input_size, float score_threshold = 0.9f, float nms_threshold = 0.3f, int top_k = 5000, int backend_id = 0, int target_id = 0 ) ``` Path to the requested model file Path to the config file for compatibility (not requested for ONNX models) Size of the input image Threshold to filter out bounding boxes of score less than the given value Threshold to suppress bounding boxes that have IoU greater than the given value Number of bounding boxes to preserve from top rank based on score before NMS ID of the backend (DNN backend) ID of the target device **Returns:** Pointer to FaceDetectorYN instance #### create (from buffer) ```cpp theme={null} static Ptr create( const String& framework, const std::vector& bufferModel, const std::vector& bufferConfig, const Size& input_size, float score_threshold = 0.9f, float nms_threshold = 0.3f, int top_k = 5000, int backend_id = 0, int target_id = 0 ) ``` Name of origin framework Buffer with content of binary file with model weights Buffer with content of text file containing network configuration ### Methods #### setInputSize Sets the size for the network input. ```cpp theme={null} void setInputSize(const Size& input_size) ``` Size of the input image. This overwrites the input size used when creating the model. Call this method when the size of the input image does not match the input size when creating the model. #### getInputSize Gets the current input size. ```cpp theme={null} Size getInputSize() ``` **Returns:** Current input size #### setScoreThreshold Sets the score threshold to filter out bounding boxes. ```cpp theme={null} void setScoreThreshold(float score_threshold) ``` Threshold for filtering out bounding boxes #### getScoreThreshold Gets the current score threshold. ```cpp theme={null} float getScoreThreshold() ``` **Returns:** Current score threshold #### setNMSThreshold Sets the Non-maximum-suppression threshold. ```cpp theme={null} void setNMSThreshold(float nms_threshold) ``` Threshold for NMS operation to suppress bounding boxes that have IoU greater than the given value #### getNMSThreshold Gets the current NMS threshold. ```cpp theme={null} float getNMSThreshold() ``` **Returns:** Current NMS threshold #### setTopK Sets the number of bounding boxes preserved before NMS. ```cpp theme={null} void setTopK(int top_k) ``` Number of bounding boxes to preserve from top rank based on score #### getTopK Gets the current top K value. ```cpp theme={null} int getTopK() ``` **Returns:** Current top K value #### detect Detects faces in the input image. ```cpp theme={null} int detect(InputArray image, OutputArray faces) ``` Input image to detect faces in Detection results stored in a 2D cv::Mat of shape \[num\_faces, 15]: * 0-1: x, y of bbox top left corner * 2-3: width, height of bbox * 4-5: x, y of right eye * 6-7: x, y of left eye * 8-9: x, y of nose tip * 10-11: x, y of right corner of mouth * 12-13: x, y of left corner of mouth * 14: face score **Returns:** Number of faces detected ### Example Usage ```cpp theme={null} #include #include // Create face detector auto detector = cv::FaceDetectorYN::create( "face_detection_yunet_2023mar.onnx", "", cv::Size(320, 320), 0.9f, 0.3f, 5000 ); // Set input size to match image detector->setInputSize(image.size()); // Detect faces cv::Mat faces; detector->detect(image, faces); // Draw bounding boxes and landmarks for (int i = 0; i < faces.rows; i++) { // Get bounding box int x = faces.at(i, 0); int y = faces.at(i, 1); int w = faces.at(i, 2); int h = faces.at(i, 3); cv::rectangle(image, cv::Rect(x, y, w, h), cv::Scalar(0, 255, 0), 2); // Draw facial landmarks for (int j = 4; j < 14; j += 2) { cv::circle(image, cv::Point(faces.at(i, j), faces.at(i, j+1)), 2, cv::Scalar(255, 0, 0), -1); } } ``` ```python theme={null} import cv2 # Create face detector detector = cv2.FaceDetectorYN.create( 'face_detection_yunet_2023mar.onnx', '', (320, 320), 0.9, 0.3, 5000 ) # Set input size to match image detector.setInputSize(image.shape[1::-1]) # Detect faces _, faces = detector.detect(image) if faces is not None: for face in faces: # Get bounding box x, y, w, h = face[:4].astype(int) cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2) # Draw facial landmarks for i in range(4, 14, 2): cv2.circle(image, (int(face[i]), int(face[i+1])), 2, (255, 0, 0), -1) ``` *** ## FaceRecognizerSF DNN-based face recognizer class. Model download link: [Face Recognition SFace](https://github.com/opencv/opencv_zoo/tree/master/models/face_recognition_sface) ### Constructor Use the static `create()` method to create an instance. #### create (from file) ```cpp theme={null} static Ptr create( const String& model, const String& config, int backend_id = 0, int target_id = 0 ) ``` Path to the ONNX model used for face recognition Path to the config file for compatibility (not requested for ONNX models) ID of the backend ID of the target device **Returns:** Pointer to FaceRecognizerSF instance #### create (from buffer) ```cpp theme={null} static Ptr create( const String& framework, const std::vector& bufferModel, const std::vector& bufferConfig, int backend_id = 0, int target_id = 0 ) ``` Name of the framework (ONNX, etc.) Buffer containing the binary model weights Buffer containing the network configuration ### Enums #### DisType Distance types for calculating distance between face features. ```cpp theme={null} enum DisType { FR_COSINE = 0, // Cosine distance FR_NORM_L2 = 1 // L2 norm distance } ``` ### Methods #### alignCrop Aligns detected face with the source input image and crops it. ```cpp theme={null} void alignCrop( InputArray src_img, InputArray face_box, OutputArray aligned_img ) const ``` Input image Detected face result from the input image (from FaceDetectorYN) Output aligned and cropped face image #### feature Extracts face feature from aligned image. ```cpp theme={null} void feature( InputArray aligned_img, OutputArray face_feature ) ``` Input aligned face image Output face feature vector #### match Calculates the distance between two face features. ```cpp theme={null} double match( InputArray face_feature1, InputArray face_feature2, int dis_type = FaceRecognizerSF::FR_COSINE ) const ``` First input feature vector Second input feature vector of the same size and type as face\_feature1 Distance calculation method: FR\_COSINE (cosine distance) or FR\_NORM\_L2 (L2 norm distance) **Returns:** Distance between the two face features. Lower values indicate more similar faces. ### Example Usage ```cpp theme={null} #include // Create face recognizer auto recognizer = cv::FaceRecognizerSF::create( "face_recognition_sface_2021dec.onnx", "" ); // Assume we have detected faces using FaceDetectorYN cv::Mat face1_aligned, face2_aligned; // Align and crop faces recognizer->alignCrop(image1, face1_box, face1_aligned); recognizer->alignCrop(image2, face2_box, face2_aligned); // Extract features cv::Mat feature1, feature2; recognizer->feature(face1_aligned, feature1); recognizer->feature(face2_aligned, feature2); // Calculate similarity double cosine_score = recognizer->match( feature1, feature2, cv::FaceRecognizerSF::FR_COSINE ); // Threshold for face matching (typical: 0.363 for cosine) bool is_same_person = cosine_score >= 0.363; std::cout << "Cosine similarity: " << cosine_score << std::endl; std::cout << "Same person: " << (is_same_person ? "Yes" : "No") << std::endl; ``` ```python theme={null} import cv2 # Create face recognizer recognizer = cv2.FaceRecognizerSF.create( 'face_recognition_sface_2021dec.onnx', '' ) # Assume we have detected faces using FaceDetectorYN # Align and crop faces face1_aligned = recognizer.alignCrop(image1, face1_box) face2_aligned = recognizer.alignCrop(image2, face2_box) # Extract features feature1 = recognizer.feature(face1_aligned) feature2 = recognizer.feature(face2_aligned) # Calculate similarity cosine_score = recognizer.match( feature1, feature2, cv2.FaceRecognizerSF_FR_COSINE ) # Threshold for face matching (typical: 0.363 for cosine) is_same_person = cosine_score >= 0.363 print(f"Cosine similarity: {cosine_score}") print(f"Same person: {'Yes' if is_same_person else 'No'}") ``` ## Complete Workflow Example ```cpp theme={null} #include #include #include int main() { // Load images cv::Mat img1 = cv::imread("person1.jpg"); cv::Mat img2 = cv::imread("person2.jpg"); // Create detector and recognizer auto detector = cv::FaceDetectorYN::create( "face_detection_yunet_2023mar.onnx", "", cv::Size(320, 320) ); auto recognizer = cv::FaceRecognizerSF::create( "face_recognition_sface_2021dec.onnx", "" ); // Detect faces in both images detector->setInputSize(img1.size()); cv::Mat faces1; detector->detect(img1, faces1); detector->setInputSize(img2.size()); cv::Mat faces2; detector->detect(img2, faces2); if (faces1.rows > 0 && faces2.rows > 0) { // Get first face from each image cv::Mat face1_box = faces1.row(0); cv::Mat face2_box = faces2.row(0); // Align and extract features cv::Mat aligned1, aligned2; recognizer->alignCrop(img1, face1_box, aligned1); recognizer->alignCrop(img2, face2_box, aligned2); cv::Mat feature1, feature2; recognizer->feature(aligned1, feature1); recognizer->feature(aligned2, feature2); // Compare faces double score = recognizer->match(feature1, feature2); std::cout << "Match score: " << score << std::endl; } return 0; } ``` ```python theme={null} import cv2 # Load images img1 = cv2.imread('person1.jpg') img2 = cv2.imread('person2.jpg') # Create detector and recognizer detector = cv2.FaceDetectorYN.create( 'face_detection_yunet_2023mar.onnx', '', (320, 320) ) recognizer = cv2.FaceRecognizerSF.create( 'face_recognition_sface_2021dec.onnx', '' ) # Detect faces in both images detector.setInputSize((img1.shape[1], img1.shape[0])) _, faces1 = detector.detect(img1) detector.setInputSize((img2.shape[1], img2.shape[0])) _, faces2 = detector.detect(img2) if faces1 is not None and faces2 is not None: # Get first face from each image face1_box = faces1[0] face2_box = faces2[0] # Align and extract features aligned1 = recognizer.alignCrop(img1, face1_box) aligned2 = recognizer.alignCrop(img2, face2_box) feature1 = recognizer.feature(aligned1) feature2 = recognizer.feature(aligned2) # Compare faces score = recognizer.match(feature1, feature2) print(f"Match score: {score}") ``` ## See Also * [Cascade Classifier](/api/objdetect/cascade) * [ArUco Detection](/api/objdetect/aruco) * [QR Code Detection](/api/objdetect/qrcode) # QR Code and Barcode Detection Source: https://opencv-opencv.mintlify.app/api/objdetect/qrcode API reference for QR code and barcode detection and decoding # QR Code and Barcode Detection API reference for detecting and decoding QR codes, barcodes, and other graphical codes. ## GraphicalCodeDetector Base class for graphical code detection and decoding. ### Methods #### detect Detects graphical code in image and returns the quadrangle containing the code. ```cpp theme={null} bool detect(InputArray img, OutputArray points) const ``` Grayscale or color (BGR) image containing (or not) graphical code Output vector of vertices of the minimum-area quadrangle containing the code **Returns:** `true` if a code is detected #### decode Decodes graphical code once it's found by the detect() method. ```cpp theme={null} std::string decode( InputArray img, InputArray points, OutputArray straight_code = noArray() ) const ``` Grayscale or color (BGR) image containing graphical code Quadrangle vertices found by detect() method Optional output image containing binarized code **Returns:** UTF8-encoded output string or empty string if the code cannot be decoded #### detectAndDecode Both detects and decodes graphical code. ```cpp theme={null} std::string detectAndDecode( InputArray img, OutputArray points = noArray(), OutputArray straight_code = noArray() ) const ``` Grayscale or color (BGR) image containing graphical code Optional output array of vertices of the found graphical code quadrangle Optional output image containing binarized code **Returns:** Decoded string or empty string if not found #### detectMulti Detects multiple graphical codes in image. ```cpp theme={null} bool detectMulti(InputArray img, OutputArray points) const ``` Grayscale or color (BGR) image Output vector of vector of vertices of the minimum-area quadrangles containing the codes **Returns:** `true` if at least one code is detected #### decodeMulti Decodes multiple graphical codes. ```cpp theme={null} bool decodeMulti( InputArray img, InputArray points, std::vector& decoded_info, OutputArrayOfArrays straight_code = noArray() ) const ``` Grayscale or color (BGR) image containing graphical codes Vector of quadrangle vertices found by detect() method UTF8-encoded output vector of strings or empty vector if codes cannot be decoded Optional output vector of images containing binarized codes **Returns:** `true` if at least one code is decoded #### detectAndDecodeMulti Both detects and decodes multiple graphical codes. ```cpp theme={null} bool detectAndDecodeMulti( InputArray img, std::vector& decoded_info, OutputArray points = noArray(), OutputArrayOfArrays straight_code = noArray() ) const ``` UTF8-encoded output vector of strings or empty vector if codes cannot be decoded If there are QR codes encoded with Structured Append mode and all are detected and decoded correctly, the method writes the full message to the position corresponding to the 0-th code in the sequence. The rest of the QR codes from the same sequence have empty strings. *** ## QRCodeDetector QR code detector and decoder. ### Constructor ```cpp theme={null} cv::QRCodeDetector::QRCodeDetector() ``` ### Methods #### setEpsX Sets the epsilon used during horizontal scan of QR code stop marker detection. ```cpp theme={null} QRCodeDetector& setEpsX(double epsX) ``` Epsilon neighborhood for determining the horizontal pattern of the scheme 1:1:3:1:1 according to QR code standard **Returns:** Reference to this QRCodeDetector #### setEpsY Sets the epsilon used during vertical scan of QR code stop marker detection. ```cpp theme={null} QRCodeDetector& setEpsY(double epsY) ``` Epsilon neighborhood for determining the vertical pattern of the scheme 1:1:3:1:1 according to QR code standard **Returns:** Reference to this QRCodeDetector #### setUseAlignmentMarkers Enables or disables use of alignment markers to improve corner position. ```cpp theme={null} QRCodeDetector& setUseAlignmentMarkers(bool useAlignmentMarkers) ``` Flag to enable alignment markers (enabled by default) **Returns:** Reference to this QRCodeDetector #### decodeCurved Decodes QR code on a curved surface once found by detect(). ```cpp theme={null} std::string decodeCurved( InputArray img, InputArray points, OutputArray straight_qrcode = noArray() ) ``` Grayscale or color (BGR) image containing QR code Quadrangle vertices found by detect() method Optional output image containing rectified and binarized QR code **Returns:** UTF8-encoded output string or empty string if the code cannot be decoded #### detectAndDecodeCurved Both detects and decodes QR code on a curved surface. ```cpp theme={null} std::string detectAndDecodeCurved( InputArray img, OutputArray points = noArray(), OutputArray straight_qrcode = noArray() ) ``` #### getEncoding Returns the encoding type for the decoded info from the latest decode call. ```cpp theme={null} QRCodeEncoder::ECIEncodings getEncoding(int codeIdx = 0) ``` Index of the previously decoded QR code. For single code detection, use 0. **Returns:** Encoding type (e.g., ECI\_UTF8, ECI\_SHIFT\_JIS) ### Example Usage ```cpp theme={null} #include #include #include // Create QR code detector cv::QRCodeDetector detector; // Detect and decode QR code std::vector points; std::string decoded_info = detector.detectAndDecode(image, points); if (!decoded_info.empty()) { std::cout << "QR Code decoded: " << decoded_info << std::endl; // Draw quadrangle around QR code for (size_t i = 0; i < points.size(); i++) { cv::line(image, points[i], points[(i+1) % points.size()], cv::Scalar(0, 255, 0), 3); } } else { std::cout << "QR Code not detected" << std::endl; } ``` ```python theme={null} import cv2 # Create QR code detector detector = cv2.QRCodeDetector() # Detect and decode QR code decoded_info, points, _ = detector.detectAndDecode(image) if decoded_info: print(f"QR Code decoded: {decoded_info}") # Draw quadrangle around QR code if points is not None: points = points.astype(int) for i in range(len(points)): cv2.line(image, tuple(points[i][0]), tuple(points[(i+1) % len(points)][0]), (0, 255, 0), 3) else: print("QR Code not detected") ``` *** ## QRCodeEncoder QR code encoder for generating QR codes. ### Constructor Use the static `create()` method. ```cpp theme={null} static Ptr create( const QRCodeEncoder::Params& parameters = QRCodeEncoder::Params() ) ``` ### Enums #### EncodeMode ```cpp theme={null} enum EncodeMode { MODE_AUTO = -1, MODE_NUMERIC = 1, MODE_ALPHANUMERIC = 2, MODE_BYTE = 4, MODE_ECI = 7, MODE_KANJI = 8, MODE_STRUCTURED_APPEND = 3 } ``` #### CorrectionLevel ```cpp theme={null} enum CorrectionLevel { CORRECT_LEVEL_L = 0, // ~7% error correction CORRECT_LEVEL_M = 1, // ~15% error correction CORRECT_LEVEL_Q = 2, // ~25% error correction CORRECT_LEVEL_H = 3 // ~30% error correction } ``` #### ECIEncodings ```cpp theme={null} enum ECIEncodings { ECI_SHIFT_JIS = 20, ECI_UTF8 = 26 } ``` ### Params Structure ```cpp theme={null} struct Params { int version; // QR code version CorrectionLevel correction_level; // Error correction level EncodeMode mode; // Encoding mode int structure_number; // Number of QR codes in Structured Append } ``` ### Methods #### encode Generates QR code from input string. ```cpp theme={null} void encode(const String& encoded_info, OutputArray qrcode) ``` Input string to encode Generated QR code image #### encodeStructuredAppend Generates QR code in Structured Append mode, splitting the message over multiple QR codes. ```cpp theme={null} void encodeStructuredAppend( const String& encoded_info, OutputArrayOfArrays qrcodes ) ``` Input string to encode Vector of generated QR code images ### Example Usage ```cpp theme={null} #include #include // Create encoder with parameters cv::QRCodeEncoder::Params params; params.version = 5; // or -1 for automatic params.correction_level = cv::QRCodeEncoder::CORRECT_LEVEL_M; params.mode = cv::QRCodeEncoder::MODE_BYTE; auto encoder = cv::QRCodeEncoder::create(params); // Generate QR code cv::Mat qrcode; encoder->encode("https://opencv.org", qrcode); // Save or display cv::imwrite("qrcode.png", qrcode); ``` ```python theme={null} import cv2 # Create encoder with parameters params = cv2.QRCodeEncoder_Params() params.version = 5 # or -1 for automatic params.correction_level = cv2.QRCodeEncoder_CORRECT_LEVEL_M params.mode = cv2.QRCodeEncoder_MODE_BYTE encoder = cv2.QRCodeEncoder.create(params) # Generate QR code qrcode = encoder.encode("https://opencv.org") # Save or display cv2.imwrite("qrcode.png", qrcode) ``` *** ## BarcodeDetector Barcode detector and decoder. Inherits from GraphicalCodeDetector. ### Constructor ```cpp theme={null} cv::barcode::BarcodeDetector::BarcodeDetector() cv::barcode::BarcodeDetector::BarcodeDetector( const std::string& prototxt_path, const std::string& model_path ) ``` Prototxt file path for the super resolution model (optional) Model file path for the super resolution model (optional) ### Methods #### decodeWithType Decodes barcode in image once found by detect(). ```cpp theme={null} bool decodeWithType( InputArray img, InputArray points, std::vector& decoded_info, std::vector& decoded_type ) const ``` Grayscale or color (BGR) image containing barcode Vector of rotated rectangle vertices found by detect(). Order: bottomLeft, topLeft, topRight, bottomRight. UTF8-encoded output vector of strings or empty if codes cannot be decoded Vector of strings specifying the type of barcodes (e.g., "EAN\_13", "CODE\_128") **Returns:** `true` if at least one valid barcode is found #### detectAndDecodeWithType Both detects and decodes barcode. ```cpp theme={null} bool detectAndDecodeWithType( InputArray img, std::vector& decoded_info, std::vector& decoded_type, OutputArray points = noArray() ) const ``` Grayscale or color (BGR) image containing barcode UTF8-encoded output vector of strings Vector of strings specifying the barcode types Optional output vector of vertices of the found barcode rectangle **Returns:** `true` if at least one valid barcode is found #### setDownsamplingThreshold Sets detector downsampling threshold. ```cpp theme={null} BarcodeDetector& setDownsamplingThreshold(double thresh) ``` Downsampling limit to apply (default 512). The detect method resizes the input image to this limit if the smallest dimension is greater than the threshold. **Returns:** Reference to this BarcodeDetector #### getDownsamplingThreshold Gets detector downsampling threshold. ```cpp theme={null} double getDownsamplingThreshold() const ``` **Returns:** Current downsampling threshold #### setDetectorScales Sets detector box filter sizes. ```cpp theme={null} BarcodeDetector& setDetectorScales(const std::vector& sizes) ``` Box filter sizes relative to minimum dimension of the image (default \[0.01, 0.03, 0.06, 0.08]) Filter sizes directly correlate with the expected line widths for a barcode. If the downsampling limit is increased, filter sizes need to be adjusted inversely. **Returns:** Reference to this BarcodeDetector #### getDetectorScales Gets detector box filter sizes. ```cpp theme={null} void getDetectorScales(std::vector& sizes) const ``` Output parameter for returning the sizes #### setGradientThreshold Sets detector gradient magnitude threshold. ```cpp theme={null} BarcodeDetector& setGradientThreshold(double thresh) ``` Gradient magnitude threshold (default 64). Values between 16 and 1024 generally work. **Returns:** Reference to this BarcodeDetector #### getGradientThreshold Gets detector gradient magnitude threshold. ```cpp theme={null} double getGradientThreshold() const ``` **Returns:** Current gradient threshold ### Example Usage ```cpp theme={null} #include #include #include // Create barcode detector cv::barcode::BarcodeDetector detector; // Optionally adjust parameters detector.setDownsamplingThreshold(800); detector.setGradientThreshold(64); // Detect and decode barcodes std::vector decoded_info; std::vector decoded_type; std::vector points; bool found = detector.detectAndDecodeWithType( image, decoded_info, decoded_type, points ); if (found) { for (size_t i = 0; i < decoded_info.size(); i++) { std::cout << "Barcode " << i << ": " << decoded_info[i] << " (Type: " << decoded_type[i] << ")" << std::endl; } } else { std::cout << "No barcodes detected" << std::endl; } ``` ```python theme={null} import cv2 # Create barcode detector detector = cv2.barcode.BarcodeDetector() # Optionally adjust parameters detector.setDownsamplingThreshold(800) detector.setGradientThreshold(64) # Detect and decode barcodes retval, decoded_info, decoded_type, points = \ detector.detectAndDecodeWithType(image) if retval: for i, (info, type_) in enumerate(zip(decoded_info, decoded_type)): print(f"Barcode {i}: {info} (Type: {type_})") else: print("No barcodes detected") ``` ## Supported Barcode Types The BarcodeDetector supports various 1D and 2D barcode formats: * EAN-8, EAN-13 * UPC-A, UPC-E * Code 39, Code 93, Code 128 * ITF (Interleaved 2 of 5) * Codabar * QR Code * DataMatrix * PDF417 ## See Also * [Cascade Classifier](/api/objdetect/cascade) * [Face Detection](/api/objdetect/face) * [ArUco Detection](/api/objdetect/aruco) # Motion Analysis Functions Source: https://opencv-opencv.mintlify.app/api/video/motion Core motion analysis algorithms including CamShift, MeanShift, and ECC image alignment The motion analysis module provides fundamental algorithms for tracking object motion, estimating transformations between images, and computing similarity measures. ## CamShift Finds an object center, size, and orientation using the CAMSHIFT algorithm. ```cpp theme={null} RotatedRect CamShift( InputArray probImage, Rect& window, TermCriteria criteria ); ``` Back projection of the object histogram. See `calcBackProject` for details. Initial search window. The function updates this parameter with the new window position. Stop criteria for the underlying meanShift algorithm. **Returns:** `RotatedRect` structure that includes the object position, size, and orientation. The function implements the CAMSHIFT object tracking algorithm. It first finds an object center using `meanShift`, then adjusts the window size and finds the optimal rotation. The next position of the search window can be obtained with `RotatedRect::boundingRect()`. ### Example ```cpp theme={null} Mat hsv, backproj; cvtColor(frame, hsv, COLOR_BGR2HSV); calcBackProject(&hsv, 1, channels, hist, backproj, ranges); Rect trackWindow = Rect(x, y, w, h); TermCriteria criteria(TermCriteria::EPS | TermCriteria::COUNT, 10, 1); RotatedRect trackBox = CamShift(backproj, trackWindow, criteria); ``` ## meanShift Finds an object on a back projection image using iterative search. ```cpp theme={null} int meanShift( InputArray probImage, Rect& window, TermCriteria criteria ); ``` Back projection of the object histogram. See `calcBackProject` for details. Initial search window. Updated with the final window position. Stop criteria for the iterative search algorithm. **Returns:** Number of iterations the algorithm took to converge. The function implements the iterative object search algorithm. It computes the mass center in the window of the back projection image and shifts the search window center to the mass center. The procedure repeats until the specified number of iterations is reached or the window center shifts by less than the epsilon threshold. Unlike `CamShift`, the search window size and orientation do not change during the search. For better results, pre-filter the back projection to remove noise using techniques like morphological operations or connected components analysis. ### Example ```cpp theme={null} Mat hsv, backproj; cvtColor(frame, hsv, COLOR_BGR2HSV); calcBackProject(&hsv, 1, channels, hist, backproj, ranges); Rect trackWindow = Rect(x, y, w, h); TermCriteria criteria(TermCriteria::EPS | TermCriteria::COUNT, 10, 1); int iterations = meanShift(backproj, trackWindow, criteria); ``` ## computeECC Computes the Enhanced Correlation Coefficient (ECC) value between two images. ```cpp theme={null} double computeECC( InputArray templateImage, InputArray inputImage, InputArray inputMask = noArray() ); ``` Input template image; must have 1 or 3 channels and be of type CV\_8U, CV\_16U, CV\_32F, or CV\_64F. Input image to be compared with the template; must have the same type and number of channels as templateImage. Optional single-channel mask to specify the valid region of interest. **Returns:** The ECC similarity coefficient in the range \[-1, 1], where 1 indicates perfect similarity, 0 indicates no correlation, and -1 indicates perfect negative correlation. The Enhanced Correlation Coefficient (ECC) is a normalized measure of similarity between two images. For single-channel images: $$ \mathrm{ECC}(I, T) = \frac{\sum_{x} (I(x) - \mu_I)(T(x) - \mu_T)} {\sqrt{\sum_{x} (I(x) - \mu_I)^2} \cdot \sqrt{\sum_{x} (T(x) - \mu_T)^2}} $$ ## findTransformECC Finds the geometric transform (warp) between two images in terms of the ECC criterion. ```cpp theme={null} double findTransformECC( InputArray templateImage, InputArray inputImage, InputOutputArray warpMatrix, int motionType = MOTION_AFFINE, TermCriteria criteria = TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 50, 0.001), InputArray inputMask = noArray() ); ``` Template image; 1 or 3 channels, CV\_8U, CV\_16U, CV\_32F, or CV\_64F type. Input image to be warped; same type as templateImage. Floating-point 2×3 or 3×3 mapping matrix (warp). Should be initialized with a rough alignment estimate. Type of motion model. Default: `MOTION_AFFINE` Termination criteria of the ECC algorithm. Optional mask indicating valid values of inputImage. **Returns:** The final enhanced correlation coefficient. ### Motion Type Constants Translational motion model. The warpMatrix is 2×3 with the first 2×2 part being the identity matrix. ```cpp theme={null} // 2 parameters estimated [1 0 tx] [0 1 ty] ``` Euclidean (rigid) transformation. Rotation and translation only. ```cpp theme={null} // 3 parameters estimated (rotation angle, tx, ty) [cos(θ) -sin(θ) tx] [sin(θ) cos(θ) ty] ``` Affine motion model (default). 6 parameters estimated. ```cpp theme={null} [a11 a12 tx] [a21 a22 ty] ``` Homography as motion model. 8 parameters estimated. The warpMatrix is 3×3. ```cpp theme={null} [h11 h12 h13] [h21 h22 h23] [h31 h32 h33] ``` The function implements an area-based alignment that builds on intensity similarities. If images undergo strong displacements or rotations, provide a rough initial transformation. Use the identity matrix if no prior information is available. ### Example ```cpp theme={null} Mat warpMatrix = Mat::eye(2, 3, CV_32F); TermCriteria criteria(TermCriteria::COUNT+TermCriteria::EPS, 50, 0.001); double ecc = findTransformECC( templateImage, inputImage, warpMatrix, MOTION_AFFINE, criteria ); Mat aligned; warpAffine(inputImage, aligned, warpMatrix, templateImage.size(), INTER_LINEAR + WARP_INVERSE_MAP); ``` ## findTransformECCWithMask Extended version of `findTransformECC` that supports validity masks for both template and input images. ```cpp theme={null} double findTransformECCWithMask( InputArray templateImage, InputArray inputImage, InputArray templateMask, InputArray inputMask, InputOutputArray warpMatrix, int motionType = MOTION_AFFINE, TermCriteria criteria = TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 50, 1e-6), int gaussFiltSize = 5 ); ``` Single-channel 8-bit mask for templateImage indicating valid pixels. Must have the same size as templateImage. Single-channel 8-bit mask for inputImage indicating valid pixels before warping. Must have the same size as inputImage. Size of the Gaussian blur filter used for smoothing images and masks before computing alignment. Default: 5 The ECC is evaluated only over pixels that are valid in both images. On each iteration, inputMask is warped into the template frame and combined with templateMask. ## estimateRigidTransform (Deprecated) Computes an optimal affine transformation between two 2D point sets. ```cpp theme={null} Mat estimateRigidTransform( InputArray src, InputArray dst, bool fullAffine ); ``` This function is deprecated. Use `cv::estimateAffine2D` or `cv::estimateAffinePartial2D` instead. If using with images, extract points using `cv::calcOpticalFlowPyrLK` first, then use the estimation functions. First input 2D point set stored in std::vector or Mat, or an image stored in Mat. Second input 2D point set of the same size and type as src, or another image. If true, finds an optimal affine transformation with no restrictions (6 DOF). If false, limits transformations to translation, rotation, and uniform scaling (4 DOF). **Returns:** 2×3 floating-point matrix representing the affine transform \[A|b]. # Optical Flow Source: https://opencv-opencv.mintlify.app/api/video/optical-flow Dense and sparse optical flow algorithms including Lucas-Kanade, Farneback, DIS, and variational refinement methods Optical flow algorithms estimate motion between two consecutive frames by analyzing pixel displacement patterns. OpenCV provides both sparse (feature-based) and dense (per-pixel) optical flow methods. ## calcOpticalFlowPyrLK Calculates sparse optical flow using the iterative Lucas-Kanade method with pyramids. ```cpp theme={null} void calcOpticalFlowPyrLK( InputArray prevImg, InputArray nextImg, InputArray prevPts, InputOutputArray nextPts, OutputArray status, OutputArray err, Size winSize = Size(21, 21), int maxLevel = 3, TermCriteria criteria = TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 30, 0.01), int flags = 0, double minEigThreshold = 1e-4 ); ``` First 8-bit input image or pyramid constructed by `buildOpticalFlowPyramid`. Second input image or pyramid of the same size and type as prevImg. Vector of 2D points for which the flow needs to be found. Point coordinates must be single-precision floating-point. Output vector of 2D points containing the calculated new positions of input features in the second image. Output status vector (unsigned chars). Each element is set to 1 if flow was found for the corresponding feature, otherwise 0. Output vector of errors for each feature. The error type depends on the flags parameter. Size of the search window at each pyramid level. Default: Size(21, 21) 0-based maximal pyramid level number. 0 means no pyramid (single level), 1 means two levels, etc. Default: 3 Termination criteria for the iterative search algorithm. Default: 30 iterations or epsilon of 0.01 Operation flags: * `OPTFLOW_USE_INITIAL_FLOW`: Use initial estimations stored in nextPts * `OPTFLOW_LK_GET_MIN_EIGENVALS`: Use minimum eigen values as error measure Default: 0 Minimum eigen value threshold. Features with smaller values are filtered out. Default: 1e-4 The function implements a sparse iterative version of the Lucas-Kanade optical flow in pyramids. It is parallelized with TBB for better performance. The algorithm calculates the minimum eigen value of a 2×2 spatial gradient matrix; if this value is less than minEigThreshold, the feature is filtered out. ### Example ```cpp theme={null} vector prevPts, nextPts; vector status; vector err; // Detect features in first frame goodFeaturesToTrack(prevGray, prevPts, 100, 0.3, 7); // Calculate optical flow calcOpticalFlowPyrLK(prevGray, nextGray, prevPts, nextPts, status, err); // Draw tracked points for (size_t i = 0; i < prevPts.size(); i++) { if (status[i]) { line(frame, prevPts[i], nextPts[i], Scalar(0, 255, 0), 2); circle(frame, nextPts[i], 3, Scalar(0, 255, 0), -1); } } ``` ## buildOpticalFlowPyramid Constructs an image pyramid for use with calcOpticalFlowPyrLK. ```cpp theme={null} int buildOpticalFlowPyramid( InputArray img, OutputArrayOfArrays pyramid, Size winSize, int maxLevel, bool withDerivatives = true, int pyrBorder = BORDER_REFLECT_101, int derivBorder = BORDER_CONSTANT, bool tryReuseInputImage = true ); ``` 8-bit input image. Output pyramid. Window size of optical flow algorithm. Must be at least as large as the winSize argument of calcOpticalFlowPyrLK. 0-based maximal pyramid level number. Set to precompute gradients for every pyramid level. If false, calcOpticalFlowPyrLK will compute them internally. Default: true Border mode for pyramid layers. Default: BORDER\_REFLECT\_101 Border mode for gradients. Default: BORDER\_CONSTANT Put ROI of input image into the pyramid if possible. Set to false to force data copying. Default: true **Returns:** Number of levels in the constructed pyramid (can be less than maxLevel). ## calcOpticalFlowFarneback Computes dense optical flow using the Gunnar Farneback algorithm. ```cpp theme={null} void calcOpticalFlowFarneback( InputArray prev, InputArray next, InputOutputArray flow, double pyr_scale, int levels, int winsize, int iterations, int poly_n, double poly_sigma, int flags ); ``` First 8-bit single-channel input image. Second input image of the same size and type as prev. Computed flow image with the same size as prev and type CV\_32FC2. Image scale (\<1) to build pyramids. 0.5 means a classical pyramid where each next layer is twice smaller. Number of pyramid layers including the initial image. levels=1 means no extra layers. Averaging window size. Larger values increase robustness to noise and detect fast motion better, but yield more blurred motion fields. Number of iterations the algorithm does at each pyramid level. Size of pixel neighborhood used to find polynomial expansion. Larger values mean smoother surfaces. Typically 5 or 7. Standard deviation of the Gaussian used to smooth derivatives. For poly\_n=5, use poly\_sigma=1.1; for poly\_n=7, use poly\_sigma=1.5. Operation flags: * `OPTFLOW_USE_INITIAL_FLOW`: Use input flow as initial approximation * `OPTFLOW_FARNEBACK_GAUSSIAN`: Use Gaussian filter instead of box filter (more accurate but slower) The function finds optical flow for each pixel using the Farneback algorithm: $$ \texttt{prev}(y,x) \sim \texttt{next}(y + \texttt{flow}(y,x)[1], x + \texttt{flow}(y,x)[0]) $$ ### Example ```cpp theme={null} Mat flow; calcOpticalFlowFarneback( prevGray, nextGray, flow, 0.5, // pyr_scale 3, // levels 15, // winsize 3, // iterations 5, // poly_n 1.2, // poly_sigma 0 // flags ); // Visualize flow for (int y = 0; y < flow.rows; y += 10) { for (int x = 0; x < flow.cols; x += 10) { Point2f f = flow.at(y, x); line(frame, Point(x, y), Point(x + f.x, y + f.y), Scalar(0, 255, 0)); } } ``` ## readOpticalFlow / writeOpticalFlow Read and write optical flow files in .flo format. ```cpp theme={null} Mat readOpticalFlow(const String& path); bool writeOpticalFlow(const String& path, InputArray flow); ``` Path to the .flo file. Flow field to be stored. Must be 2-channel, floating-point (CV\_32FC2). First channel is horizontal (u), second is vertical (v). ## DenseOpticalFlow Interface Base class for dense optical flow algorithms. ```cpp theme={null} class DenseOpticalFlow : public Algorithm { public: virtual void calc(InputArray I0, InputArray I1, InputOutputArray flow) = 0; virtual void collectGarbage() = 0; }; ``` ### calc Calculates optical flow between two frames. First 8-bit single-channel input image. Second input image of the same size and type. Computed flow image that has the same size as I0 and type CV\_32FC2. ### collectGarbage Releases all inner buffers to free memory. ## SparseOpticalFlow Interface Base interface for sparse optical flow algorithms. ```cpp theme={null} class SparseOpticalFlow : public Algorithm { public: virtual void calc( InputArray prevImg, InputArray nextImg, InputArray prevPts, InputOutputArray nextPts, OutputArray status, OutputArray err = cv::noArray() ) = 0; }; ``` ## FarnebackOpticalFlow Class computing dense optical flow using the Gunnar Farneback algorithm. ```cpp theme={null} class FarnebackOpticalFlow : public DenseOpticalFlow { public: static Ptr create( int numLevels = 5, double pyrScale = 0.5, bool fastPyramids = false, int winSize = 13, int numIters = 10, int polyN = 5, double polySigma = 1.1, int flags = 0 ); virtual int getNumLevels() const = 0; virtual void setNumLevels(int numLevels) = 0; virtual double getPyrScale() const = 0; virtual void setPyrScale(double pyrScale) = 0; // ... additional getters/setters for all parameters }; ``` ### Example ```cpp theme={null} Ptr farneback = FarnebackOpticalFlow::create(); farneback->setNumLevels(3); farneback->setPyrScale(0.5); farneback->setWinSize(15); Mat flow; farneback->calc(prevGray, nextGray, flow); ``` ## SparsePyrLKOpticalFlow Class for calculating sparse optical flow using the iterative Lucas-Kanade method with pyramids. ```cpp theme={null} class SparsePyrLKOpticalFlow : public SparseOpticalFlow { public: static Ptr create( Size winSize = Size(21, 21), int maxLevel = 3, TermCriteria crit = TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 30, 0.01), int flags = 0, double minEigThreshold = 1e-4 ); virtual Size getWinSize() const = 0; virtual void setWinSize(Size winSize) = 0; virtual int getMaxLevel() const = 0; virtual void setMaxLevel(int maxLevel) = 0; // ... additional getters/setters }; ``` ### Example ```cpp theme={null} Ptr lk = SparsePyrLKOpticalFlow::create(); lk->setWinSize(Size(21, 21)); lk->setMaxLevel(3); vector prevPts, nextPts; vector status; vector err; goodFeaturesToTrack(prevGray, prevPts, 100, 0.3, 7); lk->calc(prevGray, nextGray, prevPts, nextPts, status, err); ``` ## DISOpticalFlow Dense Inverse Search (DIS) optical flow algorithm with configurable speed/quality presets. ```cpp theme={null} class DISOpticalFlow : public DenseOpticalFlow { public: enum { PRESET_ULTRAFAST = 0, PRESET_FAST = 1, PRESET_MEDIUM = 2 }; static Ptr create(int preset = PRESET_FAST); virtual int getFinestScale() const = 0; virtual void setFinestScale(int val) = 0; virtual int getPatchSize() const = 0; virtual void setPatchSize(int val) = 0; virtual int getPatchStride() const = 0; virtual void setPatchStride(int val) = 0; virtual bool getUseSpatialPropagation() const = 0; virtual void setUseSpatialPropagation(bool val) = 0; // ... additional parameters }; ``` DIS includes several enhancements over the paper implementation, including spatial propagation of flow vectors and support for initial flow approximation. Even the slowest preset is relatively fast; use DeepFlow if you need better quality and don't care about speed. ### Presets Fastest preset with basic quality. Suitable for real-time applications where speed is critical. Default preset offering good balance between speed and quality. Recommended for most applications. Higher quality at the cost of some speed. Still relatively fast compared to other dense methods. ### Example ```cpp theme={null} Ptr dis = DISOpticalFlow::create(DISOpticalFlow::PRESET_FAST); dis->setFinestScale(2); dis->setPatchStride(4); Mat flow; dis->calc(prevGray, nextGray, flow); ``` ## VariationalRefinement Variational optical flow refinement for improving existing flow fields. ```cpp theme={null} class VariationalRefinement : public DenseOpticalFlow { public: static Ptr create(); virtual void calcUV(InputArray I0, InputArray I1, InputOutputArray flow_u, InputOutputArray flow_v) = 0; virtual int getFixedPointIterations() const = 0; virtual void setFixedPointIterations(int val) = 0; virtual int getSorIterations() const = 0; virtual void setSorIterations(int val) = 0; virtual float getAlpha() const = 0; // Smoothness weight virtual void setAlpha(float val) = 0; virtual float getDelta() const = 0; // Color constancy weight virtual void setDelta(float val) = 0; virtual float getGamma() const = 0; // Gradient constancy weight virtual void setGamma(float val) = 0; }; ``` This class implements variational refinement of input flow fields. It uses the input flow to initialize minimization of the following functional: $$ E(U) = \int_{\Omega} \delta \Psi(E_I) + \gamma \Psi(E_G) + \alpha \Psi(E_S) $$ where $E_I$, $E_G$, $E_S$ are color constancy, gradient constancy, and smoothness terms respectively. ### Example ```cpp theme={null} // First compute initial flow with DIS Ptr dis = DISOpticalFlow::create(); Mat flow; dis->calc(prevGray, nextGray, flow); // Refine the flow Ptr variational = VariationalRefinement::create(); variational->setAlpha(20.0f); variational->setDelta(5.0f); variational->setGamma(10.0f); variational->calc(prevGray, nextGray, flow); ``` ## Algorithm Comparison **Lucas-Kanade (calcOpticalFlowPyrLK)** * Type: Sparse (feature points) * Speed: Very fast * Accuracy: Good for well-textured features * Use case: Feature tracking, structure from motion **Farneback (calcOpticalFlowFarneback)** * Type: Dense * Speed: Medium * Accuracy: Good * Use case: General dense flow estimation **DIS (Dense Inverse Search)** * Type: Dense * Speed: Fast (with presets) * Accuracy: Good to excellent (preset-dependent) * Use case: Real-time dense flow with quality/speed tradeoff **VariationalRefinement** * Type: Post-processing * Speed: Medium * Accuracy: Improves existing flow * Use case: Refining flow from other algorithms # Object Tracking Source: https://opencv-opencv.mintlify.app/api/video/tracking Object tracking algorithms including Tracker classes, KalmanFilter, and modern deep learning-based trackers The tracking module provides various algorithms for tracking objects across video frames, from classical methods like Kalman filtering to modern deep learning-based approaches. ## Tracker Base Class Base abstract class for long-term object trackers. ```cpp theme={null} class Tracker { public: virtual void init(InputArray image, const Rect& boundingBox) = 0; virtual bool update(InputArray image, Rect& boundingBox) = 0; }; ``` ### init Initialize the tracker with a known bounding box that surrounds the target. ```cpp theme={null} virtual void init( InputArray image, const Rect& boundingBox ); ``` The initial frame containing the object to track. The initial bounding box surrounding the target object. ### update Update the tracker and find the new most likely bounding box for the target. ```cpp theme={null} virtual bool update( InputArray image, Rect& boundingBox ); ``` The current frame to process. Output parameter for the new target location. Updated only if the function returns true. **Returns:** `true` if the target was located, `false` if the tracker cannot locate the target. Note that `false` does not necessarily mean the tracker has failed—the target may be temporarily out of view. ## KalmanFilter Implements a standard Kalman filter for state estimation. ```cpp theme={null} class KalmanFilter { public: KalmanFilter(); KalmanFilter(int dynamParams, int measureParams, int controlParams = 0, int type = CV_32F); void init(int dynamParams, int measureParams, int controlParams = 0, int type = CV_32F); const Mat& predict(const Mat& control = Mat()); const Mat& correct(const Mat& measurement); // State vectors and matrices Mat statePre; // Predicted state (x'(k)) Mat statePost; // Corrected state (x(k)) Mat transitionMatrix; // State transition matrix (A) Mat controlMatrix; // Control matrix (B) Mat measurementMatrix; // Measurement matrix (H) Mat processNoiseCov; // Process noise covariance (Q) Mat measurementNoiseCov;// Measurement noise covariance (R) Mat errorCovPre; // Priori error covariance (P'(k)) Mat gain; // Kalman gain (K(k)) Mat errorCovPost; // Posteriori error covariance (P(k)) }; ``` ### Constructor ```cpp theme={null} KalmanFilter( int dynamParams, int measureParams, int controlParams = 0, int type = CV_32F ); ``` Dimensionality of the state vector. Dimensionality of the measurement vector. Dimensionality of the control vector. Default: 0 (no control). Type of the created matrices. Should be CV\_32F or CV\_64F. Default: CV\_32F. ### predict Computes a predicted state. ```cpp theme={null} const Mat& predict(const Mat& control = Mat()); ``` Optional input control vector. **Returns:** Reference to the predicted state vector. ### correct Updates the predicted state from the measurement. ```cpp theme={null} const Mat& correct(const Mat& measurement); ``` The measured system parameters. **Returns:** Reference to the corrected state vector. The Kalman filter operates in two steps: prediction (using the system model) and correction (using measurements). The filter maintains estimates of the state and its uncertainty through covariance matrices. ### Example ```cpp theme={null} // Create Kalman filter: 4D state (x, y, dx, dy), 2D measurement (x, y) KalmanFilter kf(4, 2, 0); // Initialize state transition matrix (constant velocity model) kf.transitionMatrix = (Mat_(4, 4) << 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1); // Initialize measurement matrix kf.measurementMatrix = (Mat_(2, 4) << 1, 0, 0, 0, 0, 1, 0, 0); // Set process and measurement noise setIdentity(kf.processNoiseCov, Scalar::all(1e-5)); setIdentity(kf.measurementNoiseCov, Scalar::all(1e-1)); // Tracking loop while (true) { Mat prediction = kf.predict(); Mat measurement = getMeasurement(); // Your measurement function Mat estimated = kf.correct(measurement); } ``` ## TrackerMIL Multiple Instance Learning (MIL) tracker that trains a classifier online to separate object from background. ```cpp theme={null} class TrackerMIL : public Tracker { public: struct Params { float samplerInitInRadius; // Radius for positive samples during init int samplerInitMaxNegNum; // # negative samples during init float samplerSearchWinSize; // Search window size float samplerTrackInRadius; // Radius for positive samples during tracking int samplerTrackMaxPosNum; // # positive samples during tracking int samplerTrackMaxNegNum; // # negative samples during tracking int featureSetNumFeatures; // # features }; static Ptr create(const Params& parameters = Params()); }; ``` MIL avoids the drift problem for robust tracking. The implementation is based on "Visual Tracking with Online Multiple Instance Learning" by Babenko et al. ### Example ```cpp theme={null} Ptr tracker = TrackerMIL::create(); Rect bbox = selectROI(frame); // User selects initial bounding box tracker->init(frame, bbox); while (true) { cap >> frame; if (tracker->update(frame, bbox)) { rectangle(frame, bbox, Scalar(0, 255, 0), 2); } imshow("Tracking", frame); } ``` ## TrackerGOTURN Generic Object Tracking Using Regression Networks - a CNN-based tracker trained offline. ```cpp theme={null} class TrackerGOTURN : public Tracker { public: struct Params { std::string modelTxt; // Path to .prototxt file std::string modelBin; // Path to .caffemodel file }; static Ptr create(const Params& parameters = Params()); }; ``` GOTURN is much faster than online-training CNN trackers due to its offline training approach. It handles viewpoint changes, lighting changes, and deformations well, but does not handle occlusions. Requires pre-trained models (goturn.prototxt and goturn.caffemodel). ### Example ```cpp theme={null} TrackerGOTURN::Params params; params.modelTxt = "goturn.prototxt"; params.modelBin = "goturn.caffemodel"; Ptr tracker = TrackerGOTURN::create(params); Rect bbox = selectROI(frame); tracker->init(frame, bbox); while (true) { cap >> frame; if (tracker->update(frame, bbox)) { rectangle(frame, bbox, Scalar(255, 0, 0), 2); } imshow("GOTURN Tracking", frame); } ``` ## TrackerDaSiamRPN Deep learning-based tracker using Siamese Region Proposal Networks. ```cpp theme={null} class TrackerDaSiamRPN : public Tracker { public: struct Params { std::string model; // SiamRPN model path std::string kernel_cls1; // CLS kernel path std::string kernel_r1; // R1 kernel path int backend; // DNN backend int target; // DNN target device }; static Ptr create(const Params& parameters = Params()); virtual float getTrackingScore() = 0; }; ``` Returns the tracking confidence score for the current frame. ## TrackerNano Super lightweight DNN-based tracker with model size of only 1.9 MB. ```cpp theme={null} class TrackerNano : public Tracker { public: struct Params { std::string backbone; // Backbone model for feature extraction std::string neckhead; // Neckhead model for localization int backend; // DNN backend int target; // DNN target device }; static Ptr create(const Params& parameters = Params()); virtual float getTrackingScore() = 0; }; ``` Nano tracker is extremely lightweight and fast due to its special model structure. Requires two models: one for feature extraction (backbone) and another for localization (neckhead). ## TrackerVit Vision Transformer (ViT) based tracker, extremely lightweight at approximately 767KB. ```cpp theme={null} class TrackerVit : public Tracker { public: struct Params { std::string net; // Model path int backend; // DNN backend int target; // DNN target device Scalar meanvalue; // Mean for preprocessing Scalar stdvalue; // Std for preprocessing float tracking_score_threshold; // Score threshold }; static Ptr create(const Params& parameters = Params()); virtual float getTrackingScore() = 0; }; ``` Mean values for image preprocessing. Default: (0.485, 0.456, 0.406) Standard deviation values for image preprocessing. Default: (0.229, 0.224, 0.225) Minimum confidence threshold for tracking. Default: 0.20 ## Comparison of Trackers **MIL (Multiple Instance Learning)** * Pros: Robust, handles appearance changes * Cons: Slower than modern methods * Use case: General purpose tracking **GOTURN** * Pros: Fast, no online training * Cons: Doesn't handle occlusions * Model size: \~500MB * Use case: Real-time tracking without occlusions **Nano, Vit** * Pros: Extremely lightweight, very fast * Model size: 1-2MB * Use case: Embedded systems, mobile devices **DaSiamRPN** * Pros: High accuracy, robust * Cons: Larger model size * Use case: High-accuracy tracking # C++ API Source: https://opencv-opencv.mintlify.app/bindings/cpp Learn how to use OpenCV's native C++ API, build from source, and write high-performance computer vision applications ## Overview The C++ API is OpenCV's native interface, providing direct access to all functionality with optimal performance. All OpenCV modules are written in C++ and offer the most complete feature set. ## Installation ### Building from Source OpenCV uses CMake as its build system. Here's how to build and install OpenCV from source: ```bash theme={null} # Install dependencies sudo apt-get install build-essential cmake git libgtk2.0-dev pkg-config \ libavcodec-dev libavformat-dev libswscale-dev # Clone the repository git clone https://github.com/opencv/opencv.git cd opencv mkdir build && cd build # Configure with CMake cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local .. # Build (use -j flag for parallel compilation) make -j$(nproc) # Install sudo make install ``` ```bash theme={null} # Install dependencies using Homebrew brew install cmake pkg-config # Clone the repository git clone https://github.com/opencv/opencv.git cd opencv mkdir build && cd build # Configure with CMake cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local .. # Build make -j$(sysctl -n hw.ncpu) # Install sudo make install ``` ```bash theme={null} # Clone the repository git clone https://github.com/opencv/opencv.git cd opencv mkdir build cd build # Configure with CMake (adjust paths as needed) cmake -G "Visual Studio 17 2022" -A x64 \ -DCMAKE_BUILD_TYPE=Release .. # Build using CMake or Visual Studio cmake --build . --config Release # Install cmake --install . ``` ### Key CMake Options Customize your build with these important CMake flags: * `BUILD_EXAMPLES=ON` - Build example applications * `BUILD_opencv_world=ON` - Build single combined library (Windows) * `WITH_CUDA=ON` - Enable CUDA support for GPU acceleration * `WITH_TBB=ON` - Enable Intel TBB for parallel processing * `OPENCV_EXTRA_MODULES_PATH=` - Add opencv\_contrib modules ## Including OpenCV in Your Project ### Using CMake Create a `CMakeLists.txt` file: ```cmake theme={null} cmake_minimum_required(VERSION 3.5) project(MyOpenCVApp) # Find OpenCV find_package(OpenCV REQUIRED) # Add your executable add_executable(myapp main.cpp) # Link OpenCV libraries target_link_libraries(myapp ${OpenCV_LIBS}) ``` Build your project: ```bash theme={null} mkdir build && cd build cmake .. make ``` ### Manual Compilation ```bash theme={null} g++ main.cpp -o myapp `pkg-config --cflags --libs opencv4` ``` ## Core Concepts ### Including Headers The main header includes all modules: ```cpp theme={null} #include ``` Or include specific modules for faster compilation: ```cpp theme={null} #include #include #include #include ``` ### Namespace All OpenCV C++ functions and classes are in the `cv` namespace: ```cpp theme={null} using namespace cv; // Or use explicit namespace cv::Mat image; ``` ## Code Examples ### Reading and Displaying an Image ```cpp theme={null} #include #include using namespace cv; using namespace std; int main(int argc, char** argv) { // Read an image Mat image = imread("image.jpg", IMREAD_COLOR); if (image.empty()) { cout << "Could not open or find the image" << endl; return -1; } // Display the image namedWindow("Display Image", WINDOW_AUTOSIZE); imshow("Display Image", image); // Wait for a keystroke waitKey(0); return 0; } ``` ### Face Detection with Cascade Classifier ```cpp theme={null} #include #include #include #include #include using namespace std; using namespace cv; void detectAndDraw(Mat& img, CascadeClassifier& cascade, double scale) { vector faces; Mat gray, smallImg; // Convert to grayscale cvtColor(img, gray, COLOR_BGR2GRAY); // Resize for faster detection double fx = 1 / scale; resize(gray, smallImg, Size(), fx, fx, INTER_LINEAR_EXACT); equalizeHist(smallImg, smallImg); // Detect faces cascade.detectMultiScale(smallImg, faces, 1.1, 2, 0 | CASCADE_SCALE_IMAGE, Size(30, 30)); // Draw rectangles around detected faces for (size_t i = 0; i < faces.size(); i++) { Rect r = faces[i]; Point center(cvRound((r.x + r.width*0.5)*scale), cvRound((r.y + r.height*0.5)*scale)); int radius = cvRound((r.width + r.height)*0.25*scale); circle(img, center, radius, Scalar(255, 0, 0), 3, 8, 0); } imshow("Detection", img); } int main(int argc, const char** argv) { VideoCapture capture; Mat frame; CascadeClassifier cascade; double scale = 1.3; // Load the cascade classifier if (!cascade.load("haarcascade_frontalface_alt.xml")) { cerr << "ERROR: Could not load classifier cascade" << endl; return -1; } // Open camera if (!capture.open(0)) { cout << "Capture from camera failed" << endl; return 1; } cout << "Video capturing has been started..." << endl; for (;;) { capture >> frame; if (frame.empty()) break; detectAndDraw(frame, cascade, scale); char c = (char)waitKey(10); if (c == 27 || c == 'q') break; } return 0; } ``` ### Image Processing Pipeline ```cpp theme={null} #include using namespace cv; int main() { // Read image Mat src = imread("input.jpg"); // Convert to grayscale Mat gray; cvtColor(src, gray, COLOR_BGR2GRAY); // Apply Gaussian blur Mat blurred; GaussianBlur(gray, blurred, Size(5, 5), 0); // Edge detection Mat edges; Canny(blurred, edges, 50, 150); // Find contours vector> contours; findContours(edges, contours, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE); // Draw contours on original image Mat result = src.clone(); drawContours(result, contours, -1, Scalar(0, 255, 0), 2); // Save result imwrite("output.jpg", result); return 0; } ``` ### Working with Matrices ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { // Create a 3x3 matrix Mat M = (Mat_(3, 3) << 1, 2, 3, 4, 5, 6, 7, 8, 9); cout << "M = " << endl << M << endl; // Create identity matrix Mat I = Mat::eye(4, 4, CV_64F); cout << "I = " << endl << I << endl; // Create matrix filled with zeros Mat Z = Mat::zeros(2, 3, CV_8UC1); cout << "Z = " << endl << Z << endl; // Matrix operations Mat A = Mat::eye(3, 3, CV_64F); Mat B = Mat::ones(3, 3, CV_64F); Mat C = A + B; // Addition Mat D = A * B; // Multiplication Mat E = A.t(); // Transpose // Element access double value = M.at(1, 2); M.at(0, 0) = 10; return 0; } ``` ## Best Practices **Memory Management**: OpenCV uses reference counting for Mat objects. No need for manual memory management in most cases. ### Performance Tips 1. **Use appropriate data types**: Choose the smallest data type that fits your needs 2. **Avoid unnecessary copies**: Use references and ROI (Region of Interest) operations 3. **Enable parallel processing**: Build with TBB or OpenMP support 4. **Use GPU acceleration**: Enable CUDA modules for compute-intensive operations ### Mat Operations ```cpp theme={null} // Efficient: No data copy, just header copy Mat A = imread("image.jpg"); Mat B = A; // Shares data with A // Deep copy when needed Mat C = A.clone(); // Region of Interest (ROI) - no data copy Rect roi(10, 10, 100, 100); Mat imageROI = A(roi); ``` Modifying `B` will also modify `A` since they share the same data. Use `.clone()` or `.copyTo()` for independent copies. ## Module Organization OpenCV is organized into several modules: * **core**: Basic data structures and operations * **imgproc**: Image processing functions * **imgcodecs**: Image file reading and writing * **videoio**: Video I/O operations * **highgui**: GUI functionality * **video**: Video analysis * **calib3d**: Camera calibration and 3D reconstruction * **features2d**: 2D feature detection and description * **objdetect**: Object detection * **dnn**: Deep neural network module * **ml**: Machine learning ## Resources * [OpenCV C++ API Reference](https://docs.opencv.org/4.x/) * [OpenCV Tutorials](https://docs.opencv.org/4.x/d9/df8/tutorial_root.html) * [OpenCV GitHub Repository](https://github.com/opencv/opencv) * [Sample Code](https://github.com/opencv/opencv/tree/4.x/samples/cpp) ## Next Steps * Explore the [Image Processing](/modules/imgproc) module * Learn about [Video Analysis](/modules/video) * Try [Object Detection](/modules/objdetect) features # Java Bindings Source: https://opencv-opencv.mintlify.app/bindings/java Use OpenCV in Java applications and Android apps with automatically generated Java bindings ## Overview OpenCV provides Java bindings through an automatic code generation system. The bindings include both the Java API (JAR file) and native JNI libraries, enabling OpenCV functionality in standard Java applications and Android apps. ## Installation ### Desktop Java Applications Add OpenCV to your `pom.xml`: ```xml theme={null} org.openpnp opencv 4.9.0-0 ``` Add to your `build.gradle`: ```groovy theme={null} dependencies { implementation 'org.openpnp:opencv:4.9.0-0' } ``` 1. Download OpenCV from [opencv.org](https://opencv.org/releases/) 2. Extract the archive 3. Find the Java bindings: * JAR file: `build/bin/opencv-4xx.jar` * Native library: `build/lib/libopencv_java4xx.so` (Linux), `.dylib` (macOS), or `.dll` (Windows) 4. Add JAR to your classpath 5. Load native library at runtime ### Android Applications Add to your app's `build.gradle`: ```groovy theme={null} android { defaultConfig { // ... } } dependencies { implementation 'org.opencv:opencv:4.9.0' } ``` 1. Download [OpenCV Android SDK](https://opencv.org/releases/) 2. Extract the SDK 3. Import as module in Android Studio: * File → New → Import Module * Select `sdk/java` directory 4. Add module dependency in `build.gradle`: ```groovy theme={null} dependencies { implementation project(':opencv') } ``` ### Building from Source For custom builds: ```bash theme={null} # Clone repository git clone https://github.com/opencv/opencv.git cd opencv mkdir build && cd build # Configure for Java cmake -DBUILD_SHARED_LIBS=OFF \ -DBUILD_TESTS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DBUILD_opencv_java=ON \ .. # Build make -j$(nproc) # Find outputs in build/bin/ ``` ## Quick Start ### Loading the Native Library Before using OpenCV in Java, load the native library: ```java theme={null} import org.opencv.core.Core; public class OpenCVExample { static { // Load native library System.loadLibrary(Core.NATIVE_LIBRARY_NAME); } public static void main(String[] args) { System.out.println("OpenCV version: " + Core.getVersionString()); } } ``` On Android, use `OpenCVLoader` to load the library asynchronously or through the OpenCV Manager. ### Android Initialization ```java theme={null} import org.opencv.android.OpenCVLoader; import android.app.Activity; import android.os.Bundle; public class MainActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); if (OpenCVLoader.initLocal()) { // OpenCV loaded successfully System.out.println("OpenCV loaded"); } else { // Handle initialization error System.out.println("OpenCV not loaded"); } } } ``` ## Core Concepts ### Mat Class The `Mat` class represents images and matrices: ```java theme={null} import org.opencv.core.Mat; import org.opencv.core.CvType; import org.opencv.core.Scalar; // Create a 3x3 matrix Mat mat = new Mat(3, 3, CvType.CV_8UC1); // Create identity matrix Mat identity = Mat.eye(3, 3, CvType.CV_64FC1); // Create matrix filled with zeros Mat zeros = Mat.zeros(480, 640, CvType.CV_8UC3); // Create matrix filled with ones Mat ones = Mat.ones(100, 100, CvType.CV_32FC1); // Set all elements to a value mat.setTo(new Scalar(255)); // Remember to release Mat objects mat.release(); ``` **Memory Management**: Unlike Python, Java OpenCV requires manual memory management. Always call `mat.release()` when done with Mat objects to prevent memory leaks. ## Code Examples ### Reading and Writing Images ```java theme={null} import org.opencv.core.Mat; import org.opencv.imgcodecs.Imgcodecs; import org.opencv.core.Core; public class ImageIO { static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); } public static void main(String[] args) { // Read image Mat image = Imgcodecs.imread("input.jpg"); if (image.empty()) { System.out.println("Could not open or find the image"); return; } System.out.println("Image loaded: " + image.rows() + "x" + image.cols()); // Read in grayscale Mat gray = Imgcodecs.imread("input.jpg", Imgcodecs.IMREAD_GRAYSCALE); // Write image Imgcodecs.imwrite("output.jpg", image); // Clean up image.release(); gray.release(); } } ``` ### Image Processing ```java theme={null} import org.opencv.core.*; import org.opencv.imgproc.Imgproc; import org.opencv.imgcodecs.Imgcodecs; public class ImageProcessing { static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); } public static void main(String[] args) { // Load image Mat src = Imgcodecs.imread("input.jpg"); // Convert to grayscale Mat gray = new Mat(); Imgproc.cvtColor(src, gray, Imgproc.COLOR_BGR2GRAY); // Apply Gaussian blur Mat blurred = new Mat(); Imgproc.GaussianBlur(gray, blurred, new Size(5, 5), 0); // Edge detection Mat edges = new Mat(); Imgproc.Canny(blurred, edges, 50, 150); // Find contours java.util.List contours = new java.util.ArrayList<>(); Mat hierarchy = new Mat(); Imgproc.findContours(edges, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE); // Draw contours Mat result = src.clone(); Imgproc.drawContours(result, contours, -1, new Scalar(0, 255, 0), 2); // Save result Imgcodecs.imwrite("output.jpg", result); // Clean up src.release(); gray.release(); blurred.release(); edges.release(); hierarchy.release(); result.release(); } } ``` ### Android Camera Processing ```java theme={null} import org.opencv.android.CameraBridgeViewBase; import org.opencv.android.OpenCVLoader; import org.opencv.core.Mat; import org.opencv.core.Core; import org.opencv.core.Scalar; import org.opencv.core.Size; import org.opencv.imgproc.Imgproc; import android.app.Activity; import android.os.Bundle; public class Puzzle15Processor { private static final int GRID_SIZE = 4; private Mat mRgba15; private Mat[] mCells15; public void prepareGameSize(int width, int height) { mRgba15 = new Mat(height, width, CvType.CV_8UC4); mCells15 = new Mat[GRID_SIZE * GRID_SIZE]; for (int i = 0; i < GRID_SIZE; i++) { for (int j = 0; j < GRID_SIZE; j++) { int k = i * GRID_SIZE + j; mCells15[k] = mRgba15.submat( i * height / GRID_SIZE, (i + 1) * height / GRID_SIZE, j * width / GRID_SIZE, (j + 1) * width / GRID_SIZE ); } } } public synchronized Mat puzzleFrame(Mat inputPicture) { int rows = inputPicture.rows(); int cols = inputPicture.cols(); rows = rows - rows % 4; cols = cols - cols % 4; Mat[] cells = new Mat[GRID_SIZE * GRID_SIZE]; for (int i = 0; i < GRID_SIZE; i++) { for (int j = 0; j < GRID_SIZE; j++) { int k = i * GRID_SIZE + j; cells[k] = inputPicture.submat( i * inputPicture.rows() / GRID_SIZE, (i + 1) * inputPicture.rows() / GRID_SIZE, j * inputPicture.cols() / GRID_SIZE, (j + 1) * inputPicture.cols() / GRID_SIZE ); } } // Copy cells to output for (int i = 0; i < GRID_SIZE * GRID_SIZE; i++) { cells[i].copyTo(mCells15[i]); } // Draw grid lines drawGrid(cols, rows, mRgba15); // Release temporary cells for (Mat cell : cells) { cell.release(); } return mRgba15; } private void drawGrid(int cols, int rows, Mat drawMat) { for (int i = 1; i < GRID_SIZE; i++) { Imgproc.line(drawMat, new Point(0, i * rows / GRID_SIZE), new Point(cols, i * rows / GRID_SIZE), new Scalar(0, 255, 0, 255), 3); Imgproc.line(drawMat, new Point(i * cols / GRID_SIZE, 0), new Point(i * cols / GRID_SIZE, rows), new Scalar(0, 255, 0, 255), 3); } } } ``` ### Face Detection ```java theme={null} import org.opencv.core.*; import org.opencv.imgproc.Imgproc; import org.opencv.objdetect.CascadeClassifier; import org.opencv.imgcodecs.Imgcodecs; public class FaceDetection { static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); } public static void main(String[] args) { // Load cascade classifier CascadeClassifier faceCascade = new CascadeClassifier(); faceCascade.load("haarcascade_frontalface_alt.xml"); // Load image Mat image = Imgcodecs.imread("faces.jpg"); Mat gray = new Mat(); // Convert to grayscale Imgproc.cvtColor(image, gray, Imgproc.COLOR_BGR2GRAY); Imgproc.equalizeHist(gray, gray); // Detect faces MatOfRect faces = new MatOfRect(); faceCascade.detectMultiScale(gray, faces, 1.1, 2, 0, new Size(30, 30), new Size()); // Draw rectangles around faces for (Rect rect : faces.toArray()) { Imgproc.rectangle(image, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(0, 255, 0), 2); } System.out.println("Detected " + faces.toArray().length + " faces"); // Save result Imgcodecs.imwrite("output.jpg", image); // Clean up image.release(); gray.release(); faces.release(); } } ``` ### Working with Matrices ```java theme={null} import org.opencv.core.*; public class MatrixOperations { static { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); } public static void main(String[] args) { // Create matrices Mat A = new Mat(3, 3, CvType.CV_64FC1); Mat B = new Mat(3, 3, CvType.CV_64FC1); // Fill with random values Core.randu(A, 0, 10); Core.randu(B, 0, 10); System.out.println("Matrix A:\n" + A.dump()); // Matrix operations Mat C = new Mat(); Core.add(A, B, C); // Addition Core.subtract(A, B, C); // Subtraction Core.gemm(A, B, 1, new Mat(), 0, C); // Multiplication Core.transpose(A, C); // Transpose // Element access double[] data = new double[1]; A.get(0, 0, data); System.out.println("Element [0,0]: " + data[0]); // Set element A.put(0, 0, 42.0); // Clean up A.release(); B.release(); C.release(); } } ``` ## Android Best Practices ### Using CameraBridgeViewBase ```java theme={null} import org.opencv.android.CameraBridgeViewBase; import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame; import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2; public class MainActivity extends Activity implements CvCameraViewListener2 { private CameraBridgeViewBase mOpenCvCameraView; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); mOpenCvCameraView = findViewById(R.id.camera_view); mOpenCvCameraView.setCvCameraViewListener(this); } @Override public void onCameraViewStarted(int width, int height) { // Initialize processing } @Override public void onCameraViewStopped() { // Clean up } @Override public Mat onCameraFrame(CvCameraViewFrame inputFrame) { Mat rgba = inputFrame.rgba(); // Process frame here // Return the Mat to be displayed return rgba; } @Override protected void onResume() { super.onResume(); if (OpenCVLoader.initLocal()) { mOpenCvCameraView.enableView(); } } @Override protected void onPause() { super.onPause(); if (mOpenCvCameraView != null) { mOpenCvCameraView.disableView(); } } } ``` Always enable and disable the camera view in `onResume()` and `onPause()` to properly manage resources. ## Common Patterns ### Resource Management ```java theme={null} // Use try-finally for cleanup Mat mat = new Mat(); try { // Use mat Imgcodecs.imread("image.jpg", mat); // Process... } finally { mat.release(); } // Or create a helper method public void processImage(String path) { Mat image = Imgcodecs.imread(path); if (image.empty()) return; try { // Processing logic } finally { image.release(); } } ``` ## Resources * [OpenCV Java API Reference](https://docs.opencv.org/4.x/javadoc/) * [Android Tutorials](https://docs.opencv.org/4.x/d9/d52/tutorial_android_dev_intro.html) * [Sample Code](https://github.com/opencv/opencv/tree/4.x/samples/android) * [OpenCV Android SDK](https://opencv.org/releases/) ## Next Steps * Learn about [Image Processing](/modules/imgproc) in Java * Explore [Object Detection](/modules/objdetect) features * Build Android apps with OpenCV camera integration # JavaScript Bindings (opencv.js) Source: https://opencv-opencv.mintlify.app/bindings/javascript Use OpenCV directly in web browsers with opencv.js compiled via Emscripten ## Overview OpenCV.js brings OpenCV functionality to web browsers through WebAssembly (WASM). It's compiled from C++ using Emscripten and provides a JavaScript API that closely mirrors the C++ interface. ## Installation ### Using Pre-built opencv.js The easiest way to get started: ```html CDN theme={null} ``` ```html Local File theme={null} ``` The `async` attribute allows the page to load while opencv.js downloads. Use the `onRuntimeInitialized` callback to run code after OpenCV is ready. ### Waiting for OpenCV to Load ```html theme={null} ``` Or use the runtime callback: ```html theme={null} ``` ### Building from Source For custom builds with specific modules: ```bash theme={null} # Install Emscripten git clone https://github.com/emscripten-core/emsdk.git cd emsdk ./emsdk install latest ./emsdk activate latest source ./emsdk_env.sh # Clone OpenCV git clone https://github.com/opencv/opencv.git cd opencv # Build opencv.js emcmake python ./platforms/js/build_js.py build_js \ --build_wasm \ --cmake_option="-DCMAKE_CXX_STANDARD=17" # Output: build_js/bin/opencv.js ``` Emscripten 4.0.20+ requires C++17 or newer. The build process typically takes 5-10 minutes. ## Quick Start ### Basic HTML Template ```html theme={null} OpenCV.js Example
OpenCV.js Demo

``` ## Core Concepts ### Mat Objects All images in OpenCV.js are represented by `cv.Mat`: ```javascript theme={null} // Create Mat from canvas or image element let img = cv.imread('imageId'); // Create empty Mat let mat = new cv.Mat(rows, cols, cv.CV_8UC3); // Create with specific values let zeros = cv.Mat.zeros(480, 640, cv.CV_8UC1); let ones = cv.Mat.ones(100, 100, cv.CV_8UC3); let eye = cv.Mat.eye(3, 3, cv.CV_64FC1); // IMPORTANT: Always delete Mat objects when done mat.delete(); ``` **Memory Management**: JavaScript garbage collection doesn't handle WebAssembly memory. Always call `.delete()` on Mat objects to prevent memory leaks. ### Reading Images ```javascript theme={null} // From img element let img = document.getElementById('myImage'); let mat = cv.imread(img); // From canvas let canvas = document.getElementById('myCanvas'); let mat = cv.imread(canvas); // From video element (current frame) let video = document.getElementById('myVideo'); let mat = cv.imread(video); ``` ### Displaying Images ```javascript theme={null} // Display Mat on canvas cv.imshow('canvasOutputId', mat); // Or get canvas element first let canvas = document.getElementById('canvasOutput'); cv.imshow(canvas, mat); ``` ## Code Examples ### Image Processing Pipeline ```html theme={null} ``` ### Face Detection in Browser ```html theme={null}
Face Detection Demo
``` ### Real-time Webcam Processing ```javascript theme={null} let video = document.getElementById('videoInput'); let src = new cv.Mat(video.height, video.width, cv.CV_8UC4); let dst = new cv.Mat(video.height, video.width, cv.CV_8UC1); let cap = new cv.VideoCapture(video); const FPS = 30; function processVideo() { let begin = Date.now(); // Capture frame cap.read(src); // Convert to grayscale cv.cvtColor(src, dst, cv.COLOR_RGBA2GRAY); // Display cv.imshow('canvasOutput', dst); // Schedule next frame let delay = 1000/FPS - (Date.now() - begin); setTimeout(processVideo, delay); } // Start camera navigator.mediaDevices.getUserMedia({video: true, audio: false}) .then(stream => { video.srcObject = stream; video.onloadedmetadata = () => { video.play(); processVideo(); }; }) .catch(err => { console.error('Camera error:', err); }); ``` ### Working with Data ```javascript theme={null} // Access pixel values let mat = cv.imread('imageId'); let row = 10, col = 20; let pixel = mat.ucharPtr(row, col); // pixel is Uint8Array with [R, G, B, A] values // Modify pixels pixel[0] = 255; // Red pixel[1] = 0; // Green pixel[2] = 0; // Blue pixel[3] = 255; // Alpha // Get Mat data as typed array let data = mat.data; // Uint8Array // Create Mat from array let dataArray = new Uint8Array([255, 0, 0, 255, 0, 255, 0, 255]); let mat2 = new cv.Mat(2, 1, cv.CV_8UC4); mat2.data.set(dataArray); ``` ### Image Transformations ```javascript theme={null} function transformImage() { let src = cv.imread('inputImage'); let dst = new cv.Mat(); // Resize let dsize = new cv.Size(320, 240); cv.resize(src, dst, dsize, 0, 0, cv.INTER_LINEAR); // Rotate let center = new cv.Point(src.cols/2, src.rows/2); let M = cv.getRotationMatrix2D(center, 45, 1.0); cv.warpAffine(src, dst, M, new cv.Size(src.cols, src.rows)); // Flip cv.flip(src, dst, 1); // 1 = horizontal, 0 = vertical, -1 = both // Crop (using ROI) let rect = new cv.Rect(10, 10, 100, 100); let cropped = src.roi(rect); cv.imshow('canvasOutput', dst); // Clean up src.delete(); dst.delete(); M.delete(); cropped.delete(); } ``` ## Advanced Features ### Using DNN Module ```javascript theme={null} // Load ONNX model let net = cv.readNet('model.onnx'); // Prepare input let img = cv.imread('imageId'); let blob = cv.blobFromImage( img, 1.0, // scale factor new cv.Size(224, 224), // size new cv.Scalar(0, 0, 0, 0), // mean true, // swapRB false // crop ); // Set input and run inference net.setInput(blob); let output = net.forward(); console.log('Output shape:', output.size()); // Clean up img.delete(); blob.delete(); output.delete(); net.delete(); ``` ### Performance Optimization ```javascript theme={null} // Good: Clean up immediately function process() { let mat = cv.imread('img'); let result = new cv.Mat(); try { cv.cvtColor(mat, result, cv.COLOR_RGBA2GRAY); cv.imshow('output', result); } finally { mat.delete(); result.delete(); } } // Bad: Memory leak function processLeaky() { let mat = cv.imread('img'); let result = new cv.Mat(); cv.cvtColor(mat, result, cv.COLOR_RGBA2GRAY); // Forgot to delete! } ``` ```javascript theme={null} // Good: Reuse Mat objects let src = new cv.Mat(); let dst = new cv.Mat(); function processFrame() { cap.read(src); cv.cvtColor(src, dst, cv.COLOR_RGBA2GRAY); cv.imshow('output', dst); } // Call many times without recreating setInterval(processFrame, 33); // Clean up when done // src.delete(); // dst.delete(); ``` **Performance Tip**: Reuse Mat objects in loops instead of creating new ones. Only delete when completely done. ## Common Issues ### Memory Usage ```javascript theme={null} // Check memory usage (Emscripten specific) console.log('Memory:', cv.getBuildInformation()); // Force garbage collection (if available) if (typeof gc === 'function') { gc(); } ``` ### Loading Files OpenCV.js uses Emscripten's virtual filesystem: ```javascript theme={null} // Create file in virtual FS let utils = new Utils(''); utils.createFileFromUrl('model.xml', 'path/to/model.xml', () => { // File loaded, can use it now let cascade = new cv.CascadeClassifier(); cascade.load('model.xml'); }); ``` ## Build Configuration Customize which modules to include: ```bash theme={null} python ./platforms/js/build_js.py build_js \ --build_wasm \ --disable_single_file \ --cmake_option="-DBUILD_LIST=core,imgproc,objdetect,dnn" ``` Common build flags: * `--build_wasm`: Build WebAssembly version * `--disable_single_file`: Separate .wasm file * `--enable_exception`: Enable C++ exceptions * `--build_test`: Include test utilities ## Resources * [OpenCV.js Tutorials](https://docs.opencv.org/4.x/d5/d10/tutorial_js_root.html) * [OpenCV.js API Reference](https://docs.opencv.org/4.x/d5/d10/tutorial_js_root.html) * [Building opencv.js](https://docs.opencv.org/4.x/d4/da1/tutorial_js_setup.html) * [Sample Code](https://github.com/opencv/opencv/tree/4.x/samples/dnn) * [Emscripten Documentation](https://emscripten.org/docs/) ## Next Steps * Learn about [DNN Module](/modules/dnn) for deep learning * Explore [Image Processing](/modules/imgproc) functions * Try [Video Analysis](/modules/video) in the browser # Python Bindings Source: https://opencv-opencv.mintlify.app/bindings/python Install and use OpenCV with Python, including NumPy integration and idiomatic Python examples ## Overview The OpenCV Python bindings provide a Pythonic interface to OpenCV's C++ API. The bindings are automatically generated and offer excellent performance while maintaining ease of use. All functions work seamlessly with NumPy arrays. ## Installation ### Using pip (Recommended) The easiest way to install OpenCV for Python: ```bash Standard Package theme={null} pip install opencv-python ``` ```bash Full Package (with contrib modules) theme={null} pip install opencv-contrib-python ``` ```bash Headless (no GUI) theme={null} pip install opencv-python-headless ``` The `opencv-python` package includes prebuilt binaries for Windows, macOS, and Linux. No compilation required. ### Version Requirements * Python 3.6 or higher * NumPy (automatically installed as a dependency) ### Building from Source For custom builds or the latest development version: ```bash theme={null} # Clone the repository git clone https://github.com/opencv/opencv.git cd opencv # Create build directory mkdir build && cd build # Configure with Python support cmake -DBUILD_opencv_python3=ON \ -DPYTHON3_EXECUTABLE=$(which python3) \ -DPYTHON3_NUMPY_INCLUDE_DIRS=$(python3 -c "import numpy; print(numpy.get_include())") \ .. # Build make -j$(nproc) # Install sudo make install ``` ## Quick Start ### Importing OpenCV ```python theme={null} import cv2 as cv import numpy as np ``` The module is imported as `cv2` for historical reasons. This naming convention is standard across all OpenCV Python code. ### Verify Installation ```python theme={null} import cv2 as cv print(f"OpenCV version: {cv.__version__}") print(f"NumPy version: {np.__version__}") # Check available modules print(cv.getBuildInformation()) ``` ## Core Concepts ### NumPy Integration OpenCV images are represented as NumPy arrays: ```python theme={null} import cv2 as cv import numpy as np # Load image as NumPy array img = cv.imread('image.jpg') print(f"Image shape: {img.shape}") # (height, width, channels) print(f"Data type: {img.dtype}") # uint8 print(f"Image size: {img.size}") # total pixels # Create blank image blank = np.zeros((480, 640, 3), dtype=np.uint8) # All NumPy operations work on images img_float = img.astype(np.float32) / 255.0 mean_color = np.mean(img, axis=(0, 1)) ``` ### Image Format: BGR vs RGB OpenCV uses BGR color order by default, not RGB. Convert when working with other libraries like Matplotlib or PIL. ```python theme={null} import cv2 as cv import matplotlib.pyplot as plt # Read image (BGR format) img_bgr = cv.imread('image.jpg') # Convert to RGB for matplotlib img_rgb = cv.cvtColor(img_bgr, cv.COLOR_BGR2RGB) plt.imshow(img_rgb) plt.show() ``` ## Code Examples ### Reading and Writing Images ```python theme={null} import cv2 as cv # Read image img = cv.imread('input.jpg') if img is None: print('Could not open or find the image') exit(0) # Read in grayscale gray = cv.imread('input.jpg', cv.IMREAD_GRAYSCALE) # Save image cv.imwrite('output.jpg', img) # Save with quality settings (JPEG) cv.imwrite('output.jpg', img, [cv.IMWRITE_JPEG_QUALITY, 90]) ``` ### Video Capture and Display ```python theme={null} import cv2 as cv # Open webcam cap = cv.VideoCapture(0) # Or open video file # cap = cv.VideoCapture('video.mp4') if not cap.isOpened(): print("Cannot open camera") exit() while True: # Capture frame-by-frame ret, frame = cap.read() if not ret: print("Can't receive frame. Exiting...") break # Convert to grayscale gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) # Display the frame cv.imshow('frame', gray) # Press 'q' to quit if cv.waitKey(1) == ord('q'): break # Release resources cap.release() cv.destroyAllWindows() ``` ### Face Detection ```python theme={null} import cv2 as cv import numpy as np def detect_faces(img, cascade): """Detect faces in an image using Haar Cascade.""" gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) gray = cv.equalizeHist(gray) faces = cascade.detectMultiScale( gray, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30), flags=cv.CASCADE_SCALE_IMAGE ) return faces def draw_faces(img, faces): """Draw rectangles around detected faces.""" for (x, y, w, h) in faces: cv.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2) # Load cascade classifier cascade = cv.CascadeClassifier( cv.samples.findFile('haarcascades/haarcascade_frontalface_alt.xml') ) # Open camera cam = cv.VideoCapture(0) while True: ret, img = cam.read() if not ret: break faces = detect_faces(img, cascade) draw_faces(img, faces) cv.imshow('facedetect', img) if cv.waitKey(5) == 27: # ESC key break cam.release() cv.destroyAllWindows() ``` ### Image Processing Pipeline ```python theme={null} import cv2 as cv import numpy as np # Read image src = cv.imread('input.jpg') # Convert to grayscale gray = cv.cvtColor(src, cv.COLOR_BGR2GRAY) # Apply Gaussian blur blurred = cv.GaussianBlur(gray, (5, 5), 0) # Edge detection edges = cv.Canny(blurred, 50, 150) # Find contours contours, hierarchy = cv.findContours( edges, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE ) # Draw contours on original image result = src.copy() cv.drawContours(result, contours, -1, (0, 255, 0), 2) # Save result cv.imwrite('output.jpg', result) ``` ### Histogram Calculation and Visualization ```python theme={null} import cv2 as cv import numpy as np import matplotlib.pyplot as plt # Load image src = cv.imread('image.jpg') # Split into color channels bgr_planes = cv.split(src) # Calculate histograms histSize = 256 histRange = (0, 256) b_hist = cv.calcHist(bgr_planes, [0], None, [histSize], histRange) g_hist = cv.calcHist(bgr_planes, [1], None, [histSize], histRange) r_hist = cv.calcHist(bgr_planes, [2], None, [histSize], histRange) # Plot histograms plt.figure(figsize=(10, 6)) plt.plot(b_hist, color='b', label='Blue') plt.plot(g_hist, color='g', label='Green') plt.plot(r_hist, color='r', label='Red') plt.xlabel('Pixel Value') plt.ylabel('Frequency') plt.legend() plt.show() ``` ### Working with ROI (Region of Interest) ```python theme={null} import cv2 as cv import numpy as np # Load image img = cv.imread('image.jpg') # Define ROI using NumPy slicing height, width = img.shape[:2] roi = img[100:300, 200:400] # [y1:y2, x1:x2] # Modify ROI roi[:] = (0, 255, 0) # Fill with green # Copy ROI to another location img[50:250, 450:650] = roi # Create ROI mask mask = np.zeros(img.shape[:2], dtype=np.uint8) cv.circle(mask, (width//2, height//2), 100, 255, -1) # Apply mask masked_img = cv.bitwise_and(img, img, mask=mask) cv.imshow('Result', masked_img) cv.waitKey(0) ``` ### Image Transformations ```python theme={null} import cv2 as cv import numpy as np img = cv.imread('image.jpg') # Resize resized = cv.resize(img, (640, 480)) # Or scale by factor scaled = cv.resize(img, None, fx=0.5, fy=0.5) # Rotate height, width = img.shape[:2] center = (width // 2, height // 2) angle = 45 scale = 1.0 rotation_matrix = cv.getRotationMatrix2D(center, angle, scale) rotated = cv.warpAffine(img, rotation_matrix, (width, height)) # Flip flipped_horizontal = cv.flip(img, 1) flipped_vertical = cv.flip(img, 0) flipped_both = cv.flip(img, -1) # Crop using NumPy slicing cropped = img[100:400, 200:500] ``` ## Advanced Features ### Working with Multiple Images ```python theme={null} import cv2 as cv import glob # Process all images in a directory image_paths = glob.glob('images/*.jpg') for path in image_paths: img = cv.imread(path) # Apply processing gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) blurred = cv.GaussianBlur(gray, (5, 5), 0) # Save result output_path = path.replace('images/', 'output/') cv.imwrite(output_path, blurred) ``` ```python theme={null} import cv2 as cv import numpy as np # Read multiple images img1 = cv.imread('image1.jpg') img2 = cv.imread('image2.jpg') img3 = cv.imread('image3.jpg') # Horizontal stack h_stack = np.hstack([img1, img2, img3]) # Vertical stack v_stack = np.vstack([img1, img2, img3]) # Grid layout row1 = np.hstack([img1, img2]) row2 = np.hstack([img3, img1]) grid = np.vstack([row1, row2]) cv.imshow('Grid', grid) cv.waitKey(0) ``` ### Performance Tips ```python theme={null} import cv2 as cv import numpy as np import time # Use optimized NumPy operations img = cv.imread('large_image.jpg') # Efficient: Vectorized operation start = time.time() result = img * 0.5 print(f"Vectorized: {time.time() - start:.4f}s") # Inefficient: Loop over pixels (avoid this!) start = time.time() result = img.copy() for i in range(img.shape[0]): for j in range(img.shape[1]): result[i, j] = img[i, j] * 0.5 print(f"Loop: {time.time() - start:.4f}s") # Use in-place operations when possible img *= 0.5 # Modifies img directly, no new array # Pre-allocate arrays output = np.empty_like(img) cv.cvtColor(img, cv.COLOR_BGR2GRAY, dst=output[:,:,0]) ``` ## Package Structure The Python bindings are organized to mirror the C++ API: ```python theme={null} import cv2 as cv # Core functionality mat = cv.Mat() version = cv.__version__ # Image processing blurred = cv.GaussianBlur(img, (5, 5), 0) # Video I/O cap = cv.VideoCapture(0) # Feature detection orb = cv.ORB_create() # DNN module net = cv.dnn.readNet('model.onnx') # Find sample data files path = cv.samples.findFile('lena.jpg') ``` ## Common Issues **AttributeError**: If you get "module 'cv2' has no attribute", make sure you're using the correct module name and that the feature is included in your OpenCV build. ### Virtual Environments ```bash theme={null} # Create virtual environment python -m venv opencv_env # Activate source opencv_env/bin/activate # Linux/macOS # or opencv_env\Scripts\activate # Windows # Install OpenCV pip install opencv-python ``` ## Resources * [OpenCV-Python Tutorials](https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html) * [PyPI Package](https://pypi.org/project/opencv-python/) * [Python API Reference](https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html) * [Sample Code](https://github.com/opencv/opencv/tree/4.x/samples/python) ## Next Steps * Learn about [Image Processing](/modules/imgproc) with Python * Explore [Deep Learning with DNN](/modules/dnn) module * Try [Video Analysis](/modules/video) examples # Color Spaces Source: https://opencv-opencv.mintlify.app/concepts/color-spaces Understanding and converting between different color representations in OpenCV ## Overview Color spaces are different ways of representing colors numerically. OpenCV supports numerous color space conversions through the `cvtColor()` function. ## BGR Color Space ### Default in OpenCV OpenCV uses **BGR** (Blue-Green-Red) as its default color format: ```cpp theme={null} Mat img = imread("image.jpg"); // Loaded as BGR // Access pixel Vec3b pixel = img.at(y, x); uchar blue = pixel[0]; uchar green = pixel[1]; uchar red = pixel[2]; ``` Most other libraries (including matplotlib, PIL) use RGB order. Always convert when needed. ## Common Color Spaces ### RGB/BGR **Use case**: Display, most common representation ```cpp theme={null} // BGR to RGB Mat rgb; cvtColor(bgr, rgb, COLOR_BGR2RGB); // RGB to BGR cvtColor(rgb, bgr, COLOR_RGB2BGR); ``` ### Grayscale **Use case**: Simplify processing, reduce computation ```cpp theme={null} // Color to grayscale Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); // Grayscale to color (creates 3 identical channels) cvtColor(gray, img, COLOR_GRAY2BGR); ``` ### HSV (Hue-Saturation-Value) **Use case**: Color-based segmentation, lighting-independent processing * **Hue**: Color type (0-179 in OpenCV) * **Saturation**: Color intensity (0-255) * **Value**: Brightness (0-255) ```cpp theme={null} Mat hsv; cvtColor(img, hsv, COLOR_BGR2HSV); // Color range detection Mat mask; inRange(hsv, Scalar(0, 100, 100), Scalar(10, 255, 255), mask); ``` ### LAB (L*a*b\*) **Use case**: Perceptually uniform, skin detection * **L**: Lightness (0-100) * **a**: Green-Red axis * **b**: Blue-Yellow axis ```cpp theme={null} Mat lab; cvtColor(img, lab, COLOR_BGR2Lab); // Useful for color-based operations vector channels; split(lab, channels); Mat L = channels[0]; // Lightness only ``` ### YCrCb **Use case**: Video compression, skin detection * **Y**: Luminance * **Cr**: Red-difference * **Cb**: Blue-difference ```cpp theme={null} Mat ycrcb; cvtColor(img, ycrcb, COLOR_BGR2YCrCb); ``` ## Color Conversion ### Basic Conversion ```cpp theme={null} // Function signature void cvtColor(InputArray src, OutputArray dst, int code); // Examples cvtColor(bgr, gray, COLOR_BGR2GRAY); cvtColor(bgr, hsv, COLOR_BGR2HSV); cvtColor(hsv, bgr, COLOR_HSV2BGR); ``` ### Common Conversion Codes | From | To | Code | | ---- | ----- | ----------------- | | BGR | Gray | `COLOR_BGR2GRAY` | | BGR | RGB | `COLOR_BGR2RGB` | | BGR | HSV | `COLOR_BGR2HSV` | | BGR | LAB | `COLOR_BGR2Lab` | | BGR | YCrCb | `COLOR_BGR2YCrCb` | | HSV | BGR | `COLOR_HSV2BGR` | | Gray | BGR | `COLOR_GRAY2BGR` | ## Practical Examples ### Color Detection ```cpp theme={null} // Detect red objects Mat hsv, mask; cvtColor(img, hsv, COLOR_BGR2HSV); // Red is at both ends of hue range Mat mask1, mask2; inRange(hsv, Scalar(0, 100, 100), Scalar(10, 255, 255), mask1); inRange(hsv, Scalar(170, 100, 100), Scalar(180, 255, 255), mask2); mask = mask1 | mask2; ``` ### Lighting Normalization ```cpp theme={null} // Separate intensity from color Mat lab; cvtColor(img, lab, COLOR_BGR2Lab); vector channels; split(lab, channels); // Normalize L channel equalizeHist(channels[0], channels[0]); merge(channels, lab); cvtColor(lab, img, COLOR_Lab2BGR); ``` ### Skin Detection ```cpp theme={null} Mat ycrcb; cvtColor(img, ycrcb, COLOR_BGR2YCrCb); // Skin color range in YCrCb Mat skinMask; inRange(ycrcb, Scalar(0, 133, 77), Scalar(255, 173, 127), skinMask); ``` ## Color Space Selection * Color-based segmentation * Lighting-independent tracking * Hue provides rotation invariance * Perceptually uniform * Separate luminance from color * Better for color difference calculations * Video compression * Skin detection * Chroma subsampling * Simplest representation * Fastest processing * Use when color not needed ## Best Practices ### Conversion Tips 1. **Minimize conversions**: Convert once, cache result 2. **Choose appropriate space**: Match algorithm requirements 3. **Remember value ranges**: HSV hue is 0-179, not 0-255 4. **Consider precision**: Use CV\_32F for sensitive operations ### Performance ```cpp theme={null} // Avoid repeated conversion in loops Mat hsv; cvtColor(img, hsv, COLOR_BGR2HSV); // Convert once for(int i = 0; i < 1000; i++) { // Use hsv directly processImage(hsv); } ``` ## See Also * [Image Basics](/concepts/image-basics) - Working with images * [ImgProc Module](/modules/imgproc) - Image processing functions * [cvtColor Reference](https://docs.opencv.org/master/d8/d01/group__imgproc__color__conversions.html) # Image Basics Source: https://opencv-opencv.mintlify.app/concepts/image-basics Understanding image representation and manipulation in OpenCV ## Image as Matrix In OpenCV, images are represented as `Mat` objects - multi-dimensional arrays where: * **Grayscale images**: Single-channel 2D arrays (CV\_8UC1) * **Color images**: Multi-channel 2D arrays (typically CV\_8UC3 for BGR) * **Pixel values**: Usually 8-bit unsigned integers (0-255) ## Image Properties ### Dimensions ```cpp theme={null} Mat img = imread("image.jpg"); int height = img.rows; // Image height int width = img.cols; // Image width int channels = img.channels(); // Number of channels Size size = img.size(); // Size(width, height) ``` ### Data Type ```cpp theme={null} int type = img.type(); // e.g., CV_8UC3 int depth = img.depth(); // e.g., CV_8U bool empty = img.empty(); // Check if image is empty ``` ## Color Spaces OpenCV uses **BGR** (not RGB) as the default color order: ```cpp theme={null} // Load color image (BGR by default) Mat bgr = imread("image.jpg"); // Access BGR values Vec3b pixel = bgr.at(y, x); uchar blue = pixel[0]; uchar green = pixel[1]; uchar red = pixel[2]; ``` Most other libraries use RGB order. Always convert when interfacing with external systems. See [Color Spaces](/concepts/color-spaces) for conversion details. ## Pixel Access ### Single Pixel ```cpp theme={null} // Grayscale image uchar intensity = grayImg.at(y, x); // Color image Vec3b color = colorImg.at(y, x); color[0] = 255; // Modify blue channel colorImg.at(y, x) = color; ``` ### Efficient Row Access ```cpp theme={null} for(int i = 0; i < img.rows; i++) { uchar* row = img.ptr(i); for(int j = 0; j < img.cols; j++) { row[j] = /* process pixel */; } } ``` ### Multi-Channel Access ```cpp theme={null} for(int i = 0; i < img.rows; i++) { Vec3b* row = img.ptr(i); for(int j = 0; j < img.cols; j++) { row[j][0] = /* blue */; row[j][1] = /* green */; row[j][2] = /* red */; } } ``` ## Image Operations ### Creating Images ```cpp theme={null} // Create blank image Mat img(480, 640, CV_8UC3, Scalar(0, 0, 0)); // Black // Create from size Mat img = Mat::zeros(Size(640, 480), CV_8UC3); Mat img = Mat::ones(Size(640, 480), CV_8UC1); ``` ### Copying Images ```cpp theme={null} // Shallow copy (shares data) Mat img2 = img1; // Deep copy Mat img2 = img1.clone(); img1.copyTo(img2); // Copy with mask img1.copyTo(img2, mask); ``` ### Image ROI ```cpp theme={null} // Select rectangular region Rect roi(x, y, width, height); Mat region = img(roi); // Modify ROI (affects original) region = Scalar(0, 255, 0); // Fill with green ``` ## Channel Operations ### Split and Merge ```cpp theme={null} // Split into channels vector channels; split(img, channels); // channels[0]=B, channels[1]=G, channels[2]=R // Merge channels Mat merged; merge(channels, merged); ``` ### Extract Single Channel ```cpp theme={null} // Extract blue channel Mat blueChannel; extractChannel(img, blueChannel, 0); // Set channel to zero Mat channels[3]; split(img, channels); channels[0] = Mat::zeros(img.size(), CV_8UC1); merge(channels, 3, img); // Blue channel now zero ``` ## Image I/O ### Reading Images ```cpp theme={null} // Read color image Mat img = imread("image.jpg"); // Read grayscale Mat gray = imread("image.jpg", IMREAD_GRAYSCALE); // Read with alpha channel Mat rgba = imread("image.png", IMREAD_UNCHANGED); ``` ### Writing Images ```cpp theme={null} // Save image imwrite("output.jpg", img); // Save with parameters vector params = {IMWRITE_JPEG_QUALITY, 95}; imwrite("output.jpg", img, params); ``` ## Common Patterns ### Convert to Grayscale ```cpp theme={null} Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); ``` ### Resize Image ```cpp theme={null} Mat resized; resize(img, resized, Size(320, 240)); // Scale by factor resize(img, resized, Size(), 0.5, 0.5); ``` ### Crop Image ```cpp theme={null} Rect cropRegion(x, y, width, height); Mat cropped = img(cropRegion).clone(); ``` ## Performance Tips Prefer ptr\() over at\() for pixel-level operations Continuous matrices enable faster processing Use references and ROI instead of cloning Use OpenCV functions instead of pixel loops ## See Also * [Matrices](/concepts/matrices) - Understanding Mat class * [Color Spaces](/concepts/color-spaces) - Color conversion * [ImgProc Module](/modules/imgproc) - Image processing functions # Matrices (Mat) Source: https://opencv-opencv.mintlify.app/concepts/matrices Understanding OpenCV's Mat class for n-dimensional dense array representation ## Overview The `Mat` class is the fundamental data structure in OpenCV, representing an n-dimensional dense numerical array. It can store images, matrices, vectors, histograms, point clouds, and other multi-dimensional data. ## Mat Structure ### Key Properties A Mat object consists of: * **Header**: Contains metadata (dimensions, type, reference counter) * **Data pointer**: Points to the actual pixel/element data * **Step array**: Defines memory layout for multi-dimensional indexing ### Memory Layout For a 2D matrix, element `M(i,j)` is located at: ``` addr(M[i,j]) = M.data + M.step[0]*i + M.step[1]*j ``` Matrices are stored **row-by-row** (row-major order), making row access more efficient than column access. ## Creating Matrices ### Basic Creation ```cpp theme={null} // Create 7x7 complex matrix filled with 1+3j Mat M(7, 7, CV_32FC2, Scalar(1, 3)); // Create and initialize later Mat img(Size(320, 240), CV_8UC3); // Multi-dimensional array int sz[] = {100, 100, 100}; Mat bigCube(3, sz, CV_8U, Scalar::all(0)); ``` ### From Existing Data ```cpp theme={null} // Wrap user-allocated data (no copy) unsigned char* pixels = /* your data */; Mat img(height, width, CV_8UC3, pixels, step); ``` ## Data Types OpenCV supports various data types: * `CV_8U`: 8-bit unsigned integer * `CV_8S`: 8-bit signed integer * `CV_16U`, `CV_16S`: 16-bit integers * `CV_32S`: 32-bit signed integer * `CV_32F`: 32-bit floating point * `CV_64F`: 64-bit floating point Channel specification: `CV_8UC1` (1 channel), `CV_8UC3` (3 channels), `CV_8UC(n)` (n channels, max 512) ## Reference Counting Mat uses **shallow copying** by default: ```cpp theme={null} Mat A = Mat::eye(10, 10, CV_32S); Mat B = A; // B points to same data as A ``` For a deep copy: ```cpp theme={null} Mat C = A.clone(); ``` ## Element Access ### Direct Access ```cpp theme={null} // Single element access M.at(i, j) += 1.0; // Row pointer access (faster) for(int i = 0; i < M.rows; i++) { const double* Mi = M.ptr(i); for(int j = 0; j < M.cols; j++) sum += std::max(Mi[j], 0.); } ``` ### Iterator Access ```cpp theme={null} MatConstIterator_ it = M.begin(); MatConstIterator_ it_end = M.end(); for(; it != it_end; ++it) sum += std::max(*it, 0.); ``` ## ROI (Region of Interest) ```cpp theme={null} // Select rectangular region Mat roi(img, Rect(10, 10, 100, 100)); // Row/column selection Mat row3 = M.row(3); Mat col7 = M.col(7); // Range selection Mat B = A(Range::all(), Range(1, 3)); // All rows, columns 1-2 ``` ROI operations are O(1) as they only create a new header pointing to the same data. ## Continuous Matrices Check if matrix data is continuous (no gaps between rows): ```cpp theme={null} if(M.isContinuous()) { // Process as single row for better performance cols *= rows; rows = 1; } ``` ## Best Practices Work with sub-matrices using ROI instead of copying data Access rows sequentially for better cache performance Optimize processing for continuous matrices Use clone() when you need independent data ## See Also * [Image Basics](/concepts/image-basics) - Working with images as matrices * [Memory Management](/concepts/memory-management) - Understanding Mat memory model * [Core Module](/modules/core) - Basic Mat operations # Memory Management Source: https://opencv-opencv.mintlify.app/concepts/memory-management Understanding OpenCV's automatic memory management and reference counting ## Overview OpenCV uses **automatic memory management** with reference counting for Mat objects. This eliminates most manual memory management while providing efficiency through shallow copying. ## Reference Counting ### How It Works Each Mat object has: * **Header**: Small, cheap to copy (\~100 bytes) * **Data block**: Large, expensive to copy * **Reference counter**: Tracks how many Mat objects share the data ```cpp theme={null} Mat A(100, 100, CV_8UC1); // Allocates data, refcount = 1 Mat B = A; // Shares data, refcount = 2 Mat C = A; // Shares data, refcount = 3 // When C goes out of scope, refcount = 2 // When B goes out of scope, refcount = 1 // When A goes out of scope, refcount = 0, data freed ``` ### Shallow vs Deep Copy **Shallow Copy** (default): ```cpp theme={null} Mat A = imread("image.jpg"); Mat B = A; // Only header copied B.at(0,0) = 0; // Modifies A too! ``` **Deep Copy**: ```cpp theme={null} Mat A = imread("image.jpg"); Mat B = A.clone(); // Data copied B.at(0,0) = 0; // A unchanged // Alternative Mat C; A.copyTo(C); ``` ## Memory Allocation ### Automatic Allocation ```cpp theme={null} // create() allocates only if needed Mat A; A.create(100, 100, CV_8UC3); // Allocates A.create(100, 100, CV_8UC3); // No reallocation (same size/type) A.create(200, 200, CV_8UC3); // Reallocates (different size) ``` ### Constructor Allocation ```cpp theme={null} // Allocates on construction Mat A(480, 640, CV_8UC3); Mat B(Size(640, 480), CV_8UC3, Scalar(0)); // No allocation Mat C; // Empty matrix ``` ## UMat and Memory Allocators ### MatAllocator Custom memory allocation through `MatAllocator` class: ```cpp theme={null} class CV_EXPORTS MatAllocator { public: virtual UMatData* allocate(int dims, const int* sizes, int type, void* data, size_t* step, AccessFlag flags, UMatUsageFlags usageFlags) const = 0; virtual bool allocate(UMatData* data, AccessFlag accessflags, UMatUsageFlags usageFlags) const = 0; virtual void deallocate(UMatData* data) const = 0; }; ``` ### Memory Pools OpenCV supports buffer pooling for frequent allocations: ```cpp theme={null} // Get buffer pool controller BufferPoolController* pool = allocator->getBufferPoolController(); ``` ## Best Practices ### Avoid Unnecessary Copies ```cpp Good theme={null} void processImage(const Mat& img) { // Use reference, no copy Mat result; filter2D(img, result, -1, kernel); return result; } ``` ```cpp Bad theme={null} void processImage(Mat img) { // Copies header // Process... } ``` ### Return Values ```cpp theme={null} // Efficient: No copy due to return value optimization Mat createImage() { Mat img(480, 640, CV_8UC3); // ... initialize return img; // No actual copy } Mat result = createImage(); // Moves or shallow copy ``` ### Pre-allocation ```cpp theme={null} // Pre-allocate output Mat dst; dst.create(src.size(), src.type()); // OpenCV functions allocate if needed cvtColor(src, dst, COLOR_BGR2GRAY); // Will allocate dst if needed ``` ## Manual Memory Management ### External Data Wrap user-allocated memory: ```cpp theme={null} // User manages memory unsigned char* data = new unsigned char[640*480*3]; // OpenCV wraps it (no copy, no ownership) Mat img(480, 640, CV_8UC3, data); // Process with OpenCV cvtColor(img, gray, COLOR_BGR2GRAY); // User must free delete[] data; ``` ### Release Memory ```cpp theme={null} Mat A(1000, 1000, CV_8UC3); // Manually release (rarely needed) A.release(); // Decrements refcount, frees if zero // Check if empty if(A.empty()) { // A has no data } ``` ## Memory Continuity ### Continuous Storage ```cpp theme={null} // Check if continuous (no padding between rows) if(img.isContinuous()) { // Can treat as 1D array size_t total = img.total() * img.elemSize(); processData(img.ptr(), total); } ``` ### ROI and Continuity ```cpp theme={null} Mat img = imread("image.jpg"); // Continuous Mat roi = img(Rect(10, 10, 100, 100)); // NOT continuous // Make continuous Mat roiCopy = roi.clone(); // Now continuous ``` ## Common Pitfalls **Dangling Pointers** ```cpp theme={null} Mat getROI() { Mat img(480, 640, CV_8UC3); return img(Rect(0, 0, 100, 100)); // Dangerous! } // img destroyed, ROI points to freed memory Mat roi = getROI(); // roi is invalid! ``` Fix: Return a clone or the full image. **Shared Data Modification** ```cpp theme={null} Mat A = imread("img.jpg"); Mat B = A; // Shares data GaussianBlur(B, B, Size(5,5), 0); // Modifies A too! ``` Fix: Use `B = A.clone()` if independent data needed. ## Performance Considerations Always pass Mat as const reference to avoid header copies ROI creates header only, no data copy Use shallow copies when possible Reuse matrices in loops to avoid reallocation ## Memory Debugging ```cpp theme={null} // Check properties std::cout << "Size: " << img.total() * img.elemSize() << " bytes\n"; std::cout << "Continuous: " << img.isContinuous() << "\n"; std::cout << "Submatrix: " << img.isSubmatrix() << "\n"; // Check reference count (internal) // Mat doesn't expose refcount directly ``` ## See Also * [Matrices](/concepts/matrices) - Understanding Mat class * [Image Basics](/concepts/image-basics) - Working with images * [Core Module](/modules/core) - Core data structures # Color Space Conversions Source: https://opencv-opencv.mintlify.app/examples/color-spaces Learn how to convert between different color spaces and apply thresholding techniques This guide covers color space conversions between BGR, RGB, HSV, grayscale, and other formats, along with thresholding techniques for image segmentation. ## Overview Color spaces represent colors in different ways for various purposes: * **BGR/RGB**: Standard color representation for displays * **HSV**: Hue, Saturation, Value - intuitive for color filtering * **Grayscale**: Single channel intensity values * **LAB**: Perceptually uniform color space * **YCrCb**: Luminance and chrominance separation ## BGR to RGB Conversion OpenCV reads images in BGR format by default, but many libraries expect RGB. ```python theme={null} import cv2 as cv import matplotlib.pyplot as plt # Read image (loaded in BGR format) img_bgr = cv.imread("image.jpg") # Convert BGR to RGB img_rgb = cv.cvtColor(img_bgr, cv.COLOR_BGR2RGB) # Display using matplotlib (expects RGB) plt.imshow(img_rgb) plt.title("RGB Image") plt.axis('off') plt.show() # Display using OpenCV (expects BGR) cv.imshow("BGR Image", img_bgr) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; // Read image (loaded in BGR format) Mat img_bgr = imread("image.jpg"); // Convert BGR to RGB Mat img_rgb; cvtColor(img_bgr, img_rgb, COLOR_BGR2RGB); // Display (OpenCV expects BGR) imshow("BGR Image", img_bgr); waitKey(0); ``` OpenCV uses BGR format by default for historical reasons related to early camera standards. Always convert to RGB when interfacing with other libraries like Matplotlib, PIL, or TensorFlow. ## BGR to Grayscale Conversion Convert color images to grayscale for simplified processing. ```python theme={null} import cv2 as cv # Read color image img = cv.imread("image.jpg") # Convert to grayscale gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) cv.imshow("Original", img) cv.imshow("Grayscale", gray) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; // Read color image Mat img = imread("image.jpg"); // Convert to grayscale Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); imshow("Original", img); imshow("Grayscale", gray); waitKey(0); ``` You can also load images directly in grayscale using `imread("image.jpg", IMREAD_GRAYSCALE)` to skip the conversion step. ## BGR to HSV Conversion HSV (Hue, Saturation, Value) is ideal for color-based object detection and filtering. ```python theme={null} import cv2 as cv import numpy as np # Read image img = cv.imread("image.jpg") # Convert BGR to HSV hsv = cv.cvtColor(img, cv.COLOR_BGR2HSV) # Split into channels h, s, v = cv.split(hsv) cv.imshow("Original", img) cv.imshow("Hue", h) cv.imshow("Saturation", s) cv.imshow("Value", v) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; using namespace std; // Read image Mat img = imread("image.jpg"); // Convert BGR to HSV Mat hsv; cvtColor(img, hsv, COLOR_BGR2HSV); // Split into channels vector channels; split(hsv, channels); imshow("Original", img); imshow("Hue", channels[0]); imshow("Saturation", channels[1]); imshow("Value", channels[2]); waitKey(0); ``` ## Color Range Detection with HSV Detect specific colors by defining HSV ranges. ```python theme={null} import cv2 as cv import numpy as np # Read image frame = cv.imread("image.jpg") # Convert to HSV frame_HSV = cv.cvtColor(frame, cv.COLOR_BGR2HSV) # Define range for blue color in HSV low_H = 100 low_S = 50 low_V = 50 high_H = 130 high_S = 255 high_V = 255 # Create mask for blue color mask = cv.inRange(frame_HSV, (low_H, low_S, low_V), (high_H, high_S, high_V)) # Apply mask to original image result = cv.bitwise_and(frame, frame, mask=mask) cv.imshow("Original", frame) cv.imshow("Mask", mask) cv.imshow("Result", result) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; // Read image Mat frame = imread("image.jpg"); // Convert to HSV Mat frame_HSV; cvtColor(frame, frame_HSV, COLOR_BGR2HSV); // Define range for blue color in HSV int low_H = 100, low_S = 50, low_V = 50; int high_H = 130, high_S = 255, high_V = 255; // Create mask for blue color Mat mask; inRange(frame_HSV, Scalar(low_H, low_S, low_V), Scalar(high_H, high_S, high_V), mask); // Apply mask to original image Mat result; bitwise_and(frame, frame, result, mask); imshow("Original", frame); imshow("Mask", mask); imshow("Result", result); waitKey(0); ``` ### Common HSV Color Ranges | Color | Hue Range (H) | Saturation (S) | Value (V) | | ------ | ------------- | -------------- | --------- | | Red | 0-10, 170-180 | 50-255 | 50-255 | | Orange | 10-25 | 50-255 | 50-255 | | Yellow | 25-35 | 50-255 | 50-255 | | Green | 35-85 | 50-255 | 50-255 | | Blue | 100-130 | 50-255 | 50-255 | | Purple | 130-160 | 50-255 | 50-255 | In OpenCV, Hue values range from 0-179 (not 0-359) to fit in a single byte. Adjust your ranges accordingly. ## Thresholding Thresholding converts grayscale images to binary images by applying a threshold value. ### Basic Thresholding ```python theme={null} import cv2 as cv # Read image in grayscale src = cv.imread("image.jpg") src_gray = cv.cvtColor(src, cv.COLOR_BGR2GRAY) # Apply different threshold types threshold_value = 127 max_value = 255 # Binary threshold _, binary = cv.threshold(src_gray, threshold_value, max_value, cv.THRESH_BINARY) # Binary inverted threshold _, binary_inv = cv.threshold(src_gray, threshold_value, max_value, cv.THRESH_BINARY_INV) # Truncate threshold _, truncate = cv.threshold(src_gray, threshold_value, max_value, cv.THRESH_TRUNC) # To zero threshold _, to_zero = cv.threshold(src_gray, threshold_value, max_value, cv.THRESH_TOZERO) # To zero inverted threshold _, to_zero_inv = cv.threshold(src_gray, threshold_value, max_value, cv.THRESH_TOZERO_INV) cv.imshow("Original", src_gray) cv.imshow("Binary", binary) cv.imshow("Binary Inverted", binary_inv) cv.imshow("Truncate", truncate) cv.imshow("To Zero", to_zero) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; // Read image in grayscale Mat src = imread("image.jpg"); Mat src_gray; cvtColor(src, src_gray, COLOR_BGR2GRAY); // Apply different threshold types int threshold_value = 127; int max_value = 255; Mat binary, binary_inv, truncate, to_zero, to_zero_inv; // Binary threshold threshold(src_gray, binary, threshold_value, max_value, THRESH_BINARY); // Binary inverted threshold threshold(src_gray, binary_inv, threshold_value, max_value, THRESH_BINARY_INV); // Truncate threshold threshold(src_gray, truncate, threshold_value, max_value, THRESH_TRUNC); // To zero threshold threshold(src_gray, to_zero, threshold_value, max_value, THRESH_TOZERO); // To zero inverted threshold threshold(src_gray, to_zero_inv, threshold_value, max_value, THRESH_TOZERO_INV); imshow("Original", src_gray); imshow("Binary", binary); imshow("Binary Inverted", binary_inv); waitKey(0); ``` ### Adaptive Thresholding Adaptive thresholding calculates different thresholds for different regions, useful for varying lighting conditions. ```python theme={null} import cv2 as cv # Read image in grayscale img = cv.imread("image.jpg") gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Global threshold _, global_thresh = cv.threshold(gray, 127, 255, cv.THRESH_BINARY) # Adaptive threshold (mean) adaptive_mean = cv.adaptiveThreshold(gray, 255, cv.ADAPTIVE_THRESH_MEAN_C, cv.THRESH_BINARY, 11, 2) # Adaptive threshold (gaussian) adaptive_gaussian = cv.adaptiveThreshold(gray, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY, 11, 2) cv.imshow("Original", gray) cv.imshow("Global Threshold", global_thresh) cv.imshow("Adaptive Mean", adaptive_mean) cv.imshow("Adaptive Gaussian", adaptive_gaussian) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; // Read image in grayscale Mat img = imread("image.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); Mat global_thresh, adaptive_mean, adaptive_gaussian; // Global threshold threshold(gray, global_thresh, 127, 255, THRESH_BINARY); // Adaptive threshold (mean) adaptiveThreshold(gray, adaptive_mean, 255, ADAPTIVE_THRESH_MEAN_C, THRESH_BINARY, 11, 2); // Adaptive threshold (gaussian) adaptiveThreshold(gray, adaptive_gaussian, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2); imshow("Original", gray); imshow("Global Threshold", global_thresh); imshow("Adaptive Mean", adaptive_mean); imshow("Adaptive Gaussian", adaptive_gaussian); waitKey(0); ``` ## Other Color Space Conversions ```python theme={null} import cv2 as cv img = cv.imread("image.jpg") # BGR to LAB lab = cv.cvtColor(img, cv.COLOR_BGR2LAB) # BGR to YCrCb ycrcb = cv.cvtColor(img, cv.COLOR_BGR2YCrCb) # BGR to XYZ xyz = cv.cvtColor(img, cv.COLOR_BGR2XYZ) # BGR to HLS hls = cv.cvtColor(img, cv.COLOR_BGR2HLS) # HSV back to BGR hsv = cv.cvtColor(img, cv.COLOR_BGR2HSV) bgr_from_hsv = cv.cvtColor(hsv, cv.COLOR_HSV2BGR) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat lab, ycrcb, xyz, hls, hsv, bgr_from_hsv; // BGR to LAB cvtColor(img, lab, COLOR_BGR2LAB); // BGR to YCrCb cvtColor(img, ycrcb, COLOR_BGR2YCrCb); // BGR to XYZ cvtColor(img, xyz, COLOR_BGR2XYZ); // BGR to HLS cvtColor(img, hls, COLOR_BGR2HLS); // HSV back to BGR cvtColor(img, hsv, COLOR_BGR2HSV); cvtColor(hsv, bgr_from_hsv, COLOR_HSV2BGR); ``` ## Key Functions | Function | Description | | --------------------- | ---------------------------------------------------- | | `cvtColor()` | Convert image between color spaces | | `split()` | Split multi-channel image into separate channels | | `merge()` | Merge separate channels into multi-channel image | | `inRange()` | Create binary mask for pixels within specified range | | `threshold()` | Apply global threshold to grayscale image | | `adaptiveThreshold()` | Apply adaptive threshold for varying lighting | ## Common Color Space Codes | Conversion | Code | | ----------- | ---------------- | | BGR to RGB | `COLOR_BGR2RGB` | | BGR to Gray | `COLOR_BGR2GRAY` | | BGR to HSV | `COLOR_BGR2HSV` | | BGR to LAB | `COLOR_BGR2LAB` | | HSV to BGR | `COLOR_HSV2BGR` | | Gray to BGR | `COLOR_GRAY2BGR` | For color-based object detection, HSV color space is generally more robust than BGR/RGB because it separates color information (Hue) from lighting conditions (Value). # Face Detection and Recognition Source: https://opencv-opencv.mintlify.app/examples/face-recognition Complete guide to face detection using Haar Cascades and deep learning models with facial landmark detection and recognition ## Overview OpenCV provides multiple approaches for face detection and recognition: * **Haar Cascade Classifiers**: Fast, CPU-friendly classical method * **DNN-based Detection**: Modern deep learning approach with YuNet * **Face Recognition**: Feature extraction and matching with SFace * **Facial Landmarks**: Detect eyes, nose, and mouth positions ## Haar Cascade Face Detection ```python theme={null} import cv2 as cv import numpy as np from video import create_capture from common import clock, draw_str def detect(img, cascade): """Detect faces in image using cascade classifier""" rects = cascade.detectMultiScale(img, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30), flags=cv.CASCADE_SCALE_IMAGE) if len(rects) == 0: return [] rects[:,2:] += rects[:,:2] return rects def draw_rects(img, rects, color): """Draw rectangles around detected faces""" for x1, y1, x2, y2 in rects: cv.rectangle(img, (x1, y1), (x2, y2), color, 2) # Load cascade classifiers cascade_fn = "haarcascades/haarcascade_frontalface_alt.xml" nested_fn = "haarcascades/haarcascade_eye.xml" cascade = cv.CascadeClassifier(cv.samples.findFile(cascade_fn)) nested = cv.CascadeClassifier(cv.samples.findFile(nested_fn)) # Initialize video capture cam = create_capture(0, fallback='synth:bg={}:noise=0.05'.format( cv.samples.findFile('lena.jpg'))) while True: _ret, img = cam.read() gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) gray = cv.equalizeHist(gray) t = clock() rects = detect(gray, cascade) vis = img.copy() draw_rects(vis, rects, (0, 255, 0)) # Detect eyes within face regions if not nested.empty(): for x1, y1, x2, y2 in rects: roi = gray[y1:y2, x1:x2] vis_roi = vis[y1:y2, x1:x2] subrects = detect(roi.copy(), nested) draw_rects(vis_roi, subrects, (255, 0, 0)) dt = clock() - t draw_str(vis, (20, 20), 'time: %.1f ms' % (dt*1000)) cv.imshow('facedetect', vis) if cv.waitKey(5) == 27: break cv.destroyAllWindows() ``` ```cpp theme={null} #include "opencv2/objdetect.hpp" #include "opencv2/highgui.hpp" #include "opencv2/imgproc.hpp" #include "opencv2/videoio.hpp" #include using namespace std; using namespace cv; void detectAndDraw(Mat& img, CascadeClassifier& cascade, CascadeClassifier& nestedCascade, double scale, bool tryflip) { vector faces, faces2; const static Scalar colors[] = { Scalar(255,0,0), Scalar(255,128,0), Scalar(255,255,0), Scalar(0,255,0), Scalar(0,128,255), Scalar(0,255,255), Scalar(0,0,255), Scalar(255,0,255) }; Mat gray, smallImg; cvtColor(img, gray, COLOR_BGR2GRAY); double fx = 1 / scale; resize(gray, smallImg, Size(), fx, fx, INTER_LINEAR_EXACT); equalizeHist(smallImg, smallImg); double t = (double)getTickCount(); cascade.detectMultiScale(smallImg, faces, 1.1, 2, 0 | CASCADE_SCALE_IMAGE, Size(30, 30)); if(tryflip) { flip(smallImg, smallImg, 1); cascade.detectMultiScale(smallImg, faces2, 1.1, 2, 0 | CASCADE_SCALE_IMAGE, Size(30, 30)); for(vector::const_iterator r = faces2.begin(); r != faces2.end(); ++r) { faces.push_back(Rect(smallImg.cols - r->x - r->width, r->y, r->width, r->height)); } } t = (double)getTickCount() - t; printf("detection time = %g ms\n", t*1000/getTickFrequency()); for(size_t i = 0; i < faces.size(); i++) { Rect r = faces[i]; Mat smallImgROI; vector nestedObjects; Point center; Scalar color = colors[i%8]; int radius; double aspect_ratio = (double)r.width/r.height; if(0.75 < aspect_ratio && aspect_ratio < 1.3) { center.x = cvRound((r.x + r.width*0.5)*scale); center.y = cvRound((r.y + r.height*0.5)*scale); radius = cvRound((r.width + r.height)*0.25*scale); circle(img, center, radius, color, 3, 8, 0); } else { rectangle(img, Point(cvRound(r.x*scale), cvRound(r.y*scale)), Point(cvRound((r.x + r.width-1)*scale), cvRound((r.y + r.height-1)*scale)), color, 3, 8, 0); } if(nestedCascade.empty()) continue; smallImgROI = smallImg(r); nestedCascade.detectMultiScale(smallImgROI, nestedObjects, 1.1, 2, 0 | CASCADE_SCALE_IMAGE, Size(30, 30)); for(size_t j = 0; j < nestedObjects.size(); j++) { Rect nr = nestedObjects[j]; center.x = cvRound((r.x + nr.x + nr.width*0.5)*scale); center.y = cvRound((r.y + nr.y + nr.height*0.5)*scale); radius = cvRound((nr.width + nr.height)*0.25*scale); circle(img, center, radius, color, 3, 8, 0); } } imshow("result", img); } int main(int argc, const char** argv) { VideoCapture capture; Mat frame; CascadeClassifier cascade, nestedCascade; double scale = 1.0; string cascadeName = "data/haarcascades/haarcascade_frontalface_alt.xml"; string nestedCascadeName = "data/haarcascades/haarcascade_eye_tree_eyeglasses.xml"; if(!cascade.load(samples::findFile(cascadeName))) { cerr << "ERROR: Could not load classifier cascade" << endl; return -1; } if(!nestedCascade.load(samples::findFileOrKeep(nestedCascadeName))) cerr << "WARNING: Could not load nested cascade" << endl; if(!capture.open(0)) { cout << "Capture from camera didn't work" << endl; return 1; } cout << "Video capturing has been started ..." << endl; for(;;) { capture >> frame; if(frame.empty()) break; Mat frame1 = frame.clone(); detectAndDraw(frame1, cascade, nestedCascade, scale, false); char c = (char)waitKey(10); if(c == 27 || c == 'q' || c == 'Q') break; } return 0; } ``` ## DNN-based Face Detection with YuNet Modern deep learning approach using the YuNet model for accurate face detection with facial landmarks. ```python theme={null} import cv2 as cv import numpy as np # Initialize YuNet face detector detector = cv.FaceDetectorYN.create( 'face_detection_yunet_2021dec.onnx', "", (320, 320), score_threshold=0.9, nms_threshold=0.3, top_k=5000 ) def visualize(input, faces, fps, thickness=2): """Draw detected faces with landmarks""" if faces[1] is not None: for idx, face in enumerate(faces[1]): coords = face[:-1].astype(np.int32) # Draw bounding box cv.rectangle(input, (coords[0], coords[1]), (coords[0]+coords[2], coords[1]+coords[3]), (0, 255, 0), thickness) # Draw facial landmarks cv.circle(input, (coords[4], coords[5]), 2, (255, 0, 0), thickness) # right eye cv.circle(input, (coords[6], coords[7]), 2, (0, 0, 255), thickness) # left eye cv.circle(input, (coords[8], coords[9]), 2, (0, 255, 0), thickness) # nose cv.circle(input, (coords[10], coords[11]), 2, (255, 0, 255), thickness) # right mouth cv.circle(input, (coords[12], coords[13]), 2, (0, 255, 255), thickness) # left mouth cv.putText(input, f'FPS: {fps:.2f}', (1, 16), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) # Video capture cap = cv.VideoCapture(0) frameWidth = int(cap.get(cv.CAP_PROP_FRAME_WIDTH)) frameHeight = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT)) detector.setInputSize([frameWidth, frameHeight]) tm = cv.TickMeter() while True: hasFrame, frame = cap.read() if not hasFrame: break # Detect faces tm.start() faces = detector.detect(frame) tm.stop() # Draw results visualize(frame, faces, tm.getFPS()) cv.imshow('Face Detection', frame) if cv.waitKey(1) == 27: break cap.release() cv.destroyAllWindows() ``` ## Face Recognition with SFace Compare faces and determine if they belong to the same person. ```python theme={null} import cv2 as cv # Initialize detector and recognizer detector = cv.FaceDetectorYN.create( 'face_detection_yunet_2021dec.onnx', "", (320, 320), 0.9, 0.3, 5000 ) recognizer = cv.FaceRecognizerSF.create( 'face_recognition_sface_2021dec.onnx', "" ) # Load two images img1 = cv.imread('person1.jpg') img2 = cv.imread('person2.jpg') # Detect faces detector.setInputSize((img1.shape[1], img1.shape[0])) faces1 = detector.detect(img1) detector.setInputSize((img2.shape[1], img2.shape[0])) faces2 = detector.detect(img2) if faces1[1] is None or faces2[1] is None: print("No face detected") else: # Align and extract features face1_align = recognizer.alignCrop(img1, faces1[1][0]) face2_align = recognizer.alignCrop(img2, faces2[1][0]) face1_feature = recognizer.feature(face1_align) face2_feature = recognizer.feature(face2_align) # Compare faces cosine_score = recognizer.match( face1_feature, face2_feature, cv.FaceRecognizerSF_FR_COSINE ) l2_score = recognizer.match( face1_feature, face2_feature, cv.FaceRecognizerSF_FR_NORM_L2 ) # Thresholds cosine_threshold = 0.363 l2_threshold = 1.128 if cosine_score >= cosine_threshold: print(f"Same person (Cosine: {cosine_score:.3f})") else: print(f"Different person (Cosine: {cosine_score:.3f})") if l2_score <= l2_threshold: print(f"Same person (L2: {l2_score:.3f})") else: print(f"Different person (L2: {l2_score:.3f})") ``` ## Key Parameters ### Haar Cascade Detection | Parameter | Description | Typical Value | | -------------- | ----------------------------------- | ------------- | | `scaleFactor` | Image scale reduction between scans | 1.1 - 1.3 | | `minNeighbors` | Minimum neighbors for detection | 3 - 6 | | `minSize` | Minimum object size | (30, 30) | | `maxSize` | Maximum object size | Image size | ### YuNet Detection | Parameter | Description | Typical Value | | ----------------- | ------------------------- | ------------- | | `score_threshold` | Confidence threshold | 0.6 - 0.9 | | `nms_threshold` | Non-maximum suppression | 0.3 - 0.5 | | `top_k` | Max detections before NMS | 5000 | **Model Downloads**: YuNet and SFace models can be downloaded from the [OpenCV Zoo](https://github.com/opencv/opencv_zoo): * [YuNet Face Detection](https://github.com/opencv/opencv_zoo/tree/master/models/face_detection_yunet) * [SFace Recognition](https://github.com/opencv/opencv_zoo/tree/master/models/face_recognition_sface) ## Performance Tips Reduce input image size for faster processing: ```python theme={null} fx = 0.5 # 50% scale small = cv.resize(gray, None, fx=fx, fy=fx) ``` Improve detection under varying lighting: ```python theme={null} gray = cv.equalizeHist(gray) ``` Tune `minNeighbors` to balance speed vs accuracy: * Lower values = faster, more false positives * Higher values = slower, fewer false positives Detect nested features only within face regions to improve performance **Privacy Considerations**: Face recognition technology should be used responsibly. Always obtain consent when processing personal biometric data and comply with relevant privacy regulations. ## Next Steps * Explore [Object Detection](/modules/objdetect) for other detection methods * Learn about [DNN Module](/modules/dnn) for deep learning models * Check [Video I/O](/api/videoio) for camera handling * See [Image Processing](/modules/imgproc) for preprocessing techniques # Image Classification with DNN Source: https://opencv-opencv.mintlify.app/examples/image-classification Learn how to perform image classification using OpenCV DNN module with popular models like ResNet, MobileNet, and GoogLeNet Image classification is the task of assigning a label or category to an entire image. OpenCV's DNN module provides support for running pre-trained classification models from various frameworks. ## Supported Models The following classification models are commonly used: * **ResNet** - Deep residual networks with skip connections * **MobileNet** - Lightweight models optimized for mobile devices * **GoogLeNet** - Inception architecture from Google * **SqueezeNet** - Compact model with high accuracy * **VGG** - Very deep convolutional networks ## Python Implementation ```python theme={null} import cv2 as cv import numpy as np ``` ```python theme={null} # Load the pre-trained model model = 'bvlc_googlenet.caffemodel' config = 'bvlc_googlenet.prototxt' net = cv.dnn.readNet(model, config) # Set computation backend and target net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) ``` You can use `DNN_BACKEND_CUDA` and `DNN_TARGET_CUDA` for GPU acceleration if available. ```python theme={null} # Load class labels classes = None with open('classification_classes_ILSVRC2012.txt', 'rt') as f: classes = f.read().rstrip('\n').split('\n') ``` ```python theme={null} # Read the input image frame = cv.imread('image.jpg') # Create a 4D blob from the image # GoogLeNet uses 224x224 input with mean [104, 117, 123] blob = cv.dnn.blobFromImage(frame, 1.0, (224, 224), [104, 117, 123], False, crop=False) ``` The `blobFromImage` function performs: * Mean subtraction * Scaling * Optional channel swapping (BGR to RGB) * Resizing to target dimensions ```python theme={null} # Set the input blob net.setInput(blob) # Forward pass to get predictions out = net.forward() # Get the class with highest score out = out.flatten() classId = np.argmax(out) confidence = out[classId] # Print the result label = f'{classes[classId]}: {confidence:.4f}' print(label) ``` ```python theme={null} # Get inference time t, _ = net.getPerfProfile() inference_time = t * 1000.0 / cv.getTickFrequency() # Put text on image label = f'Inference time: {inference_time:.2f} ms' cv.putText(frame, label, (0, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0)) label = f'{classes[classId]}: {confidence:.4f}' cv.putText(frame, label, (0, 40), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0)) # Display the result cv.imshow('Classification', frame) cv.waitKey(0) ``` ## C++ Implementation ```cpp theme={null} #include #include #include #include #include using namespace cv; using namespace dnn; int main() { // Load the network String model = "bvlc_googlenet.caffemodel"; String config = "bvlc_googlenet.prototxt"; Net net = readNet(model, config); net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // Read input image Mat frame = imread("image.jpg"); // Create a 4D blob from the frame Mat blob; Scalar mean(104, 117, 123); blobFromImage(frame, blob, 1.0, Size(224, 224), mean, false, false); // Set input blob net.setInput(blob); // Make forward pass Mat prob = net.forward(); // Get the class with highest score Point classIdPoint; double confidence; minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint); int classId = classIdPoint.x; std::cout << "Class ID: " << classId << ", Confidence: " << confidence << std::endl; return 0; } ``` ```cpp theme={null} // Open video capture VideoCapture cap; cap.open("video.mp4"); // or use 0 for camera Mat frame, blob; while (waitKey(1) < 0) { cap >> frame; if (frame.empty()) { break; } // Create blob from frame blobFromImage(frame, blob, 1.0, Size(224, 224), mean, swapRB, false); // Run inference net.setInput(blob); Mat prob = net.forward(); // Get classification result Point classIdPoint; double confidence; minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint); int classId = classIdPoint.x; // Display results std::string label = format("%s: %.4f", classes[classId].c_str(), confidence); putText(frame, label, Point(0, 40), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0)); imshow("Classification", frame); } ``` ## Model Download and Configuration ### GoogLeNet (Caffe) ```yaml theme={null} googlenet: model: "bvlc_googlenet.caffemodel" config: "bvlc_googlenet.prototxt" mean: [104, 117, 123] scale: 1.0 width: 224 height: 224 rgb: false classes: "classification_classes_ILSVRC2012.txt" ``` **Download:** [http://dl.caffe.berkeleyvision.org/bvlc\_googlenet.caffemodel](http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel) ### SqueezeNet (Caffe) ```yaml theme={null} squeezenet: model: "squeezenet_v1.1.caffemodel" config: "squeezenet_v1.1.prototxt" mean: [0, 0, 0] scale: 1.0 width: 227 height: 227 rgb: false classes: "classification_classes_ILSVRC2012.txt" ``` **Download:** [https://github.com/DeepScale/SqueezeNet](https://github.com/DeepScale/SqueezeNet) (SqueezeNet v1.1) ## Preprocessing Parameters Different models require different preprocessing parameters: | Model | Input Size | Mean | Scale | RGB Order | | ---------- | ---------- | ------------------------- | -------- | --------- | | GoogLeNet | 224x224 | \[104, 117, 123] | 1.0 | BGR | | SqueezeNet | 227x227 | \[0, 0, 0] | 1.0 | BGR | | ResNet | 224x224 | \[103.94, 116.78, 123.68] | 1.0 | BGR | | MobileNet | 224x224 | \[127.5, 127.5, 127.5] | 0.007843 | RGB | ## Backend and Target Options ### Available Backends ```python theme={null} # Computation backends cv.dnn.DNN_BACKEND_DEFAULT # Automatic selection cv.dnn.DNN_BACKEND_OPENCV # OpenCV implementation cv.dnn.DNN_BACKEND_INFERENCE_ENGINE # Intel OpenVINO cv.dnn.DNN_BACKEND_CUDA # NVIDIA CUDA cv.dnn.DNN_BACKEND_VKCOM # Vulkan ``` ### Available Targets ```python theme={null} # Target devices cv.dnn.DNN_TARGET_CPU # CPU cv.dnn.DNN_TARGET_OPENCL # OpenCL (GPU) cv.dnn.DNN_TARGET_OPENCL_FP16 # OpenCL with FP16 cv.dnn.DNN_TARGET_CUDA # CUDA (GPU) cv.dnn.DNN_TARGET_CUDA_FP16 # CUDA with FP16 ``` When using CUDA backend, ensure you have compiled OpenCV with CUDA support and the appropriate CUDA toolkit installed. ## Complete Example Here's a complete classification example that processes video frames: ```python theme={null} import cv2 as cv import numpy as np import argparse def main(): # Parse arguments parser = argparse.ArgumentParser() parser.add_argument('--model', required=True, help='Path to model file') parser.add_argument('--config', help='Path to config file') parser.add_argument('--classes', help='Path to classes file') parser.add_argument('--input', help='Path to input image or video') parser.add_argument('--backend', type=int, default=cv.dnn.DNN_BACKEND_OPENCV) parser.add_argument('--target', type=int, default=cv.dnn.DNN_TARGET_CPU) args = parser.parse_args() # Load class names classes = None if args.classes: with open(args.classes, 'rt') as f: classes = f.read().rstrip('\n').split('\n') # Load network net = cv.dnn.readNet(args.model, args.config) net.setPreferableBackend(args.backend) net.setPreferableTarget(args.target) # Open video capture cap = cv.VideoCapture(args.input if args.input else 0) while cv.waitKey(1) < 0: hasFrame, frame = cap.read() if not hasFrame: break # Create blob blob = cv.dnn.blobFromImage(frame, 1.0, (224, 224), [104, 117, 123], False) # Run model net.setInput(blob) out = net.forward() # Get result out = out.flatten() classId = np.argmax(out) confidence = out[classId] # Display t, _ = net.getPerfProfile() label = f'Inference time: {t * 1000.0 / cv.getTickFrequency():.2f} ms' cv.putText(frame, label, (0, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0)) if classes: label = f'{classes[classId]}: {confidence:.4f}' cv.putText(frame, label, (0, 40), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0)) cv.imshow('Classification', frame) if __name__ == '__main__': main() ``` ## Source Code The complete source code for classification examples can be found in the OpenCV repository: * Python: `samples/dnn/classification.py` * C++: `samples/dnn/classification.cpp` # Image Transformations Source: https://opencv-opencv.mintlify.app/examples/image-transformations Learn how to resize, rotate, flip, crop, and apply geometric transformations to images This guide covers essential geometric transformations including resizing, rotation, flipping, cropping, and more advanced affine and perspective transformations. ## Overview Image transformations allow you to modify the geometry of images. Common operations include: * Resizing images to different dimensions * Rotating images by any angle * Flipping images horizontally or vertically * Cropping regions of interest * Applying affine and perspective transformations ## Resizing Images Resize images to specific dimensions or by a scaling factor. ```python theme={null} import cv2 as cv import numpy as np # Load image img = cv.imread("image.jpg") height, width = img.shape[:2] # Resize to specific dimensions resized = cv.resize(img, (800, 600)) # Resize by scaling factor scaled = cv.resize(img, None, fx=0.5, fy=0.5, interpolation=cv.INTER_LINEAR) # Resize maintaining aspect ratio new_width = 640 aspect_ratio = new_width / width new_height = int(height * aspect_ratio) resized_aspect = cv.resize(img, (new_width, new_height)) cv.imshow("Original", img) cv.imshow("Resized", resized) cv.imshow("Scaled", scaled) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); int height = img.rows; int width = img.cols; // Resize to specific dimensions Mat resized; resize(img, resized, Size(800, 600)); // Resize by scaling factor Mat scaled; resize(img, scaled, Size(), 0.5, 0.5, INTER_LINEAR); // Resize maintaining aspect ratio int new_width = 640; double aspect_ratio = (double)new_width / width; int new_height = (int)(height * aspect_ratio); Mat resized_aspect; resize(img, resized_aspect, Size(new_width, new_height)); imshow("Original", img); imshow("Resized", resized); imshow("Scaled", scaled); waitKey(0); ``` ### Interpolation Methods | Method | Description | Use Case | | ---------------- | --------------------------- | ------------------------ | | `INTER_NEAREST` | Nearest neighbor | Fastest, lowest quality | | `INTER_LINEAR` | Bilinear interpolation | Good balance (default) | | `INTER_CUBIC` | Bicubic interpolation | Slower, higher quality | | `INTER_AREA` | Resampling using pixel area | Best for downsampling | | `INTER_LANCZOS4` | Lanczos interpolation | Highest quality, slowest | ## Rotating Images Rotate images by 90-degree increments or arbitrary angles. ### Simple 90-Degree Rotations ```python theme={null} import cv2 as cv img = cv.imread("image.jpg") # Rotate 90 degrees clockwise rotated_90_cw = cv.rotate(img, cv.ROTATE_90_CLOCKWISE) # Rotate 90 degrees counter-clockwise rotated_90_ccw = cv.rotate(img, cv.ROTATE_90_COUNTERCLOCKWISE) # Rotate 180 degrees rotated_180 = cv.rotate(img, cv.ROTATE_180) cv.imshow("Original", img) cv.imshow("90° CW", rotated_90_cw) cv.imshow("180°", rotated_180) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat rotated_90_cw, rotated_90_ccw, rotated_180; // Rotate 90 degrees clockwise rotate(img, rotated_90_cw, ROTATE_90_CLOCKWISE); // Rotate 90 degrees counter-clockwise rotate(img, rotated_90_ccw, ROTATE_90_COUNTERCLOCKWISE); // Rotate 180 degrees rotate(img, rotated_180, ROTATE_180); imshow("Original", img); imshow("90° CW", rotated_90_cw); imshow("180°", rotated_180); waitKey(0); ``` ### Arbitrary Angle Rotation ```python theme={null} import cv2 as cv import numpy as np img = cv.imread("image.jpg") height, width = img.shape[:2] # Define rotation center (image center) center = (width // 2, height // 2) # Rotation angle in degrees (positive = counter-clockwise) angle = 45 # Scale factor (1.0 = no scaling) scale = 1.0 # Get rotation matrix rotation_matrix = cv.getRotationMatrix2D(center, angle, scale) # Apply rotation rotated = cv.warpAffine(img, rotation_matrix, (width, height)) cv.imshow("Original", img) cv.imshow("Rotated 45°", rotated) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); int height = img.rows; int width = img.cols; // Define rotation center (image center) Point2f center(width / 2.0, height / 2.0); // Rotation angle in degrees (positive = counter-clockwise) double angle = 45.0; // Scale factor (1.0 = no scaling) double scale = 1.0; // Get rotation matrix Mat rotation_matrix = getRotationMatrix2D(center, angle, scale); // Apply rotation Mat rotated; warpAffine(img, rotated, rotation_matrix, Size(width, height)); imshow("Original", img); imshow("Rotated 45°", rotated); waitKey(0); ``` When rotating by arbitrary angles, parts of the image may be cropped. To preserve the entire rotated image, calculate new dimensions and adjust the rotation matrix accordingly. ## Flipping Images Flip images horizontally, vertically, or both. ```python theme={null} import cv2 as cv img = cv.imread("image.jpg") # Flip horizontally (mirror) flipped_h = cv.flip(img, 1) # Flip vertically flipped_v = cv.flip(img, 0) # Flip both horizontally and vertically flipped_both = cv.flip(img, -1) cv.imshow("Original", img) cv.imshow("Horizontal Flip", flipped_h) cv.imshow("Vertical Flip", flipped_v) cv.imshow("Both", flipped_both) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat flipped_h, flipped_v, flipped_both; // Flip horizontally (mirror) flip(img, flipped_h, 1); // Flip vertically flip(img, flipped_v, 0); // Flip both horizontally and vertically flip(img, flipped_both, -1); imshow("Original", img); imshow("Horizontal Flip", flipped_h); imshow("Vertical Flip", flipped_v); imshow("Both", flipped_both); waitKey(0); ``` ## Cropping Images Crop a region of interest from an image using array slicing. ```python theme={null} import cv2 as cv img = cv.imread("image.jpg") # Define crop region (y1:y2, x1:x2) y1, y2 = 100, 400 x1, x2 = 200, 600 # Crop the image cropped = img[y1:y2, x1:x2] cv.imshow("Original", img) cv.imshow("Cropped", cropped) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); // Define crop region using Rect(x, y, width, height) Rect roi(200, 100, 400, 300); // Crop the image Mat cropped = img(roi); imshow("Original", img); imshow("Cropped", cropped); waitKey(0); ``` ## Affine Transformations Affine transformations preserve parallel lines and include translation, rotation, scaling, and shearing. ```python theme={null} import cv2 as cv import numpy as np img = cv.imread("image.jpg") rows, cols = img.shape[:2] # Define source points (3 points from original image) src_points = np.float32([[50, 50], [200, 50], [50, 200]]) # Define destination points (where those points should move to) dst_points = np.float32([[10, 100], [200, 50], [100, 250]]) # Get affine transformation matrix affine_matrix = cv.getAffineTransform(src_points, dst_points) # Apply affine transformation transformed = cv.warpAffine(img, affine_matrix, (cols, rows)) cv.imshow("Original", img) cv.imshow("Affine Transform", transformed) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; using namespace std; Mat img = imread("image.jpg"); int rows = img.rows; int cols = img.cols; // Define source points (3 points from original image) Point2f src_points[3]; src_points[0] = Point2f(50, 50); src_points[1] = Point2f(200, 50); src_points[2] = Point2f(50, 200); // Define destination points Point2f dst_points[3]; dst_points[0] = Point2f(10, 100); dst_points[1] = Point2f(200, 50); dst_points[2] = Point2f(100, 250); // Get affine transformation matrix Mat affine_matrix = getAffineTransform(src_points, dst_points); // Apply affine transformation Mat transformed; warpAffine(img, transformed, affine_matrix, Size(cols, rows)); imshow("Original", img); imshow("Affine Transform", transformed); waitKey(0); ``` ## Perspective Transformations Perspective transformations correct for camera angle and viewpoint changes. ```python theme={null} import cv2 as cv import numpy as np img = cv.imread("image.jpg") rows, cols = img.shape[:2] # Define source points (4 corners from original image) src_points = np.float32([[56, 65], [368, 52], [28, 387], [389, 390]]) # Define destination points (rectangle) dst_points = np.float32([[0, 0], [300, 0], [0, 300], [300, 300]]) # Get perspective transformation matrix perspective_matrix = cv.getPerspectiveTransform(src_points, dst_points) # Apply perspective transformation warped = cv.warpPerspective(img, perspective_matrix, (300, 300)) cv.imshow("Original", img) cv.imshow("Perspective Transform", warped) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; using namespace std; Mat img = imread("image.jpg"); // Define source points (4 corners from original image) vector src_points; src_points.push_back(Point2f(56, 65)); src_points.push_back(Point2f(368, 52)); src_points.push_back(Point2f(28, 387)); src_points.push_back(Point2f(389, 390)); // Define destination points (rectangle) vector dst_points; dst_points.push_back(Point2f(0, 0)); dst_points.push_back(Point2f(300, 0)); dst_points.push_back(Point2f(0, 300)); dst_points.push_back(Point2f(300, 300)); // Get perspective transformation matrix Mat perspective_matrix = getPerspectiveTransform(src_points, dst_points); // Apply perspective transformation Mat warped; warpPerspective(img, warped, perspective_matrix, Size(300, 300)); imshow("Original", img); imshow("Perspective Transform", warped); waitKey(0); ``` Perspective transformations are commonly used for document scanning, where you select the four corners of a document in a photo and transform it to a flat, rectangular view. ## Key Functions | Function | Description | | --------------------------- | -------------------------------------------------------- | | `resize()` | Resize image to specific dimensions or scale | | `rotate()` | Rotate image by 90, 180, or 270 degrees | | `flip()` | Flip image horizontally, vertically, or both | | `getRotationMatrix2D()` | Get rotation matrix for arbitrary angles | | `getAffineTransform()` | Get affine transformation matrix from 3 point pairs | | `getPerspectiveTransform()` | Get perspective transformation matrix from 4 point pairs | | `warpAffine()` | Apply affine transformation | | `warpPerspective()` | Apply perspective transformation | When applying transformations, pixels that fall outside the destination image are cropped. Use appropriate border modes or adjust output dimensions to preserve all data. # Object Detection with DNN Source: https://opencv-opencv.mintlify.app/examples/object-detection-dnn Detect objects in images and videos using YOLO, SSD, and Faster R-CNN models with OpenCV DNN module Object detection identifies and localizes multiple objects in an image, providing both class labels and bounding box coordinates. OpenCV's DNN module supports popular detection models like YOLO, SSD, and Faster R-CNN. ## Supported Models **YOLO (You Only Look Once)** - Real-time object detection * YOLOv3, YOLOv4 (Darknet) * YOLOv5 (PyTorch/ONNX) * YOLOv8, YOLOv9, YOLOv10 (Ultralytics) * YOLOX, YOLO-NAS YOLO models are single-stage detectors optimized for speed. **SSD (Single Shot MultiBox Detector)** * MobileNet-SSD (Caffe) * SSD with various backbones (TensorFlow) SSD models balance speed and accuracy with multi-scale feature maps. **Faster R-CNN** - Two-stage detector * Faster R-CNN with Inception v2 * Faster R-CNN with ResNet Two-stage detectors prioritize accuracy over speed. ## YOLO Object Detection (Python) ```python theme={null} import cv2 as cv import numpy as np ``` ```python theme={null} # Load YOLOv8 model model = 'yolov8n.onnx' net = cv.dnn.readNet(model) net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # Load class names with open('object_detection_classes_yolo.txt', 'rt') as f: classes = f.read().rstrip('\n').split('\n') ``` ```python theme={null} # Read image frame = cv.imread('image.jpg') frameHeight, frameWidth = frame.shape[:2] # Create blob from image # YOLOv8 uses 640x640 input with scale 1/255 blob = cv.dnn.blobFromImage(frame, 1/255.0, (640, 640), [0, 0, 0], True, crop=False) ``` ```python theme={null} # Set input and run forward pass net.setInput(blob) outNames = net.getUnconnectedOutLayersNames() outs = net.forward(outNames) ``` ```python theme={null} def postprocess(frame, outs, confThreshold=0.5, nmsThreshold=0.4): frameHeight, frameWidth = frame.shape[:2] classIds = [] confidences = [] boxes = [] # For YOLOv8, output shape is [1, 84, 8400] -> transpose to [1, 8400, 84] for out in outs: out = out[0].transpose(1, 0) # [8400, 84] for detection in out: scores = detection[4:] # class scores classId = np.argmax(scores) confidence = scores[classId] if confidence > confThreshold: # YOLOv8 uses center format: [cx, cy, w, h] center_x = int(detection[0] * frameWidth / 640) center_y = int(detection[1] * frameHeight / 640) width = int(detection[2] * frameWidth / 640) height = int(detection[3] * frameHeight / 640) left = int(center_x - width / 2) top = int(center_y - height / 2) classIds.append(classId) confidences.append(float(confidence)) boxes.append([left, top, width, height]) # Apply Non-Maximum Suppression indices = cv.dnn.NMSBoxes(boxes, confidences, confThreshold, nmsThreshold) return classIds, confidences, boxes, indices classIds, confidences, boxes, indices = postprocess(frame, outs) ``` ```python theme={null} def drawPred(frame, classId, conf, left, top, right, bottom, classes): # Draw bounding box cv.rectangle(frame, (left, top), (right, bottom), (0, 255, 0), 2) # Create label label = f'{classes[classId]}: {conf:.2f}' # Draw label background labelSize, baseLine = cv.getTextSize(label, cv.FONT_HERSHEY_SIMPLEX, 0.5, 1) top = max(top, labelSize[1]) cv.rectangle(frame, (left, top - labelSize[1]), (left + labelSize[0], top + baseLine), (255, 255, 255), cv.FILLED) # Draw label text cv.putText(frame, label, (left, top), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0)) # Draw all detections for i in indices: box = boxes[i] left, top, width, height = box drawPred(frame, classIds[i], confidences[i], left, top, left + width, top + height, classes) # Display result cv.imshow('Object Detection', frame) cv.waitKey(0) ``` ## YOLO Object Detection (C++) ```cpp theme={null} #include #include #include #include #include using namespace cv; using namespace cv::dnn; std::vector classes; void getClasses(std::string classesFile) { std::ifstream ifs(classesFile.c_str()); if (!ifs.is_open()) CV_Error(Error::StsError, "File " + classesFile + " not found"); std::string line; while (std::getline(ifs, line)) classes.push_back(line); } void drawPrediction(int classId, float conf, int left, int top, int right, int bottom, Mat& frame) { rectangle(frame, Point(left, top), Point(right, bottom), Scalar(0, 255, 0)); std::string label = format("%.2f", conf); if (!classes.empty()) { CV_Assert(classId < (int)classes.size()); label = classes[classId] + ": " + label; } int baseLine; Size labelSize = getTextSize(label, FONT_HERSHEY_SIMPLEX, 0.5, 1, &baseLine); top = max(top, labelSize.height); rectangle(frame, Point(left, top - labelSize.height), Point(left + labelSize.width, top + baseLine), Scalar::all(255), FILLED); putText(frame, label, Point(left, top), FONT_HERSHEY_SIMPLEX, 0.5, Scalar()); } void yoloPostProcessing(std::vector& outs, std::vector& keep_classIds, std::vector& keep_confidences, std::vector& keep_boxes, float conf_threshold, float iou_threshold, const std::string& model_name, const int nc = 80) { std::vector classIds; std::vector confidences; std::vector boxes; // For YOLOv8/v9/v10, transpose output if (model_name == "yolov8" || model_name == "yolov10" || model_name == "yolov9") { cv::transposeND(outs[0], {0, 2, 1}, outs[0]); } for (auto preds : outs) { preds = preds.reshape(1, preds.size[1]); for (int i = 0; i < preds.rows; ++i) { // Filter out non-objects float obj_conf = (model_name == "yolov8" || model_name == "yolov9" || model_name == "yolov10") ? 1.0f : preds.at(i, 4); if (obj_conf < conf_threshold) continue; Mat scores = preds.row(i).colRange( (model_name == "yolov8" || model_name == "yolov9" || model_name == "yolov10") ? 4 : 5, preds.cols); double conf; Point maxLoc; minMaxLoc(scores, 0, &conf, 0, &maxLoc); conf = (model_name == "yolov8" || model_name == "yolov9" || model_name == "yolov10") ? conf : conf * obj_conf; if (conf < conf_threshold) continue; // Get bbox coordinates float* det = preds.ptr(i); double cx = det[0]; double cy = det[1]; double w = det[2]; double h = det[3]; if (model_name == "yolov10") { boxes.push_back(Rect2d(cx, cy, w, h)); } else { boxes.push_back(Rect2d(cx - 0.5 * w, cy - 0.5 * h, cx + 0.5 * w, cy + 0.5 * h)); } classIds.push_back(maxLoc.x); confidences.push_back(static_cast(conf)); } } // Apply NMS std::vector keep_idx; NMSBoxes(boxes, confidences, conf_threshold, iou_threshold, keep_idx); for (auto i : keep_idx) { keep_classIds.push_back(classIds[i]); keep_confidences.push_back(confidences[i]); keep_boxes.push_back(boxes[i]); } } int main(int argc, char** argv) { // Load model std::string weightPath = "yolov8n.onnx"; Net net = readNet(weightPath); net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // Load classes getClasses("object_detection_classes_yolo.txt"); // Read image Mat img = imread("image.jpg"); // Preprocess Mat blob; Size size(640, 640); Scalar mean(0, 0, 0); Scalar scale(1.0/255.0, 1.0/255.0, 1.0/255.0); blobFromImage(img, blob, 1.0/255.0, size, mean, true, false); // Forward pass net.setInput(blob); std::vector outs; net.forward(outs, net.getUnconnectedOutLayersNames()); // Postprocess std::vector keep_classIds; std::vector keep_confidences; std::vector keep_boxes; yoloPostProcessing(outs, keep_classIds, keep_confidences, keep_boxes, 0.5, 0.4, "yolov8", 80); // Draw results std::vector boxes; for (auto box : keep_boxes) { boxes.push_back(Rect(cvFloor(box.x), cvFloor(box.y), cvFloor(box.width - box.x), cvFloor(box.height - box.y))); } for (size_t idx = 0; idx < boxes.size(); ++idx) { Rect box = boxes[idx]; drawPrediction(keep_classIds[idx], keep_confidences[idx], box.x, box.y, box.width + box.x, box.height + box.y, img); } imshow("YOLO Object Detection", img); waitKey(0); return 0; } ``` ```cpp theme={null} // Open video capture VideoCapture cap; cap.open(0); // or video file path Mat img; while (waitKey(1) < 0) { cap >> img; if (img.empty()) break; // Preprocess Mat inp = blobFromImageWithParams(img, imgParams); // Forward net.setInput(inp); std::vector outs; net.forward(outs, net.getUnconnectedOutLayersNames()); // Postprocess std::vector keep_classIds; std::vector keep_confidences; std::vector keep_boxes; yoloPostProcessing(outs, keep_classIds, keep_confidences, keep_boxes, confThreshold, nmsThreshold, yolo_model, nc); // Draw boxes for (size_t idx = 0; idx < keep_boxes.size(); ++idx) { Rect2d box = keep_boxes[idx]; drawPrediction(keep_classIds[idx], keep_confidences[idx], box.x, box.y, box.width + box.x, box.height + box.y, img); } imshow("YOLO Object Detector", img); } ``` ## SSD Object Detection ### MobileNet-SSD (Caffe) ```python theme={null} import cv2 as cv # Load MobileNet-SSD model model = 'MobileNetSSD_deploy.caffemodel' config = 'MobileNetSSD_deploy.prototxt' net = cv.dnn.readNetFromCaffe(config, model) # Prepare input frame = cv.imread('image.jpg') blob = cv.dnn.blobFromImage(frame, 0.007843, (300, 300), [127.5, 127.5, 127.5], False) # Run detection net.setInput(blob) detections = net.forward() # Process detections for i in range(detections.shape[2]): confidence = detections[0, 0, i, 2] if confidence > 0.5: # Extract bounding box box = detections[0, 0, i, 3:7] * np.array([width, height, width, height]) (left, top, right, bottom) = box.astype("int") # Draw detection cv.rectangle(frame, (left, top), (right, bottom), (0, 255, 0), 2) ``` ## Model Configurations ### YOLOv8 (ONNX) ```yaml theme={null} yolov8: model: "yolov8n.onnx" mean: [0, 0, 0] scale: 0.00392 # 1/255 width: 640 height: 640 rgb: true classes: "object_detection_classes_yolo.txt" ``` **Download:** [https://github.com/ultralytics/ultralytics](https://github.com/ultralytics/ultralytics) ### YOLOv4 (Darknet) ```yaml theme={null} yolov4: model: "yolov4.weights" config: "yolov4.cfg" mean: [0, 0, 0] scale: 0.00392 width: 416 height: 416 rgb: true classes: "object_detection_classes_yolo.txt" ``` **Download:** [https://github.com/AlexeyAB/darknet/releases](https://github.com/AlexeyAB/darknet/releases) ### MobileNet-SSD (Caffe) ```yaml theme={null} ssd_caffe: model: "MobileNetSSD_deploy.caffemodel" config: "MobileNetSSD_deploy.prototxt" mean: [127.5, 127.5, 127.5] scale: 0.007843 width: 300 height: 300 rgb: false classes: "object_detection_classes_pascal_voc.txt" ``` ### Faster R-CNN (TensorFlow) ```yaml theme={null} faster_rcnn_tf: model: "faster_rcnn_inception_v2_coco_2018_01_28.pb" config: "faster_rcnn_inception_v2_coco_2018_01_28.pbtxt" mean: [0, 0, 0] scale: 1.0 width: 800 height: 600 rgb: true ``` **Download:** [http://download.tensorflow.org/models/object\_detection/](http://download.tensorflow.org/models/object_detection/) ## Non-Maximum Suppression (NMS) NMS removes duplicate detections by suppressing boxes with high overlap. ```python theme={null} # Apply NMS confidence_threshold = 0.5 nms_threshold = 0.4 indices = cv.dnn.NMSBoxes(boxes, confidences, confidence_threshold, nms_threshold) for i in indices: box = boxes[i] # Draw detection drawPrediction(frame, classIds[i], confidences[i], box[0], box[1], box[0] + box[2], box[1] + box[3]) ``` ## Performance Tips ```python theme={null} net.setPreferableBackend(cv.dnn.DNN_BACKEND_CUDA) net.setPreferableTarget(cv.dnn.DNN_TARGET_CUDA) ``` ```python theme={null} net.setPreferableTarget(cv.dnn.DNN_TARGET_CUDA_FP16) ``` * YOLOv8n (nano): Fastest, lowest accuracy * YOLOv8s (small): Balanced * YOLOv8m (medium): Higher accuracy * YOLOv8l/x (large/xlarge): Best accuracy, slowest Different YOLO versions (v3, v4, v5, v8) have different output formats and require different post-processing. ## Source Code Complete source code for object detection: * Python: `samples/dnn/object_detection.py` * C++ (YOLO): `samples/dnn/yolo_detector.cpp` * C++ (Generic): `samples/dnn/object_detection.cpp` # Real-Time Object Tracking Source: https://opencv-opencv.mintlify.app/examples/object-tracking Track objects across video frames using various tracking algorithms including Lucas-Kanade, MIL, GOTURN, DaSiamRPN, and NanoTrack ## Overview OpenCV provides multiple tracking algorithms for different use cases: * **Optical Flow (Lucas-Kanade)**: Track sparse feature points * **MIL Tracker**: Multiple Instance Learning, CPU-friendly * **GOTURN**: Deep learning tracker using Caffe models * **DaSiamRPN**: State-of-the-art Siamese network tracker * **NanoTrack**: Lightweight deep learning tracker * **Planar Tracking**: Track planar objects using feature matching ## Lucas-Kanade Optical Flow Tracker Sparse optical flow tracking with automatic feature detection and back-tracking for verification. ```python theme={null} import numpy as np import cv2 as cv from video import create_capture from common import anorm2, draw_str # Lucas-Kanade parameters lk_params = dict( winSize=(15, 15), maxLevel=2, criteria=(cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 0.03) ) # Feature detection parameters feature_params = dict( maxCorners=500, qualityLevel=0.3, minDistance=7, blockSize=7 ) class LKTracker: def __init__(self, video_src): self.track_len = 10 self.detect_interval = 5 self.tracks = [] self.cam = create_capture(video_src) self.frame_idx = 0 def run(self): while True: _ret, frame = self.cam.read() frame_gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) vis = frame.copy() if len(self.tracks) > 0: img0, img1 = self.prev_gray, frame_gray p0 = np.float32([tr[-1] for tr in self.tracks]).reshape(-1, 1, 2) # Forward optical flow p1, _st, _err = cv.calcOpticalFlowPyrLK(img0, img1, p0, None, **lk_params) # Backward optical flow for verification p0r, _st, _err = cv.calcOpticalFlowPyrLK(img1, img0, p1, None, **lk_params) # Compute back-tracking error d = abs(p0 - p0r).reshape(-1, 2).max(-1) good = d < 1 new_tracks = [] for tr, (x, y), good_flag in zip(self.tracks, p1.reshape(-1, 2), good): if not good_flag: continue tr.append((x, y)) if len(tr) > self.track_len: del tr[0] new_tracks.append(tr) cv.circle(vis, (int(x), int(y)), 2, (0, 255, 0), -1) self.tracks = new_tracks cv.polylines(vis, [np.int32(tr) for tr in self.tracks], False, (0, 255, 0)) draw_str(vis, (20, 20), f'track count: {len(self.tracks)}') # Detect new features periodically if self.frame_idx % self.detect_interval == 0: mask = np.zeros_like(frame_gray) mask[:] = 255 for x, y in [np.int32(tr[-1]) for tr in self.tracks]: cv.circle(mask, (x, y), 5, 0, -1) p = cv.goodFeaturesToTrack(frame_gray, mask=mask, **feature_params) if p is not None: for x, y in np.float32(p).reshape(-1, 2): self.tracks.append([(x, y)]) self.frame_idx += 1 self.prev_gray = frame_gray cv.imshow('lk_track', vis) if cv.waitKey(1) == 27: break # Run tracker tracker = LKTracker(0) tracker.run() cv.destroyAllWindows() ``` ## Modern DNN-Based Trackers High-performance trackers using deep learning models. ```python theme={null} import cv2 as cv import numpy as np from video import create_capture class ObjectTracker: def __init__(self, tracker_type='nanotrack'): self.tracker_type = tracker_type self.tracker = self.create_tracker() def create_tracker(self): """Create tracker based on type""" if self.tracker_type == 'mil': # Multiple Instance Learning tracker return cv.TrackerMIL_create() elif self.tracker_type == 'goturn': # GOTURN deep learning tracker params = cv.TrackerGOTURN_Params() params.modelTxt = 'goturn.prototxt' params.modelBin = 'goturn.caffemodel' return cv.TrackerGOTURN_create(params) elif self.tracker_type == 'dasiamrpn': # DaSiamRPN tracker params = cv.TrackerDaSiamRPN_Params() params.model = 'dasiamrpn_model.onnx' params.kernel_cls1 = 'dasiamrpn_kernel_cls1.onnx' params.kernel_r1 = 'dasiamrpn_kernel_r1.onnx' params.backend = cv.dnn.DNN_BACKEND_OPENCV params.target = cv.dnn.DNN_TARGET_CPU return cv.TrackerDaSiamRPN_create(params) elif self.tracker_type == 'nanotrack': # NanoTrack lightweight tracker params = cv.TrackerNano_Params() params.backbone = 'nanotrack_backbone_sim.onnx' params.neckhead = 'nanotrack_head_sim.onnx' params.backend = cv.dnn.DNN_BACKEND_OPENCV params.target = cv.dnn.DNN_TARGET_CPU return cv.TrackerNano_create(params) elif self.tracker_type == 'vittrack': # Vision Transformer tracker params = cv.TrackerVit_Params() params.net = 'vitTracker.onnx' params.tracking_score_threshold = 0.3 params.backend = cv.dnn.DNN_BACKEND_OPENCV params.target = cv.dnn.DNN_TARGET_CPU return cv.TrackerVit_create(params) else: raise ValueError(f"Unknown tracker: {self.tracker_type}") def initialize_tracker(self, image): """Select ROI and initialize tracker""" print('Select object ROI for tracker...') bbox = cv.selectROI('tracking', image) print(f'ROI: {bbox}') if bbox[2] <= 0 or bbox[3] <= 0: raise ValueError("Invalid ROI selected") self.tracker.init(image, bbox) def run(self, video_path=0): """Run tracking on video""" camera = create_capture(video_path) if not camera.isOpened(): raise RuntimeError(f"Can't open video: {video_path}") # Read first frame and initialize ok, image = camera.read() if not ok: raise RuntimeError("Can't read first frame") cv.namedWindow('tracking') self.initialize_tracker(image) print("Tracking started. Press SPACE to re-init, ESC to exit...") while camera.isOpened(): ok, image = camera.read() if not ok: print("Can't read frame") break # Update tracker ok, newbox = self.tracker.update(image) if ok: # Draw bounding box p1 = (int(newbox[0]), int(newbox[1])) p2 = (int(newbox[0] + newbox[2]), int(newbox[1] + newbox[3])) cv.rectangle(image, p1, p2, (0, 255, 0), 2) else: # Tracking failure cv.putText(image, "Tracking failure", (10, 80), cv.FONT_HERSHEY_SIMPLEX, 0.75, (0, 0, 255), 2) # Display tracker type cv.putText(image, f"Tracker: {self.tracker_type}", (10, 20), cv.FONT_HERSHEY_SIMPLEX, 0.75, (50, 170, 50), 2) cv.imshow("tracking", image) k = cv.waitKey(1) if k == 32: # SPACE - reinitialize self.initialize_tracker(image) if k == 27: # ESC - exit break camera.release() cv.destroyAllWindows() # Example usage tracker = ObjectTracker('nanotrack') tracker.run('video.mp4') ``` ## Planar Object Tracker Track planar objects using feature matching with ORB and FLANN. ```python theme={null} import numpy as np import cv2 as cv from collections import namedtuple FLANN_INDEX_LSH = 6 flann_params = dict( algorithm=FLANN_INDEX_LSH, table_number=6, key_size=12, multi_probe_level=1 ) MIN_MATCH_COUNT = 10 PlanarTarget = namedtuple('PlaneTarget', 'image, rect, keypoints, descrs, data') TrackedTarget = namedtuple('TrackedTarget', 'target, p0, p1, H, quad') class PlaneTracker: def __init__(self): self.detector = cv.ORB_create(nfeatures=1000) self.matcher = cv.FlannBasedMatcher(flann_params, {}) self.targets = [] def add_target(self, image, rect, data=None): """Add new tracking target""" x0, y0, x1, y1 = rect raw_points, raw_descrs = self.detector.detectAndCompute(image, None) # Filter keypoints within rect points, descs = [], [] for kp, desc in zip(raw_points, raw_descrs): x, y = kp.pt if x0 <= x <= x1 and y0 <= y <= y1: points.append(kp) descs.append(desc) descs = np.uint8(descs) self.matcher.add([descs]) target = PlanarTarget( image=image, rect=rect, keypoints=points, descrs=descs, data=data ) self.targets.append(target) def track(self, frame): """Track targets in frame""" frame_points, frame_descrs = self.detector.detectAndCompute(frame, None) if len(frame_points) < MIN_MATCH_COUNT: return [] # Match features matches = self.matcher.knnMatch(frame_descrs, k=2) matches = [m[0] for m in matches if len(m) == 2 and m[0].distance < m[1].distance * 0.75] if len(matches) < MIN_MATCH_COUNT: return [] # Group matches by target matches_by_id = [[] for _ in range(len(self.targets))] for m in matches: matches_by_id[m.imgIdx].append(m) tracked = [] for imgIdx, matches in enumerate(matches_by_id): if len(matches) < MIN_MATCH_COUNT: continue target = self.targets[imgIdx] p0 = [target.keypoints[m.trainIdx].pt for m in matches] p1 = [frame_points[m.queryIdx].pt for m in matches] p0, p1 = np.float32((p0, p1)) # Find homography H, status = cv.findHomography(p0, p1, cv.RANSAC, 3.0) status = status.ravel() != 0 if status.sum() < MIN_MATCH_COUNT: continue p0, p1 = p0[status], p1[status] # Transform target rectangle x0, y0, x1, y1 = target.rect quad = np.float32([[x0, y0], [x1, y0], [x1, y1], [x0, y1]]) quad = cv.perspectiveTransform(quad.reshape(1, -1, 2), H).reshape(-1, 2) track = TrackedTarget(target=target, p0=p0, p1=p1, H=H, quad=quad) tracked.append(track) tracked.sort(key=lambda t: len(t.p0), reverse=True) return tracked # Example usage import video from common import RectSelector cap = video.create_capture(0) tracker = PlaneTracker() rect_sel = RectSelector('plane', tracker.add_target) while True: ret, frame = cap.read() if not ret: break vis = frame.copy() tracked = tracker.track(frame) for tr in tracked: cv.polylines(vis, [np.int32(tr.quad)], True, (255, 255, 255), 2) for (x, y) in np.int32(tr.p1): cv.circle(vis, (x, y), 2, (255, 255, 255)) cv.imshow('plane', vis) if cv.waitKey(1) == 27: break cv.destroyAllWindows() ``` ## Tracker Comparison | Tracker | Speed | Accuracy | CPU/GPU | Use Case | | ---------------- | --------- | --------- | ------- | --------------------------------- | | **Lucas-Kanade** | Very Fast | Medium | CPU | Feature tracking, motion analysis | | **MIL** | Fast | Good | CPU | General object tracking | | **GOTURN** | Fast | Good | CPU/GPU | Real-time tracking | | **DaSiamRPN** | Medium | Excellent | CPU/GPU | High accuracy requirements | | **NanoTrack** | Fast | Very Good | CPU/GPU | Mobile/embedded | | **Planar** | Medium | Excellent | CPU | Textured planar objects | ## Key Parameters ### Lucas-Kanade ```python theme={null} lk_params = dict( winSize=(15, 15), # Search window size maxLevel=2, # Pyramid levels criteria=(cv.TERM_CRITERIA_EPS | cv.TERM_CRITERIA_COUNT, 10, 0.03) ) ``` ### Feature Detection ```python theme={null} feature_params = dict( maxCorners=500, # Maximum number of corners qualityLevel=0.3, # Quality threshold (0-1) minDistance=7, # Minimum distance between corners blockSize=7 # Size of averaging block ) ``` **Model Downloads**: Deep learning tracker models are available from: * [GOTURN Models](https://github.com/opencv/opencv_extra/tree/master/testdata/tracking) * [DaSiamRPN](https://github.com/opencv/opencv_zoo) * [NanoTrack](https://github.com/HonglinChu/SiamTrackers/tree/master/NanoTrack) ## Best Practices Select based on your requirements: * **Speed critical**: Lucas-Kanade or NanoTrack * **Accuracy critical**: DaSiamRPN * **CPU-only**: MIL or Lucas-Kanade * **Planar objects**: Planar tracker Implement recovery mechanisms: ```python theme={null} if not ok: # Reinitialize or use detection tracker.init(frame, new_bbox) ``` Use detector periodically to recover from failures: ```python theme={null} if frame_idx % 30 == 0: bbox = detector.detect(frame) tracker.init(frame, bbox) ``` * Resize frames for faster processing * Use GPU backend when available * Reduce detection frequency in optical flow **Tracker Initialization**: All trackers require a good initial bounding box. Poor initialization will lead to immediate tracking failure. ## Next Steps * Explore [Video I/O](/api/videoio) for video handling * Learn about [Feature Detection](/modules/features2d) for custom trackers * Check [Optical Flow](/api/video/optical-flow) for motion estimation * See [DNN Module](/modules/dnn) for deep learning models # Panorama Stitching Source: https://opencv-opencv.mintlify.app/examples/panorama-stitching Create seamless panoramas from multiple images using feature matching, homography estimation, and image blending ## Overview OpenCV's Stitcher API provides a complete pipeline for creating panoramas: * **Feature Detection**: Find distinctive points in images * **Feature Matching**: Establish correspondences between images * **Homography Estimation**: Calculate geometric transformations * **Image Warping**: Transform images to common coordinate system * **Seam Finding**: Minimize visible boundaries * **Blending**: Create smooth transitions between images ## Basic Stitching Simple panorama creation with minimal code. ```python theme={null} import cv2 as cv import sys import numpy as np def stitch_images(image_paths, mode='panorama', output='result.jpg'): """ Stitch multiple images into a panorama Args: image_paths: List of image file paths mode: 'panorama' or 'scans' output: Output filename """ # Read input images imgs = [] for img_path in image_paths: img = cv.imread(cv.samples.findFile(img_path)) if img is None: print(f"Can't read image {img_path}") return False imgs.append(img) print(f"Stitching {len(imgs)} images...") # Create stitcher if mode == 'panorama': stitcher = cv.Stitcher.create(cv.Stitcher_PANORAMA) else: stitcher = cv.Stitcher.create(cv.Stitcher_SCANS) # Perform stitching status, pano = stitcher.stitch(imgs) if status != cv.Stitcher_OK: print(f"Can't stitch images, error code = {status}") print("Error codes:") print(" ERR_NEED_MORE_IMGS = 1") print(" ERR_HOMOGRAPHY_EST_FAIL = 2") print(" ERR_CAMERA_PARAMS_ADJUST_FAIL = 3") return False # Save result cv.imwrite(output, pano) print(f"Stitching completed successfully!") print(f"Result saved to {output}") print(f"Panorama size: {pano.shape[1]} x {pano.shape[0]}") return True # Example usage if __name__ == '__main__': image_files = [ 'images/panorama1.jpg', 'images/panorama2.jpg', 'images/panorama3.jpg' ] stitch_images(image_files, mode='panorama', output='panorama.jpg') ``` ```cpp theme={null} #include "opencv2/imgcodecs.hpp" #include "opencv2/highgui.hpp" #include "opencv2/stitching.hpp" #include using namespace std; using namespace cv; int main(int argc, char* argv[]) { vector imgs; Stitcher::Mode mode = Stitcher::PANORAMA; string result_name = "result.jpg"; // Parse command line arguments for (int i = 1; i < argc; ++i) { if (string(argv[i]) == "--mode") { if (string(argv[i + 1]) == "panorama") mode = Stitcher::PANORAMA; else if (string(argv[i + 1]) == "scans") mode = Stitcher::SCANS; i++; } else if (string(argv[i]) == "--output") { result_name = argv[i + 1]; i++; } else { // Read image Mat img = imread(samples::findFile(argv[i])); if (img.empty()) { cout << "Can't read image '" << argv[i] << "'\n"; return EXIT_FAILURE; } imgs.push_back(img); } } if (imgs.size() < 2) { cout << "Need at least 2 images\n"; return EXIT_FAILURE; } // Create stitcher and perform stitching Mat pano; Ptr stitcher = Stitcher::create(mode); Stitcher::Status status = stitcher->stitch(imgs, pano); if (status != Stitcher::OK) { cout << "Can't stitch images, error code = " << int(status) << endl; return EXIT_FAILURE; } // Save result imwrite(result_name, pano); cout << "Stitching completed successfully\n"; cout << result_name << " saved!" << endl; return EXIT_SUCCESS; } ``` ## Advanced Stitching Configuration Customize the stitching pipeline for better control. ```python theme={null} import cv2 as cv import numpy as np def advanced_stitching(image_paths, output='panorama.jpg'): """ Advanced panorama stitching with custom parameters """ # Read images imgs = [] for path in image_paths: img = cv.imread(path) if img is None: raise ValueError(f"Cannot read {path}") imgs.append(img) # Create stitcher with custom settings stitcher = cv.Stitcher.create(cv.Stitcher_PANORAMA) # Configure feature finder # Options: ORB, AKAZE, SIFT, SURF finder = cv.ORB.create() stitcher.setFeaturesFinder(cv.detail.OrbFeaturesFinder()) # Configure matcher # Best results with large number of features stitcher.setFeaturesMatcher( cv.detail_BestOf2NearestMatcher(False, 0.3) ) # Set bundle adjuster # Options: reproj, ray, affine, no stitcher.setBundleAdjuster( cv.detail_BundleAdjusterRay() ) # Set warper type # Options: spherical, cylindrical, plane, fisheye stitcher.setWarper( cv.PyRotationWarper('spherical', 1.0) ) # Set seam finder # Options: no, voronoi, gc_color, gc_colorgrad, dp_color, dp_colorgrad stitcher.setSeamFinder( cv.detail.SeamFinder_createDefault(cv.detail.SeamFinder_VORONOI_SEAM) ) # Set blender # Options: no, feather, multiband stitcher.setBlender( cv.detail.Blender_createDefault(cv.detail.Blender_MULTI_BAND) ) # Set composition resolution stitcher.setCompositingResol(1.0) # Use -1 for original resolution # Set confidence threshold for feature matching stitcher.setPanoConfidenceThresh(1.0) # Perform stitching print("Stitching...") status, pano = stitcher.stitch(imgs) if status == cv.Stitcher_OK: cv.imwrite(output, pano) print(f"Success! Saved to {output}") return pano else: error_messages = { cv.Stitcher_ERR_NEED_MORE_IMGS: "Need more images", cv.Stitcher_ERR_HOMOGRAPHY_EST_FAIL: "Homography estimation failed", cv.Stitcher_ERR_CAMERA_PARAMS_ADJUST_FAIL: "Camera parameters adjustment failed" } print(f"Error: {error_messages.get(status, 'Unknown error')}") return None # Usage image_files = ['img1.jpg', 'img2.jpg', 'img3.jpg', 'img4.jpg'] panorama = advanced_stitching(image_files) ``` ## Panorama with Rotating Camera Specialized approach for images taken with a rotating camera around its optical center. ```python theme={null} import cv2 as cv import numpy as np def rotating_camera_panorama(image_paths): """ Stitch images from rotating camera using homography Assumes camera rotates around optical center """ # Read images imgs = [cv.imread(path) for path in image_paths] if len(imgs) < 2: raise ValueError("Need at least 2 images") # Initialize with first image panorama = imgs[0] # Feature detector and matcher detector = cv.SIFT_create() matcher = cv.BFMatcher(cv.NORM_L2) for i in range(1, len(imgs)): print(f"Stitching image {i+1}/{len(imgs)}...") # Detect features kp1, des1 = detector.detectAndCompute(panorama, None) kp2, des2 = detector.detectAndCompute(imgs[i], None) # Match features matches = matcher.knnMatch(des1, des2, k=2) # Apply ratio test good_matches = [] for m, n in matches: if m.distance < 0.7 * n.distance: good_matches.append(m) if len(good_matches) < 10: print(f"Not enough matches for image {i}") continue # Extract matched keypoints src_pts = np.float32([kp1[m.queryIdx].pt for m in good_matches]) dst_pts = np.float32([kp2[m.trainIdx].pt for m in good_matches]) # Find homography H, mask = cv.findHomography(dst_pts, src_pts, cv.RANSAC, 5.0) # Warp image h1, w1 = panorama.shape[:2] h2, w2 = imgs[i].shape[:2] # Transform corners to find output size corners = np.float32([[0, 0], [w2, 0], [w2, h2], [0, h2]]).reshape(-1, 1, 2) corners_transformed = cv.perspectiveTransform(corners, H) # Combine with panorama corners all_corners = np.concatenate([ np.float32([[0, 0], [w1, 0], [w1, h1], [0, h1]]).reshape(-1, 1, 2), corners_transformed ]) [x_min, y_min] = np.int32(all_corners.min(axis=0).ravel() - 0.5) [x_max, y_max] = np.int32(all_corners.max(axis=0).ravel() + 0.5) # Translation for positive coordinates translation = np.array([ [1, 0, -x_min], [0, 1, -y_min], [0, 0, 1] ]) # Warp images output_size = (x_max - x_min, y_max - y_min) panorama_warped = cv.warpPerspective( panorama, translation, output_size ) img_warped = cv.warpPerspective( imgs[i], translation.dot(H), output_size ) # Blend images mask1 = (panorama_warped > 0).astype(np.uint8) * 255 mask2 = (img_warped > 0).astype(np.uint8) * 255 overlap = cv.bitwise_and(mask1, mask2) # Simple alpha blending in overlap region panorama = np.where(overlap[..., None] > 0, panorama_warped * 0.5 + img_warped * 0.5, panorama_warped + img_warped).astype(np.uint8) return panorama # Usage images = ['rotate1.jpg', 'rotate2.jpg', 'rotate3.jpg'] pano = rotating_camera_panorama(images) cv.imwrite('rotating_panorama.jpg', pano) ``` ## Stitching Modes ### Panorama Mode Optimized for photo panoramas with rotation around camera center. ```python theme={null} stitcher = cv.Stitcher.create(cv.Stitcher_PANORAMA) ``` **Best for:** * Landscape photography * 360° panoramas * Handheld camera rotation ### Scans Mode Optimized for scanning documents or materials under affine transformation. ```python theme={null} stitcher = cv.Stitcher.create(cv.Stitcher_SCANS) ``` **Best for:** * Document scanning * Flat surface imaging * Parallel camera motion ## Configuration Options | Component | Options | Description | | ------------------- | -------------------------------------- | -------------------------------- | | **Feature Finder** | ORB, AKAZE, SIFT, SURF | Detect distinctive points | | **Matcher** | BestOf2Nearest, Affine | Match features between images | | **Bundle Adjuster** | Reproj, Ray, Affine | Refine camera parameters | | **Warper** | Spherical, Cylindrical, Plane, Fisheye | Project images to common surface | | **Seam Finder** | Voronoi, Graph Cut, DP | Find optimal seam location | | **Blender** | Feather, Multiband | Blend images at seams | ## Troubleshooting Not enough images or too little overlap **Solutions:** * Add more images * Increase overlap between images (aim for 30-50%) * Reduce `setPanoConfidenceThresh()` value Cannot find valid geometric transformation **Solutions:** * Ensure sufficient texture in images * Check image quality and focus * Try different feature detector (SIFT instead of ORB) * Increase number of features detected Bundle adjustment failed **Solutions:** * Check for extreme distortion * Try different bundle adjuster * Reduce number of images * Use SCANS mode for planar scenes Obvious boundaries between images **Solutions:** * Use multiband blending: `Blender_MULTI_BAND` * Try different seam finder: `SeamFinder_DP_COLOR` * Ensure consistent exposure across images * Avoid moving objects in overlap regions ## Best Practices **Image Capture Tips:** * Overlap images by 30-50% * Keep camera level and rotate around optical center * Use consistent exposure settings * Avoid moving objects in scene * Capture in good lighting conditions * Use tripod for best results * Take images in sequence (left to right or top to bottom) **Performance**: Stitching high-resolution images is computationally intensive. Consider: * Reducing image resolution before stitching * Using ORB instead of SIFT for faster processing * Limiting number of features detected * Processing in batches for large panoramas ## Performance Optimization ```python theme={null} def fast_stitching(images, scale=0.5): """ Fast stitching with downscaled images """ # Downscale images small_imgs = [] for img in images: small = cv.resize(img, None, fx=scale, fy=scale) small_imgs.append(small) # Stitch downscaled images stitcher = cv.Stitcher.create(cv.Stitcher_PANORAMA) stitcher.setFeaturesFinder(cv.detail.OrbFeaturesFinder()) status, pano = stitcher.stitch(small_imgs) if status == cv.Stitcher_OK: # Optionally upscale result pano_large = cv.resize(pano, None, fx=1/scale, fy=1/scale) return pano_large return None ``` ## Next Steps * Learn about [Feature Detection](/modules/features2d) for custom pipelines * Explore [Camera Calibration](/examples/pose-estimation) for better results * Check [Homography](/api/calib3d/homography) for geometric transformations * See [Image Blending](/tutorials/blending) for seamless compositing # Camera Pose Estimation Source: https://opencv-opencv.mintlify.app/examples/pose-estimation Estimate camera position and orientation for augmented reality, 3D reconstruction, and robotics applications ## Overview Camera pose estimation determines the position and orientation of a camera relative to a scene or object: * **Camera Calibration**: Determine intrinsic camera parameters * **PnP (Perspective-n-Point)**: Estimate pose from 2D-3D correspondences * **Homography-based**: Estimate pose from planar objects * **Augmented Reality**: Overlay 3D graphics on video * **Visual Odometry**: Track camera motion over time ## Camera Calibration Calibrate camera to obtain intrinsic parameters needed for accurate pose estimation. ```python theme={null} import numpy as np import cv2 as cv import glob def calibrate_camera(images, pattern_size, square_size): """ Calibrate camera using chessboard pattern Args: images: List of calibration image paths pattern_size: Chessboard size (width, height) in inner corners square_size: Size of chessboard square in mm or cm Returns: Camera matrix, distortion coefficients, RMS error """ # Prepare object points objp = np.zeros((pattern_size[0] * pattern_size[1], 3), np.float32) objp[:, :2] = np.indices(pattern_size).T.reshape(-1, 2) objp *= square_size obj_points = [] # 3D points in real world space img_points = [] # 2D points in image plane # Find chessboard corners for fname in images: img = cv.imread(fname) gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Find corners ret, corners = cv.findChessboardCorners(gray, pattern_size, None) if ret: # Refine corner locations criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) corners_refined = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria) obj_points.append(objp) img_points.append(corners_refined) # Draw and display corners cv.drawChessboardCorners(img, pattern_size, corners_refined, ret) cv.imshow('Calibration', img) cv.waitKey(100) cv.destroyAllWindows() if len(obj_points) == 0: raise ValueError("No valid calibration images found") # Calibrate camera h, w = gray.shape[:2] rms, camera_matrix, dist_coefs, rvecs, tvecs = cv.calibrateCamera( obj_points, img_points, (w, h), None, None ) print(f"\nCalibration RMS error: {rms:.3f}") print(f"\nCamera Matrix:\n{camera_matrix}") print(f"\nDistortion Coefficients:\n{dist_coefs.ravel()}") return camera_matrix, dist_coefs, rms # Example usage images = glob.glob('calibration_images/*.jpg') pattern_size = (9, 6) # 9x6 inner corners square_size = 25.0 # 25mm squares K, dist, rms = calibrate_camera(images, pattern_size, square_size) # Save calibration np.savez('camera_calibration.npz', camera_matrix=K, dist_coefs=dist, rms=rms) ``` ## Pose Estimation with solvePnP Estimate camera pose from known 3D-2D point correspondences. ```python theme={null} import numpy as np import cv2 as cv def estimate_pose_pnp(object_points, image_points, camera_matrix, dist_coefs): """ Estimate camera pose using PnP Args: object_points: 3D points in world coordinates (Nx3) image_points: Corresponding 2D points in image (Nx2) camera_matrix: Camera intrinsic matrix dist_coefs: Distortion coefficients Returns: Rotation vector, translation vector, success flag """ # Solve PnP success, rvec, tvec = cv.solvePnP( object_points, image_points, camera_matrix, dist_coefs, flags=cv.SOLVEPNP_ITERATIVE ) if not success: return None, None, False # Convert rotation vector to matrix R, _ = cv.Rodrigues(rvec) print(f"Rotation vector:\n{rvec.ravel()}") print(f"\nTranslation vector:\n{tvec.ravel()}") print(f"\nRotation matrix:\n{R}") return rvec, tvec, True # Example: Define 3D object points (e.g., corners of a square) object_points = np.array([ [0, 0, 0], # Origin [100, 0, 0], # 100mm along X [100, 100, 0], # 100mm along X and Y [0, 100, 0] # 100mm along Y ], dtype=np.float32) # Corresponding 2D image points (detected in image) image_points = np.array([ [320, 240], [420, 240], [420, 340], [320, 340] ], dtype=np.float32) # Load camera calibration calib = np.load('camera_calibration.npz') K = calib['camera_matrix'] dist = calib['dist_coefs'] # Estimate pose rvec, tvec, success = estimate_pose_pnp(object_points, image_points, K, dist) ``` ## Augmented Reality Application Overlay 3D graphics on tracked planar objects. ```python theme={null} import numpy as np import cv2 as cv from plane_tracker import PlaneTracker import video # Define 3D model (cube with pyramid roof) ar_verts = np.float32([ [0, 0, 0], [0, 1, 0], [1, 1, 0], [1, 0, 0], # Base [0, 0, 1], [0, 1, 1], [1, 1, 1], [1, 0, 1], # Top [0, 0.5, 2], [1, 0.5, 2] # Roof ]) ar_edges = [ (0, 1), (1, 2), (2, 3), (3, 0), # Base edges (4, 5), (5, 6), (6, 7), (7, 4), # Top edges (0, 4), (1, 5), (2, 6), (3, 7), # Vertical edges (4, 8), (5, 8), (6, 9), (7, 9), (8, 9) # Roof edges ] class ARApp: def __init__(self, video_src=0): self.cap = video.create_capture(video_src) self.tracker = PlaneTracker() self.frame = None self.paused = False cv.namedWindow('AR Demo') cv.createTrackbar('focal', 'AR Demo', 25, 50, lambda x: None) def draw_3d_overlay(self, img, tracked): """Draw 3D model on tracked plane""" x0, y0, x1, y1 = tracked.target.rect quad_3d = np.float32([ [x0, y0, 0], [x1, y0, 0], [x1, y1, 0], [x0, y1, 0] ]) # Estimate camera intrinsics from focal length fx = 0.5 + cv.getTrackbarPos('focal', 'AR Demo') / 50.0 h, w = img.shape[:2] K = np.float64([ [fx*w, 0, 0.5*(w-1)], [0, fx*w, 0.5*(h-1)], [0.0, 0.0, 1.0] ]) dist_coef = np.zeros(4) # Solve PnP to get camera pose _ret, rvec, tvec = cv.solvePnP( quad_3d, tracked.quad, K, dist_coef ) # Transform and project 3D points verts = ar_verts * [(x1-x0), (y1-y0), -(x1-x0)*0.3] + (x0, y0, 0) verts_2d = cv.projectPoints( verts, rvec, tvec, K, dist_coef )[0].reshape(-1, 2) # Draw 3D model for i, j in ar_edges: pt1 = tuple(map(int, verts_2d[i])) pt2 = tuple(map(int, verts_2d[j])) cv.line(img, pt1, pt2, (255, 255, 0), 2) def run(self): """Main AR loop""" while True: if not self.paused: ret, self.frame = self.cap.read() if not ret: break vis = self.frame.copy() if not self.paused: # Track planar objects tracked = self.tracker.track(self.frame) for tr in tracked: # Draw tracking quad cv.polylines(vis, [np.int32(tr.quad)], True, (255, 255, 255), 2) # Draw 3D overlay self.draw_3d_overlay(vis, tr) cv.imshow('AR Demo', vis) ch = cv.waitKey(1) if ch == ord(' '): self.paused = not self.paused if ch == 27: # ESC break self.cap.release() cv.destroyAllWindows() # Run AR application if __name__ == '__main__': app = ARApp(0) app.run() ``` ## Pose from Homography Extract pose information from planar object homography. ```python theme={null} import numpy as np import cv2 as cv def decompose_homography_to_pose(H, K): """ Decompose homography to rotation and translation Args: H: Homography matrix (3x3) K: Camera intrinsic matrix (3x3) Returns: List of possible (R, t, n) tuples """ # Normalize homography H_norm = np.linalg.inv(K) @ H @ K # Decompose homography num_solutions, Rs, ts, normals = cv.decomposeHomographyMat(H, K) print(f"Found {num_solutions} possible solutions") solutions = [] for i in range(num_solutions): R = Rs[i] t = ts[i] n = normals[i] # Check if rotation matrix is valid if np.linalg.det(R) > 0: solutions.append((R, t, n)) print(f"\nSolution {len(solutions)}:") print(f"Rotation:\n{R}") print(f"Translation:\n{t.ravel()}") print(f"Normal:\n{n.ravel()}") return solutions # Example usage # Assume H is computed from matched points src_pts = np.float32([[0, 0], [100, 0], [100, 100], [0, 100]]) dst_pts = np.float32([[120, 150], [250, 140], [260, 280], [110, 290]]) H, mask = cv.findHomography(src_pts, dst_pts) # Load camera matrix calib = np.load('camera_calibration.npz') K = calib['camera_matrix'] # Decompose solutions = decompose_homography_to_pose(H, K) ``` ## Visual Odometry Track camera motion over time using feature tracking. ```python theme={null} import numpy as np import cv2 as cv class VisualOdometry: def __init__(self, camera_matrix, dist_coefs): self.K = camera_matrix self.dist = dist_coefs self.detector = cv.ORB_create(1000) self.prev_frame = None self.prev_kp = None self.prev_des = None # Camera pose (accumulated) self.R = np.eye(3) self.t = np.zeros((3, 1)) def process_frame(self, frame): """Process new frame and update pose""" gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) # Detect features kp, des = self.detector.detectAndCompute(gray, None) if self.prev_frame is None: self.prev_frame = gray self.prev_kp = kp self.prev_des = des return self.R, self.t # Match features matcher = cv.BFMatcher(cv.NORM_HAMMING) matches = matcher.knnMatch(self.prev_des, des, k=2) # Filter matches good = [] for match in matches: if len(match) == 2: m, n = match if m.distance < 0.7 * n.distance: good.append(m) if len(good) < 10: print("Not enough matches") return self.R, self.t # Extract matched points pts1 = np.float32([self.prev_kp[m.queryIdx].pt for m in good]) pts2 = np.float32([kp[m.trainIdx].pt for m in good]) # Compute essential matrix E, mask = cv.findEssentialMat(pts2, pts1, self.K, method=cv.RANSAC) # Recover pose _, R, t, mask = cv.recoverPose(E, pts2, pts1, self.K, mask=mask) # Update accumulated pose self.t = self.t + self.R @ t self.R = R @ self.R # Update previous frame self.prev_frame = gray self.prev_kp = kp self.prev_des = des return self.R, self.t # Example usage calib = np.load('camera_calibration.npz') vo = VisualOdometry(calib['camera_matrix'], calib['dist_coefs']) cap = cv.VideoCapture('video.mp4') trajectory = [] while True: ret, frame = cap.read() if not ret: break R, t = vo.process_frame(frame) trajectory.append(t.copy()) # Visualize trajectory traj_img = np.zeros((600, 600, 3), dtype=np.uint8) for i in range(1, len(trajectory)): pt1 = (int(trajectory[i-1][0]) + 300, int(trajectory[i-1][2]) + 500) pt2 = (int(trajectory[i][0]) + 300, int(trajectory[i][2]) + 500) cv.line(traj_img, pt1, pt2, (0, 255, 0), 2) cv.imshow('Trajectory', traj_img) if cv.waitKey(1) == 27: break cv.destroyAllWindows() ``` ## PnP Algorithms Comparison | Algorithm | Speed | Accuracy | Min Points | Use Case | | ------------- | ------ | --------- | ---------- | --------------- | | **ITERATIVE** | Medium | Good | 4 | General purpose | | **P3P** | Fast | Good | 3 | Minimal case | | **EPNP** | Fast | Good | 4+ | Many points | | **DLS** | Medium | Very Good | 4+ | High accuracy | | **UPNP** | Fast | Good | 4+ | Fast processing | | **IPPE** | Fast | Good | 4 (planar) | Planar objects | | **SQPNP** | Medium | Excellent | 3+ | Best accuracy | ## Best Practices Always calibrate for accurate pose estimation: ```python theme={null} K, dist = calibrate_camera(images, (9, 6), 25.0) ``` More points = better accuracy: * Minimum: 4 points for general case * Recommended: 10+ points * Use RANSAC for outlier rejection Some configurations have multiple solutions: ```python theme={null} # Check determinant of rotation matrix if np.linalg.det(R) < 0: R = -R # Flip if improper rotation ``` Check reprojection error: ```python theme={null} projected, _ = cv.projectPoints(object_pts, rvec, tvec, K, dist) error = cv.norm(image_pts, projected, cv.NORM_L2) / len(projected) ``` **Coordinate Systems**: OpenCV uses right-handed coordinate system: * X-axis: right * Y-axis: down * Z-axis: forward (into scene) Rotation vectors use Rodrigues representation, convertible to matrices with `cv.Rodrigues()`. **Calibration Quality**: Poor calibration leads to inaccurate pose estimation. Always: * Use at least 10-20 calibration images * Vary target orientation and position * Check RMS error (should be \< 1.0 pixel) * Test on held-out validation images ## Troubleshooting ### Unstable Pose ```python theme={null} # Use more robust PnP algorithm success, rvec, tvec = cv.solvePnP( obj_pts, img_pts, K, dist, flags=cv.SOLVEPNP_SQPNP # Most accurate ) # Or use RANSAC for outliers success, rvec, tvec, inliers = cv.solvePnPRansac( obj_pts, img_pts, K, dist, reprojectionError=8.0 ) ``` ### Incorrect Pose ```python theme={null} # Verify point correspondences for i, (obj_pt, img_pt) in enumerate(zip(obj_pts, img_pts)): projected = cv.projectPoints( obj_pt.reshape(1, 1, 3), rvec, tvec, K, dist )[0].ravel() error = np.linalg.norm(img_pt - projected) print(f"Point {i}: error = {error:.2f} pixels") ``` ## Next Steps * Learn [Camera Calibration](/api/calib3d/calibration) in detail * Explore [3D Reconstruction](/tutorials/3d-reconstruction) techniques * Check [Feature Matching](/modules/features2d) for point correspondences * See [Video Stabilization](/examples/stabilization) for motion estimation # Read and Display Images Source: https://opencv-opencv.mintlify.app/examples/read-display Learn how to load, display, and save images using OpenCV This guide demonstrates the fundamental operations of reading images from disk, displaying them in windows, and saving them to files. ## Overview Image I/O is the foundation of computer vision applications. OpenCV provides simple functions to: * Read images from various formats (JPEG, PNG, BMP, etc.) * Display images in GUI windows * Save processed images to disk * Handle errors when files are not found ## Basic Image Loading and Display Use `imread()` to load an image file. Always check if the image was loaded successfully. Use `imshow()` to display the image in a named window, followed by `waitKey()` to keep the window open. Use `imwrite()` to save the image to a file. ## Complete Example ```python theme={null} import cv2 as cv import sys # Read the image img = cv.imread(cv.samples.findFile("starry_night.jpg")) # Check if image was loaded successfully if img is None: sys.exit("Could not read the image.") # Display the image in a window cv.imshow("Display window", img) k = cv.waitKey(0) # Wait for a keystroke # Save image if 's' key is pressed if k == ord("s"): cv.imwrite("starry_night.png", img) ``` ```cpp theme={null} #include #include #include #include using namespace cv; int main() { // Read the image std::string image_path = samples::findFile("starry_night.jpg"); Mat img = imread(image_path, IMREAD_COLOR); // Check if image was loaded successfully if(img.empty()) { std::cout << "Could not read the image: " << image_path << std::endl; return 1; } // Display the image in a window imshow("Display window", img); int k = waitKey(0); // Wait for a keystroke in the window // Save image if 's' key is pressed if(k == 's') { imwrite("starry_night.png", img); } return 0; } ``` ## Window Operations OpenCV provides several functions to manage display windows: ### Creating and Managing Windows ```python theme={null} import cv2 as cv # Create a named window with specific properties cv.namedWindow("My Window", cv.WINDOW_NORMAL) # Resize the window cv.resizeWindow("My Window", 800, 600) # Move the window to a specific position cv.moveWindow("My Window", 100, 100) # Display an image img = cv.imread("image.jpg") cv.imshow("My Window", img) cv.waitKey(0) # Destroy specific window cv.destroyWindow("My Window") # Or destroy all windows cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; // Create a named window with specific properties namedWindow("My Window", WINDOW_NORMAL); // Resize the window resizeWindow("My Window", 800, 600); // Move the window to a specific position moveWindow("My Window", 100, 100); // Display an image Mat img = imread("image.jpg"); imshow("My Window", img); waitKey(0); // Destroy specific window destroyWindow("My Window"); // Or destroy all windows destroyAllWindows(); ``` ## Image Reading Flags The `imread()` function accepts flags to control how images are loaded: ```python theme={null} import cv2 as cv # Read image in color (default) img_color = cv.imread("image.jpg", cv.IMREAD_COLOR) # Read image in grayscale img_gray = cv.imread("image.jpg", cv.IMREAD_GRAYSCALE) # Read image with alpha channel img_alpha = cv.imread("image.png", cv.IMREAD_UNCHANGED) # Read image and reduce it to 1 channel grayscale img_reduced = cv.imread("image.jpg", cv.IMREAD_REDUCED_GRAYSCALE_2) ``` ```cpp theme={null} #include using namespace cv; // Read image in color (default) Mat img_color = imread("image.jpg", IMREAD_COLOR); // Read image in grayscale Mat img_gray = imread("image.jpg", IMREAD_GRAYSCALE); // Read image with alpha channel Mat img_alpha = imread("image.png", IMREAD_UNCHANGED); // Read image and reduce it to 1 channel grayscale Mat img_reduced = imread("image.jpg", IMREAD_REDUCED_GRAYSCALE_2); ``` Always check if an image was loaded successfully before processing it. An empty/null image will cause your program to crash. ## Key Functions | Function | Description | | --------------------- | --------------------------------------------- | | `imread()` | Loads an image from a file | | `imshow()` | Displays an image in a window | | `imwrite()` | Saves an image to a file | | `waitKey()` | Waits for a key press (0 = wait indefinitely) | | `namedWindow()` | Creates a window with a specific name | | `destroyWindow()` | Closes a specific window | | `destroyAllWindows()` | Closes all OpenCV windows | Use `cv.samples.findFile()` to locate sample images that come with OpenCV. This ensures your code works across different platforms and installations. # Semantic Segmentation with DNN Source: https://opencv-opencv.mintlify.app/examples/semantic-segmentation Perform pixel-wise classification using semantic segmentation models like FCN, ENet, and DeepLab with OpenCV DNN module Semantic segmentation assigns a class label to every pixel in an image, enabling detailed scene understanding. OpenCV's DNN module supports various segmentation architectures trained on datasets like PASCAL VOC, Cityscapes, and COCO. ## Supported Models * **FCN (Fully Convolutional Networks)** - FCN-8s, FCN-ResNet101 * **ENet** - Efficient neural network for real-time segmentation * **DeepLab** - State-of-the-art segmentation with atrous convolution * **U-Net** - Popular architecture for medical image segmentation * **PSPNet** - Pyramid Scene Parsing Network ## Python Implementation ```python theme={null} import cv2 as cv import numpy as np ``` ```python theme={null} # Load segmentation model (ENet example) model = 'Enet-model-best.net' net = cv.dnn.readNet(model) net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # Load class names classes = None with open('enet-classes.txt', 'rt') as f: classes = f.read().rstrip('\n').split('\n') ``` ```python theme={null} # Generate random colors for each class np.random.seed(324) colors = None # Option 1: Generate colors automatically def generate_colors(num_classes): colors = [np.array([0, 0, 0], np.uint8)] for i in range(1, num_classes): colors.append((colors[i - 1] + np.random.randint(0, 256, [3], np.uint8)) / 2) return colors # Option 2: Load predefined colors from file colors_file = 'colors.txt' with open(colors_file, 'rt') as f: colors = [np.array(color.split(' '), np.uint8) for color in f.read().rstrip('\n').split('\n')] ``` ```python theme={null} # Read input image frame = cv.imread('image.jpg') frameHeight, frameWidth = frame.shape[:2] # Create blob from image # ENet uses 512x256 input with scale 1/255 blob = cv.dnn.blobFromImage(frame, 1.0/255.0, (512, 256), [0, 0, 0], True, crop=False) ``` Different segmentation models require different input sizes: * ENet: 512x256 * FCN-8s: 500x500 * FCN-ResNet101: 500x500 ```python theme={null} # Set input blob net.setInput(blob) # Forward pass to get score map score = net.forward() # score shape: [1, num_classes, height, width] numClasses = score.shape[1] height = score.shape[2] width = score.shape[3] ``` ```python theme={null} # Generate colors if not loaded if not colors: colors = [np.array([0, 0, 0], np.uint8)] for i in range(1, numClasses): colors.append((colors[i - 1] + np.random.randint(0, 256, [3], np.uint8)) / 2) # Get class ID for each pixel classIds = np.argmax(score[0], axis=0) # Create colored segmentation mask segm = np.stack([colors[idx] for idx in classIds.flatten()]) segm = segm.reshape(height, width, 3) # Resize to original frame size segm = cv.resize(segm, (frameWidth, frameHeight), interpolation=cv.INTER_NEAREST) ``` ```python theme={null} # Blend segmentation with original image frame = (0.1 * frame + 0.9 * segm).astype(np.uint8) # Add inference time t, _ = net.getPerfProfile() label = f'Inference time: {t * 1000.0 / cv.getTickFrequency():.2f} ms' cv.putText(frame, label, (0, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0)) # Display result cv.imshow('Semantic Segmentation', frame) cv.waitKey(0) ``` ## C++ Implementation ```cpp theme={null} #include #include #include #include #include using namespace cv; using namespace dnn; std::vector classes; std::vector colors; void colorizeSegmentation(const Mat &score, Mat &segm) { const int rows = score.size[2]; const int cols = score.size[3]; const int chns = score.size[1]; if (colors.empty()) { // Generate colors colors.push_back(Vec3b()); for (int i = 1; i < chns; ++i) { Vec3b color; for (int j = 0; j < 3; ++j) color[j] = (colors[i - 1][j] + rand() % 256) / 2; colors.push_back(color); } } // Find class with maximum score for each pixel Mat maxCl = Mat::zeros(rows, cols, CV_8UC1); Mat maxVal(rows, cols, CV_32FC1, score.data); for (int ch = 1; ch < chns; ch++) { for (int row = 0; row < rows; row++) { const float *ptrScore = score.ptr(0, ch, row); uint8_t *ptrMaxCl = maxCl.ptr(row); float *ptrMaxVal = maxVal.ptr(row); for (int col = 0; col < cols; col++) { if (ptrScore[col] > ptrMaxVal[col]) { ptrMaxVal[col] = ptrScore[col]; ptrMaxCl[col] = (uchar)ch; } } } } // Create colored segmentation mask segm.create(rows, cols, CV_8UC3); for (int row = 0; row < rows; row++) { const uchar *ptrMaxCl = maxCl.ptr(row); Vec3b *ptrSegm = segm.ptr(row); for (int col = 0; col < cols; col++) { ptrSegm[col] = colors[ptrMaxCl[col]]; } } } int main(int argc, char** argv) { // Load model String model = "Enet-model-best.net"; Net net = readNet(model); net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // Read input image Mat frame = imread("image.jpg"); // Create blob Mat blob; Scalar mean(0, 0, 0); blobFromImage(frame, blob, 1.0/255.0, Size(512, 256), mean, true, false); // Set input and forward net.setInput(blob); Mat score = net.forward(); // Colorize segmentation Mat segm; colorizeSegmentation(score, segm); // Resize to original size resize(segm, segm, frame.size(), 0, 0, INTER_NEAREST); // Blend with original image addWeighted(frame, 0.1, segm, 0.9, 0.0, frame); // Display imshow("Semantic Segmentation", frame); waitKey(0); return 0; } ``` ```cpp theme={null} // Open video capture VideoCapture cap; cap.open(0); // or video file Mat frame, blob; while (waitKey(1) < 0) { cap >> frame; if (frame.empty()) break; // Create blob blobFromImage(frame, blob, scale, Size(inpWidth, inpHeight), mean, swapRB, false); // Forward pass net.setInput(blob); Mat score = net.forward(); // Colorize Mat segm; colorizeSegmentation(score, segm); // Resize and blend resize(segm, segm, frame.size(), 0, 0, INTER_NEAREST); addWeighted(frame, 0.1, segm, 0.9, 0.0, frame); // Display performance std::vector layersTimes; double freq = getTickFrequency() / 1000; double t = net.getPerfProfile(layersTimes) / freq; std::string label = format("Inference time: %.2f ms", t); putText(frame, label, Point(0, 15), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0)); imshow("Semantic Segmentation", frame); } ``` ## Creating a Legend Display a legend showing class names and colors: ```python theme={null} def showLegend(classes, colors): if classes is None or len(classes) == 0: return blockHeight = 30 legend = np.zeros((blockHeight * len(colors), 200, 3), np.uint8) for i in range(len(classes)): block = legend[i * blockHeight:(i + 1) * blockHeight] block[:, :] = colors[i] cv.putText(block, classes[i], (0, blockHeight // 2), cv.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255)) cv.namedWindow('Legend', cv.WINDOW_NORMAL) cv.imshow('Legend', legend) # Call in main loop showLegend(classes, colors) ``` C++ version: ```cpp theme={null} void showLegend() { static const int kBlockHeight = 30; static Mat legend; if (legend.empty()) { const int numClasses = (int)classes.size(); legend.create(kBlockHeight * numClasses, 200, CV_8UC3); for (int i = 0; i < numClasses; i++) { Mat block = legend.rowRange(i * kBlockHeight, (i + 1) * kBlockHeight); block.setTo(colors[i]); putText(block, classes[i], Point(0, kBlockHeight / 2), FONT_HERSHEY_SIMPLEX, 0.5, Vec3b(255, 255, 255)); } namedWindow("Legend", WINDOW_NORMAL); imshow("Legend", legend); } } ``` ## Model Configurations ### ENet (Torch) ```yaml theme={null} enet: model: "Enet-model-best.net" mean: [0, 0, 0] scale: 0.00392 # 1/255 width: 512 height: 256 rgb: true classes: "enet-classes.txt" ``` **Download:** [https://github.com/e-lab/ENet-training](https://github.com/e-lab/ENet-training) **Classes:** 20 road scene classes (Cityscapes-style) ### FCN-8s (Caffe) ```yaml theme={null} fcn8s: model: "fcn8s-heavy-pascal.caffemodel" config: "fcn8s-heavy-pascal.prototxt" mean: [0, 0, 0] scale: 1.0 width: 500 height: 500 rgb: false ``` **Download:** [http://dl.caffe.berkeleyvision.org/fcn8s-heavy-pascal.caffemodel](http://dl.caffe.berkeleyvision.org/fcn8s-heavy-pascal.caffemodel) **Classes:** 21 PASCAL VOC classes ### FCN-ResNet101 (ONNX) ```yaml theme={null} fcnresnet101: model: "fcn-resnet101-11.onnx" mean: [103.5, 116.2, 123.6] scale: 0.019 width: 500 height: 500 rgb: false ``` **Download:** [https://github.com/onnx/models](https://github.com/onnx/models) (ONNX Model Zoo) ## Common Segmentation Classes Road scene segmentation with 20 classes: * road * sidewalk * building * wall * fence * pole * traffic light * traffic sign * vegetation * terrain * sky * person * rider * car * truck * bus * train * motorcycle * bicycle General object segmentation with 21 classes: * background * aeroplane * bicycle * bird * boat * bottle * bus * car * cat * chair * cow * dining table * dog * horse * motorbike * person * potted plant * sheep * sofa * train * tv/monitor ## Performance Optimization ```python theme={null} net.setPreferableBackend(cv.dnn.DNN_BACKEND_CUDA) net.setPreferableTarget(cv.dnn.DNN_TARGET_CUDA) ``` Smaller input sizes process faster but may lose detail: ```python theme={null} # Original: 512x256 blob = cv.dnn.blobFromImage(frame, 1.0/255.0, (512, 256), [0, 0, 0], True) # Faster: 256x128 blob = cv.dnn.blobFromImage(frame, 1.0/255.0, (256, 128), [0, 0, 0], True) ``` Choose models based on speed/accuracy tradeoff: * **ENet**: Real-time, good for road scenes * **FCN-8s**: Moderate speed, high accuracy * **DeepLab**: Best accuracy, slower ## Blending Segmentation with Original Image Adjust the blend ratio for different visualization effects: ```python theme={null} # Heavy segmentation overlay (90% segmentation, 10% original) frame = (0.1 * frame + 0.9 * segm).astype(np.uint8) # Balanced overlay (50% each) frame = (0.5 * frame + 0.5 * segm).astype(np.uint8) # Light segmentation overlay (30% segmentation, 70% original) frame = (0.7 * frame + 0.3 * segm).astype(np.uint8) # Using OpenCV addWeighted (C++/Python) output = cv.addWeighted(frame, 0.3, segm, 0.7, 0.0) ``` ## Complete Example with Video ```python theme={null} import cv2 as cv import numpy as np def main(): # Load model model = 'Enet-model-best.net' net = cv.dnn.readNet(model) net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # Load classes with open('enet-classes.txt', 'rt') as f: classes = f.read().rstrip('\n').split('\n') # Generate colors np.random.seed(324) colors = [np.array([0, 0, 0], np.uint8)] for i in range(1, len(classes)): colors.append((colors[i - 1] + np.random.randint(0, 256, [3], np.uint8)) / 2) # Open video cap = cv.VideoCapture(0) # or video file cv.namedWindow('Segmentation', cv.WINDOW_NORMAL) while cv.waitKey(1) < 0: hasFrame, frame = cap.read() if not hasFrame: break frameHeight, frameWidth = frame.shape[:2] # Create blob blob = cv.dnn.blobFromImage(frame, 1.0/255.0, (512, 256), [0, 0, 0], True, crop=False) # Run segmentation net.setInput(blob) score = net.forward() numClasses = score.shape[1] height = score.shape[2] width = score.shape[3] # Get class for each pixel classIds = np.argmax(score[0], axis=0) # Create colored mask segm = np.stack([colors[idx] for idx in classIds.flatten()]) segm = segm.reshape(height, width, 3) segm = cv.resize(segm, (frameWidth, frameHeight), interpolation=cv.INTER_NEAREST) # Blend frame = (0.1 * frame + 0.9 * segm).astype(np.uint8) # Add timing info t, _ = net.getPerfProfile() label = f'Inference time: {t * 1000.0 / cv.getTickFrequency():.2f} ms' cv.putText(frame, label, (0, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0)) cv.imshow('Segmentation', frame) if __name__ == '__main__': main() ``` Semantic segmentation is computationally expensive. For real-time applications on CPU, use lightweight models like ENet or reduce input resolution. ## Source Code Complete source code for semantic segmentation: * Python: `samples/dnn/segmentation.py` * C++: `samples/dnn/segmentation.cpp` # Installing OpenCV Source: https://opencv-opencv.mintlify.app/installation Complete installation guide for OpenCV on Linux, Windows, and macOS with support for C++, Python, and Java ## Installation Overview OpenCV can be installed in two ways: using prebuilt binaries or building from source. This guide covers both methods across different platforms. For most users, especially Python developers, using prebuilt packages (pip, conda, or system package managers) is the quickest way to get started. ## Quick Install For rapid setup, use these package managers: ```bash theme={null} # Using pip pip install opencv-python # With contrib modules (extra features) pip install opencv-contrib-python # Using conda conda install -c conda-forge opencv ``` The `opencv-python` package includes prebuilt binaries for most platforms and is the easiest way to get started with Python. ```bash theme={null} # Ubuntu/Debian sudo apt-get update sudo apt-get install libopencv-dev python3-opencv # Fedora sudo dnf install opencv opencv-devel python3-opencv # Arch Linux sudo pacman -S opencv ``` ```bash theme={null} # Using Homebrew brew install opencv # Using pip pip install opencv-python ``` ```powershell theme={null} # Using pip pip install opencv-python # Or download prebuilt binaries from: # https://github.com/opencv/opencv/releases ``` ## Building from Source Building OpenCV from source gives you full control over features, optimizations, and dependencies. ### Prerequisites Install a C++ compiler and build tools for your platform: **Linux:** ```bash theme={null} # GCC/G++ sudo apt-get install build-essential # Or Clang sudo apt-get install clang ``` **Windows:** * Visual Studio 2015 or later (Community Edition is free) * Or MinGW-w64 compiler **macOS:** ```bash theme={null} xcode-select --install ``` CMake 3.9 or higher is required: ```bash theme={null} # Linux (Ubuntu/Debian) sudo apt-get install cmake # macOS brew install cmake # Windows: Download from https://cmake.org/download/ ``` Verify installation: ```bash theme={null} cmake --version ``` Required to download OpenCV source: ```bash theme={null} # Linux (Ubuntu/Debian) sudo apt-get install git # macOS (if not already installed) brew install git # Windows: Download from https://git-scm.com/ ``` ### Download OpenCV Source ```bash Git Clone theme={null} # Clone main repository git clone https://github.com/opencv/opencv.git cd opencv git checkout 4.x # or specific version like 4.8.0 # Optional: Clone contrib modules cd .. git clone https://github.com/opencv/opencv_contrib.git cd opencv_contrib git checkout 4.x # must match opencv version ``` ```bash Download Archive theme={null} # Download from GitHub releases wget https://github.com/opencv/opencv/archive/4.x.zip unzip 4.x.zip # Optional: Download contrib modules wget https://github.com/opencv/opencv_contrib/archive/4.x.zip unzip 4.x.zip ``` When using both `opencv` and `opencv_contrib`, ensure both repositories are at the same version/tag to avoid compatibility issues. ## Platform-Specific Build Instructions ### Building on Linux ```bash theme={null} # Required dependencies sudo apt-get install build-essential cmake git pkg-config \ libgtk-3-dev libavcodec-dev libavformat-dev libswscale-dev \ libv4l-dev libxvidcore-dev libx264-dev libjpeg-dev libpng-dev \ libtiff-dev gfortran openexr libatlas-base-dev python3-dev \ python3-numpy libtbb2 libtbb-dev libdc1394-22-dev # Optional: for Python bindings pip install numpy ``` ```bash theme={null} cd opencv mkdir build && cd build # Basic configuration cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local \ -DBUILD_EXAMPLES=ON \ .. # With contrib modules cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local \ -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \ -DBUILD_EXAMPLES=ON \ .. ``` Use `cmake -DCMAKE_BUILD_TYPE=Release` for optimized builds. Use `Debug` for development. ```bash theme={null} # Build using all CPU cores make -j$(nproc) # This may take 15-60 minutes depending on your system ``` ```bash theme={null} # Install to system directories (requires sudo) sudo make install sudo ldconfig ``` Installation locations: * Binaries: `/usr/local/bin` * Libraries: `/usr/local/lib` * Headers: `/usr/local/include/opencv4` * CMake config: `/usr/local/lib/cmake/opencv4` ### Verify Installation ```bash theme={null} # Check OpenCV version pkg-config --modversion opencv4 # Test Python bindings python3 -c "import cv2; print(cv2.__version__)" ``` ### Building on Windows Download and install Visual Studio 2015 or later: * Community Edition (free): [https://visualstudio.microsoft.com/](https://visualstudio.microsoft.com/) * Select "Desktop development with C++" workload * CMake: [https://cmake.org/download/](https://cmake.org/download/) * Git: [https://git-scm.com/download/win](https://git-scm.com/download/win) During CMake installation, select "Add CMake to system PATH for all users" Open Git Bash or Command Prompt: ```bash theme={null} cd C:\\lib git clone https://github.com/opencv/opencv.git git clone https://github.com/opencv/opencv_contrib.git ``` 1. Open CMake GUI 2. Set source directory: `C:/lib/opencv` 3. Set build directory: `C:/lib/opencv/build` 4. Click "Configure" and select your Visual Studio version 5. Set configuration options: * `OPENCV_EXTRA_MODULES_PATH`: `C:/lib/opencv_contrib/modules` * `BUILD_EXAMPLES`: ON * `BUILD_opencv_python3`: ON (if you want Python support) 6. Click "Generate" ```cmd theme={null} cd C:\\lib\\opencv\\build REM Build Release version cmake --build . --config Release REM Build Debug version cmake --build . --config Debug REM Install (optional) cmake --build . --target install --config Release ``` Add OpenCV to system PATH: ```cmd theme={null} setx OpenCV_DIR C:\\lib\\opencv\\build\\x64\\vc16 setx PATH "%PATH%;%OpenCV_DIR%\\bin" ``` Replace `vc16` with your Visual Studio version: * VS 2015: vc14 * VS 2017: vc15 * VS 2019: vc16 * VS 2022: vc17 ### Building on macOS ```bash theme={null} xcode-select --install ``` ```bash theme={null} # Using Homebrew (recommended) brew install cmake # Or download from https://cmake.org/download/ ``` ```bash theme={null} cd ~/workspace git clone https://github.com/opencv/opencv.git git clone https://github.com/opencv/opencv_contrib.git ``` ```bash theme={null} cd opencv mkdir build && cd build # Configure cmake -DCMAKE_BUILD_TYPE=Release \ -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \ -DBUILD_EXAMPLES=ON \ -DPYTHON3_EXECUTABLE=$(which python3) \ .. # Build using all CPU cores make -j$(sysctl -n hw.ncpu) ``` ```bash theme={null} # Install system-wide sudo make install # Update library cache sudo update_dyld_shared_cache ``` ### Verify Installation ```bash theme={null} # Check version python3 -c "import cv2; print(cv2.__version__)" # Test in C++ pkg-config --cflags --libs opencv4 ``` ## Configuration Options Customize your OpenCV build with these CMake options: ### Common Options ```bash theme={null} # Performance optimizations -DENABLE_FAST_MATH=ON -DWITH_TBB=ON # Threading Building Blocks -DWITH_OPENMP=ON # OpenMP support -DWITH_IPP=ON # Intel Performance Primitives # Hardware acceleration -DWITH_CUDA=ON # NVIDIA CUDA support -DWITH_OPENCL=ON # OpenCL support # Module options -DBUILD_EXAMPLES=ON # Build example applications -DBUILD_TESTS=OFF # Skip tests (faster build) -DBUILD_PERF_TESTS=OFF # Skip performance tests -DBUILD_DOCS=ON # Build documentation # Language bindings -DBUILD_opencv_python3=ON # Python 3 bindings -DBUILD_opencv_java=ON # Java bindings # Extra modules -DOPENCV_EXTRA_MODULES_PATH=/path/to/opencv_contrib/modules ``` For a list of all available options, run `cmake -L` in your build directory after initial configuration. ## Language-Specific Setup ### Python Verify Python bindings: ```python theme={null} import cv2 print(f"OpenCV version: {cv2.__version__}") print(f"Build info:\n{cv2.getBuildInformation()}") ``` ### C++ Create a CMakeLists.txt for your project: ```cmake theme={null} cmake_minimum_required(VERSION 3.9) project(MyOpenCVApp) find_package(OpenCV REQUIRED) include_directories(${OpenCV_INCLUDE_DIRS}) add_executable(myapp main.cpp) target_link_libraries(myapp ${OpenCV_LIBS}) ``` ### Java Add OpenCV to your Java project: ```java theme={null} // Load native library System.loadLibrary(Core.NATIVE_LIBRARY_NAME); // Verify installation System.out.println("OpenCV version: " + Core.VERSION); ``` ## Troubleshooting Install missing packages or specify paths manually: ```bash theme={null} cmake -DPYTHON3_EXECUTABLE=/usr/bin/python3 \ -DPYTHON3_INCLUDE_DIR=/usr/include/python3.8 \ -DPYTHON3_NUMPY_INCLUDE_DIRS=/usr/local/lib/python3.8/site-packages/numpy/core/include \ .. ``` Reduce parallel jobs: ```bash theme={null} make -j2 # Use only 2 cores instead of all ``` Check installation path: ```python theme={null} import sys print(sys.path) ``` Add OpenCV to Python path: ```bash theme={null} export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3.8/site-packages ``` Ensure OpenCV bin directory is in PATH: ```cmd theme={null} echo %PATH% ``` Copy DLLs to your application directory as a workaround. If you encounter errors, check the [OpenCV forum](https://forum.opencv.org) or [GitHub issues](https://github.com/opencv/opencv/issues) for solutions. ## Next Steps Now that OpenCV is installed, you're ready to write your first computer vision application! Build your first OpenCV application with step-by-step examples # Introduction to OpenCV Source: https://opencv-opencv.mintlify.app/introduction Learn about OpenCV, the world's leading open-source computer vision library, its capabilities, and real-world applications ## What is OpenCV? OpenCV (Open Source Computer Vision Library) is an open-source computer vision and machine learning software library. It provides a comprehensive set of tools and algorithms for real-time computer vision applications, image processing, and AI development. OpenCV is released under the Apache 2 License and is free for both academic and commercial use. ## Key Features OpenCV offers extensive functionality across multiple domains: * **Matrix Operations**: Efficient data structures and operations for image manipulation * **Image Processing**: Filtering, morphological operations, color space conversions * **Feature Detection**: SIFT, SURF, ORB, and other feature extractors * **Object Detection**: Haar cascades, HOG descriptors, DNN-based detection * **Machine Learning**: Support for various ML algorithms including SVM, decision trees, neural networks * **Deep Learning**: Integration with TensorFlow, PyTorch, and other frameworks * **Video Analysis**: Optical flow, background subtraction, object tracking * **Camera Calibration**: 3D reconstruction, stereo vision, pose estimation * **Hardware Acceleration**: CUDA, OpenCL, Intel IPP support * **Parallel Processing**: TBB (Threading Building Blocks) integration * **Optimized Algorithms**: Hand-tuned implementations for maximum performance * **Cross-Platform**: Runs on Windows, Linux, macOS, iOS, and Android ## Use Cases OpenCV powers computer vision applications across diverse industries: ### Robotics and Automation * Vision-guided robots for manufacturing and warehouses * Autonomous navigation and obstacle detection * Quality control and defect inspection ### Medical Imaging * Medical image analysis and diagnosis * Surgical assistance and planning * Pathology image processing ### Security and Surveillance * Face recognition and detection * License plate recognition * Intrusion detection and monitoring ### Augmented Reality * Marker-based and markerless AR * Real-time object tracking * 3D pose estimation ### Automotive * Advanced driver assistance systems (ADAS) * Lane detection and traffic sign recognition * Pedestrian detection ## Library Architecture OpenCV is organized into multiple modules, each focused on specific functionality: ```cpp theme={null} // Core modules included in most applications #include // Basic data structures and operations #include // Image file reading and writing #include // Image processing functions #include // GUI and display functions ``` ### Main Modules * **core**: Basic data structures (Mat, Vec, etc.) and fundamental operations * **imgproc**: Image processing (filtering, geometric transformations, color space conversions) * **imgcodecs**: Image file I/O (JPEG, PNG, TIFF, etc.) * **videoio**: Video capture and encoding * **highgui**: UI creation and display functions * **video**: Video analysis (optical flow, background subtraction, tracking) * **calib3d**: Camera calibration and 3D reconstruction * **features2d**: Feature detection and description * **objdetect**: Object detection (face, pedestrian, etc.) * **dnn**: Deep neural networks module * **ml**: Machine learning algorithms You can extend OpenCV's functionality by building with the **opencv\_contrib** repository, which contains experimental and non-free algorithms. ## Language Bindings OpenCV supports multiple programming languages: ```python Python theme={null} import cv2 as cv import numpy as np # Load and display an image img = cv.imread('image.jpg') cv.imshow('Image', img) cv.waitKey(0) ``` ```cpp C++ theme={null} #include using namespace cv; int main() { // Load and display an image Mat img = imread("image.jpg"); imshow("Image", img); waitKey(0); return 0; } ``` ```java Java theme={null} import org.opencv.core.Core; import org.opencv.core.Mat; import org.opencv.imgcodecs.Imgcodecs; import org.opencv.highgui.HighGui; public class Main { public static void main(String[] args) { System.loadLibrary(Core.NATIVE_LIBRARY_NAME); Mat img = Imgcodecs.imread("image.jpg"); HighGui.imshow("Image", img); HighGui.waitKey(); } } ``` ## History and Community OpenCV was initially developed by Intel in 1999 and has grown into one of the most widely-used computer vision libraries: * **1999**: Initial release by Intel Research * **2006**: First stable release (OpenCV 1.0) * **2009**: OpenCV 2.0 with C++ API * **2012**: Non-profit OpenCV Foundation established * **2015**: OpenCV 3.0 with refactored architecture * **2018**: OpenCV 4.0 with C++11 baseline * **Present**: Active development with regular releases ### Community Resources Comprehensive API reference and tutorials Q\&A forum for community support Source code and issue tracking Official training and certification OpenCV is actively maintained with regular updates. Always check the [official website](https://opencv.org) for the latest releases and security updates. ## Next Steps Ready to get started with OpenCV? Continue to the installation guide to set up OpenCV on your system. Learn how to install OpenCV on Linux, Windows, or macOS # Camera Calibration and 3D Reconstruction Source: https://opencv-opencv.mintlify.app/modules/calib3d Camera calibration, stereo vision, 3D reconstruction, and pose estimation algorithms The Camera Calibration and 3D Reconstruction (calib3d) module provides algorithms for camera calibration, stereo vision, 3D reconstruction, and geometric transformations for computer vision applications. ## Overview From opencv2/calib3d.hpp:54-277, detailed mathematical background: > The functions in this section use a pinhole camera model with lens distortion for camera calibration, stereo calibration and rectification, 3D reconstruction from stereo, and pose estimation. Determine intrinsic and extrinsic camera parameters Calibrate stereo camera pairs and rectify images Reconstruct 3D points from multiple views Estimate object position and orientation ## Pinhole Camera Model From calib3d.hpp:64-72, the fundamental projection equation: \[ s \begin u \ v \ 1 \end = \mathbf \begin \mathbf | \mathbf \end \begin X\_w \ Y\_w \ Z\_w \ 1 \end ] Where: * **A** - Camera intrinsic matrix (focal length, principal point) * **R** - Rotation matrix (3x3) * **t** - Translation vector (3x1) * **(X\_w, Y\_w, Z\_w)** - 3D world coordinates * **(u, v)** - 2D image coordinates ### Camera Intrinsic Matrix From calib3d.hpp:79-83: \[ \mathbf = \begin f\_x & 0 & c\_x \ 0 & f\_y & c\_y \ 0 & 0 & 1 \end ] * **f\_x, f\_y** - Focal lengths in pixel units * **c\_x, c\_y** - Principal point (optical center) ## Lens Distortion From calib3d.hpp:225-264, real lenses have distortion: ### Distortion Coefficients ```cpp theme={null} // Distortion vector (OpenCV format) vector distCoeffs = { k1, k2, p1, p2, k3, // Standard 5 parameters k4, k5, k6, // Optional: radial distortion s1, s2, s3, s4 // Optional: thin prism }; ``` * **k1, k2, k3, k4, k5, k6** - Radial distortion coefficients * **p1, p2** - Tangential distortion coefficients * **s1, s2, s3, s4** - Thin prism distortion coefficients ### Distortion Model From calib3d.hpp:228-244: $x'' = x' \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + 2p_1 x'y' + p_2(r^2 + 2x'^2)$ $y'' = y' \frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + p_1(r^2 + 2y'^2) + 2p_2 x'y'$ Where ( r^2 = x'^2 + y'^2 ) ## Camera Calibration Example from samples/cpp/calibration.cpp: ### Chessboard Calibration ```cpp theme={null} #include #include #include using namespace cv; using namespace std; int main() { // Calibration pattern size (inner corners) Size boardSize(9, 6); // 9x6 chessboard float squareSize = 25.0; // mm // Collect calibration images vector> imagePoints; vector> objectPoints; VideoCapture cap(0); Mat frame, gray; cout << "Press SPACE to capture, ESC to finish\n"; while (cap.read(frame)) { cvtColor(frame, gray, COLOR_BGR2GRAY); // Find chessboard corners vector corners; bool found = findChessboardCorners( gray, boardSize, corners, CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE | CALIB_CB_FAST_CHECK ); if (found) { // Refine corner positions cornerSubPix(gray, corners, Size(11, 11), Size(-1, -1), TermCriteria(TermCriteria::EPS + TermCriteria::MAX_ITER, 30, 0.1)); // Draw corners drawChessboardCorners(frame, boardSize, corners, found); } imshow("Calibration", frame); int key = waitKey(30); if (key == ' ' && found) { // Capture image imagePoints.push_back(corners); // Generate 3D object points vector obj; for (int i = 0; i < boardSize.height; i++) { for (int j = 0; j < boardSize.width; j++) { obj.push_back(Point3f( j * squareSize, i * squareSize, 0 )); } } objectPoints.push_back(obj); cout << "Captured " << imagePoints.size() << " images\n"; } else if (key == 27) break; // ESC } // Calibrate camera if (imagePoints.size() >= 10) { Mat cameraMatrix, distCoeffs; vector rvecs, tvecs; double rms = calibrateCamera( objectPoints, imagePoints, gray.size(), cameraMatrix, distCoeffs, rvecs, tvecs ); cout << "\nCalibration complete!\n"; cout << "RMS error: " << rms << "\n"; cout << "Camera matrix:\n" << cameraMatrix << "\n"; cout << "Distortion coefficients:\n" << distCoeffs << "\n"; // Save calibration FileStorage fs("calibration.yml", FileStorage::WRITE); fs << "camera_matrix" << cameraMatrix; fs << "distortion_coefficients" << distCoeffs; fs << "image_width" << gray.cols; fs << "image_height" << gray.rows; fs << "rms_error" << rms; fs.release(); cout << "Saved to calibration.yml\n"; } else { cout << "Not enough images for calibration\n"; } return 0; } ``` ### Calibration Flags ```cpp theme={null} // Calibration flags int flags = 0; flags |= CALIB_FIX_ASPECT_RATIO; // Fix fx/fy ratio flags |= CALIB_ZERO_TANGENT_DIST; // Assume p1=p2=0 flags |= CALIB_FIX_PRINCIPAL_POINT; // Fix cx, cy at center flags |= CALIB_FIX_K1; // Fix k1=0 flags |= CALIB_FIX_K2; // Fix k2=0 flags |= CALIB_FIX_K3; // Fix k3=0 flags |= CALIB_RATIONAL_MODEL; // Enable k4, k5, k6 calibrateCamera(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, rvecs, tvecs, flags); ``` ## Undistortion ### Undistort Images ```cpp theme={null} // Load calibration FileStorage fs("calibration.yml", FileStorage::READ); Mat cameraMatrix, distCoeffs; fs["camera_matrix"] >> cameraMatrix; fs["distortion_coefficients"] >> distCoeffs; // Undistort image Mat img = imread("distorted.jpg"); Mat undistorted; undistort(img, undistorted, cameraMatrix, distCoeffs); // Or get optimal camera matrix Mat newCameraMatrix = getOptimalNewCameraMatrix( cameraMatrix, distCoeffs, img.size(), 1.0); undistort(img, undistorted, cameraMatrix, distCoeffs, newCameraMatrix); ``` ### Remap for Efficiency ```cpp theme={null} // Compute undistortion maps once Mat map1, map2; initUndistortRectifyMap( cameraMatrix, distCoeffs, Mat(), // No rectification newCameraMatrix, imageSize, CV_16SC2, map1, map2 ); // Apply to multiple images efficiently for (const auto& img : images) { Mat undistorted; remap(img, undistorted, map1, map2, INTER_LINEAR); } ``` ## Stereo Calibration ### Calibrate Stereo Pair ```cpp theme={null} // Collect image points from both cameras vector> leftPoints, rightPoints; vector> objectPoints; // ... collect points from stereo pairs ... // Individual camera matrices and distortion Mat K1, K2, D1, D2; Mat R, T, E, F; // Stereo parameters // Stereo calibration double rms = stereoCalibrate( objectPoints, leftPoints, rightPoints, K1, D1, // Left camera K2, D2, // Right camera imageSize, R, // Rotation between cameras T, // Translation between cameras E, // Essential matrix F, // Fundamental matrix CALIB_FIX_INTRINSIC // Use pre-calibrated cameras ); cout << "Stereo calibration RMS: " << rms << endl; cout << "Baseline: " << norm(T) << " mm\n"; ``` ### Stereo Rectification ```cpp theme={null} // Compute rectification transforms Mat R1, R2, P1, P2, Q; Rect validRoi[2]; stereoRectify( K1, D1, // Left camera K2, D2, // Right camera imageSize, R, T, // Stereo parameters R1, R2, // Output: rectification rotations P1, P2, // Output: projection matrices Q, // Output: disparity-to-depth mapping CALIB_ZERO_DISPARITY, 1.0, // Alpha (0=crop, 1=all pixels) imageSize, &validRoi[0], &validRoi[1] ); // Create rectification maps Mat map1L, map2L, map1R, map2R; initUndistortRectifyMap(K1, D1, R1, P1, imageSize, CV_16SC2, map1L, map2L); initUndistortRectifyMap(K2, D2, R2, P2, imageSize, CV_16SC2, map1R, map2R); // Rectify stereo pair Mat leftImg, rightImg; Mat rectLeft, rectRight; remap(leftImg, rectLeft, map1L, map2L, INTER_LINEAR); remap(rightImg, rectRight, map1R, map2R, INTER_LINEAR); // Now rectLeft and rectRight have aligned epipolar lines ``` ## Disparity and Depth ### Stereo Matching ```cpp theme={null} #include #include // Create stereo matcher Ptr stereo = StereoBM::create(16*5, 21); // Or use StereoSGBM for better quality Ptr stereo = StereoSGBM::create( 0, // minDisparity 16*5, // numDisparities (must be divisible by 16) 21, // blockSize 8*21*21, // P1 32*21*21, // P2 1, // disp12MaxDiff 63, // preFilterCap 10, // uniquenessRatio 100, // speckleWindowSize 32, // speckleRange StereoSGBM::MODE_SGBM_3WAY ); // Compute disparity Mat disparity; stereo->compute(rectLeft, rectRight, disparity); // Normalize for visualization Mat disp8; disparity.convertTo(disp8, CV_8U, 255.0/(16*5*16)); imshow("Disparity", disp8); ``` ### Reconstruct 3D Points ```cpp theme={null} // Compute 3D points from disparity Mat points3D; reprojectImageTo3D(disparity, points3D, Q); // Access 3D coordinates for (int y = 0; y < points3D.rows; y++) { for (int x = 0; x < points3D.cols; x++) { Vec3f point = points3D.at(y, x); float X = point[0]; float Y = point[1]; float Z = point[2]; // Filter invalid points if (abs(Z) < 10000) { // Valid 3D point } } } ``` ## Pose Estimation ### solvePnP - Estimate Camera Pose ```cpp theme={null} // Known 3D object points vector objectPoints = { Point3f(0, 0, 0), Point3f(100, 0, 0), Point3f(100, 100, 0), Point3f(0, 100, 0) }; // Corresponding 2D image points vector imagePoints = { Point2f(234, 456), Point2f(567, 445), Point2f(589, 234), Point2f(245, 267) }; // Camera calibration Mat cameraMatrix, distCoeffs; // ... load calibration ... // Solve for pose Mat rvec, tvec; bool success = solvePnP( objectPoints, imagePoints, cameraMatrix, distCoeffs, rvec, // Output: rotation vector tvec, // Output: translation vector false, // useExtrinsicGuess SOLVEPNP_ITERATIVE // Method ); if (success) { cout << "Rotation: " << rvec.t() << "\n"; cout << "Translation: " << tvec.t() << "\n"; // Convert rotation vector to matrix Mat R; Rodrigues(rvec, R); } ``` ### PnP Methods ```cpp theme={null} enum SolvePnPMethod { SOLVEPNP_ITERATIVE, // Iterative method SOLVEPNP_EPNP, // Efficient PnP SOLVEPNP_P3P, // 3-point algorithm SOLVEPNP_DLS, // Direct Least Squares SOLVEPNP_UPNP, // Unified PnP SOLVEPNP_AP3P, // Alternative P3P SOLVEPNP_IPPE, // Infinitesimal Plane-based Pose SOLVEPNP_IPPE_SQUARE // IPPE for square markers }; ``` ### Draw 3D Axes ```cpp theme={null} void drawAxes(Mat& img, const Mat& cameraMatrix, const Mat& distCoeffs, const Mat& rvec, const Mat& tvec, float length) { // 3D axis points vector axisPoints = { Point3f(0, 0, 0), // Origin Point3f(length, 0, 0), // X axis Point3f(0, length, 0), // Y axis Point3f(0, 0, length) // Z axis }; // Project to 2D vector imagePoints; projectPoints(axisPoints, rvec, tvec, cameraMatrix, distCoeffs, imagePoints); // Draw axes line(img, imagePoints[0], imagePoints[1], Scalar(0, 0, 255), 3); // X - Red line(img, imagePoints[0], imagePoints[2], Scalar(0, 255, 0), 3); // Y - Green line(img, imagePoints[0], imagePoints[3], Scalar(255, 0, 0), 3); // Z - Blue } ``` ## Homography ### Find Homography ```cpp theme={null} // Match points between two images vector srcPoints, dstPoints; // ... find corresponding points ... // Compute homography Mat H = findHomography(srcPoints, dstPoints, RANSAC, 3.0); // Warp image Mat warped; warpPerspective(srcImage, warped, H, dstImage.size()); ``` ### Decompose Homography ```cpp theme={null} // Decompose into rotation and translation vector rotations, translations, normals; int solutions = decomposeHomographyMat( H, cameraMatrix, rotations, translations, normals ); cout << "Found " << solutions << " solutions\n"; ``` ## Triangulation ```cpp theme={null} // Points from two calibrated cameras vector points1, points2; Mat P1, P2; // Projection matrices // Triangulate points Mat points4D; triangulatePoints(P1, P2, points1, points2, points4D); // Convert from homogeneous coordinates vector points3D; for (int i = 0; i < points4D.cols; i++) { float w = points4D.at(3, i); Point3f pt( points4D.at(0, i) / w, points4D.at(1, i) / w, points4D.at(2, i) / w ); points3D.push_back(pt); } ``` ## Best Practices **Calibration Quality:** * Use at least 10-20 images from different angles * Cover the entire image area with calibration pattern * RMS error should be \< 1 pixel for good calibration * Check reprojection errors for outliers **Calibration Pattern:** * Chessboard is most common and reliable * Ensure pattern is perfectly flat * Use high-quality printing * Good lighting without glare **Stereo Vision:** * Baseline (distance between cameras) affects depth range * Larger baseline = better depth accuracy at distance * Cameras should be well-aligned (\< 5° rotation) * Synchronized capture for moving scenes **Performance:** ```cpp theme={null} // Cache undistortion maps Mat map1, map2; initUndistortRectifyMap(K, D, Mat(), K, size, CV_16SC2, map1, map2); // Reuse for all images for (auto& img : images) { remap(img, undistorted, map1, map2, INTER_LINEAR); } ``` ## Related Modules * [Features 2D](/modules/features2d) - Feature detection for matching * [Image Processing](/modules/imgproc) - Image transformations * [Video Analysis](/modules/video) - Optical flow for tracking ## Source Reference Main header: `~/workspace/source/modules/calib3d/include/opencv2/calib3d.hpp` Examples: * `samples/cpp/calibration.cpp` - Camera calibration * `samples/cpp/stereo_calib.cpp` - Stereo calibration * `samples/cpp/stereo_match.cpp` - Stereo matching # Core Module Source: https://opencv-opencv.mintlify.app/modules/core Fundamental data structures, matrix operations, and utility functions that form the backbone of OpenCV The Core module is the foundation of OpenCV, providing essential data structures, basic operations, and utilities that other modules depend on. ## Overview From the OpenCV source (\~/workspace/source/modules/core/include/opencv2/core.hpp:62-67): > The Core module is the backbone of OpenCV, offering fundamental data structures, matrix operations, and utility functions that other modules depend on. It's essential for handling image data, performing mathematical computations, and managing memory efficiently within the OpenCV ecosystem. ## Key Components N-dimensional dense array for storing images and matrices Mathematical operations on arrays and matrices XML/YAML/JSON file I/O for data structures System functions, logging, and error handling ## Mat - The Core Data Structure The `Mat` class is OpenCV's primary container for images and matrices. ### Creating Mat Objects ```cpp theme={null} #include using namespace cv; // Create empty matrix Mat img; // Create with size and type Mat img1(480, 640, CV_8UC3); // 8-bit, 3-channel (BGR) Mat img2(Size(640, 480), CV_8UC1); // 8-bit, single channel // Create and initialize Mat zeros = Mat::zeros(100, 100, CV_8UC1); Mat ones = Mat::ones(100, 100, CV_32F); Mat eye = Mat::eye(3, 3, CV_64F); // Create from data float data[] = {1, 2, 3, 4}; Mat m(2, 2, CV_32F, data); ``` ### Mat Properties ```cpp theme={null} Mat img = imread("image.jpg"); // Dimensions int rows = img.rows; int cols = img.cols; Size size = img.size(); // width x height int channels = img.channels(); // Type information int type = img.type(); // e.g., CV_8UC3 int depth = img.depth(); // e.g., CV_8U // Memory size_t step = img.step; // bytes per row bool continuous = img.isContinuous(); ``` ## Array Operations The core module provides extensive array operations defined in opencv2/core.hpp. ### Arithmetic Operations ```cpp theme={null} // Addition (core.hpp:298) void add(InputArray src1, InputArray src2, OutputArray dst); Mat a, b, result; add(a, b, result); // Also supports operator overloading Mat c = a + b; Mat d = a - b; Mat e = a * 2.5; ``` ### Element-wise Operations ```cpp theme={null} // Multiplication multiply(src1, src2, dst); // Division divide(src1, src2, dst); // Absolute value absdiff(src1, src2, dst); // Power pow(src, power, dst); // Square root sqrt(src, dst); // Logarithm log(src, dst); // Exponential exp(src, dst); ``` ### Matrix Operations ```cpp theme={null} // Matrix multiplication Mat A, B, C; C = A * B; // Matrix product gemm(A, B, 1, Mat(), 0, C); // General matrix multiply // Transpose Mat At = A.t(); transpose(A, At); // Inversion Mat invA = A.inv(); invert(A, invA); // Determinant double det = determinant(A); // Eigenvalues and eigenvectors Mat eigenvalues, eigenvectors; eigen(A, eigenvalues, eigenvectors); // SVD (Singular Value Decomposition) SVD svd(A); Mat U = svd.u; Mat W = svd.w; Mat Vt = svd.vt; ``` ## Reduction Operations From core.hpp:210-215, OpenCV provides reduction types: ```cpp theme={null} enum ReduceTypes { REDUCE_SUM = 0, // Sum of all rows/columns REDUCE_AVG = 1, // Mean of all rows/columns REDUCE_MAX = 2, // Maximum value REDUCE_MIN = 3, // Minimum value REDUCE_SUM2 = 4 // Sum of squares }; // Example usage Mat src, dst; reduce(src, dst, 0, REDUCE_SUM); // Sum along columns reduce(src, dst, 1, REDUCE_AVG); // Average along rows ``` ## Statistical Functions ```cpp theme={null} // Min/Max double minVal, maxVal; Point minLoc, maxLoc; minMaxLoc(src, &minVal, &maxVal, &minLoc, &maxLoc); // Mean and standard deviation Scalar meanVal = mean(src); Scalar meanVal, stddevVal; meanStdDev(src, meanVal, stddevVal); // Sum Scalar sum = sum(src); // Count non-zero int count = countNonZero(src); // Norm double n = norm(src, NORM_L2); double n2 = norm(src1, src2, NORM_INF); ``` ## Border Handling From core.hpp:223-244, OpenCV provides `borderInterpolate()` for extrapolation: ```cpp theme={null} // Border types enum BorderTypes { BORDER_CONSTANT, // iiiiii|abcdefgh|iiiiiii BORDER_REPLICATE, // aaaaaa|abcdefgh|hhhhhhh BORDER_REFLECT, // fedcba|abcdefgh|hgfedcb BORDER_WRAP, // cdefgh|abcdefgh|abcdefg BORDER_REFLECT_101, // gfedcb|abcdefgh|gfedcba BORDER_TRANSPARENT, // Not modified BORDER_ISOLATED // Do not look outside ROI }; // Create border (core.hpp:294) void copyMakeBorder(InputArray src, OutputArray dst, int top, int bottom, int left, int right, int borderType, const Scalar& value = Scalar()); ``` ## Data Persistence Save and load data structures to XML/YAML/JSON files. ```cpp theme={null} // Writing FileStorage fs("data.yml", FileStorage::WRITE); fs << "image" << img; fs << "matrix" << mat; fs << "number" << 42; fs.release(); // Reading FileStorage fs("data.yml", FileStorage::READ); Mat img, mat; int number; fs["image"] >> img; fs["matrix"] >> mat; fs["number"] >> number; fs.release(); ``` ## Utility Functions ### Exception Handling From core.hpp:112-156: ```cpp theme={null} // Exception class class Exception : public std::exception { public: int code; // Error code String err; // Error description String func; // Function name String file; // Source file int line; // Line number }; // Error handling try { // OpenCV operations } catch(const cv::Exception& e) { std::cerr << "Error: " << e.what() << std::endl; } ``` ### System Information ```cpp theme={null} // OpenCV version String version = getBuildInformation(); // Number of CPU cores int cores = getNumberOfCPUs(); // Number of threads int threads = getNumThreads(); setNumThreads(4); // Tick count (for performance measurement) int64 t1 = getTickCount(); // ... operations ... int64 t2 = getTickCount(); double time = (t2 - t1) / getTickFrequency(); ``` ## Data Types OpenCV supports various data types: ```cpp theme={null} // Depth types CV_8U // 8-bit unsigned (0-255) CV_8S // 8-bit signed (-128-127) CV_16U // 16-bit unsigned CV_16S // 16-bit signed CV_32S // 32-bit signed integer CV_32F // 32-bit floating point CV_64F // 64-bit floating point // Type macros: CV_C CV_8UC1 // 8-bit, 1 channel (grayscale) CV_8UC3 // 8-bit, 3 channels (BGR color) CV_32FC1 // 32-bit float, 1 channel CV_64FC3 // 64-bit float, 3 channels ``` ## Sorting From core.hpp:158-167: ```cpp theme={null} enum SortFlags { SORT_EVERY_ROW = 0, // Sort each row independently SORT_EVERY_COLUMN = 1, // Sort each column independently SORT_ASCENDING = 0, // Ascending order SORT_DESCENDING = 16 // Descending order }; // Sort array sort(src, dst, SORT_EVERY_ROW | SORT_ASCENDING); // Sort with indices sortIdx(src, dst, SORT_EVERY_ROW | SORT_DESCENDING); ``` ## Memory Management ```cpp theme={null} // Mat uses reference counting Mat a = imread("image.jpg"); Mat b = a; // Shallow copy (same data) Mat c = a.clone(); // Deep copy (new data) // Check reference count int refs = a.u->refcount; // Release (automatic with scope) a.release(); // ROI (Region of Interest) - shares data Rect roi(10, 10, 100, 100); Mat roiMat = img(roi); // Check if data is continuous if (img.isContinuous()) { // Can treat as 1D array } ``` ## Example: Matrix Operations From samples/cpp/cout\_mat.cpp: ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { // Create matrices Mat A = (Mat_(3,3) << 1, 2, 3, 4, 5, 6, 7, 8, 9); Mat B = Mat::eye(3, 3, CV_64F); // Operations Mat C = A + B; Mat D = A * B; Mat At = A.t(); // Display cout << "A = " << endl << A << endl; cout << "A + I = " << endl << C << endl; cout << "A * I = " << endl << D << endl; cout << "A^T = " << endl << At << endl; // Statistics Scalar mean_val = mean(A); cout << "Mean: " << mean_val[0] << endl; return 0; } ``` ## Best Practices **Memory Efficiency:** * Use `Mat::clone()` only when you need a deep copy * Prefer ROI over copying when working with image regions * Let Mat destructor handle memory cleanup automatically **Type Safety:** * Always check `Mat::type()` before operations * Use `convertTo()` for type conversion * Verify dimensions match before matrix operations ## Related Modules * [Image Processing](/modules/imgproc) - Uses Mat for all operations * [Image I/O](/modules/imgcodecs) - Reads/writes images as Mat * [Video Analysis](/modules/video) - Processes video frames as Mat ## Source Reference Key header file: `~/workspace/source/modules/core/include/opencv2/core.hpp` See also: * `opencv2/core/mat.hpp` - Mat class implementation * `opencv2/core/operations.hpp` - Array operations * `opencv2/core/persistence.hpp` - File I/O # DNN Module Source: https://opencv-opencv.mintlify.app/modules/dnn Deep Neural Networks inference with OpenCV's DNN module ## Overview The DNN (Deep Neural Networks) module provides: * Loading models from popular frameworks (TensorFlow, PyTorch, ONNX, Caffe, Darknet) * Forward inference (no training) * Multiple backend support (CPU, OpenCL, CUDA) * Pre-trained model zoo The DNN module is for **inference only**. For training, use frameworks like TensorFlow or PyTorch. ## Quick Start ```cpp theme={null} #include using namespace cv::dnn; // Load model Net net = readNet("model.onnx"); // Prepare input Mat blob = blobFromImage(img, 1.0/255, Size(224, 224), Scalar(), true, false); // Set input net.setInput(blob); // Forward pass Mat output = net.forward(); ``` ## Net Class ### Loading Models ```cpp theme={null} // From ONNX Net net = readNetFromONNX("model.onnx"); // From TensorFlow Net net = readNetFromTensorflow("model.pb", "config.pbtxt"); // From Caffe Net net = readNetFromCaffe("deploy.prototxt", "model.caffemodel"); // From Darknet (YOLO) Net net = readNetFromDarknet("yolov4.cfg", "yolov4.weights"); // From PyTorch (via ONNX) Net net = readNetFromONNX("model.onnx"); // Auto-detect format Net net = readNet("model.onnx"); // Detects format automatically ``` ### Setting Backend and Target ```cpp theme={null} // CPU backend net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // OpenCL (GPU) net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_OPENCL); // CUDA (NVIDIA GPU) net.setPreferableBackend(DNN_BACKEND_CUDA); net.setPreferableTarget(DNN_TARGET_CUDA); // Intel OpenVINO net.setPreferableBackend(DNN_BACKEND_INFERENCE_ENGINE); net.setPreferableTarget(DNN_TARGET_CPU); ``` ### Inference ```cpp theme={null} // Single output net.setInput(blob, "input_name"); Mat output = net.forward("output_name"); // Multiple outputs std::vector outNames = {"output1", "output2"}; std::vector outputs; net.forward(outputs, outNames); // All outputs std::vector outNames = net.getUnconnectedOutLayersNames(); std::vector outputs; net.forward(outputs, outNames); ``` ## Blob Preparation ### blobFromImage ```cpp theme={null} Mat blob = blobFromImage( image, // Input image 1.0/255.0, // Scale factor Size(224, 224), // Target size Scalar(0, 0, 0), // Mean subtraction true, // swapRB (BGR to RGB) false, // crop CV_32F // Output type ); ``` ### blobFromImages (Batch) ```cpp theme={null} std::vector images = {img1, img2, img3}; Mat blob = blobFromImages(images, 1.0/255.0, Size(224, 224), Scalar(), true); ``` ### Blob Format Blobs use **NCHW** format: * **N**: Batch size * **C**: Channels * **H**: Height * **W**: Width ```cpp theme={null} // Blob shape: [1, 3, 224, 224] // 1 image, 3 channels (RGB), 224x224 pixels ``` ## Common Tasks ### Image Classification ```cpp theme={null} // Load model Net net = readNet("mobilenet_v2.onnx"); net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // Prepare input Mat img = imread("image.jpg"); Mat blob = blobFromImage(img, 1.0/255, Size(224, 224), Scalar(), true, false); // Inference net.setInput(blob); Mat prob = net.forward(); // Get top class Point classIdPoint; minMaxLoc(prob.reshape(1, 1), 0, 0, 0, &classIdPoint); int classId = classIdPoint.x; ``` ### Object Detection (YOLO) ```cpp theme={null} // Load YOLO Net net = readNetFromDarknet("yolov4.cfg", "yolov4.weights"); // Prepare input Mat blob = blobFromImage(img, 1/255.0, Size(416, 416), Scalar(), true, false); // Forward net.setInput(blob); std::vector outputs; net.forward(outputs, net.getUnconnectedOutLayersNames()); // Process detections for(Mat& output : outputs) { for(int i = 0; i < output.rows; i++) { float* data = output.ptr(i); float confidence = data[4]; if(confidence > 0.5) { int classId = max_element(data+5, data+output.cols) - (data+5); float x = data[0] * img.cols; float y = data[1] * img.rows; float w = data[2] * img.cols; float h = data[3] * img.rows; // Draw bounding box } } } ``` ### Semantic Segmentation ```cpp theme={null} Net net = readNet("fcn-resnet50.onnx"); Mat blob = blobFromImage(img, 1.0, Size(500, 500)); net.setInput(blob); Mat score = net.forward(); // score shape: [1, num_classes, H, W] // Get class per pixel Mat classMap; for(int h = 0; h < score.size[2]; h++) { for(int w = 0; w < score.size[3]; w++) { // Get class with max score // ... } } ``` ## Model Zoo OpenCV provides pre-trained models: ```cpp theme={null} // Face detection Net faceNet = readNet("opencv_face_detector.caffemodel", "opencv_face_detector.prototxt"); // Age/Gender Net ageNet = readNet("age_net.caffemodel", "age_deploy.prototxt"); Net genderNet = readNet("gender_net.caffemodel", "gender_deploy.prototxt"); // OpenPose (pose estimation) Net poseNet = readNet("pose_iter_440000.caffemodel", "pose_deploy_linevec.prototxt"); ``` ## Performance Optimization ### Backend Selection ```cpp CPU theme={null} net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); ``` ```cpp CUDA theme={null} net.setPreferableBackend(DNN_BACKEND_CUDA); net.setPreferableTarget(DNN_TARGET_CUDA); ``` ```cpp OpenCL theme={null} net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_OPENCL); ``` ### Input Size ```cpp theme={null} // Smaller input = faster inference Mat blob = blobFromImage(img, 1.0/255, Size(320, 320), // Reduce from 640x640 Scalar(), true); ``` ### Batch Processing ```cpp theme={null} // Process multiple images at once std::vector images; for(int i = 0; i < 4; i++) { images.push_back(imread(files[i])); } Mat blob = blobFromImages(images, 1.0/255, Size(224, 224)); net.setInput(blob); Mat output = net.forward(); // Batch inference ``` ## Best Practices ONNX provides best compatibility across frameworks Use CUDA backend for 5-10x speedup on NVIDIA GPUs Smaller inputs trade accuracy for speed Batch processing improves GPU utilization ## Troubleshooting ### Model Loading Issues ```cpp theme={null} if(net.empty()) { std::cerr << "Failed to load model\n"; return -1; } ``` ### Check Backend Support ```cpp theme={null} auto backends = getAvailableBackends(); for(auto& backend : backends) { std::cout << "Backend: " << backend.first << ", Target: " << backend.second << "\n"; } ``` ### Enable Diagnostic Mode ```cpp theme={null} enableModelDiagnostics(true); Net net = readNet("model.onnx"); // Verbose loading ``` ## See Also * [DNN Network API](/api/dnn/network) - Detailed Net class reference * [DNN Layers](/api/dnn/layers) - Layer types and custom layers * [DNN Inference](/api/dnn/inference) - Advanced inference techniques * [Object Detection Module](/modules/objdetect) - Classic detection methods # Features2D Module Source: https://opencv-opencv.mintlify.app/modules/features2d 2D feature detection, description, and matching in OpenCV ## Overview The `features2d` module provides algorithms for detecting, describing, and matching 2D features in images. Features are distinctive points (keypoints) that can be reliably found across different views of the same scene. ## Key Concepts ### Feature Detection Detecting distinctive points (corners, blobs) in images that are stable under various transformations. ### Feature Description Computing numerical descriptors for each keypoint that capture local appearance. ### Feature Matching Finding correspondences between features in different images. ## Main Classes ### Feature2D Base class for feature detectors and descriptor extractors: ```cpp theme={null} class Feature2D : public Algorithm { public: // Detect keypoints virtual void detect(InputArray image, std::vector& keypoints, InputArray mask = noArray()); // Compute descriptors virtual void compute(InputArray image, std::vector& keypoints, OutputArray descriptors); // Detect and compute in one call virtual void detectAndCompute(InputArray image, InputArray mask, std::vector& keypoints, OutputArray descriptors, bool useProvidedKeypoints = false); }; ``` ### KeyPoint Structure ```cpp theme={null} struct KeyPoint { Point2f pt; // Coordinates float size; // Diameter of meaningful neighborhood float angle; // Orientation (-1 if not applicable) float response; // Detector response (strength) int octave; // Pyramid layer int class_id; // Object class (for tracking) }; ``` ## Feature Detectors ### SIFT (Scale-Invariant Feature Transform) ```cpp theme={null} // Create detector Ptr sift = SIFT::create( 0, // nfeatures (0 = all) 3, // nOctaveLayers 0.04, // contrastThreshold 10, // edgeThreshold 1.6 // sigma ); // Detect and compute std::vector keypoints; Mat descriptors; sift->detectAndCompute(img, noArray(), keypoints, descriptors); ``` **Properties**: * Scale invariant * Rotation invariant * 128-dimensional float descriptors * Patented (free for research) ### ORB (Oriented FAST and Rotated BRIEF) ```cpp theme={null} Ptr orb = ORB::create( 500, // nfeatures 1.2f, // scaleFactor 8, // nlevels 31, // edgeThreshold 0, // firstLevel 2, // WTA_K ORB::HARRIS_SCORE, // scoreType 31, // patchSize 20 // fastThreshold ); orb->detectAndCompute(img, noArray(), keypoints, descriptors); ``` **Properties**: * Very fast * Binary descriptors (32 bytes) * Free to use * Good for real-time applications ### BRISK ```cpp theme={null} Ptr brisk = BRISK::create( 30, // threshold 3, // octaves 1.0f // patternScale ); ``` ### AKAZE ```cpp theme={null} Ptr akaze = AKAZE::create( AKAZE::DESCRIPTOR_MLDB, // descriptor_type 0, // descriptor_size 3, // descriptor_channels 0.001f, // threshold 4, // nOctaves 4, // nOctaveLayers KAZE::DIFF_PM_G2 // diffusivity ); ``` ### FAST Corner Detector ```cpp theme={null} std::vector keypoints; FAST(img, keypoints, 10, // threshold true); // nonmaxSuppression // Or using detector class Ptr fast = FastFeatureDetector::create(10, true); fast->detect(img, keypoints); ``` ## Descriptor Matchers ### BFMatcher (Brute Force) ```cpp theme={null} // For binary descriptors (ORB, BRISK) BFMatcher matcher(NORM_HAMMING, true); // crossCheck // For float descriptors (SIFT, SURF) BFMatcher matcher(NORM_L2, true); // Match std::vector matches; matcher.match(descriptors1, descriptors2, matches); ``` ### FLANN Matcher ```cpp theme={null} // Faster for large datasets FlannBasedMatcher matcher; std::vector matches; matcher.match(descriptors1, descriptors2, matches); ``` ### KNN Matching ```cpp theme={null} std::vector> knn_matches; matcher.knnMatch(descriptors1, descriptors2, knn_matches, 2); // Lowe's ratio test std::vector good_matches; for(size_t i = 0; i < knn_matches.size(); i++) { if(knn_matches[i][0].distance < 0.75f * knn_matches[i][1].distance) { good_matches.push_back(knn_matches[i][0]); } } ``` ## Complete Example ```cpp theme={null} #include #include int main() { // Load images Mat img1 = imread("image1.jpg", IMREAD_GRAYSCALE); Mat img2 = imread("image2.jpg", IMREAD_GRAYSCALE); // Create detector Ptr detector = ORB::create(1000); // Detect and compute std::vector kp1, kp2; Mat desc1, desc2; detector->detectAndCompute(img1, noArray(), kp1, desc1); detector->detectAndCompute(img2, noArray(), kp2, desc2); // Match BFMatcher matcher(NORM_HAMMING); std::vector matches; matcher.match(desc1, desc2, matches); // Draw matches Mat img_matches; drawMatches(img1, kp1, img2, kp2, matches, img_matches); imshow("Matches", img_matches); waitKey(0); return 0; } ``` ## Algorithm Comparison | Algorithm | Speed | Descriptor | License | Best For | | --------- | --------- | ------------ | -------- | --------- | | **SIFT** | Slow | Float 128D | Patented | Accuracy | | **ORB** | Very Fast | Binary 32B | Free | Real-time | | **AKAZE** | Fast | Binary/Float | Free | General | | **BRISK** | Fast | Binary 64B | Free | Real-time | ## Best Practices Fastest detector/descriptor, good for video Best accuracy but slower, patented Filter matches using Lowe's ratio test Convert to grayscale for better performance ## See Also * [Calib3D Module](/modules/calib3d) - Camera calibration using features * [Video Module](/modules/video) - Object tracking with features * [Feature Detection Tutorial](https://docs.opencv.org/master/df/d0c/tutorial_py_fast.html) # G-API Module Source: https://opencv-opencv.mintlify.app/modules/gapi Graph-based API for efficient image processing pipelines ## Overview G-API (Graph API) is OpenCV's graph-based framework for building efficient, portable image processing pipelines. It provides: * **Lazy evaluation**: Build computation graph, execute later * **Backend abstraction**: CPU, GPU, neural network accelerators * **Performance optimization**: Automatic fusion and optimization * **Heterogeneous execution**: Mix different backends ## Key Concepts ### Computation Graph G-API separates graph **construction** from **execution**: 1. **Build graph** - Define operations 2. **Compile** - Optimize for target backend 3. **Execute** - Run on actual data ### Data Types * **GMat**: Graph matrix (image/matrix) * **GScalar**: Graph scalar value * **GArray**: Graph array of values * **GOpaque**: Graph opaque type * **GFrame**: Graph video frame ## Basic Example ```cpp theme={null} #include #include #include using namespace cv; int main() { // 1. Declare computation GMat in; // Input placeholder GMat gray = gapi::RGB2Gray(in); GMat blurred = gapi::blur(gray, Size(5, 5)); GMat edges = gapi::Canny(blurred, 32, 128); // 2. Compile computation GComputation ac(in, edges); auto compiled = ac.compile(descr_of(input_mat)); // 3. Execute on actual data Mat input_mat = imread("image.jpg"); Mat output_mat; compiled(input_mat, output_mat); return 0; } ``` ## Available Operations ### Image Processing ```cpp theme={null} // Color conversion GMat gray = gapi::RGB2Gray(rgb); GMat hsv = gapi::RGB2HSV(rgb); // Filtering GMat blurred = gapi::blur(src, Size(5, 5)); GMat gaussian = gapi::gaussianBlur(src, Size(5, 5), 1.5); GMat median = gapi::medianBlur(src, 5); // Edge detection GMat edges = gapi::Canny(src, 50, 150); GMat sobel = gapi::Sobel(src, CV_8U, 1, 1); // Morphology Mat kernel = getStructuringElement(MORPH_RECT, Size(5,5)); GMat dilated = gapi::dilate(src, kernel); GMat eroded = gapi::erode(src, kernel); // Geometric GMat resized = gapi::resize(src, Size(320, 240)); GMat flipped = gapi::flip(src, 1); ``` ### Core Operations ```cpp theme={null} // Arithmetic GMat sum = gapi::add(src1, src2); GMat diff = gapi::sub(src1, src2); GMat prod = gapi::mul(src1, src2); GMat quot = gapi::div(src1, src2); // Scalar operations GMat added = gapi::addC(src, Scalar(10)); GMat scaled = gapi::mulC(src, 1.5); // Bitwise GMat anded = gapi::bitwise_and(src1, src2); GMat ored = gapi::bitwise_or(src1, src2); GMat inverted = gapi::bitwise_not(src); // Comparison GMat mask = gapi::cmpGT(src, Scalar(128)); GMat result = gapi::select(mask, src1, src2); // Normalization GMat normalized = gapi::normalize(src, 0, 255, NORM_MINMAX); ``` ## Computation ### GComputation Class ```cpp theme={null} // Single input, single output GMat in; GMat out = gapi::blur(in, Size(5,5)); GComputation c(in, out); // Multiple inputs GMat in1, in2; GMat out = gapi::add(in1, in2); GComputation c(GIn(in1, in2), GOut(out)); // Multiple outputs GMat in; GMat out1 = gapi::blur(in, Size(3,3)); GMat out2 = gapi::blur(in, Size(5,5)); GComputation c(GIn(in), GOut(out1, out2)); ``` ### Compilation ```cpp theme={null} // Compile for specific input GComputation c(in, out); Mat input_data = imread("img.jpg"); auto compiled = c.compile(descr_of(input_data)); // Execute Mat output_data; compiled(input_data, output_data); // Reuse compiled graph for(Mat frame : frames) { compiled(frame, result); } ``` ## Backends ### CPU Backend (Default) ```cpp theme={null} GComputation c(in, out); auto compiled = c.compile(descr_of(input)); // Uses CPU by default ``` ### OpenCL Backend ```cpp theme={null} #include GComputation c(in, out); auto compiled = c.compile( descr_of(input), compile_args(gapi::use_only{gapi::ocl::kernels()}) ); ``` ### Fluid Backend (Cache-Efficient) ```cpp theme={null} #include GComputation c(in, out); auto compiled = c.compile( descr_of(input), compile_args(gapi::use_only{gapi::fluid::kernels()}) ); ``` ### Heterogeneous Execution ```cpp theme={null} // Mix CPU and OpenCL auto compiled = c.compile( descr_of(input), compile_args( gapi::kernels(), // CPU blur gapi::kernels() // OpenCL Canny ) ); ``` ## Streaming Mode ### Video Processing ```cpp theme={null} #include GMat in; GMat out = gapi::blur(in, Size(5,5)); GComputation c(in, out); auto pipeline = c.compileStreaming(); // Set source pipeline.setSource("video.mp4"); // Process frames pipeline.start(); while(pipeline.pull(output_mat)) { imshow("Output", output_mat); if(waitKey(1) == 27) break; } pipeline.stop(); ``` ### Camera Processing ```cpp theme={null} pipeline.setSource(0); // Camera 0 pipeline.start(); while(pipeline.pull(frame)) { // Process frame } ``` ## Custom Operations ### Define Custom Kernel ```cpp theme={null} // 1. Declare operation G_TYPED_KERNEL(GCustomOp, , "custom.op") { static GMatDesc outMeta(GMatDesc in) { return in; // Same as input } }; // 2. Implement for CPU GAPI_OCV_KERNEL(GCPUCustomOp, GCustomOp) { static void run(const Mat& in, Mat& out) { // Custom processing out = in * 2; } }; // 3. Use in graph GMat in; GMat out = GCustomOp::on(in); GComputation c(in, out); auto compiled = c.compile( descr_of(input), compile_args(gapi::kernels()) ); ``` ## Performance Optimization ### Operation Fusion G-API automatically fuses operations: ```cpp theme={null} // These operations may be fused GMat gray = gapi::RGB2Gray(in); GMat blurred = gapi::blur(gray, Size(5,5)); GMat edges = gapi::Canny(blurred, 50, 150); // Compiled as optimized pipeline auto compiled = c.compile(descr_of(input)); ``` ### Memory Optimization ```cpp theme={null} // Fluid backend minimizes memory usage auto compiled = c.compile( descr_of(input), compile_args( gapi::use_only{gapi::fluid::kernels()} ) ); ``` ## Complete Example: Edge Detection Pipeline ```cpp theme={null} #include #include #include #include #include using namespace cv; int main() { // Build graph GMat in; GMat gray = gapi::RGB2Gray(in); GMat blurred = gapi::gaussianBlur(gray, Size(5,5), 1.5); GMat edges = gapi::Canny(blurred, 32, 128, 3); // Compile for streaming GComputation c(GIn(in), GOut(edges)); auto pipeline = c.compileStreaming(); // Set video source pipeline.setSource("video.mp4"); // Process stream pipeline.start(); Mat output; while(pipeline.pull(output)) { imshow("Edges", output); if(waitKey(1) == 27) break; } pipeline.stop(); return 0; } ``` ## Best Practices Compile graph once, execute on multiple inputs Use Fluid for cache-efficiency, OpenCL for GPU Streaming mode for video processing Implement custom operations when needed ## Advantages Over Traditional API | Feature | Traditional API | G-API | | ---------------- | ------------------------------ | ---------------- | | **Optimization** | Manual | Automatic | | **Portability** | Backend-specific | Backend-agnostic | | **Efficiency** | Per-operation | Graph-level | | **Memory** | Allocates intermediate buffers | Optimizes memory | ## See Also * [Core Module](/modules/core) - Basic operations * [ImgProc Module](/modules/imgproc) - Image processing * [G-API Tutorial](https://docs.opencv.org/master/d0/d1e/gapi.html) # High-Level GUI Module Source: https://opencv-opencv.mintlify.app/modules/highgui Window management, image display, and user interaction tools for creating simple graphical interfaces The High-Level GUI (highgui) module provides easy-to-use interfaces for creating windows, displaying images, and handling user interaction through keyboard, mouse, and trackbars. ## Overview From opencv2/highgui.hpp:55-66: > While OpenCV was designed for use in full-scale applications and can be used within functionally rich UI frameworks (such as Qt, WinForms, or Cocoa) or without any UI at all, sometimes there it is required to try functionality quickly and visualize the results. This is what the HighGUI module has been designed for. Create and manage display windows Show images with imshow() function Handle keyboard and mouse events Interactive parameter adjustment ## Window Management ### Creating Windows From highgui.hpp:142-152: ```cpp theme={null} #include using namespace cv; // Create window with auto-sizing namedWindow("My Window", WINDOW_AUTOSIZE); // Create resizable window namedWindow("Resizable", WINDOW_NORMAL); // Create fullscreen window namedWindow("Fullscreen", WINDOW_FULLSCREEN); // Create window with OpenGL support namedWindow("OpenGL", WINDOW_OPENGL); ``` ### Window Flags ```cpp theme={null} enum WindowFlags { WINDOW_NORMAL = 0x00000000, // User can resize WINDOW_AUTOSIZE = 0x00000001, // Size constrained by image WINDOW_OPENGL = 0x00001000, // OpenGL support WINDOW_FULLSCREEN = 1, // Fullscreen mode WINDOW_FREERATIO = 0x00000100, // No aspect ratio constraint WINDOW_KEEPRATIO = 0x00000000 // Respect aspect ratio }; ``` ### Window Operations ```cpp theme={null} // Move window moveWindow("My Window", 100, 100); // Resize window (only for WINDOW_NORMAL) resizeWindow("My Window", 800, 600); // Destroy specific window destroyWindow("My Window"); // Destroy all windows destroyAllWindows(); // Get window property int isVisible = getWindowProperty("My Window", WND_PROP_VISIBLE); int isFullscreen = getWindowProperty("My Window", WND_PROP_FULLSCREEN); // Set window property setWindowProperty("My Window", WND_PROP_FULLSCREEN, WINDOW_FULLSCREEN); setWindowProperty("My Window", WND_PROP_TOPMOST, 1); // Always on top ``` ## Displaying Images ### Basic Image Display ```cpp theme={null} #include #include using namespace cv; int main() { // Load image Mat img = imread("photo.jpg"); if (img.empty()) { cerr << "Could not load image" << endl; return -1; } // Create window and display namedWindow("Display", WINDOW_AUTOSIZE); imshow("Display", img); // Wait for key press waitKey(0); // Cleanup destroyAllWindows(); return 0; } ``` ### Multiple Images ```cpp theme={null} // Display multiple images Mat img1 = imread("image1.jpg"); Mat img2 = imread("image2.jpg"); Mat img3 = imread("image3.jpg"); namedWindow("Image 1", WINDOW_NORMAL); namedWindow("Image 2", WINDOW_NORMAL); namedWindow("Image 3", WINDOW_NORMAL); // Position windows moveWindow("Image 1", 0, 0); moveWindow("Image 2", 650, 0); moveWindow("Image 3", 1300, 0); // Display imshow("Image 1", img1); imshow("Image 2", img2); imshow("Image 3", img3); waitKey(0); ``` ### Updating Display ```cpp theme={null} // Create window once namedWindow("Animation", WINDOW_AUTOSIZE); // Update display in loop for (int i = 0; i < 100; i++) { Mat frame = generateFrame(i); imshow("Animation", frame); // Wait 30ms between frames if (waitKey(30) >= 0) break; } ``` ## Keyboard Input ### waitKey Function ```cpp theme={null} // Wait indefinitely for key int key = waitKey(0); // Wait with timeout (milliseconds) int key = waitKey(30); // Wait 30ms // Check specific keys if (key == 'q' || key == 27) { // 'q' or ESC break; } // Arrow keys (platform-dependent) const int KEY_UP = 2490368; const int KEY_DOWN = 2621440; const int KEY_LEFT = 2424832; const int KEY_RIGHT = 2555904; // Special keys if (key == 's') { imwrite("screenshot.png", image); } if (key == ' ') { // Space paused = !paused; } ``` ### Interactive Key Handling From samples/cpp/edge.cpp: ```cpp theme={null} #include using namespace cv; int main() { Mat image = imread("photo.jpg"); namedWindow("Controls", WINDOW_AUTOSIZE); imshow("Controls", image); cout << "Commands:\n" << " q/ESC - quit\n" << " s - save\n" << " r - reset\n"; while (true) { int key = waitKey(0); if (key == 'q' || key == 27) { break; } else if (key == 's') { imwrite("saved.png", image); cout << "Image saved\n"; } else if (key == 'r') { image = imread("photo.jpg"); imshow("Controls", image); cout << "Image reset\n"; } } return 0; } ``` ## Mouse Events From highgui.hpp:166-189: ### Mouse Callback ```cpp theme={null} // Mouse event types enum MouseEventTypes { EVENT_MOUSEMOVE = 0, // Mouse moved EVENT_LBUTTONDOWN = 1, // Left button pressed EVENT_RBUTTONDOWN = 2, // Right button pressed EVENT_MBUTTONDOWN = 3, // Middle button pressed EVENT_LBUTTONUP = 4, // Left button released EVENT_RBUTTONUP = 5, // Right button released EVENT_MBUTTONUP = 6, // Middle button released EVENT_LBUTTONDBLCLK = 7, // Left button double-click EVENT_RBUTTONDBLCLK = 8, // Right button double-click EVENT_MBUTTONDBLCLK = 9, // Middle button double-click EVENT_MOUSEWHEEL = 10, // Mouse wheel scrolled EVENT_MOUSEHWHEEL = 11 // Horizontal wheel scrolled }; // Mouse event flags enum MouseEventFlags { EVENT_FLAG_LBUTTON = 1, // Left button down EVENT_FLAG_RBUTTON = 2, // Right button down EVENT_FLAG_MBUTTON = 4, // Middle button down EVENT_FLAG_CTRLKEY = 8, // Ctrl key pressed EVENT_FLAG_SHIFTKEY = 16, // Shift key pressed EVENT_FLAG_ALTKEY = 32 // Alt key pressed }; ``` ### Implementing Mouse Callback ```cpp theme={null} // Global or class member variables Mat image; vector points; // Mouse callback function void onMouse(int event, int x, int y, int flags, void* userdata) { if (event == EVENT_LBUTTONDOWN) { // Left click - add point points.push_back(Point(x, y)); circle(image, Point(x, y), 3, Scalar(0, 0, 255), -1); imshow("Image", image); } else if (event == EVENT_RBUTTONDOWN) { // Right click - clear points points.clear(); image = imread("original.jpg"); imshow("Image", image); } else if (event == EVENT_MOUSEMOVE) { // Show coordinates in window title if (flags & EVENT_FLAG_LBUTTON) { // Drawing while left button held circle(image, Point(x, y), 2, Scalar(255, 0, 0), -1); imshow("Image", image); } } } int main() { image = imread("photo.jpg"); namedWindow("Image"); setMouseCallback("Image", onMouse, nullptr); imshow("Image", image); waitKey(0); return 0; } ``` ### Interactive Drawing ```cpp theme={null} Mat canvas; bool drawing = false; Point prevPt(-1, -1); void drawCallback(int event, int x, int y, int flags, void*) { if (event == EVENT_LBUTTONDOWN) { drawing = true; prevPt = Point(x, y); } else if (event == EVENT_MOUSEMOVE && drawing) { Point pt(x, y); line(canvas, prevPt, pt, Scalar(0, 255, 0), 2); prevPt = pt; imshow("Drawing", canvas); } else if (event == EVENT_LBUTTONUP) { drawing = false; } } int main() { canvas = Mat::zeros(480, 640, CV_8UC3); namedWindow("Drawing"); setMouseCallback("Drawing", drawCallback); imshow("Drawing", canvas); waitKey(0); return 0; } ``` ## Trackbars Example from samples/cpp/edge.cpp: ### Creating Trackbars ```cpp theme={null} // Global variables for trackbar values int threshold1 = 50; int threshold2 = 150; // Trackbar callback function void onThresholdChange(int, void*) { Mat edges; Canny(gray, edges, threshold1, threshold2); imshow("Edges", edges); } int main() { Mat image = imread("photo.jpg"); Mat gray; cvtColor(image, gray, COLOR_BGR2GRAY); // Create window namedWindow("Edges", WINDOW_AUTOSIZE); // Create trackbars (edge.cpp:75-76) createTrackbar("Threshold 1", "Edges", &threshold1, 255, onThresholdChange); createTrackbar("Threshold 2", "Edges", &threshold2, 255, onThresholdChange); // Initial display onThresholdChange(0, nullptr); waitKey(0); return 0; } ``` ### Multiple Trackbars ```cpp theme={null} Mat image, result; int blur_size = 1; int threshold_val = 128; int morph_size = 1; void processImage(int, void*) { Mat blurred, binary, morphed; // Apply blur int ksize = blur_size * 2 + 1; GaussianBlur(image, blurred, Size(ksize, ksize), 0); // Threshold threshold(blurred, binary, threshold_val, 255, THRESH_BINARY); // Morphology int msize = morph_size * 2 + 1; Mat element = getStructuringElement(MORPH_RECT, Size(msize, msize)); morphologyEx(binary, result, MORPH_CLOSE, element); imshow("Result", result); } int main() { image = imread("document.jpg", IMREAD_GRAYSCALE); namedWindow("Result"); createTrackbar("Blur", "Result", &blur_size, 15, processImage); createTrackbar("Threshold", "Result", &threshold_val, 255, processImage); createTrackbar("Morph", "Result", &morph_size, 10, processImage); processImage(0, nullptr); waitKey(0); return 0; } ``` ### Getting Trackbar Position ```cpp theme={null} // Get current trackbar position int pos = getTrackbarPos("Threshold", "Window"); // Set trackbar position setTrackbarPos("Threshold", "Window", 100); // Set min/max values (requires Qt backend) setTrackbarMin("Threshold", "Window", 10); setTrackbarMax("Threshold", "Window", 200); ``` ## Practical Examples ### Image Viewer with Controls ```cpp theme={null} #include #include #include using namespace cv; using namespace std; Mat originalImage, displayImage; int brightness = 50; int contrast = 50; void updateImage(int, void*) { double alpha = contrast / 50.0; // 0.0 to 2.0 int beta = brightness - 50; // -50 to +50 displayImage = Mat::zeros(originalImage.size(), originalImage.type()); originalImage.convertTo(displayImage, -1, alpha, beta); imshow("Image Viewer", displayImage); } int main(int argc, char** argv) { if (argc < 2) { cout << "Usage: " << argv[0] << " \n"; return -1; } originalImage = imread(argv[1]); if (originalImage.empty()) { cerr << "Error loading image\n"; return -1; } namedWindow("Image Viewer", WINDOW_NORMAL); createTrackbar("Brightness", "Image Viewer", &brightness, 100, updateImage); createTrackbar("Contrast", "Image Viewer", &contrast, 100, updateImage); updateImage(0, nullptr); cout << "Controls:\n" << " q - quit\n" << " s - save\n" << " r - reset\n"; while (true) { int key = waitKey(30); if (key == 'q' || key == 27) break; if (key == 's') { imwrite("output.jpg", displayImage); cout << "Saved output.jpg\n"; } if (key == 'r') { brightness = 50; contrast = 50; setTrackbarPos("Brightness", "Image Viewer", 50); setTrackbarPos("Contrast", "Image Viewer", 50); updateImage(0, nullptr); } } return 0; } ``` ### ROI Selector ```cpp theme={null} Mat image, roiImage; Rect roiRect; bool selecting = false; Point startPt; void mouseHandler(int event, int x, int y, int flags, void*) { if (event == EVENT_LBUTTONDOWN) { selecting = true; startPt = Point(x, y); roiRect = Rect(x, y, 0, 0); } else if (event == EVENT_MOUSEMOVE && selecting) { roiRect = Rect( min(startPt.x, x), min(startPt.y, y), abs(x - startPt.x), abs(y - startPt.y) ); Mat display = image.clone(); rectangle(display, roiRect, Scalar(0, 255, 0), 2); imshow("Select ROI", display); } else if (event == EVENT_LBUTTONUP) { selecting = false; if (roiRect.width > 0 && roiRect.height > 0) { roiImage = image(roiRect).clone(); imshow("ROI", roiImage); } } } int main() { image = imread("photo.jpg"); namedWindow("Select ROI"); setMouseCallback("Select ROI", mouseHandler); imshow("Select ROI", image); cout << "Click and drag to select region\n"; waitKey(0); return 0; } ``` ### Color Picker ```cpp theme={null} Mat image; Scalar selectedColor; void pickColor(int event, int x, int y, int flags, void*) { if (event == EVENT_LBUTTONDOWN) { Vec3b pixel = image.at(y, x); selectedColor = Scalar(pixel[0], pixel[1], pixel[2]); cout << "Selected color (BGR): " << (int)pixel[0] << ", " << (int)pixel[1] << ", " << (int)pixel[2] << endl; // Show color swatch Mat swatch(100, 100, CV_8UC3, selectedColor); imshow("Selected Color", swatch); } } int main() { image = imread("colorful.jpg"); namedWindow("Image"); namedWindow("Selected Color"); setMouseCallback("Image", pickColor); imshow("Image", image); cout << "Click on image to pick color\n"; waitKey(0); return 0; } ``` ## Best Practices **Window Lifecycle:** Always destroy windows when done: ```cpp theme={null} namedWindow("Window"); imshow("Window", img); waitKey(0); destroyAllWindows(); // Important for cleanup ``` **Event Loop:** `waitKey()` is essential for window updates: ```cpp theme={null} while (true) { imshow("Display", frame); if (waitKey(1) >= 0) break; // Must call waitKey! } ``` Without `waitKey()`, windows won't refresh or respond to events. **Trackbar Callbacks:** Trackbar callbacks can be nullptr if you handle updates elsewhere: ```cpp theme={null} createTrackbar("Value", "Window", &value, 100, nullptr); while (true) { int currentValue = getTrackbarPos("Value", "Window"); // Use currentValue... imshow("Window", result); if (waitKey(30) >= 0) break; } ``` ## Platform-Specific Features ### Qt Backend When built with Qt support, additional features are available: ```cpp theme={null} // Create buttons (Qt only) createButton("Process", buttonCallback, nullptr, QT_PUSH_BUTTON, false); // Display text overlay (Qt only) displayOverlay("Window", "Processing...", 1000); // Status bar (Qt only) displayStatusBar("Window", "Ready", 0); // Save window parameters saveWindowParameters("Window"); loadWindowParameters("Window"); ``` ## Limitations **Simple GUI Only:** HighGUI is designed for quick visualization and simple interaction. For production applications with complex UI requirements, use: * Qt * wxWidgets * GTK * Platform-native frameworks **Thread Safety:** HighGUI functions should be called from the main thread. For multi-threaded applications, handle GUI updates carefully. ## Related Modules * [Image I/O](/modules/imgcodecs) - Load images to display * [Video I/O](/modules/videoio) - Display video streams * [Image Processing](/modules/imgproc) - Process images before display ## Source Reference Main header: `~/workspace/source/modules/highgui/include/opencv2/highgui.hpp` Examples: * `samples/cpp/edge.cpp` - Trackbar usage * Various tutorial samples demonstrate GUI features # Image Codecs Module Source: https://opencv-opencv.mintlify.app/modules/imgcodecs Image file reading and writing with support for multiple formats including JPEG, PNG, TIFF, WebP, AVIF, and animated formats The Image Codecs (imgcodecs) module handles reading and writing images in various formats, supporting both static images and animations with metadata. ## Overview From opencv2/imgcodecs.hpp:48-55: > This module provides image file reading and writing capabilities with support for various formats including JPEG, PNG, TIFF, WebP, AVIF, GIF, and more. It also handles multi-page images, animations, and metadata (EXIF, XMP, ICC profiles). Load images from files with automatic format detection Save images to files with format-specific parameters Read and write animated GIF, AVIF, APNG, and WebP Handle EXIF, XMP, and ICC profile metadata ## Reading Images ### Basic Image Reading From imgcodecs.hpp:332-384: ```cpp theme={null} #include using namespace cv; // Basic usage with default flags Mat img = imread("photo.jpg"); // Check if image was loaded if (img.empty()) { std::cerr << "Error: Could not load image" << std::endl; return -1; } // Read as grayscale Mat gray = imread("photo.jpg", IMREAD_GRAYSCALE); // Read with alpha channel preserved Mat rgba = imread("image.png", IMREAD_UNCHANGED); ``` ### Image Read Flags From imgcodecs.hpp:69-85: ```cpp theme={null} enum ImreadModes { IMREAD_UNCHANGED = -1, // Load as-is, including alpha IMREAD_GRAYSCALE = 0, // Convert to grayscale IMREAD_COLOR_BGR = 1, // Convert to 3-channel BGR (default) IMREAD_COLOR = 1, // Same as IMREAD_COLOR_BGR IMREAD_ANYDEPTH = 2, // Load 16-bit/32-bit when available IMREAD_ANYCOLOR = 4, // Read in any color format IMREAD_LOAD_GDAL = 8, // Use GDAL driver IMREAD_REDUCED_GRAYSCALE_2 = 16, // Grayscale at 1/2 size IMREAD_REDUCED_COLOR_2 = 17, // Color at 1/2 size IMREAD_REDUCED_GRAYSCALE_4 = 32, // Grayscale at 1/4 size IMREAD_REDUCED_COLOR_4 = 33, // Color at 1/4 size IMREAD_REDUCED_GRAYSCALE_8 = 64, // Grayscale at 1/8 size IMREAD_REDUCED_COLOR_8 = 65, // Color at 1/8 size IMREAD_IGNORE_ORIENTATION = 128, // Ignore EXIF orientation IMREAD_COLOR_RGB = 256 // Convert to RGB instead of BGR }; // Combine flags with bitwise OR Mat img = imread("hdr.tiff", IMREAD_ANYDEPTH | IMREAD_ANYCOLOR); ``` ### Reading with Metadata From imgcodecs.hpp:397-412: ```cpp theme={null} // Read image with metadata std::vector metadataTypes; std::vector metadata; Mat img = imreadWithMetadata("photo.jpg", metadataTypes, metadata, IMREAD_ANYCOLOR); // Check what metadata was found for (size_t i = 0; i < metadataTypes.size(); i++) { switch(metadataTypes[i]) { case IMAGE_METADATA_EXIF: std::cout << "Found EXIF data" << std::endl; break; case IMAGE_METADATA_XMP: std::cout << "Found XMP data" << std::endl; break; case IMAGE_METADATA_ICCP: std::cout << "Found ICC Profile" << std::endl; break; } } ``` ## Supported Formats From imgcodecs.hpp:340-355, OpenCV supports: ### Always Available * **BMP** - Windows Bitmap (\*.bmp, \*.dib) * **GIF** - Graphics Interchange Format (\*.gif) * **PBM/PGM/PPM/PXM/PNM** - Portable image formats * **Sun Rasters** - (\*.sr, \*.ras) * **HDR** - Radiance HDR (\*.hdr, \*.pic) ### With External Libraries * **JPEG** - (\*.jpeg, \*.jpg, \*.jpe) - requires libjpeg * **JPEG 2000** - (\*.jp2) - requires libjasper/OpenJPEG * **PNG** - (\*.png) - requires libpng * **WebP** - (\*.webp) - requires libwebp * **AVIF** - (\*.avif) - requires libavif * **TIFF** - (\*.tiff, \*.tif) - requires libtiff * **OpenEXR** - (\*.exr) - requires OpenEXR * **PFM** - Portable Float Map (\*.pfm) * **JPEG XL** - (\*.jxl) - requires libjxl **Format Detection:** OpenCV determines the image format by content, not by file extension. The extension is only used when writing images. ## Writing Images ### Basic Image Writing From imgcodecs.hpp:511-569: ```cpp theme={null} // Basic usage Mat img = imread("input.jpg"); bool success = imwrite("output.png", img); if (!success) { std::cerr << "Error: Could not save image" << std::endl; } // Write with compression parameters vector compression_params; compression_params.push_back(IMWRITE_JPEG_QUALITY); compression_params.push_back(95); // Quality 0-100 imwrite("output.jpg", img, compression_params); ``` ### Write Parameters From imgcodecs.hpp:88-130, format-specific parameters: ```cpp theme={null} // JPEG parameters vector jpeg_params; jpeg_params.push_back(IMWRITE_JPEG_QUALITY); jpeg_params.push_back(95); // Quality: 0-100 (default 95) jpeg_params.push_back(IMWRITE_JPEG_PROGRESSIVE); jpeg_params.push_back(1); // Enable progressive JPEG jpeg_params.push_back(IMWRITE_JPEG_OPTIMIZE); jpeg_params.push_back(1); // Optimize encoding imwrite("output.jpg", img, jpeg_params); // PNG parameters vector png_params; png_params.push_back(IMWRITE_PNG_COMPRESSION); png_params.push_back(9); // Compression: 0-9 (default 1) png_params.push_back(IMWRITE_PNG_STRATEGY); png_params.push_back(IMWRITE_PNG_STRATEGY_DEFAULT); imwrite("output.png", img, png_params); // WebP parameters vector webp_params; webp_params.push_back(IMWRITE_WEBP_QUALITY); webp_params.push_back(90); // Quality: 1-100 imwrite("output.webp", img, webp_params); // AVIF parameters vector avif_params; avif_params.push_back(IMWRITE_AVIF_QUALITY); avif_params.push_back(95); // Quality: 0-100 avif_params.push_back(IMWRITE_AVIF_SPEED); avif_params.push_back(9); // Speed: 0 (slow) to 10 (fast) imwrite("output.avif", img, avif_params); // TIFF parameters vector tiff_params; tiff_params.push_back(IMWRITE_TIFF_COMPRESSION); tiff_params.push_back(IMWRITE_TIFF_COMPRESSION_LZW); imwrite("output.tiff", img, tiff_params); ``` ### Saving Images with Alpha Channel From imgcodecs.hpp:533-536: ```cpp theme={null} // Create 4-channel BGRA image Mat bgra(height, width, CV_8UC4); // Fill with color and alpha for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { bgra.at(y, x) = Vec4b( b_value, // Blue g_value, // Green r_value, // Red alpha_value // Alpha (0=transparent, 255=opaque) ); } } // Save as PNG with alpha imwrite("transparent.png", bgra); ``` ## Multi-Page Images From imgcodecs.hpp:414-434: ```cpp theme={null} // Read all pages from multi-page TIFF vector pages; bool success = imreadmulti("document.tiff", pages); std::cout << "Loaded " << pages.size() << " pages" << std::endl; // Read specific range of pages vector selectedPages; int startPage = 2; int pageCount = 5; imreadmulti("document.tiff", selectedPages, startPage, pageCount); // Write multiple images as TIFF vector images; // ... fill images vector ... imwrite("output.tiff", images); // Count images in file size_t count = imcount("animation.avif"); ``` ## Animation Support From imgcodecs.hpp:290-330, OpenCV supports animated images: ### Reading Animations ```cpp theme={null} // Animation structure (imgcodecs.hpp:294-330) Animation anim; // Load entire animation if (imreadanimation("animated.gif", anim)) { std::cout << "Frames: " << anim.frames.size() << std::endl; std::cout << "Loop count: " << anim.loop_count << std::endl; // Process each frame for (size_t i = 0; i < anim.frames.size(); i++) { Mat frame = anim.frames[i]; int duration = anim.durations[i]; // milliseconds // Process frame... } } // Load specific range of frames Animation partialAnim; int startFrame = 10; int frameCount = 20; imreadanimation("video.avif", partialAnim, startFrame, frameCount); ``` ### Writing Animations ```cpp theme={null} // Create animation Animation anim; anim.loop_count = 0; // 0 = infinite loop anim.bgcolor = Scalar(255, 255, 255, 255); // BGRA background // Add frames for (int i = 0; i < numFrames; i++) { Mat frame = generateFrame(i); anim.frames.push_back(frame); anim.durations.push_back(100); // 100ms per frame } // Save as animated GIF vector gif_params; gif_params.push_back(IMWRITE_GIF_QUALITY); gif_params.push_back(5); // Quality 1-8 imwriteanimation("output.gif", anim, gif_params); // Save as animated AVIF vector avif_params; avif_params.push_back(IMWRITE_AVIF_QUALITY); avif_params.push_back(90); imwriteanimation("output.avif", anim, avif_params); ``` **GIF Frame Duration:** GIF durations must be multiples of 10ms due to format limitations. Values are automatically rounded down. ## Memory Buffer Operations From imgcodecs.hpp:595-694: ### Decoding from Memory ```cpp theme={null} // Decode from buffer vector buffer; // ... fill buffer with image data ... Mat img = imdecode(buffer, IMREAD_COLOR); // Decode with existing Mat (reuses memory) Mat output; imdecode(buffer, IMREAD_COLOR, &output); // Decode with metadata vector metadataTypes; vector metadata; Mat img = imdecodeWithMetadata(buffer, metadataTypes, metadata, IMREAD_ANYCOLOR); ``` ### Encoding to Memory ```cpp theme={null} // Encode to buffer vector buffer; vector params; params.push_back(IMWRITE_JPEG_QUALITY); params.push_back(90); bool success = imencode(".jpg", img, buffer, params); // Buffer now contains JPEG-encoded image std::cout << "Encoded size: " << buffer.size() << " bytes" << std::endl; // Can send over network, save to database, etc. ``` ## Format Capabilities ### Checking Format Support From imgcodecs.hpp:696-726: ```cpp theme={null} // Check if file can be read if (haveImageReader("photo.jpg")) { Mat img = imread("photo.jpg"); } // Check if format can be written if (haveImageWriter(".png")) { imwrite("output.png", img); } // Check by extension if (haveImageWriter(".avif")) { std::cout << "AVIF encoding is available" << std::endl; } ``` ### Image Collections From imgcodecs.hpp:728-758, iterate through multi-page images: ```cpp theme={null} // Create collection (lazy loading) ImageCollection collection("document.tiff"); // Iterate through pages for (auto it = collection.begin(); it != collection.end(); ++it) { Mat page = *it; // Process page... } // Random access (less efficient) Mat page5 = *collection.at(5); // Release cached pages to save memory collection.releaseCache(); ``` ## Practical Examples ### Convert Image Format ```cpp theme={null} #include int main() { // Read any format Mat img = imread("input.bmp", IMREAD_UNCHANGED); if (img.empty()) { std::cerr << "Failed to load image" << std::endl; return -1; } // Save as different format with quality settings vector params; params.push_back(IMWRITE_PNG_COMPRESSION); params.push_back(9); // Maximum compression if (imwrite("output.png", img, params)) { std::cout << "Conversion successful" << std::endl; } return 0; } ``` ### Batch Process Images ```cpp theme={null} #include #include #include namespace fs = std::filesystem; void processDirectory(const string& inputDir, const string& outputDir) { for (const auto& entry : fs::directory_iterator(inputDir)) { string path = entry.path().string(); // Check if file can be read if (!haveImageReader(path)) continue; // Load and process Mat img = imread(path, IMREAD_COLOR); if (img.empty()) continue; // Resize Mat resized; resize(img, resized, Size(800, 600)); // Save with JPEG compression string filename = entry.path().filename().string(); string outPath = outputDir + "/" + filename + ".jpg"; vector params = {IMWRITE_JPEG_QUALITY, 85}; imwrite(outPath, resized, params); } } ``` ### Create Thumbnail Grid ```cpp theme={null} #include #include Mat createThumbnailGrid(const vector& imagePaths, int thumbSize, int cols) { vector thumbs; // Load and resize images for (const auto& path : imagePaths) { Mat img = imread(path, IMREAD_COLOR); if (img.empty()) continue; Mat thumb; resize(img, thumb, Size(thumbSize, thumbSize)); thumbs.push_back(thumb); } // Calculate grid dimensions int rows = (thumbs.size() + cols - 1) / cols; Mat grid(rows * thumbSize, cols * thumbSize, CV_8UC3, Scalar(255, 255, 255)); // Place thumbnails for (size_t i = 0; i < thumbs.size(); i++) { int row = i / cols; int col = i % cols; Rect roi(col * thumbSize, row * thumbSize, thumbSize, thumbSize); thumbs[i].copyTo(grid(roi)); } return grid; } ``` ## Best Practices **Always Check Return Values:** ```cpp theme={null} Mat img = imread("photo.jpg"); if (img.empty()) { // Handle error - file might not exist or be corrupted } bool success = imwrite("output.png", img); if (!success) { // Handle error - disk full, permissions, invalid path } ``` **Choose the Right Format:** * **JPEG** - Best for photographs, lossy compression * **PNG** - Lossless, supports transparency, larger files * **WebP/AVIF** - Modern formats with better compression * **TIFF** - Multi-page, lossless, supports 16/32-bit * **EXR** - High dynamic range (HDR) images **Memory Considerations:** For large images or batch processing: ```cpp theme={null} // Read at reduced size Mat thumb = imread("large.jpg", IMREAD_REDUCED_COLOR_4); // Or use ImageCollection for multi-page ImageCollection col("huge.tiff"); for (auto it = col.begin(); it != col.end(); ++it) { processPage(*it); col.releaseCache(); // Free memory } ``` ## Color Channel Order **Important: BGR vs RGB** OpenCV uses **BGR** channel order by default (not RGB). When reading images: ```cpp theme={null} // Image is loaded as BGR Mat img = imread("photo.jpg"); // img[y][x] = [Blue, Green, Red] // To get RGB order Mat rgb = imread("photo.jpg", IMREAD_COLOR_RGB); // Or convert after loading cvtColor(img, rgb, COLOR_BGR2RGB); ``` ## Related Modules * [Core Module](/modules/core) - Mat data structure * [Image Processing](/modules/imgproc) - Process loaded images * [High-Level GUI](/modules/highgui) - Display images ## Source Reference Main header: `~/workspace/source/modules/imgcodecs/include/opencv2/imgcodecs.hpp` # Image Processing Module Source: https://opencv-opencv.mintlify.app/modules/imgproc Comprehensive image processing functions including filtering, transforms, color conversions, and feature detection The Image Processing (imgproc) module provides a comprehensive suite of image processing functions for filtering, geometric transformations, color space conversions, and feature detection. ## Overview From opencv2/imgproc.hpp:48-52: > This module offers a comprehensive suite of image processing functions, enabling tasks such as filtering, geometric transformations, color space conversions, histograms, structural analysis, and feature detection. Linear and non-linear image filtering operations Geometric transformations like resize, rotate, warp Conversions between BGR, HSV, Lab, and other formats Edge detection, corner detection, and shape analysis ## Image Filtering From imgproc.hpp:54-84, OpenCV provides various filtering operations: ### Linear Filters ```cpp theme={null} #include using namespace cv; // Gaussian blur GaussianBlur(src, dst, Size(5, 5), 1.5); // Box filter (average) boxFilter(src, dst, -1, Size(5, 5)); blur(src, dst, Size(5, 5)); // Median filter (non-linear) medianBlur(src, dst, 5); // Bilateral filter (edge-preserving) bilateralFilter(src, dst, 9, 75, 75); // Custom filter with kernel Mat kernel = (Mat_(3,3) << -1, -1, -1, -1, 9, -1, -1, -1, -1); filter2D(src, dst, -1, kernel); ``` ### Morphological Operations From imgproc.hpp:216-241: ```cpp theme={null} // Morphological operation types enum MorphTypes { MORPH_ERODE = 0, // Erosion MORPH_DILATE = 1, // Dilation MORPH_OPEN = 2, // Opening: dilate(erode(src)) MORPH_CLOSE = 3, // Closing: erode(dilate(src)) MORPH_GRADIENT = 4, // Morphological gradient MORPH_TOPHAT = 5, // Top hat MORPH_BLACKHAT = 6, // Black hat MORPH_HITMISS = 7 // Hit-or-miss }; // Create structuring element Mat element = getStructuringElement( MORPH_RECT, // Shape: RECT, CROSS, ELLIPSE Size(5, 5), // Size Point(-1, -1) // Anchor point ); // Apply morphological operations erode(src, dst, element); dilate(src, dst, element); morphologyEx(src, dst, MORPH_OPEN, element); morphologyEx(src, dst, MORPH_CLOSE, element); ``` ### Derivatives and Gradients ```cpp theme={null} // Sobel derivatives Sobel(src, dst, CV_16S, 1, 0); // x-derivative Sobel(src, dst, CV_16S, 0, 1); // y-derivative // Scharr (more accurate for 3x3) Scharr(src, dx, CV_16S, 1, 0); Scharr(src, dy, CV_16S, 0, 1); // Laplacian Laplacian(src, dst, CV_16S, 3); // Canny edge detector Canny(src, edges, 50, 150, 3); ``` ## Geometric Transformations From imgproc.hpp:90-127, geometric transformations deform the pixel grid: ### Resizing and Interpolation ```cpp theme={null} // Interpolation flags (imgproc.hpp:249-280) enum InterpolationFlags { INTER_NEAREST = 0, // Nearest neighbor INTER_LINEAR = 1, // Bilinear interpolation INTER_CUBIC = 2, // Bicubic interpolation INTER_AREA = 3, // Area interpolation (best for decimation) INTER_LANCZOS4 = 4, // Lanczos interpolation over 8x8 INTER_LINEAR_EXACT = 5, // Bit exact bilinear INTER_NEAREST_EXACT = 6 // Bit exact nearest neighbor }; // Resize image resize(src, dst, Size(640, 480), 0, 0, INTER_LINEAR); resize(src, dst, Size(), 0.5, 0.5, INTER_AREA); // Scale by 0.5 // Pyramid operations pyrDown(src, dst); // Downsample pyrUp(src, dst); // Upsample ``` ### Affine Transformations ```cpp theme={null} // Rotation Point2f center(width/2.0, height/2.0); Mat rotMat = getRotationMatrix2D(center, 45, 1.0); // 45° rotation warpAffine(src, dst, rotMat, src.size()); // Translation Mat transMat = (Mat_(2,3) << 1, 0, 50, // tx=50 0, 1, 30); // ty=30 warpAffine(src, dst, transMat, src.size()); // Affine from 3 points Point2f srcPts[3], dstPts[3]; // ... set points ... Mat affineMat = getAffineTransform(srcPts, dstPts); warpAffine(src, dst, affineMat, src.size()); ``` ### Perspective Transformations ```cpp theme={null} // Perspective transform from 4 points Point2f srcQuad[4], dstQuad[4]; // ... define source and destination points ... Mat perspMat = getPerspectiveTransform(srcQuad, dstQuad); warpPerspective(src, dst, perspMat, dst.size()); // Find homography from point correspondences vector srcPoints, dstPoints; // ... fill points ... Mat H = findHomography(srcPoints, dstPoints, RANSAC); warpPerspective(src, dst, H, dst.size()); ``` ### Remapping ```cpp theme={null} // General remapping Mat mapX, mapY; // ... create mapping functions ... remap(src, dst, mapX, mapY, INTER_LINEAR); // Polar transformations linearPolar(src, dst, center, maxRadius, INTER_LINEAR); logPolar(src, dst, center, M, INTER_LINEAR); ``` ## Color Space Conversions From imgproc.hpp:158-172: ```cpp theme={null} // Common color conversions cvtColor(src, dst, COLOR_BGR2GRAY); // BGR to Grayscale cvtColor(src, dst, COLOR_BGR2HSV); // BGR to HSV cvtColor(src, dst, COLOR_BGR2Lab); // BGR to Lab cvtColor(src, dst, COLOR_BGR2YCrCb); // BGR to YCrCb cvtColor(src, dst, COLOR_GRAY2BGR); // Grayscale to BGR cvtColor(src, dst, COLOR_HSV2BGR); // HSV to BGR // RGB vs BGR cvtColor(src, dst, COLOR_BGR2RGB); // Swap R and B channels // Alpha channel cvtColor(src, dst, COLOR_BGR2BGRA); // Add alpha channel cvtColor(src, dst, COLOR_BGRA2BGR); // Remove alpha channel ``` ## Thresholding ```cpp theme={null} // Simple thresholding threshold(src, dst, 127, 255, THRESH_BINARY); threshold(src, dst, 127, 255, THRESH_BINARY_INV); threshold(src, dst, 127, 255, THRESH_TRUNC); threshold(src, dst, 127, 255, THRESH_TOZERO); // Otsu's method (automatic threshold) threshold(src, dst, 0, 255, THRESH_BINARY | THRESH_OTSU); // Adaptive thresholding adaptiveThreshold(src, dst, 255, ADAPTIVE_THRESH_MEAN_C, THRESH_BINARY, 11, 2); adaptiveThreshold(src, dst, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2); ``` ## Edge Detection Example from samples/cpp/edge.cpp: ```cpp theme={null} #include #include Mat image, gray, blurImage, edges; // Load and convert to grayscale image = imread("image.jpg"); cvtColor(image, gray, COLOR_BGR2GRAY); // Blur to reduce noise (edge.cpp:22) blur(gray, blurImage, Size(3, 3)); // Canny edge detection (edge.cpp:25) int threshold1 = 50; int threshold2 = 150; Canny(blurImage, edges, threshold1, threshold2, 3); // Using Scharr gradient (edge.cpp:32-35) Mat dx, dy; Scharr(blurImage, dx, CV_16S, 1, 0); Scharr(blurImage, dy, CV_16S, 0, 1); Canny(dx, dy, edges, threshold1, threshold2); ``` ## Histograms ```cpp theme={null} // Calculate histogram Mat hist; int histSize = 256; float range[] = {0, 256}; const float* histRange = {range}; calcHist(&src, 1, 0, Mat(), hist, 1, &histSize, &histRange); // Histogram equalization equalizeHist(src, dst); // CLAHE (Contrast Limited Adaptive Histogram Equalization) Ptr clahe = createCLAHE(); clahe->setClipLimit(4.0); clahe->apply(src, dst); // Back projection Mat backProj; calcBackProject(&src, 1, channels, hist, backProj, ranges); ``` ## Contours and Shapes ```cpp theme={null} // Find contours vector> contours; vector hierarchy; findContours(binary, contours, hierarchy, RETR_TREE, CHAIN_APPROX_SIMPLE); // Draw contours Mat drawing = Mat::zeros(binary.size(), CV_8UC3); for (size_t i = 0; i < contours.size(); i++) { drawContours(drawing, contours, i, Scalar(0, 255, 0), 2); } // Contour properties double area = contourArea(contours[0]); double perimeter = arcLength(contours[0], true); // Approximate contour vector approx; approxPolyDP(contours[0], approx, epsilon, true); // Convex hull vector hull; convexHull(contours[0], hull); // Bounding shapes Rect bbox = boundingRect(contours[0]); RotatedRect minRect = minAreaRect(contours[0]); Point2f center; float radius; minEnclosingCircle(contours[0], center, radius); ``` ## Drawing Functions From imgproc.hpp:130-156: ```cpp theme={null} // Lines line(img, pt1, pt2, Scalar(0, 255, 0), 2); // Arrows arrowedLine(img, pt1, pt2, Scalar(255, 0, 0), 2); // Rectangles rectangle(img, pt1, pt2, Scalar(0, 0, 255), 2); rectangle(img, rect, Scalar(0, 255, 255), -1); // Filled // Circles circle(img, center, radius, Scalar(255, 255, 0), 2); // Ellipses ellipse(img, center, Size(100, 50), 45, 0, 360, Scalar(255, 0, 255), 2); // Polygons vector pts = {Point(10,10), Point(100,50), Point(50,100)}; polylines(img, pts, true, Scalar(0, 255, 0), 2); fillPoly(img, pts, Scalar(255, 255, 255)); // Text putText(img, "OpenCV", Point(50, 50), FONT_HERSHEY_SIMPLEX, 1.0, Scalar(0, 0, 0), 2); ``` ## Distance Transform ```cpp theme={null} // Distance to nearest zero pixel Mat dist; distanceTransform(binary, dist, DIST_L2, 5); // Distance transform with labels Mat labels; distanceTransform(binary, dist, labels, DIST_L2, 5, LABEL_CCOMP); ``` ## Connected Components ```cpp theme={null} // Label connected components Mat labels; int nLabels = connectedComponents(binary, labels); // With statistics Mat stats, centroids; int nLabels = connectedComponentsWithStats(binary, labels, stats, centroids); // Access statistics for (int i = 1; i < nLabels; i++) { int area = stats.at(i, CC_STAT_AREA); int left = stats.at(i, CC_STAT_LEFT); int top = stats.at(i, CC_STAT_TOP); int width = stats.at(i, CC_STAT_WIDTH); int height = stats.at(i, CC_STAT_HEIGHT); double cx = centroids.at(i, 0); double cy = centroids.at(i, 1); } ``` ## Image Moments ```cpp theme={null} // Calculate moments Moments m = moments(contour); // Centroid double cx = m.m10 / m.m00; double cy = m.m01 / m.m00; // Hu moments (rotation invariant) double hu[7]; HuMoments(m, hu); ``` ## Hough Transforms ```cpp theme={null} // Hough Line Transform vector lines; HoughLines(edges, lines, 1, CV_PI/180, 100); // Probabilistic Hough Line Transform vector linesP; HoughLinesP(edges, linesP, 1, CV_PI/180, 50, 50, 10); // Hough Circle Transform vector circles; HoughCircles(gray, circles, HOUGH_GRADIENT, 1, 50, 200, 100, 0, 0); ``` ## Watershed Segmentation ```cpp theme={null} // Prepare markers Mat markers; // ... initialize markers ... // Apply watershed watershed(image, markers); // Markers now contain segment labels // -1 indicates boundaries between segments ``` ## Template Matching ```cpp theme={null} // Match template Mat result; matchTemplate(image, templ, result, TM_CCOEFF_NORMED); // Find best match double minVal, maxVal; Point minLoc, maxLoc; minMaxLoc(result, &minVal, &maxVal, &minLoc, &maxLoc); // Draw rectangle at match location rectangle(image, maxLoc, Point(maxLoc.x + templ.cols, maxLoc.y + templ.rows), Scalar(0, 255, 0), 2); ``` ## Practical Example: Image Enhancement ```cpp theme={null} #include #include #include using namespace cv; int main() { // Load image Mat img = imread("photo.jpg"); Mat result; // Denoise fastNlMeansDenoisingColored(img, result, 10, 10, 7, 21); // Convert to Lab for better color processing Mat lab; cvtColor(result, lab, COLOR_BGR2Lab); // Split channels vector channels; split(lab, channels); // Apply CLAHE to L channel Ptr clahe = createCLAHE(2.0, Size(8, 8)); clahe->apply(channels[0], channels[0]); // Merge and convert back merge(channels, lab); cvtColor(lab, result, COLOR_Lab2BGR); // Sharpen Mat blurred; GaussianBlur(result, blurred, Size(0, 0), 3); addWeighted(result, 1.5, blurred, -0.5, 0, result); // Save result imwrite("enhanced.jpg", result); return 0; } ``` ## Best Practices **Border Handling:** Most filtering functions need to extrapolate pixels outside image boundaries. Choose the appropriate border type: * `BORDER_REPLICATE` - Good for most filtering * `BORDER_REFLECT_101` - Better for derivatives * `BORDER_CONSTANT` - When you need specific padding values **Interpolation:** Choose interpolation based on your needs: * `INTER_NEAREST` - Fastest, but lowest quality * `INTER_LINEAR` - Good balance of speed and quality * `INTER_AREA` - Best for downsampling * `INTER_CUBIC` - Best quality for upsampling * `INTER_LANCZOS4` - Highest quality, slowest ## Related Modules * [Core Module](/modules/core) - Provides Mat and basic operations * [Image I/O](/modules/imgcodecs) - Reading and writing images * [Feature Detection](/modules/features2d) - Advanced feature detection ## Source Reference Key header: `~/workspace/source/modules/imgproc/include/opencv2/imgproc.hpp` # Machine Learning Module Source: https://opencv-opencv.mintlify.app/modules/ml Statistical classification, regression, and clustering algorithms in OpenCV ## Overview The ML (Machine Learning) module provides classical machine learning algorithms for: * Classification * Regression * Clustering * Statistical modeling This module implements **traditional ML algorithms**. For deep learning, see the [DNN Module](/modules/dnn). ## Key Concepts ### StatModel Base Class All ML algorithms inherit from `StatModel`: ```cpp theme={null} class StatModel : public Algorithm { public: // Train the model virtual bool train(const Ptr& trainData, int flags=0); // Predict on new data virtual float predict(InputArray samples, OutputArray results=noArray(), int flags=0) const = 0; // Calculate error virtual float calcError(const Ptr& data, bool test, OutputArray resp) const; }; ``` ### TrainData Class Encapsulates training data: ```cpp theme={null} Ptr data = TrainData::create( samples, // Training samples (CV_32F) ROW_SAMPLE, // Each row is a sample responses // Response values ); ``` ## Classification Algorithms ### Support Vector Machines (SVM) ```cpp theme={null} // Create SVM Ptr svm = SVM::create(); svm->setType(SVM::C_SVC); svm->setKernel(SVM::LINEAR); svm->setC(1.0); // Train Ptr data = TrainData::create(samples, ROW_SAMPLE, labels); svm->train(data); // Predict float response = svm->predict(testSample); ``` **SVM Types**: * `C_SVC`: C-Support Vector Classification * `NU_SVC`: Nu-Support Vector Classification * `ONE_CLASS`: One-class SVM * `EPS_SVR`: Epsilon-Support Vector Regression * `NU_SVR`: Nu-Support Vector Regression **Kernel Types**: * `LINEAR`: Linear kernel * `POLY`: Polynomial kernel * `RBF`: Radial Basis Function (Gaussian) * `SIGMOID`: Sigmoid kernel ### K-Nearest Neighbors (KNN) ```cpp theme={null} Ptr knn = KNearest::create(); knn->setDefaultK(3); knn->setIsClassifier(true); knn->setAlgorithmType(KNearest::BRUTE_FORCE); // Train knn->train(data); // Find k nearest neighbors Mat results, neighborResponses, dists; knn->findNearest(testSample, 5, results, neighborResponses, dists); ``` ### Decision Trees ```cpp theme={null} Ptr dtree = DTrees::create(); dtree->setMaxDepth(10); dtree->setMinSampleCount(2); dtree->setUseSurrogates(false); // Train dtree->train(data); // Predict float prediction = dtree->predict(testSample); // Get tree structure std::vector nodes = dtree->getNodes(); ``` ### Random Forest ```cpp theme={null} Ptr rtrees = RTrees::create(); rtrees->setMaxDepth(10); rtrees->setMinSampleCount(2); rtrees->setActiveVarCount(4); // Features per split rtrees->setTermCriteria( TermCriteria(TermCriteria::MAX_ITER, 100, 0) ); // Train rtrees->train(data); // Predict float response = rtrees->predict(testSample); // Variable importance Mat varImportance = rtrees->getVarImportance(); ``` ### Naive Bayes ```cpp theme={null} Ptr bayes = NormalBayesClassifier::create(); // Train bayes->train(data); // Predict with probabilities Mat outputs, probs; bayes->predictProb(testSamples, outputs, probs); ``` ### Logistic Regression ```cpp theme={null} Ptr lr = LogisticRegression::create(); lr->setLearningRate(0.001); lr->setIterations(1000); lr->setRegularization(LogisticRegression::REG_L2); lr->setTrainMethod(LogisticRegression::BATCH); // Train lr->train(data); // Predict Mat predictions; lr->predict(testSamples, predictions); ``` ## Neural Networks ### ANN\_MLP (Multi-Layer Perceptron) ```cpp theme={null} Ptr ann = ANN_MLP::create(); // Define network structure Mat layers = (Mat_(1, 4) << 784, 128, 64, 10); ann->setLayerSizes(layers); // Set parameters ann->setActivationFunction(ANN_MLP::SIGMOID_SYM); ann->setTrainMethod(ANN_MLP::BACKPROP); ann->setBackpropWeightScale(0.1); ann->setBackpropMomentumScale(0.1); // Set termination criteria TermCriteria criteria( TermCriteria::MAX_ITER + TermCriteria::EPS, 1000, // Max iterations 0.01 // Min error ); ann->setTermCriteria(criteria); // Train ann->train(data); // Predict Mat output; ann->predict(testSample, output); ``` ## Clustering ### K-Means ```cpp theme={null} // K-Means clustering Mat labels, centers; int K = 3; kmeans( data, // Input samples K, // Number of clusters labels, // Output labels TermCriteria(TermCriteria::EPS + TermCriteria::MAX_ITER, 100, 0.01), 3, // Attempts KMEANS_PP_CENTERS, // Initialization method centers // Output centers ); // Visualize clusters for(int i = 0; i < data.rows; i++) { int cluster = labels.at(i); circle(img, Point(data.at(i,0), data.at(i,1)), 5, clusterColors[cluster], -1); } ``` ### EM (Expectation Maximization) ```cpp theme={null} Ptr em = EM::create(); em->setClustersNumber(3); em->setCovarianceMatrixType(EM::COV_MAT_DIAGONAL); // Train em->trainEM(samples); // Predict cluster Vec2d probs; int cluster = em->predict2(sample, probs)[1]; ``` ## Complete Example: SVM Classification ```cpp theme={null} #include #include using namespace cv; using namespace cv::ml; int main() { // Generate training data int numSamples = 100; Mat samples(numSamples, 2, CV_32F); Mat labels(numSamples, 1, CV_32S); // Class 1 for(int i = 0; i < numSamples/2; i++) { samples.at(i, 0) = randn(2.0, 1.0); samples.at(i, 1) = randn(2.0, 1.0); labels.at(i) = 0; } // Class 2 for(int i = numSamples/2; i < numSamples; i++) { samples.at(i, 0) = randn(6.0, 1.0); samples.at(i, 1) = randn(6.0, 1.0); labels.at(i) = 1; } // Create and train SVM Ptr svm = SVM::create(); svm->setType(SVM::C_SVC); svm->setKernel(SVM::RBF); svm->setGamma(0.5); svm->setC(1.0); Ptr data = TrainData::create( samples, ROW_SAMPLE, labels ); svm->train(data); // Test Mat testSample = (Mat_(1, 2) << 3.0, 3.0); float response = svm->predict(testSample); std::cout << "Predicted class: " << response << std::endl; // Save model svm->save("svm_model.xml"); return 0; } ``` ## Model Persistence ### Save Model ```cpp theme={null} // Save to file svm->save("model.xml"); svm->save("model.yml"); ``` ### Load Model ```cpp theme={null} // Load from file Ptr svm = SVM::load("model.xml"); // Use loaded model float prediction = svm->predict(sample); ``` ## Cross-Validation ```cpp theme={null} // Split data Ptr data = TrainData::create( samples, ROW_SAMPLE, responses ); data->setTrainTestSplitRatio(0.8, true); // Train on training set Ptr svm = SVM::create(); svm->train(data->getTrainSamples()); // Evaluate on test set float error = svm->calcError(data, true, noArray()); std::cout << "Test error: " << error << "%\n"; ``` ## Algorithm Selection Guide | Algorithm | Type | Pros | Cons | Best For | | ----------------- | ------------------------- | ---------------------------- | --------------------------------- | -------------------- | | **SVM** | Classification/Regression | Effective in high dimensions | Slow on large datasets | Small to medium data | | **KNN** | Classification/Regression | Simple, no training | Slow prediction, memory intensive | Small datasets | | **Random Forest** | Classification/Regression | Robust, handles non-linear | Can overfit | General purpose | | **Naive Bayes** | Classification | Fast, simple | Assumes independence | Text classification | | **ANN\_MLP** | Classification/Regression | Powerful | Needs tuning | Complex patterns | | **K-Means** | Clustering | Fast, simple | Needs K specified | Data segmentation | ## Best Practices Scale features to similar ranges for better performance Use train/test split to avoid overfitting Use grid search for optimal hyperparameters Persist trained models for reuse ## See Also * [DNN Module](/modules/dnn) - Deep learning * [Core Module](/modules/core) - Matrix operations * [ML Tutorial](https://docs.opencv.org/master/d1/d73/tutorial_introduction_to_svm.html) # Object Detection Module Source: https://opencv-opencv.mintlify.app/modules/objdetect Object detection including cascade classifiers, HOG, and QR code detection ## Overview The `objdetect` module provides tools for detecting objects in images, including: * Cascade classifiers (Haar, LBP) * HOG (Histogram of Oriented Gradients) detector * QR code and barcode detection * ArUco marker detection * Face detection ## Cascade Classifier ### Overview Cascade classifiers detect objects using Haar-like or LBP features trained with boosting algorithms. ### Basic Usage ```cpp theme={null} #include // Load pre-trained classifier CascadeClassifier face_cascade; face_cascade.load("haarcascade_frontalface_default.xml"); // Detect objects std::vector faces; Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); face_cascade.detectMultiScale( gray, faces, 1.1, // scaleFactor 3, // minNeighbors 0, // flags Size(30, 30) // minSize ); // Draw results for(const Rect& face : faces) { rectangle(img, face, Scalar(0, 255, 0), 2); } ``` ### Parameters * **scaleFactor**: Image pyramid scale (typically 1.05-1.4) * **minNeighbors**: Minimum neighbors for detection (3-6 typical) * **minSize/maxSize**: Size constraints for detected objects ### Pre-trained Models OpenCV includes cascades for: * Face detection (frontal, profile) * Eye detection * Full body detection * Upper body detection * License plate detection ## HOG Descriptor ### Overview Histogram of Oriented Gradients (HOG) is a feature descriptor used for object detection, particularly for pedestrian detection. ### Structure ```cpp theme={null} struct HOGDescriptor { Size winSize; // Detection window (64x128 default) Size blockSize; // Block size (16x16) Size blockStride; // Block stride (8x8) Size cellSize; // Cell size (8x8) int nbins; // Number of bins (9) double winSigma; // Gaussian smoothing double L2HysThreshold; // Normalization threshold bool gammaCorrection; // Gamma correction flag }; ``` ### Pedestrian Detection ```cpp theme={null} // Create HOG detector HOGDescriptor hog; hog.setSVMDetector(HOGDescriptor::getDefaultPeopleDetector()); // Detect std::vector found; std::vector weights; hog.detectMultiScale( img, found, weights, 0, // hitThreshold Size(8, 8), // winStride Size(32, 32), // padding 1.05, // scale 2.0 // groupThreshold ); // Draw detections for(const Rect& r : found) { rectangle(img, r, Scalar(0, 255, 0), 2); } ``` ### Custom Training ```cpp theme={null} // Compute HOG descriptors HOGDescriptor hog; std::vector descriptors; hog.compute(img, descriptors); // Train with SVM (external) // ... // Set trained detector hog.setSVMDetector(trained_descriptors); ``` ## QR Code Detection ### QRCodeDetector ```cpp theme={null} QRCodeDetector qrDecoder; // Detect and decode std::vector points; String data = qrDecoder.detectAndDecode(img, points); if(!data.empty()) { std::cout << "QR Code: " << data << std::endl; // Draw boundary for(size_t i = 0; i < points.size(); i++) { line(img, points[i], points[(i+1) % points.size()], Scalar(0, 255, 0), 2); } } ``` ### Multiple QR Codes ```cpp theme={null} std::vector decoded_info; std::vector> points; if(qrDecoder.detectAndDecodeMulti(img, decoded_info, points)) { for(size_t i = 0; i < decoded_info.size(); i++) { std::cout << "QR " << i << ": " << decoded_info[i] << std::endl; } } ``` ## ArUco Marker Detection ### Basic Detection ```cpp theme={null} #include // Create dictionary and detector aruco::Dictionary dictionary = aruco::getPredefinedDictionary(aruco::DICT_6X6_250); aruco::DetectorParameters params; aruco::ArucoDetector detector(dictionary, params); // Detect markers std::vector ids; std::vector> corners, rejected; detector.detectMarkers(img, corners, ids, rejected); // Draw detected markers if(!ids.empty()) { aruco::drawDetectedMarkers(img, corners, ids); } ``` ## Face Detection ### Modern DNN-based Detection ```cpp theme={null} #include // Load face detector model Ptr detector = FaceDetectorYN::create( "face_detection_yunet_2023mar.onnx", "", Size(320, 320) ); // Set input size detector->setInputSize(img.size()); // Detect faces Mat faces; detector->detect(img, faces); // Process results for(int i = 0; i < faces.rows; i++) { float confidence = faces.at(i, 14); if(confidence > 0.9) { int x = faces.at(i, 0); int y = faces.at(i, 1); int w = faces.at(i, 2); int h = faces.at(i, 3); rectangle(img, Rect(x, y, w, h), Scalar(0, 255, 0), 2); } } ``` ## Complete Example: Face Detection ```cpp theme={null} #include #include #include int main() { // Load image Mat img = imread("people.jpg"); // Load cascade CascadeClassifier face_cascade; if(!face_cascade.load("haarcascade_frontalface_default.xml")) { std::cerr << "Error loading cascade\n"; return -1; } // Convert to grayscale Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); equalizeHist(gray, gray); // Detect faces std::vector faces; face_cascade.detectMultiScale( gray, faces, 1.1, 3, 0, Size(30, 30) ); // Draw rectangles for(const Rect& face : faces) { rectangle(img, face, Scalar(255, 0, 0), 2); } // Display imshow("Faces", img); waitKey(0); return 0; } ``` ## Performance Tips Convert to grayscale before detection Resize large images for faster processing Limit detection to region of interest Tune scaleFactor and minNeighbors for speed/accuracy ## Algorithm Selection | Method | Speed | Accuracy | Use Case | | --------------- | ------ | -------- | --------------------- | | **Cascade** | Fast | Good | Real-time face/object | | **HOG** | Medium | Good | Pedestrian detection | | **DNN** | Slow | Best | High accuracy needed | | **QR Detector** | Fast | High | QR/Barcode scanning | ## Best Practices ### Cascade Classifiers 1. **Preprocess images**: Equalize histogram, reduce noise 2. **Adjust minNeighbors**: Higher = fewer false positives 3. **Set size constraints**: Filter by expected object size 4. **Use appropriate cascade**: frontal vs profile faces ### HOG Detector 1. **Standard window**: Use 64x128 for pedestrians 2. **Multi-scale detection**: Essential for varying sizes 3. **Non-maximum suppression**: Remove overlapping detections 4. **GPU acceleration**: Use cv::cuda::HOG for speed ## See Also * [DNN Module](/modules/dnn) - Deep learning based detection * [Features2D](/modules/features2d) - Feature detection * [Face Recognition Tutorial](https://docs.opencv.org/master/da/d60/tutorial_face_main.html) # OpenCV Modules Overview Source: https://opencv-opencv.mintlify.app/modules/overview Comprehensive overview of OpenCV's modular architecture and core functionality modules OpenCV is organized into a set of modules, each providing specific functionality for computer vision tasks. Understanding this modular structure helps you choose the right tools for your applications. ## Core Modules Fundamental data structures (Mat), basic operations, and utilities Image processing functions including filtering, transforms, and feature detection Image file reading and writing with support for multiple formats Video capture and writing interfaces for cameras and video files High-level GUI functions for window management and user interaction Video analysis including motion tracking and background subtraction Camera calibration and 3D reconstruction algorithms ## Module Dependencies OpenCV modules are designed with clear dependencies to maintain modularity: ```mermaid theme={null} graph TD core[core] imgproc[imgproc] imgcodecs[imgcodecs] videoio[videoio] highgui[highgui] video[video] calib3d[calib3d] imgproc --> core imgcodecs --> core videoio --> core highgui --> core highgui -.-> imgcodecs highgui -.-> videoio video --> core video --> imgproc calib3d --> core calib3d --> imgproc ``` The **core** module is the foundation - all other modules depend on it for basic data structures and operations. ## Module Descriptions ### Core Module The backbone of OpenCV providing: * **Mat** class for n-dimensional arrays * Basic array operations (add, subtract, multiply) * Mathematical functions * XML/YAML persistence * Utility functions and system information ### Image Processing (imgproc) Comprehensive image processing capabilities: * Linear and non-linear filtering * Geometric transformations * Color space conversions * Histograms * Structural analysis and shape descriptors * Motion analysis and object tracking * Feature detection ### Image Codecs (imgcodecs) Image I/O operations: * Support for JPEG, PNG, TIFF, WebP, AVIF, and more * Image reading with `imread()` * Image writing with `imwrite()` * Animation support (GIF, AVIF, APNG) * Metadata handling (EXIF, XMP, ICC) ### Video I/O (videoio) Video capture and writing: * **VideoCapture** class for reading from cameras or files * **VideoWriter** class for creating video files * Multiple backend support (FFmpeg, GStreamer, DirectShow) * Audio stream support * Hardware acceleration options ### High-Level GUI (highgui) User interface utilities: * Window creation and management * Image display with `imshow()` * Keyboard and mouse event handling * Trackbars for interactive parameter adjustment * OpenGL integration support ### Video Analysis (video) Advanced video processing: * Optical flow (Lucas-Kanade, Farneback) * Object tracking (MeanShift, CamShift) * Background subtraction (MOG2, KNN) * Motion analysis algorithms ### Camera Calibration (calib3d) 3D vision and calibration: * Camera calibration (intrinsic and extrinsic parameters) * Stereo calibration and rectification * 3D reconstruction * Pose estimation * Homography computation ## Choosing the Right Module Use **imgproc** for: * Filtering and smoothing images * Edge detection * Color transformations * Geometric transformations (resize, rotate) * Contour detection Use **video** and **videoio** for: * Reading from cameras or video files * Tracking moving objects * Detecting motion * Background/foreground segmentation * Writing processed video Use **calib3d** for: * Camera calibration * Stereo vision * 3D reconstruction * AR/VR applications * Depth estimation Use **imgcodecs** and **videoio** for: * Loading and saving images * Capturing from cameras * Recording video * Format conversion ## Getting Started To use OpenCV modules in your code: ```cpp theme={null} #include // Core functionality #include // Image processing #include // Image I/O #include // GUI functions #include // Video I/O #include // Video analysis #include // Calibration and 3D using namespace cv; ``` ## Next Steps Learn about Mat and fundamental operations Discover filtering and transformation functions Build your first OpenCV application Browse the complete API documentation # Computational Photography Module Source: https://opencv-opencv.mintlify.app/modules/photo Image inpainting, denoising, HDR imaging, and seamless cloning ## Overview The Photo module provides advanced computational photography algorithms: * Image inpainting * Denoising * HDR (High Dynamic Range) imaging * Seamless cloning * Non-photorealistic rendering ## Image Inpainting ### Overview Inpainting restores missing or damaged regions in images using information from surrounding areas. ### Basic Inpainting ```cpp theme={null} #include // Load image and mask Mat img = imread("damaged.jpg"); Mat mask = imread("mask.jpg", IMREAD_GRAYSCALE); // White pixels (255) indicate areas to inpaint Mat result; inpaint( img, // Source image mask, // Inpainting mask result, // Output 3.0, // Inpaint radius INPAINT_TELEA // Algorithm ); imwrite("restored.jpg", result); ``` ### Inpainting Methods **INPAINT\_NS** (Navier-Stokes): * Based on fluid dynamics * Better for textured regions **INPAINT\_TELEA**: * Fast Marching Method * Better for smooth regions ```cpp theme={null} // Try both methods inpaint(img, mask, result1, 3, INPAINT_NS); inpaint(img, mask, result2, 3, INPAINT_TELEA); ``` ## Image Denoising ### Non-Local Means Denoising #### Grayscale Images ```cpp theme={null} Mat noisy = imread("noisy.jpg", IMREAD_GRAYSCALE); Mat denoised; fastNlMeansDenoising( noisy, denoised, 3.0, // h: filter strength 7, // templateWindowSize 21 // searchWindowSize ); ``` #### Color Images ```cpp theme={null} Mat noisyColor = imread("noisy.jpg"); Mat denoisedColor; fastNlMeansDenoisingColored( noisyColor, denoisedColor, 3.0, // h: luminance filter strength 3.0, // hColor: color filter strength 7, // templateWindowSize 21 // searchWindowSize ); ``` ### Video Denoising ```cpp theme={null} // Denoise video frames std::vector frames; for(int i = 0; i < videoFrames.size(); i++) { frames.push_back(videoFrames[i]); } Mat denoised; fastNlMeansDenoisingMulti( frames, denoised, 2, // imgToDenoiseIndex (target frame) 5, // temporalWindowSize 3.0, // h 7, // templateWindowSize 21 // searchWindowSize ); ``` ### Parameters * **h**: Filter strength (3-10 typical) * Higher = more denoising, more blur * Lower = less denoising, preserves detail * **templateWindowSize**: Usually 7 * **searchWindowSize**: Usually 21 * Larger = better quality, slower ## HDR Imaging ### Capture HDR from Multiple Exposures ```cpp theme={null} // Load images with different exposures std::vector images; images.push_back(imread("exposure1.jpg")); images.push_back(imread("exposure2.jpg")); images.push_back(imread("exposure3.jpg")); // Exposure times (in seconds) std::vector times = {1.0/30, 1.0/15, 1.0/8}; // Merge to HDR Ptr merge = createMergeDebevec(); Mat hdr; merge->process(images, hdr, times); ``` ### Tone Mapping Convert HDR to displayable LDR (8-bit): ```cpp theme={null} // Drago tonemapping Ptr tonemap = createTonemapDrago( 1.0f, // gamma 1.0f, // saturation 0.85f // bias ); Mat ldr; tonemap->process(hdr, ldr); // Scale to 8-bit ldr = ldr * 255; ldr.convertTo(ldr, CV_8UC3); imwrite("tonemapped.jpg", ldr); ``` ### Tone Mapping Algorithms #### Reinhard ```cpp theme={null} Ptr tonemap = createTonemapReinhard( 1.0f, // gamma 0.0f, // intensity [-8, 8] 1.0f, // light_adapt [0, 1] 0.0f // color_adapt [0, 1] ); ``` #### Mantiuk ```cpp theme={null} Ptr tonemap = createTonemapMantiuk( 1.0f, // gamma 0.7f, // scale 1.0f // saturation ); ``` ### Exposure Alignment ```cpp theme={null} // Align images before merging Ptr align = createAlignMTB(); std::vector alignedImages; align->process(images, alignedImages); // Now merge aligned images Ptr merge = createMergeDebevec(); Mat hdr; merge->process(alignedImages, hdr, times); ``` ## Seamless Cloning ### Paste Object Seamlessly ```cpp theme={null} // Source: object to paste Mat src = imread("object.jpg"); // Destination: background Mat dst = imread("background.jpg"); // Mask: white region defines object Mat mask = imread("mask.jpg", IMREAD_GRAYSCALE); // Center point in destination Point center(dst.cols/2, dst.rows/2); Mat result; seamlessClone( src, dst, mask, center, result, NORMAL_CLONE // or MIXED_CLONE, MONOCHROME_TRANSFER ); imwrite("seamless.jpg", result); ``` ### Cloning Modes **NORMAL\_CLONE**: * Standard seamless cloning * Preserves source texture **MIXED\_CLONE**: * Mixes source and destination * Better for transparent objects **MONOCHROME\_TRANSFER**: * Transfer only colors * Preserves destination texture ### Illumination Change ```cpp theme={null} // Adjust lighting in specific region Mat mask = Mat::zeros(img.size(), CV_8U); circle(mask, Point(x, y), radius, 255, -1); Mat result; illuminationChange( img, mask, result, 0.9f, // alpha 0.1f // beta ); ``` ## Non-Photorealistic Rendering ### Edge Preserving Filter ```cpp theme={null} Mat filtered; edgePreservingFilter( img, filtered, RECURS_FILTER, // or NORMCONV_FILTER 60, // sigma_s 0.4 // sigma_r ); ``` ### Detail Enhancement ```cpp theme={null} Mat enhanced; detailEnhance( img, enhanced, 10, // sigma_s 0.15 // sigma_r ); ``` ### Pencil Sketch ```cpp theme={null} Mat sketch, colorSketch; pencilSketch( img, sketch, // Grayscale sketch colorSketch, // Color sketch 60, // sigma_s 0.07, // sigma_r 0.02 // shade_factor ); ``` ### Stylization ```cpp theme={null} Mat stylized; stylization( img, stylized, 60, // sigma_s 0.45 // sigma_r ); ``` ## Complete Example: HDR Processing ```cpp theme={null} #include #include using namespace cv; int main() { // Load exposure sequence std::vector images; std::vector times; images.push_back(imread("img_0.33.jpg")); images.push_back(imread("img_0.25.jpg")); images.push_back(imread("img_0.125.jpg")); times = {1.0/3, 1.0/4, 1.0/8}; // Align images Ptr align = createAlignMTB(); std::vector aligned; align->process(images, aligned); // Merge to HDR Ptr merge = createMergeDebevec(); Mat hdr; merge->process(aligned, hdr, times); // Save HDR imwrite("hdr.hdr", hdr); // Tone map for display Ptr tonemap = createTonemapDrago(1.0, 1.0, 0.85); Mat ldr; tonemap->process(hdr, ldr); // Convert to 8-bit ldr = 3 * ldr; ldr.convertTo(ldr, CV_8UC3, 255); imwrite("ldr.jpg", ldr); return 0; } ``` ## Performance Tips Downscale for faster denoising/HDR Smaller search windows = faster processing Use cv::cuda for compatible functions Process only regions of interest ## Best Practices ### Denoising * Start with h=3, increase if needed * Use colored version for color images * Multi-frame for video (better quality) ### HDR * Use 3-5 exposures with 1-2 EV spacing * Align images before merging * Experiment with tone mapping algorithms ### Seamless Cloning * Create accurate masks * Position carefully for best results * Try different cloning modes ## See Also * [ImgProc Module](/modules/imgproc) - Basic image processing * [Core Module](/modules/core) - Matrix operations * [Photo Tutorials](https://docs.opencv.org/master/d0/d86/tutorial_py_image_arithmetics.html) # Image Stitching Module Source: https://opencv-opencv.mintlify.app/modules/stitching Panorama creation and image stitching with automatic feature matching ## Overview The Stitching module provides a complete pipeline for creating panoramas from multiple images. It handles: * Feature detection and matching * Camera parameter estimation * Image warping * Exposure compensation * Seam finding * Image blending ## Quick Start ### Simple Panorama ```cpp theme={null} #include using namespace cv; int main() { // Load images std::vector images; images.push_back(imread("img1.jpg")); images.push_back(imread("img2.jpg")); images.push_back(imread("img3.jpg")); // Create stitcher Ptr stitcher = Stitcher::create( Stitcher::PANORAMA ); // Stitch images Mat pano; Stitcher::Status status = stitcher->stitch(images, pano); if(status == Stitcher::OK) { imwrite("panorama.jpg", pano); } else { std::cerr << "Stitching failed: " << status << std::endl; } return 0; } ``` ## Stitcher Modes ### PANORAMA Mode For regular camera photos: ```cpp theme={null} Ptr stitcher = Stitcher::create(Stitcher::PANORAMA); ``` **Features**: * Expects perspective transformations * Uses homography-based estimation * Projects to spherical surface * Applies exposure compensation ### SCANS Mode For scanned images or documents: ```cpp theme={null} Ptr stitcher = Stitcher::create(Stitcher::SCANS); ``` **Features**: * Expects affine transformations * No exposure compensation * Better for flat scans ## Stitching Pipeline ### Step-by-Step Process ```cpp theme={null} // 1. Create stitcher Ptr stitcher = Stitcher::create(); // 2. Configure (optional) stitcher->setRegistrationResol(0.6); // Resolution for registration stitcher->setSeamEstimationResol(0.1); // Resolution for seam finding stitcher->setCompositingResol(-1); // Use original resolution stitcher->setPanoConfidenceThresh(1); // Confidence threshold // 3. Estimate transformations Stitcher::Status status = stitcher->estimateTransform(images); if(status != Stitcher::OK) { std::cerr << "Transform estimation failed\n"; return -1; } // 4. Compose panorama Mat pano; status = stitcher->composePanorama(pano); ``` ## Advanced Configuration ### Feature Detection ```cpp theme={null} // Use ORB features (faster) Ptr orb = ORB::create(500); stitcher->setFeaturesFinder(orb); // Use SIFT features (better quality) Ptr sift = SIFT::create(); stitcher->setFeaturesFinder(sift); ``` ### Feature Matching ```cpp theme={null} using namespace cv::detail; // Best of 2 nearest matcher Ptr matcher = makePtr(false, 0.3f); stitcher->setFeaturesMatcher(matcher); // Affine matcher (for SCANS mode) Ptr affineMatcher = makePtr(); stitcher->setFeaturesMatcher(affineMatcher); ``` ### Warping ```cpp theme={null} // Spherical warper (default for PANORAMA) Ptr warper = makePtr(); stitcher->setWarper(warper); // Cylindrical warper Ptr cylindrical = makePtr(); stitcher->setWarper(cylindrical); // Plane warper Ptr plane = makePtr(); stitcher->setWarper(plane); ``` ### Exposure Compensation ```cpp theme={null} using namespace cv::detail; // Block-based gain compensation Ptr compensator = makePtr(); stitcher->setExposureCompensator(compensator); // No compensation Ptr noComp = makePtr(); stitcher->setExposureCompensator(noComp); ``` ### Seam Finding ```cpp theme={null} using namespace cv::detail; // Graph cut seam finder (best quality) Ptr seamFinder = makePtr(GraphCutSeamFinder::COST_COLOR); stitcher->setSeamFinder(seamFinder); // Voronoi seam finder (faster) Ptr voronoi = makePtr(); stitcher->setSeamFinder(voronoi); ``` ### Blending ```cpp theme={null} using namespace cv::detail; // Multi-band blending (best quality) Ptr blender = makePtr(false, 5); stitcher->setBlender(blender); // Feather blending (faster) Ptr feather = makePtr(0.01f); stitcher->setBlender(feather); ``` ## Resolution Control ```cpp theme={null} // Set resolutions (megapixels) stitcher->setRegistrationResol(0.6); // Feature detection stitcher->setSeamEstimationResol(0.1); // Seam finding stitcher->setCompositingResol(-1); // Final output (-1 = original) // Use original resolution everywhere stitcher->setRegistrationResol(Stitcher::ORIG_RESOL); stitcher->setSeamEstimationResol(Stitcher::ORIG_RESOL); stitcher->setCompositingResol(Stitcher::ORIG_RESOL); ``` ## Error Handling ```cpp theme={null} Stitcher::Status status = stitcher->stitch(images, pano); switch(status) { case Stitcher::OK: std::cout << "Stitching successful\n"; break; case Stitcher::ERR_NEED_MORE_IMGS: std::cerr << "Need more images\n"; break; case Stitcher::ERR_HOMOGRAPHY_EST_FAIL: std::cerr << "Homography estimation failed\n"; break; case Stitcher::ERR_CAMERA_PARAMS_ADJUST_FAIL: std::cerr << "Camera parameter adjustment failed\n"; break; } ``` ## Complete Example: Custom Pipeline ```cpp theme={null} #include #include using namespace cv; using namespace cv::detail; int main() { // Load images std::vector images; for(int i = 1; i <= 5; i++) { images.push_back( imread("img" + std::to_string(i) + ".jpg") ); } // Create and configure stitcher Ptr stitcher = Stitcher::create( Stitcher::PANORAMA ); // Use SIFT features Ptr sift = SIFT::create(1000); stitcher->setFeaturesFinder(sift); // Configure resolution stitcher->setRegistrationResol(0.6); stitcher->setSeamEstimationResol(0.1); stitcher->setCompositingResol(1.0); // Graph cut seam finder Ptr seamFinder = makePtr( GraphCutSeamFinder::COST_COLOR ); stitcher->setSeamFinder(seamFinder); // Multi-band blending stitcher->setBlender( makePtr(false, 5) ); // Estimate transforms Stitcher::Status status = stitcher->estimateTransform(images); if(status != Stitcher::OK) { std::cerr << "Transform estimation failed: " << status << std::endl; return -1; } // Get camera parameters std::vector cameras = stitcher->cameras(); std::cout << "Found " << cameras.size() << " cameras\n"; // Compose panorama Mat pano; status = stitcher->composePanorama(pano); if(status == Stitcher::OK) { imwrite("panorama.jpg", pano); std::cout << "Panorama size: " << pano.size() << std::endl; } return 0; } ``` ## Best Practices 30-50% overlap between adjacent images Use manual exposure or lock exposure Avoid zooming between shots Use tripod for best results ## Performance Optimization ```cpp theme={null} // Reduce resolution for speed stitcher->setRegistrationResol(0.3); // Lower = faster stitcher->setSeamEstimationResol(0.05); // Use faster algorithms stitcher->setFeaturesFinder(ORB::create(500)); stitcher->setSeamFinder( makePtr() ); stitcher->setBlender( makePtr(0.01f) ); ``` ## Troubleshooting ### Stitching Fails 1. Check image overlap (needs 30%+) 2. Ensure similar lighting 3. Try different feature detectors 4. Increase confidence threshold ### Poor Quality 1. Increase registration resolution 2. Use graph cut seam finder 3. Use multi-band blending 4. Enable exposure compensation ## See Also * [Features2D Module](/modules/features2d) - Feature detection * [Calib3D Module](/modules/calib3d) - Camera calibration * [Stitching Tutorial](https://docs.opencv.org/master/d8/d19/tutorial_stitcher.html) # Video Analysis Module Source: https://opencv-opencv.mintlify.app/modules/video Video analysis algorithms including optical flow, object tracking, and background subtraction The Video Analysis module provides algorithms for motion analysis, object tracking, and background/foreground segmentation in video streams. ## Overview From opencv2/video.hpp:47-52: > This module contains algorithms for motion analysis, object tracking, and background subtraction. It enables applications such as motion detection, object following, and foreground extraction from video sequences. Dense and sparse motion estimation between frames Track objects using MeanShift and CamShift Separate foreground from background Analyze motion patterns in video ## Module Components From opencv2/video.hpp, the module includes: * `opencv2/video/tracking.hpp` - Optical flow and tracking * `opencv2/video/background_segm.hpp` - Background subtraction ## Optical Flow Optical flow estimates the motion of pixels between consecutive frames. ### Lucas-Kanade Sparse Optical Flow From tracking.hpp:134-186, tracks sparse feature points: ```cpp theme={null} #include #include #include using namespace cv; using namespace std; // Detect good features to track vector detectFeatures(const Mat& gray) { vector points; goodFeaturesToTrack(gray, points, 100, // max corners 0.01, // quality level 10); // min distance return points; } int main() { VideoCapture cap("video.mp4"); Mat prevGray, gray, frame; cap >> frame; cvtColor(frame, prevGray, COLOR_BGR2GRAY); // Detect initial features vector prevPoints = detectFeatures(prevGray); while (cap.read(frame)) { cvtColor(frame, gray, COLOR_BGR2GRAY); // Calculate optical flow (tracking.hpp:181-186) vector nextPoints; vector status; vector err; calcOpticalFlowPyrLK( prevGray, gray, // Previous and current frames prevPoints, // Previous points nextPoints, // Output: new positions status, // Output: tracking status err, // Output: error Size(21, 21), // Window size 3 // Max pyramid level ); // Draw tracks for (size_t i = 0; i < prevPoints.size(); i++) { if (status[i]) { line(frame, prevPoints[i], nextPoints[i], Scalar(0, 255, 0), 2); circle(frame, nextPoints[i], 3, Scalar(0, 255, 0), -1); } } imshow("Optical Flow", frame); if (waitKey(30) >= 0) break; // Update for next iteration prevGray = gray.clone(); prevPoints = nextPoints; } return 0; } ``` ### Dense Optical Flow (Farneback) From tracking.hpp:188-200, computes flow for every pixel: ```cpp theme={null} #include void computeDenseFlow(const Mat& prev, const Mat& next, Mat& flow) { calcOpticalFlowFarneback( prev, next, // Input frames flow, // Output flow (CV_32FC2) 0.5, // pyr_scale 3, // levels 15, // winsize 3, // iterations 5, // poly_n 1.2, // poly_sigma 0 // flags ); } // Visualize flow Mat visualizeFlow(const Mat& flow) { Mat flowParts[2]; split(flow, flowParts); Mat magnitude, angle; cartToPolar(flowParts[0], flowParts[1], magnitude, angle, true); // Create HSV image Mat hsv = Mat::zeros(flow.size(), CV_8UC3); Mat hsvParts[3]; // Hue = direction, Value = magnitude angle.convertTo(hsvParts[0], CV_8U, 255.0/360.0); hsvParts[1] = Mat::ones(flow.size(), CV_8U) * 255; normalize(magnitude, hsvParts[2], 0, 255, NORM_MINMAX, CV_8U); merge(hsvParts, 3, hsv); Mat bgr; cvtColor(hsv, bgr, COLOR_HSV2BGR); return bgr; } int main() { VideoCapture cap("video.mp4"); Mat prevGray, gray, frame, flow; cap >> frame; cvtColor(frame, prevGray, COLOR_BGR2GRAY); while (cap.read(frame)) { cvtColor(frame, gray, COLOR_BGR2GRAY); // Compute dense optical flow computeDenseFlow(prevGray, gray, flow); // Visualize Mat flowVis = visualizeFlow(flow); imshow("Dense Optical Flow", flowVis); if (waitKey(30) >= 0) break; prevGray = gray; } return 0; } ``` ### Optical Flow Flags From tracking.hpp:59-62: ```cpp theme={null} enum OptFlowFlags { OPTFLOW_USE_INITIAL_FLOW = 4, // Use initial estimate OPTFLOW_LK_GET_MIN_EIGENVALS = 8, // Use eigenvalues for error OPTFLOW_FARNEBACK_GAUSSIAN = 256 // Use Gaussian filter }; ``` ## Object Tracking ### MeanShift Tracking From tracking.hpp:88-107, finds object center: ```cpp theme={null} #include int main() { VideoCapture cap("video.mp4"); Mat frame, hsv, backProj; cap >> frame; // Select initial ROI Rect trackWindow = selectROI(frame); // Calculate histogram of ROI Mat roi = frame(trackWindow); cvtColor(roi, hsv, COLOR_BGR2HSV); Mat hist; int hbins = 30; float hranges[] = {0, 180}; const float* ranges[] = {hranges}; int channels[] = {0}; calcHist(&hsv, 1, channels, Mat(), hist, 1, &hbins, ranges); normalize(hist, hist, 0, 255, NORM_MINMAX); // Track object while (cap.read(frame)) { cvtColor(frame, hsv, COLOR_BGR2HSV); // Calculate back projection calcBackProject(&hsv, 1, channels, hist, backProj, ranges); // Apply MeanShift (tracking.hpp:107) TermCriteria criteria(TermCriteria::EPS | TermCriteria::COUNT, 10, 1); meanShift(backProj, trackWindow, criteria); // Draw tracking rectangle rectangle(frame, trackWindow, Scalar(0, 255, 0), 2); imshow("MeanShift Tracking", frame); if (waitKey(30) >= 0) break; } return 0; } ``` ### CamShift Tracking From tracking.hpp:64-86, adaptive tracking with rotation: ```cpp theme={null} // CamShift adjusts window size and finds rotation while (cap.read(frame)) { cvtColor(frame, hsv, COLOR_BGR2HSV); calcBackProject(&hsv, 1, channels, hist, backProj, ranges); // CamShift returns rotated rectangle (tracking.hpp:82-83) RotatedRect trackBox = CamShift(backProj, trackWindow, criteria); // Draw rotated box Point2f vertices[4]; trackBox.points(vertices); for (int i = 0; i < 4; i++) { line(frame, vertices[i], vertices[(i+1)%4], Scalar(0, 255, 0), 2); } imshow("CamShift Tracking", frame); if (waitKey(30) >= 0) break; } ``` ## Background Subtraction From background\_segm.hpp:55-97, separate foreground from background: ### BackgroundSubtractor Base Class ```cpp theme={null} class BackgroundSubtractor : public Algorithm { public: // Apply background subtraction virtual void apply(InputArray image, OutputArray fgmask, double learningRate = -1) = 0; // Get background image virtual void getBackgroundImage(OutputArray backgroundImage) const = 0; }; ``` ### MOG2 Background Subtractor From background\_segm.hpp:100-150, Gaussian Mixture Model: ```cpp theme={null} #include using namespace cv; int main() { VideoCapture cap("video.mp4"); // Create MOG2 background subtractor Ptr pBackSub = createBackgroundSubtractorMOG2(); // Configure parameters pBackSub->setHistory(500); // Frames to use pBackSub->setVarThreshold(16); // Threshold pBackSub->setDetectShadows(true); // Detect shadows Mat frame, fgMask; while (cap.read(frame)) { // Apply background subtraction pBackSub->apply(frame, fgMask); // Optional: remove shadows (value 127) threshold(fgMask, fgMask, 200, 255, THRESH_BINARY); // Show results imshow("Frame", frame); imshow("Foreground Mask", fgMask); if (waitKey(30) >= 0) break; } return 0; } ``` ### KNN Background Subtractor ```cpp theme={null} // K-Nearest Neighbors background subtractor Ptr pBackSub = createBackgroundSubtractorKNN(); pBackSub->setHistory(500); pBackSub->setDist2Threshold(400.0); pBackSub->setDetectShadows(true); Mat frame, fgMask; while (cap.read(frame)) { pBackSub->apply(frame, fgMask); imshow("KNN Foreground", fgMask); if (waitKey(30) >= 0) break; } ``` ### Extracting Foreground Objects ```cpp theme={null} Ptr pBackSub = createBackgroundSubtractorMOG2(); Mat frame, fgMask, foreground; while (cap.read(frame)) { // Get foreground mask pBackSub->apply(frame, fgMask); // Clean up mask Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(5, 5)); morphologyEx(fgMask, fgMask, MORPH_OPEN, kernel); morphologyEx(fgMask, fgMask, MORPH_CLOSE, kernel); // Extract foreground frame.copyTo(foreground, fgMask); // Find contours vector> contours; findContours(fgMask, contours, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE); // Draw bounding boxes around moving objects Mat display = frame.clone(); for (const auto& contour : contours) { double area = contourArea(contour); if (area > 500) { // Filter small detections Rect bbox = boundingRect(contour); rectangle(display, bbox, Scalar(0, 255, 0), 2); } } imshow("Detection", display); imshow("Foreground", foreground); if (waitKey(30) >= 0) break; } ``` ## Kalman Filter Predict and track object positions: ```cpp theme={null} #include // Initialize Kalman filter KalmanFilter KF(4, 2, 0); // State: [x, y, vx, vy] KF.transitionMatrix = (Mat_(4, 4) << 1, 0, 1, 0, // x' = x + vx 0, 1, 0, 1, // y' = y + vy 0, 0, 1, 0, // vx' = vx 0, 0, 0, 1); // vy' = vy // Measurement matrix: measure [x, y] KF.measurementMatrix = (Mat_(2, 4) << 1, 0, 0, 0, 0, 1, 0, 0); // Process and measurement noise setIdentity(KF.processNoiseCov, Scalar::all(1e-4)); setIdentity(KF.measurementNoiseCov, Scalar::all(1e-1)); setIdentity(KF.errorCovPost, Scalar::all(1)); // Initial state KF.statePost.at(0) = initialX; KF.statePost.at(1) = initialY; // Tracking loop while (cap.read(frame)) { // Predict next position Mat prediction = KF.predict(); Point predictPt(prediction.at(0), prediction.at(1)); // Get measurement (e.g., from detection) Point measPt = detectObject(frame); // Update Kalman filter Mat measurement = (Mat_(2, 1) << measPt.x, measPt.y); KF.correct(measurement); // Visualize circle(frame, measPt, 5, Scalar(0, 0, 255), -1); // Red: measurement circle(frame, predictPt, 5, Scalar(255, 0, 0), -1); // Blue: prediction imshow("Kalman Filter", frame); waitKey(30); } ``` ## Practical Examples ### Motion Detection System ```cpp theme={null} #include #include #include using namespace cv; class MotionDetector { private: Ptr pBackSub; int minArea; public: MotionDetector(int minArea = 500) : minArea(minArea) { pBackSub = createBackgroundSubtractorMOG2(); pBackSub->setDetectShadows(false); } vector detect(const Mat& frame) { Mat fgMask; pBackSub->apply(frame, fgMask); // Clean up Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(5, 5)); morphologyEx(fgMask, fgMask, MORPH_OPEN, kernel); dilate(fgMask, fgMask, kernel); // Find contours vector> contours; findContours(fgMask, contours, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE); // Filter and create bounding boxes vector detections; for (const auto& cnt : contours) { if (contourArea(cnt) >= minArea) { detections.push_back(boundingRect(cnt)); } } return detections; } }; int main() { VideoCapture cap(0); // Camera MotionDetector detector(1000); Mat frame; while (cap.read(frame)) { auto boxes = detector.detect(frame); // Draw detections for (const auto& box : boxes) { rectangle(frame, box, Scalar(0, 255, 0), 2); string label = "Motion"; putText(frame, label, Point(box.x, box.y - 5), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0), 2); } // Show count string info = "Objects: " + to_string(boxes.size()); putText(frame, info, Point(10, 30), FONT_HERSHEY_SIMPLEX, 1, Scalar(0, 255, 0), 2); imshow("Motion Detection", frame); if (waitKey(30) >= 0) break; } return 0; } ``` ### People Counter ```cpp theme={null} class PeopleCounter { private: Ptr pBackSub; int lineY; // Counting line position map trackedObjects; int enterCount = 0; int exitCount = 0; int nextID = 0; public: PeopleCounter(int linePosition) : lineY(linePosition) { pBackSub = createBackgroundSubtractorMOG2(); } void process(Mat& frame) { Mat fgMask; pBackSub->apply(frame, fgMask); // Find current objects vector> contours; findContours(fgMask, contours, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE); map currentObjects; for (const auto& cnt : contours) { if (contourArea(cnt) > 2000) { Moments m = moments(cnt); Point center(m.m10/m.m00, m.m01/m.m00); // Match with tracked objects or create new int id = matchOrCreate(center); currentObjects[id] = center; // Check line crossing if (trackedObjects.count(id)) { Point prev = trackedObjects[id]; if (prev.y < lineY && center.y >= lineY) { enterCount++; } else if (prev.y >= lineY && center.y < lineY) { exitCount++; } } } } trackedObjects = currentObjects; // Draw counting line line(frame, Point(0, lineY), Point(frame.cols, lineY), Scalar(0, 0, 255), 2); // Display counts putText(frame, "In: " + to_string(enterCount), Point(10, 30), FONT_HERSHEY_SIMPLEX, 1, Scalar(0, 255, 0), 2); putText(frame, "Out: " + to_string(exitCount), Point(10, 70), FONT_HERSHEY_SIMPLEX, 1, Scalar(0, 0, 255), 2); } private: int matchOrCreate(const Point& center) { // Simple nearest neighbor matching double minDist = 50; int matchedID = -1; for (const auto& [id, pos] : trackedObjects) { double dist = norm(center - pos); if (dist < minDist) { minDist = dist; matchedID = id; } } return (matchedID >= 0) ? matchedID : nextID++; } }; ``` ## Best Practices **Optical Flow:** * Use sparse flow (Lucas-Kanade) when you need specific point tracking * Use dense flow (Farneback) for motion field visualization * Redetect features periodically to maintain good tracks **Background Subtraction:** * MOG2 is generally more robust than KNN * Adjust learning rate based on scene dynamics * Use morphological operations to clean up masks * Filter detections by minimum area to reduce noise **Performance:** ```cpp theme={null} // Resize frames for faster processing Mat small; resize(frame, small, Size(), 0.5, 0.5); pBackSub->apply(small, fgMask); // Use GPU acceleration if available Ptr gpu_mog = cuda::createBackgroundSubtractorMOG2(); ``` ## Related Modules * [Video I/O](/modules/videoio) - Capture video streams * [Image Processing](/modules/imgproc) - Process frames * [Object Detection](/modules/objdetect) - Detect specific objects ## Source Reference Key headers: * `~/workspace/source/modules/video/include/opencv2/video/tracking.hpp` * `~/workspace/source/modules/video/include/opencv2/video/background_segm.hpp` Examples: * `samples/cpp/lkdemo.cpp` - Lucas-Kanade optical flow * `samples/cpp/camshiftdemo.cpp` - CamShift tracking * `samples/cpp/bgfg_segm.cpp` - Background subtraction # Video I/O Module Source: https://opencv-opencv.mintlify.app/modules/videoio Video capture from cameras and files, video writing, and stream handling with support for multiple backends The Video I/O (videoio) module provides interfaces for reading video from cameras or files and writing video to files, with support for multiple backend APIs and hardware acceleration. ## Overview From opencv2/videoio.hpp:48-64: > Read and write video or image sequences with OpenCV. This module provides unified interfaces for video capture and writing across different platforms and backends including FFmpeg, GStreamer, DirectShow, AVFoundation, and more. Capture from cameras or read video files Write video files with codec selection Multiple API backend support (FFmpeg, GStreamer, etc.) GPU-accelerated encoding and decoding ## VideoCapture Class ### Capturing from Camera Example from samples/cpp/videocapture\_basic.cpp: ```cpp theme={null} #include #include using namespace cv; int main() { VideoCapture cap; // Open default camera (videocapture_basic.cpp:25-28) int deviceID = 0; // 0 = default camera int apiID = CAP_ANY; // Autodetect API cap.open(deviceID, apiID); // Check if opened successfully (videocapture_basic.cpp:30-33) if (!cap.isOpened()) { cerr << "ERROR! Unable to open camera\n"; return -1; } // Grab and process frames (videocapture_basic.cpp:38-50) Mat frame; for (;;) { cap.read(frame); // Capture frame if (frame.empty()) { cerr << "ERROR! Blank frame grabbed\n"; break; } // Display frame imshow("Live", frame); if (waitKey(5) >= 0) break; } return 0; } ``` ### Reading Video Files ```cpp theme={null} // Open video file VideoCapture cap("video.mp4"); // Or with specific backend VideoCapture cap("video.mp4", CAP_FFMPEG); // Check if opened if (!cap.isOpened()) { cerr << "Error opening video file\n"; return -1; } // Get video properties int frameCount = cap.get(CAP_PROP_FRAME_COUNT); double fps = cap.get(CAP_PROP_FPS); int width = cap.get(CAP_PROP_FRAME_WIDTH); int height = cap.get(CAP_PROP_FRAME_HEIGHT); // Read frames Mat frame; while (cap.read(frame)) { // Process frame imshow("Video", frame); if (waitKey(1000/fps) >= 0) break; } cap.release(); ``` ## Video Capture Backends From videoio.hpp:92-129, OpenCV supports multiple backends: ```cpp theme={null} enum VideoCaptureAPIs { CAP_ANY = 0, // Auto detect CAP_V4L2 = 200, // V4L/V4L2 (Linux) CAP_FIREWIRE = 300, // IEEE 1394 CAP_DSHOW = 700, // DirectShow (Windows) CAP_PVAPI = 800, // PvAPI (Prosilica GigE) CAP_OPENNI = 900, // OpenNI (Kinect) CAP_ANDROID = 1000, // MediaNDK (Android) CAP_XIAPI = 1100, // XIMEA Camera CAP_AVFOUNDATION = 1200,// AVFoundation (macOS/iOS) CAP_MSMF = 1400, // Microsoft Media Foundation CAP_REALSENSE = 1500, // Intel RealSense CAP_OPENNI2 = 1600, // OpenNI2 CAP_GPHOTO2 = 1700, // gPhoto2 CAP_GSTREAMER = 1800, // GStreamer CAP_FFMPEG = 1900, // FFmpeg CAP_IMAGES = 2000, // Image sequence CAP_ARAVIS = 2100, // Aravis SDK CAP_OPENCV_MJPEG = 2200,// OpenCV MJPEG codec CAP_INTEL_MFX = 2300, // Intel MediaSDK CAP_OBSENSOR = 2600 // Orbbec 3D sensors }; // Specify backend VideoCapture cap(0, CAP_DSHOW); // Use DirectShow on Windows VideoCapture cap("video.mp4", CAP_FFMPEG); // Use FFmpeg ``` ## Video Capture Properties From videoio.hpp:138-216, extensive property control: ```cpp theme={null} // Position properties cap.get(CAP_PROP_POS_MSEC); // Current position (ms) cap.get(CAP_PROP_POS_FRAMES); // Current frame number cap.get(CAP_PROP_POS_AVI_RATIO); // Relative position (0-1) // Frame properties int width = cap.get(CAP_PROP_FRAME_WIDTH); int height = cap.get(CAP_PROP_FRAME_HEIGHT); double fps = cap.get(CAP_PROP_FPS); int fourcc = cap.get(CAP_PROP_FOURCC); int frameCount = cap.get(CAP_PROP_FRAME_COUNT); // Camera properties (if supported) cap.set(CAP_PROP_BRIGHTNESS, 0.5); cap.set(CAP_PROP_CONTRAST, 0.5); cap.set(CAP_PROP_SATURATION, 0.5); cap.set(CAP_PROP_HUE, 0.5); cap.set(CAP_PROP_GAIN, 0.5); cap.set(CAP_PROP_EXPOSURE, 0.5); // Advanced properties cap.set(CAP_PROP_AUTOFOCUS, 1); cap.set(CAP_PROP_AUTO_WB, 1); cap.set(CAP_PROP_ZOOM, 1.5); cap.set(CAP_PROP_FOCUS, 0.5); // Backend information int backend = cap.get(CAP_PROP_BACKEND); ``` ### Seeking in Videos ```cpp theme={null} // Seek to specific frame cap.set(CAP_PROP_POS_FRAMES, 100); // Seek to time position (milliseconds) cap.set(CAP_PROP_POS_MSEC, 5000); // 5 seconds // Seek to relative position (0.0 to 1.0) cap.set(CAP_PROP_POS_AVI_RATIO, 0.5); // Middle of video // Read frame at new position Mat frame; cap.read(frame); ``` ## VideoWriter Class ### Writing Video Files ```cpp theme={null} #include using namespace cv; // Define video properties String filename = "output.mp4"; int fourcc = VideoWriter::fourcc('M','P','4','V'); double fps = 30.0; Size frameSize(640, 480); bool isColor = true; // Create VideoWriter VideoWriter writer(filename, fourcc, fps, frameSize, isColor); // Check if opened if (!writer.isOpened()) { cerr << "Could not open video writer\n"; return -1; } // Write frames Mat frame; for (int i = 0; i < 100; i++) { // Generate or capture frame frame = generateFrame(i); // Write frame writer.write(frame); // Or equivalently: writer << frame; } // Release writer (flushes and closes file) writer.release(); ``` ### FourCC Codes ```cpp theme={null} // Common FourCC codes for codecs int fourcc; // H.264 (most widely supported) fourcc = VideoWriter::fourcc('H','2','6','4'); fourcc = VideoWriter::fourcc('X','2','6','4'); // Alternative fourcc = VideoWriter::fourcc('a','v','c','1'); // Apple variant // MPEG-4 fourcc = VideoWriter::fourcc('M','P','4','V'); fourcc = VideoWriter::fourcc('M','P','4','2'); // Motion JPEG fourcc = VideoWriter::fourcc('M','J','P','G'); // VP9 (WebM) fourcc = VideoWriter::fourcc('V','P','9','0'); // HEVC (H.265) fourcc = VideoWriter::fourcc('H','E','V','C'); // Uncompressed (large files!) fourcc = VideoWriter::fourcc('D','I','B',' '); // BMP // Let backend choose fourcc = 0; // or VideoWriter::fourcc('\0','\0','\0','\0') ``` ## Hardware Acceleration From videoio.hpp:256-269: ```cpp theme={null} // Hardware acceleration types enum VideoAccelerationType { VIDEO_ACCELERATION_NONE = 0, // Software only VIDEO_ACCELERATION_ANY = 1, // Prefer hardware VIDEO_ACCELERATION_D3D11 = 2, // DirectX 11 VIDEO_ACCELERATION_VAAPI = 3, // Video Acceleration API VIDEO_ACCELERATION_MFX = 4, // Intel MediaSDK VIDEO_ACCELERATION_DRM = 5 // Raspberry Pi V4 }; // Use hardware acceleration for capture vector apiPreference = {CAP_FFMPEG}; map params = { {CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY}, {CAP_PROP_HW_DEVICE, 0} // GPU device index }; VideoCapture cap("video.mp4", CAP_FFMPEG, params); // Use hardware acceleration for writing map writerParams = { {VIDEOWRITER_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY}, {VIDEOWRITER_PROP_HW_DEVICE, 0} }; VideoWriter writer("output.mp4", CAP_FFMPEG, VideoWriter::fourcc('H','2','6','4'), 30, Size(1920, 1080), writerParams); ``` ## Image Sequences ```cpp theme={null} // Read image sequence // Files: img_0001.jpg, img_0002.jpg, etc. VideoCapture cap("img_%04d.jpg"); Mat frame; while (cap.read(frame)) { // Process frames imshow("Sequence", frame); waitKey(30); } // Write image sequence VideoWriter writer("output_%04d.png", CAP_IMAGES, 0, // FPS not used for images 30, Size(640, 480)); for (int i = 0; i < 100; i++) { Mat frame = generateFrame(i); writer.write(frame); } ``` ## Audio Support From videoio.hpp:198-206: ```cpp theme={null} // Open video with audio stream map params = { {CAP_PROP_AUDIO_STREAM, 0} // Enable audio, stream 0 }; VideoCapture cap("video.mp4", CAP_FFMPEG, params); // Get audio properties int audioPos = cap.get(CAP_PROP_AUDIO_POS); int audioBaseIndex = cap.get(CAP_PROP_AUDIO_BASE_INDEX); int audioChannels = cap.get(CAP_PROP_AUDIO_TOTAL_CHANNELS); int audioSampleRate = cap.get(CAP_PROP_AUDIO_SAMPLES_PER_SECOND); // Retrieve audio samples Mat audioFrame; cap.retrieve(audioFrame, audioBaseIndex); ``` ## Practical Examples ### Video File Converter ```cpp theme={null} #include #include using namespace cv; void convertVideo(const string& input, const string& output, int targetWidth, int targetHeight) { // Open input VideoCapture cap(input); if (!cap.isOpened()) { cerr << "Cannot open input video\n"; return; } // Get properties double fps = cap.get(CAP_PROP_FPS); // Create writer VideoWriter writer( output, VideoWriter::fourcc('H','2','6','4'), fps, Size(targetWidth, targetHeight) ); if (!writer.isOpened()) { cerr << "Cannot open output video\n"; return; } // Process frames Mat frame, resized; while (cap.read(frame)) { resize(frame, resized, Size(targetWidth, targetHeight)); writer.write(resized); } } ``` ### Extract Video Frames ```cpp theme={null} void extractFrames(const string& videoPath, const string& outputDir, int frameInterval = 30) { VideoCapture cap(videoPath); if (!cap.isOpened()) return; int frameNum = 0; int savedCount = 0; Mat frame; while (cap.read(frame)) { if (frameNum % frameInterval == 0) { string filename = outputDir + "/frame_" + to_string(savedCount) + ".jpg"; imwrite(filename, frame); savedCount++; } frameNum++; } cout << "Extracted " << savedCount << " frames\n"; } ``` ### Real-time Camera Processing ```cpp theme={null} #include #include #include using namespace cv; int main() { VideoCapture cap(0); if (!cap.isOpened()) return -1; // Set camera properties cap.set(CAP_PROP_FRAME_WIDTH, 1280); cap.set(CAP_PROP_FRAME_HEIGHT, 720); cap.set(CAP_PROP_FPS, 30); // Optional: Create video writer VideoWriter writer( "recording.mp4", VideoWriter::fourcc('H','2','6','4'), 30.0, Size(1280, 720) ); Mat frame, processed; bool recording = false; while (true) { cap >> frame; if (frame.empty()) break; // Process frame cvtColor(frame, processed, COLOR_BGR2GRAY); cvtColor(processed, processed, COLOR_GRAY2BGR); // Record if enabled if (recording) { writer.write(frame); } // Display imshow("Camera", processed); // Handle keys int key = waitKey(1); if (key == 'q') break; if (key == 'r') recording = !recording; } return 0; } ``` ### Video Stabilization ```cpp theme={null} void stabilizeVideo(const string& input, const string& output) { VideoCapture cap(input); // Get video properties int width = cap.get(CAP_PROP_FRAME_WIDTH); int height = cap.get(CAP_PROP_FRAME_HEIGHT); double fps = cap.get(CAP_PROP_FPS); VideoWriter writer( output, VideoWriter::fourcc('M','P','4','V'), fps, Size(width, height) ); Mat prevFrame, prevGray; cap >> prevFrame; cvtColor(prevFrame, prevGray, COLOR_BGR2GRAY); // Transformation accumulator Mat totalTransform = Mat::eye(2, 3, CV_64F); Mat frame, gray, transform, stabilized; while (cap.read(frame)) { cvtColor(frame, gray, COLOR_BGR2GRAY); // Estimate transform vector prevPts, currPts; goodFeaturesToTrack(prevGray, prevPts, 200, 0.01, 30); vector status; vector err; calcOpticalFlowPyrLK(prevGray, gray, prevPts, currPts, status, err); // Calculate transformation transform = estimateAffinePartial2D(prevPts, currPts); // Apply smoothing and warp // ... (smoothing logic) ... warpAffine(frame, stabilized, totalTransform, frame.size()); writer.write(stabilized); prevGray = gray.clone(); } } ``` ## Best Practices **Always Check isOpened():** ```cpp theme={null} VideoCapture cap("video.mp4"); if (!cap.isOpened()) { cerr << "Error: Cannot open video\n"; return -1; } ``` This catches missing files, unsupported formats, or codec issues. **Choose the Right Backend:** * **FFmpeg** - Best for file I/O, most format support * **GStreamer** - Good for streaming, pipelines * **Platform-specific** - MSMF (Windows), AVFoundation (macOS) for cameras ```cpp theme={null} VideoCapture cap(source, CAP_FFMPEG); // Explicit backend ``` **Frame Rate Control:** ```cpp theme={null} double fps = cap.get(CAP_PROP_FPS); int delay = cvRound(1000.0 / fps); // Delay in ms while (cap.read(frame)) { imshow("Video", frame); if (waitKey(delay) >= 0) break; // Proper playback speed } ``` **Codec Compatibility:** H.264 is the most widely supported codec: ```cpp theme={null} int fourcc = VideoWriter::fourcc('H','2','6','4'); VideoWriter writer("output.mp4", fourcc, 30, size); ``` For maximum compatibility, use .mp4 container with H.264 codec. ## Troubleshooting ### Common Issues 1. **"Cannot open video"** * Check file exists and path is correct * Verify codec support: `cap.get(CAP_PROP_FOURCC)` * Try different backend: `VideoCapture(file, CAP_FFMPEG)` 2. **Frames not written** * Ensure frame size matches VideoWriter size * Check frame type (CV\_8UC3 for color) * Verify disk space and write permissions 3. **Camera not found** * Try different camera indices (0, 1, 2...) * Specify backend: `VideoCapture(0, CAP_DSHOW)` * Check camera permissions 4. **Playback too fast/slow** * Get FPS: `cap.get(CAP_PROP_FPS)` * Use proper delay: `waitKey(1000/fps)` ## Related Modules * [Image Codecs](/modules/imgcodecs) - Image I/O operations * [High-Level GUI](/modules/highgui) - Display video frames * [Video Analysis](/modules/video) - Process video content ## Source Reference Main header: `~/workspace/source/modules/videoio/include/opencv2/videoio.hpp` Examples: * `samples/cpp/videocapture_basic.cpp` * `samples/cpp/videowriter_basic.cpp` * `samples/cpp/videocapture_camera.cpp` # Building OpenCV for Android Source: https://opencv-opencv.mintlify.app/platforms/android Complete guide for integrating OpenCV into Android applications with NDK, building AAR packages, and using Gradle OpenCV provides comprehensive Android support through native NDK libraries, prebuilt AAR packages, and Java/Kotlin bindings. ## Quick Start Get started with OpenCV on Android in minutes: Download the prebuilt OpenCV Android SDK from the [releases page](https://github.com/opencv/opencv/releases): ```bash theme={null} # Extract the archive unzip opencv-4.x.0-android-sdk.zip ``` The SDK includes: * Native libraries for all Android ABIs * Java bindings * Sample applications In Android Studio: 1. Open **File → Open** 2. Navigate to `opencv-android-sdk/samples` 3. Select a sample (e.g., `15-puzzle`) 4. Wait for Gradle sync 1. Connect your Android device (USB debugging enabled) 2. Click **Run** (Shift+F10) 3. Select your device Android 5.0 (API Level 21) or higher is required. ## Prerequisites For building OpenCV from source: ```bash theme={null} # Install JDK sudo apt install openjdk-17-jdk # Install build tools sudo apt install git cmake ninja-build # Set environment variables export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 export ANDROID_HOME=~/Android/Sdk export ANDROID_NDK=$ANDROID_HOME/ndk/25.2.9519653 ``` ```powershell theme={null} # Install Android Studio from official site # It includes JDK, SDK, and NDK # Set environment variables $env:JAVA_HOME = "C:\Program Files\Android\Android Studio\jbr" $env:ANDROID_HOME = "C:\Users\YourName\AppData\Local\Android\Sdk" $env:ANDROID_NDK = "$env:ANDROID_HOME\ndk\25.2.9519653" ``` ```bash theme={null} # Install Android Studio from official site # Set environment variables export JAVA_HOME="/Applications/Android Studio.app/Contents/jbr/Contents/Home" export ANDROID_HOME=~/Library/Android/sdk export ANDROID_NDK=$ANDROID_HOME/ndk/25.2.9519653 ``` ### Install Android SDK and NDK Using Android Studio: 1. Open **Settings → Languages & Frameworks → Android SDK** 2. Check **Show Package Details** 3. Install: * Android SDK Platform (API 21+) * Android NDK (version 25.x recommended) * CMake * Build-Tools ## Supported Architectures (ABIs) OpenCV for Android supports multiple architectures: | ABI | Architecture | Minimum API | | ------------- | -------------------- | ----------- | | `armeabi-v7a` | ARM 32-bit with NEON | 21 | | `arm64-v8a` | ARM 64-bit | 21 | | `x86` | Intel 32-bit | 21 | | `x86_64` | Intel 64-bit | 21 | The default configuration in `platforms/android/default.config.py` builds for all four ABIs. Modern devices primarily use `arm64-v8a`. ## Building OpenCV Android SDK ### Using Python Build Script ```bash theme={null} git clone https://github.com/opencv/opencv.git cd opencv ``` ```bash theme={null} export ANDROID_HOME=~/Android/Sdk export ANDROID_NDK=$ANDROID_HOME/ndk/25.2.9519653 ``` ```bash theme={null} cd platforms/android python3 build_sdk.py --ndk_path $ANDROID_NDK \ --sdk_path $ANDROID_HOME \ ../../build_android ``` Add `--config ndk-25.config.py` to use a specific NDK version configuration. The SDK will be in: ``` build_android/ OpenCV-android-sdk/ sdk/ native/ libs/ # Native .so libraries jni/ # C++ headers java/ # Java sources ``` ### Build Options ```bash theme={null} # Build with opencv_contrib modules python3 build_sdk.py --ndk_path $ANDROID_NDK \ --sdk_path $ANDROID_HOME \ --extra_modules_path ../../opencv_contrib/modules \ ../../build_android # Build specific ABIs only python3 build_sdk.py --ndk_path $ANDROID_NDK \ --sdk_path $ANDROID_HOME \ --config ndk-25.config.py \ --modules arm64-v8a \ ../../build_android # Enable extra features python3 build_sdk.py --ndk_path $ANDROID_NDK \ --sdk_path $ANDROID_HOME \ --build_doc \ ../../build_android ``` ### Custom ABI Configuration Create a custom config file: ```python theme={null} # my_config.py ABIs = [ ABI("3", "arm64-v8a", None, 21), # ARM 64-bit only ] ``` Use it: ```bash theme={null} python3 build_sdk.py --config my_config.py \ --ndk_path $ANDROID_NDK \ ../../build_android ``` ## Building AAR Packages AAR (Android Archive) packages can be imported as Gradle dependencies: ### Java + Shared Library AAR ```bash theme={null} cd platforms/android # Set environment variables export JAVA_HOME="$HOME/Android Studio/jbr" export ANDROID_HOME="$HOME/Android/Sdk" # Build AAR python3 build_java_shared_aar.py "$HOME/opencv-4.x.0-android-sdk/OpenCV-android-sdk" # Output: outputs/*.aar and Maven repository ``` ### Static Library AAR For apps that bundle all dependencies: ```bash theme={null} python3 build_static_aar.py "$HOME/opencv-4.x.0-android-sdk/OpenCV-android-sdk" ``` AAR packages include: * Compiled native libraries for all ABIs * Java wrapper classes * ProGuard rules * Manifest file ## Integrating OpenCV into Android Apps ### Method 1: Import OpenCV Module (Recommended) In Android Studio: 1. **File → New → Import Module** 2. Select `opencv-android-sdk/sdk` 3. Click **Finish** In your app's `build.gradle`: ```groovy theme={null} dependencies { implementation project(':sdk') } ``` ```groovy theme={null} include ':app', ':sdk' project(':sdk').projectDir = new File('opencv-android-sdk/sdk') ``` ### Method 2: Use AAR Package ```groovy theme={null} // app/build.gradle dependencies { implementation files('libs/opencv-4.x.0.aar') } ``` Or publish to local Maven repository: ```groovy theme={null} repositories { maven { url "$rootDir/outputs/maven_repo" } } dependencies { implementation 'org.opencv:opencv:4.x.0' } ``` ### Method 3: CMake Integration For native C++ development: ```cmake theme={null} # app/CMakeLists.txt cmake_minimum_required(VERSION 3.6) project("myapp") # Set OpenCV path set(OpenCV_DIR "${CMAKE_SOURCE_DIR}/../opencv-android-sdk/sdk/native/jni") find_package(OpenCV REQUIRED) # Your native library add_library(myapp SHARED native-lib.cpp) # Link OpenCV target_link_libraries(myapp ${OpenCV_LIBS}) ``` In `build.gradle`: ```groovy theme={null} android { defaultConfig { externalNativeBuild { cmake { cppFlags "-std=c++11" arguments "-DOpenCV_DIR=${project.projectDir}/../opencv-android-sdk/sdk/native/jni" } } } externalNativeBuild { cmake { path "CMakeLists.txt" } } } ``` ## Using OpenCV in Code ### Java/Kotlin ```kotlin theme={null} import org.opencv.android.OpenCVLoader import org.opencv.core.Mat import org.opencv.imgproc.Imgproc class MainActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) // Initialize OpenCV if (!OpenCVLoader.initDebug()) { Log.e(TAG, "OpenCV initialization failed!") return } // Use OpenCV val mat = Mat() Imgproc.cvtColor(inputMat, mat, Imgproc.COLOR_RGBA2GRAY) } } ``` ```java theme={null} import org.opencv.android.OpenCVLoader; import org.opencv.core.Mat; import org.opencv.imgproc.Imgproc; public class MainActivity extends AppCompatActivity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); // Initialize OpenCV if (!OpenCVLoader.initDebug()) { Log.e(TAG, "OpenCV initialization failed!"); return; } // Use OpenCV Mat mat = new Mat(); Imgproc.cvtColor(inputMat, mat, Imgproc.COLOR_RGBA2GRAY); } } ``` ### Native C++ with JNI ```cpp theme={null} // native-lib.cpp #include #include extern "C" JNIEXPORT void JNICALL Java_com_example_myapp_MainActivity_processImage( JNIEnv* env, jobject /* this */, jlong matAddr) { cv::Mat& mat = *(cv::Mat*)matAddr; cv::cvtColor(mat, mat, cv::COLOR_RGBA2GRAY); } ``` Java/Kotlin side: ```kotlin theme={null} external fun processImage(matAddr: Long) companion object { init { System.loadLibrary("myapp") } } ``` ## Camera Integration OpenCV provides `CameraBridgeViewBase` for easy camera access: ```kotlin theme={null} import org.opencv.android.CameraBridgeViewBase import org.opencv.android.JavaCameraView class MainActivity : AppCompatActivity(), CameraBridgeViewBase.CvCameraViewListener2 { private lateinit var cameraBridgeViewBase: CameraBridgeViewBase override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) cameraBridgeViewBase = findViewById(R.id.cameraView) cameraBridgeViewBase.visibility = SurfaceView.VISIBLE cameraBridgeViewBase.setCvCameraViewListener(this) } override fun onCameraFrame(inputFrame: CameraBridgeViewBase.CvCameraViewFrame): Mat { val mat = inputFrame.rgba() // Process frame Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGBA2GRAY) return mat } override fun onCameraViewStarted(width: Int, height: Int) {} override fun onCameraViewStopped() {} } ``` Layout XML: ```xml theme={null} ``` ## Hardware Acceleration ### OpenCL Support OpenCV can use OpenCL for GPU acceleration on Android: ```kotlin theme={null} import org.opencv.core.Core // Check OpenCL availability if (Core.useOpenCL()) { Log.i(TAG, "OpenCL is available") } else { Log.w(TAG, "OpenCL is not available") } // Use UMat for automatic OpenCL acceleration val umat = UMat() Imgproc.cvtColor(inputUMat, umat, Imgproc.COLOR_RGBA2GRAY) ``` ### Qualcomm FastCV For Qualcomm devices, enable FastCV during build: ```bash theme={null} python3 build_sdk.py --config fastcv.config.py \ --ndk_path $ANDROID_NDK \ ../../build_android ``` ### NNAPI (Neural Networks API) For DNN module on Android 8.1+: ```kotlin theme={null} import org.opencv.dnn.Net val net = Dnn.readNetFromTensorflow(modelPath) net.setPreferableBackend(Dnn.DNN_BACKEND_DEFAULT) net.setPreferableTarget(Dnn.DNN_TARGET_NNAPI) ``` ## Permissions Add required permissions to `AndroidManifest.xml`: ```xml theme={null} ``` Request runtime permissions (Android 6.0+): ```kotlin theme={null} if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) { ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.CAMERA), CAMERA_PERMISSION_CODE) } ``` ## Sample Applications The Android SDK includes several sample applications: * **15-puzzle** - Interactive puzzle game * **camera-calibration** - Camera calibration tool * **color-blob-detection** - Color detection * **face-detection** - Face detection using Haar cascades * **image-manipulations** - Basic image operations * **mobilenet-objdetect** - Object detection using MobileNet * **qr-detection** - QR code detection Explore these in `opencv-android-sdk/samples/`. ## Troubleshooting Ensure all ABIs are included in your APK: ```groovy theme={null} android { defaultConfig { ndk { abiFilters 'armeabi-v7a', 'arm64-v8a' } } } ``` 1. Check OpenCV module is properly imported 2. Verify native libraries are in APK: Analyze APK → lib/ 3. Try `OpenCVLoader.initAsync()` instead for Manager-based initialization ```bash theme={null} # Ensure correct NDK version export ANDROID_NDK=$ANDROID_HOME/ndk/25.2.9519653 # Clean and rebuild rm -rf build_android/ python3 build_sdk.py --ndk_path $ANDROID_NDK ../../build_android ``` Set camera orientation in CameraView: ```kotlin theme={null} cameraBridgeViewBase.setMaxFrameSize(1280, 720) cameraBridgeViewBase.setCameraPermissionGranted() ``` Or rotate Mat manually: ```kotlin theme={null} Core.rotate(mat, mat, Core.ROTATE_90_CLOCKWISE) ``` ## App Size Optimization ### Include Only Needed ABIs ```groovy theme={null} android { defaultConfig { ndk { abiFilters 'arm64-v8a' // Modern devices only } } } ``` ### Enable APK Splits ```groovy theme={null} android { splits { abi { enable true reset() include 'armeabi-v7a', 'arm64-v8a' universalApk true } } } ``` ### ProGuard/R8 Configuration Add to `proguard-rules.pro`: ```proguard theme={null} -keep class org.opencv.** { *; } -keep interface org.opencv.** { *; } -keepclassmembers class * { native ; } ``` ## Next Steps Learn camera processing techniques Deploy deep learning models Explore example applications Optimize Android performance # Building OpenCV for iOS Source: https://opencv-opencv.mintlify.app/platforms/ios Complete guide for building OpenCV frameworks for iOS with Swift and Objective-C support OpenCV provides native iOS support through frameworks that can be integrated into Xcode projects with both Objective-C and Swift. ## Quick Start Get OpenCV running on iOS in minutes: Install required tools: ```bash theme={null} # Install Xcode from App Store (12.2 or later) # Install command line tools xcode-select --install # Install CMake (3.19.0 or later) brew install cmake # Or download from https://cmake.org/download/ ``` ```bash theme={null} cd ~/ git clone https://github.com/opencv/opencv.git ``` ```bash theme={null} cd opencv/platforms/ios python3 build_framework.py ios ``` This builds for iOS devices (arm64) and simulators (x86\_64, arm64). Takes 15-30 minutes. The framework will be at: ``` ~/ios/opencv2.framework ``` ## System Requirements * **macOS**: 10.15 (Catalina) or later * **Xcode**: 12.2 or later * **CMake**: 3.19.0 or later (3.17+ for older Xcode) * **Python**: 3.6 or later * **iOS Deployment Target**: 9.0 or later (default) Building iOS frameworks is only supported on macOS with Xcode installed. ## Supported Architectures OpenCV iOS framework includes: | Platform | Architectures | Usage | | ----------------- | -------------------- | ---------------------------------------- | | **iOS Device** | arm64, armv7, armv7s | Physical iPhones and iPads | | **iOS Simulator** | x86\_64, arm64 | Testing on Mac (Intel and Apple Silicon) | By default, `build_framework.py` builds for arm64 (devices) and x86\_64 + arm64 (simulators). Older armv7/armv7s can be included if needed. ## Building OpenCV Framework ### Standard Build The simplest way to build OpenCV for iOS: ```bash theme={null} cd ~/ git clone https://github.com/opencv/opencv.git cd opencv/platforms/ios python3 build_framework.py ios ``` Output location: `~/ios/opencv2.framework` ### Build with opencv\_contrib Modules Include extra modules: ```bash theme={null} # Clone opencv_contrib cd ~/ git clone https://github.com/opencv/opencv_contrib.git # Build with contrib cd opencv/platforms/ios python3 build_framework.py ios --contrib ~/opencv_contrib ``` ### Custom Build Options ```bash theme={null} python3 build_framework.py ~/my_build_dir ``` ```bash theme={null} # Reduce framework size by excluding modules python3 build_framework.py ios --without video --without objc ``` ```bash theme={null} # Build only for arm64 devices and arm64 simulator python3 build_framework.py ios \ --iphoneos_archs arm64 \ --iphonesimulator_archs arm64 ``` ```bash theme={null} # Build dynamic framework (iOS 8+ only) python3 build_framework.py ios --dynamic ``` ### Complete Build Command ```bash theme={null} python3 build_framework.py ios \ --contrib ~/opencv_contrib \ --iphoneos_archs arm64 \ --iphonesimulator_archs "x86_64,arm64" \ --without optflow \ --enable_nonfree ``` ### Build Script Options Key options for `build_framework.py`: ```bash theme={null} --opencv DIR # OpenCV repository path (default: ../..) --contrib DIR # opencv_contrib path --without MODULE # Exclude module (repeat for multiple) --disable FEATURE # Disable feature (e.g., --disable tbb) --dynamic # Build dynamic framework --enable_nonfree # Enable non-free modules --iphoneos_archs # Device architectures (default: arm64) --iphonesimulator_archs # Simulator architectures (default: x86_64,arm64) --iphoneos_deployment_target # Minimum iOS version (default: 9.0) --debug # Build debug version --framework_name NAME # Framework name (default: opencv2) --disable-swift # Disable Swift wrapper generation ``` ## Building for Specific iOS Versions Set minimum deployment target: ```bash theme={null} # For iOS 12.0 and later export IPHONEOS_DEPLOYMENT_TARGET=12.0 python3 build_framework.py ios # Or specify in command python3 build_framework.py ios --iphoneos_deployment_target=12.0 ``` ## Building visionOS Framework For Apple Vision Pro: ```bash theme={null} cd opencv/platforms/ios python3 build_visionos_framework.py ~/visionos_build ``` ## Framework Structure The built framework contains: ``` opencv2.framework/ opencv2 # Binary (fat library with all architectures) Headers/ # C++ headers Modules/ # Swift module files (if enabled) Info.plist # Framework metadata ``` Verify architectures: ```bash theme={null} lipo -info ~/ios/opencv2.framework/opencv2 # Output: Architectures in the fat file: opencv2 are: arm64 x86_64 arm64 ``` ## Integrating into Xcode Projects ### Method 1: Drag and Drop (Quick) 1. Drag `opencv2.framework` into your Xcode project 2. Check "Copy items if needed" 3. Select your target Ensure framework is in **Target → General → Frameworks, Libraries, and Embedded Content** Set to "Embed & Sign" for dynamic frameworks or "Do Not Embed" for static. Add required iOS frameworks: * `Accelerate.framework` * `AVFoundation.framework` * `CoreGraphics.framework` * `CoreMedia.framework` * `CoreVideo.framework` * `UIKit.framework` ### Method 2: CocoaPods For released versions: ```ruby theme={null} # Podfile platform :ios, '11.0' target 'YourApp' do pod 'OpenCV', '~> 4.5.0' # or pod 'OpenCV2', '~> 4.5.0' # Official pod end ``` Then: ```bash theme={null} pod install ``` ### Method 3: Swift Package Manager For projects using SPM: 1. **File → Add Packages** 2. Enter OpenCV repository URL 3. Select version Official SPM support may be limited. Check OpenCV repository for latest status. ## Using OpenCV in Code ### Objective-C Simple usage: ```objectivec theme={null} #import @implementation ViewController - (void)processImage:(UIImage *)image { // Convert UIImage to cv::Mat cv::Mat mat; UIImageToMat(image, mat); // Process image cv::Mat gray; cv::cvtColor(mat, gray, cv::COLOR_BGR2GRAY); // Convert back to UIImage UIImage *result = MatToUIImage(gray); } @end ``` ### Objective-C++ Bridge for Swift Create a wrapper class: ```objectivec theme={null} // OpenCVWrapper.h #import #import @interface OpenCVWrapper : NSObject + (UIImage *)processImage:(UIImage *)image; + (UIImage *)detectEdges:(UIImage *)image; @end ``` ```objectivec theme={null} // OpenCVWrapper.mm (note .mm extension) #import "OpenCVWrapper.h" #import #import @implementation OpenCVWrapper + (UIImage *)processImage:(UIImage *)image { cv::Mat mat; UIImageToMat(image, mat); cv::Mat gray; cv::cvtColor(mat, gray, cv::COLOR_BGR2GRAY); return MatToUIImage(gray); } + (UIImage *)detectEdges:(UIImage *)image { cv::Mat mat, edges; UIImageToMat(image, mat); cv::Canny(mat, edges, 50, 150); return MatToUIImage(edges); } @end ``` ### Swift Use through Objective-C++ wrapper: ```swift theme={null} import UIKit class ViewController: UIViewController { func processImage() { guard let image = UIImage(named: "sample") else { return } // Use OpenCV through wrapper let processed = OpenCVWrapper.processImage(image) imageView.image = processed let edges = OpenCVWrapper.detectEdges(image) edgesView.image = edges } } ``` ### Advanced Objective-C++ Integration Direct Mat usage in .mm files: ```objectivec theme={null} // ImageProcessor.mm #import "ImageProcessor.h" #import #import using namespace cv; @implementation ImageProcessor + (UIImage *)applyGaussianBlur:(UIImage *)image kernelSize:(int)size { Mat mat; UIImageToMat(image, mat); Mat blurred; GaussianBlur(mat, blurred, Size(size, size), 0); return MatToUIImage(blurred); } + (NSArray *)detectFaces:(UIImage *)image { Mat mat; UIImageToMat(image, mat); // Load cascade classifier NSString *cascadePath = [[NSBundle mainBundle] pathForResource:@"haarcascade_frontalface_default" ofType:@"xml"]; CascadeClassifier face_cascade; face_cascade.load([cascadePath UTF8String]); // Detect faces std::vector faces; face_cascade.detectMultiScale(mat, faces); // Convert to NSArray NSMutableArray *result = [NSMutableArray array]; for (const auto& face : faces) { CGRect rect = CGRectMake(face.x, face.y, face.width, face.height); [result addObject:[NSValue valueWithCGRect:rect]]; } return result; } @end ``` ## Camera Integration Real-time camera processing: ```objectivec theme={null} // CameraViewController.mm #import #import #import @interface CameraViewController () @property (strong, nonatomic) AVCaptureSession *captureSession; @property (strong, nonatomic) AVCaptureVideoPreviewLayer *previewLayer; @end @implementation CameraViewController - (void)viewDidLoad { [super viewDidLoad]; [self setupCamera]; } - (void)setupCamera { self.captureSession = [[AVCaptureSession alloc] init]; AVCaptureDevice *camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:camera error:nil]; [self.captureSession addInput:input]; AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init]; [output setSampleBufferDelegate:self queue:dispatch_get_main_queue()]; [self.captureSession addOutput:output]; [self.captureSession startRunning]; } - (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection { // Convert CMSampleBuffer to cv::Mat CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); CVPixelBufferLockBaseAddress(imageBuffer, 0); void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); cv::Mat mat(height, width, CV_8UC4, baseAddress); // Process frame cv::Mat gray; cv::cvtColor(mat, gray, cv::COLOR_BGR2GRAY); CVPixelBufferUnlockBaseAddress(imageBuffer, 0); // Update UI with processed frame } @end ``` ## Core Image Integration Convert between OpenCV Mat and CIImage: ```objectivec theme={null} UIImage *MatToUIImage(const cv::Mat& mat) { NSData *data = [NSData dataWithBytes:mat.data length:mat.elemSize() * mat.total()]; CGColorSpaceRef colorSpace; if (mat.elemSize() == 1) { colorSpace = CGColorSpaceCreateDeviceGray(); } else { colorSpace = CGColorSpaceCreateDeviceRGB(); } CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data); CGImageRef imageRef = CGImageCreate( mat.cols, mat.rows, 8, 8 * mat.elemSize(), mat.step[0], colorSpace, kCGImageAlphaNone | kCGBitmapByteOrderDefault, provider, NULL, false, kCGRenderingIntentDefault ); UIImage *image = [UIImage imageWithCGImage:imageRef]; CGImageRelease(imageRef); CGDataProviderRelease(provider); CGColorSpaceRelease(colorSpace); return image; } void UIImageToMat(UIImage *image, cv::Mat& mat) { CGImageRef imageRef = image.CGImage; CGColorSpaceRef colorSpace = CGImageGetColorSpace(imageRef); size_t width = CGImageGetWidth(imageRef); size_t height = CGImageGetHeight(imageRef); mat.create(height, width, CV_8UC4); CGContextRef context = CGBitmapContextCreate( mat.data, width, height, 8, mat.step[0], colorSpace, kCGImageAlphaPremultipliedLast | kCGBitmapByteOrderDefault ); CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef); CGContextRelease(context); } ``` ## Performance Optimization ### Use Accelerate Framework OpenCV automatically uses iOS's Accelerate framework for optimized BLAS/LAPACK operations. ### Enable NEON Instructions Built by default for ARM architectures: ```bash theme={null} python3 build_framework.py ios --iphoneos_archs arm64 ``` ### Multi-threading ```objectivec theme={null} // Process images in background dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{ cv::Mat processed = processImage(inputMat); dispatch_async(dispatch_get_main_queue(), ^{ // Update UI }); }); ``` ## Troubleshooting Update CMake: ```bash theme={null} brew upgrade cmake # Or download latest from cmake.org ``` Xcode 12.2+ requires CMake 3.19.0 or later. 1. Ensure all required system frameworks are linked 2. Check framework was built for correct architecture 3. Verify bitcode settings match between app and framework ```bash theme={null} # Check framework architectures lipo -info opencv2.framework/opencv2 ``` 1. Ensure framework is added to project 2. Check **Build Settings → Framework Search Paths** 3. Verify **Header Search Paths** includes framework ``` $(PROJECT_DIR)/opencv2.framework/Headers ``` 1. Ensure .mm file is in your target 2. Add bridging header if needed: ```objectivec theme={null} // BridgingHeader.h #import "OpenCVWrapper.h" ``` 3. Set in **Build Settings → Objective-C Bridging Header** ## Sample Applications Explore example apps in the OpenCV repository: ```bash theme={null} cd opencv/samples/ios open *.xcodeproj ``` Samples include: * **HelloWorld** - Basic OpenCV integration * **FaceDetection** - Real-time face detection * **VideoFilters** - Video processing effects * **SquareDetection** - Shape detection ## App Store Submission ### Privacy Permissions Add to `Info.plist`: ```xml theme={null} NSCameraUsageDescription This app requires camera access for image processing NSPhotoLibraryUsageDescription This app needs to access your photos ``` ### Bitcode If using dynamic framework, ensure bitcode settings match: ```bash theme={null} # Build framework with bitcode enabled python3 build_framework.py ios --dynamic ``` ### Framework Size Optimization ```bash theme={null} # Build with only needed architectures python3 build_framework.py ios \ --iphoneos_archs arm64 \ --iphonesimulator_archs arm64 \ --without video --without objc ``` ## Next Steps Advanced Swift usage patterns Real-time camera applications Combine OpenCV with Core ML Explore example projects # Building OpenCV on Linux Source: https://opencv-opencv.mintlify.app/platforms/linux Complete guide for building and installing OpenCV on Linux distributions with various compilers and configurations OpenCV provides comprehensive support for Linux platforms across multiple architectures including x86, x86\_64, ARM, RISC-V, and more. ## Quick Start Get up and running with OpenCV on Linux in a few minutes: Install compiler, build tools, and CMake: ```bash theme={null} # Debian/Ubuntu sudo apt update sudo apt install build-essential cmake git pkg-config # Fedora/RHEL sudo dnf install gcc gcc-c++ cmake git # Arch Linux sudo pacman -S base-devel cmake git ``` Get the source code from GitHub: ```bash theme={null} cd ~ git clone https://github.com/opencv/opencv.git cd opencv ``` Or download a release archive: ```bash theme={null} wget -O opencv.zip https://github.com/opencv/opencv/archive/4.x.zip unzip opencv.zip cd opencv-4.x ``` Configure and compile: ```bash theme={null} mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=Release .. make -j$(nproc) ``` Install to system directories (optional): ```bash theme={null} sudo make install ``` ## Compiler Support OpenCV supports multiple compilers on Linux: GCC is the default compiler on most Linux systems: ```bash theme={null} # Install GCC sudo apt install gcc g++ # Verify version (5.x or later required) gcc --version # Build with GCC (default) cmake -DCMAKE_BUILD_TYPE=Release .. make -j$(nproc) ``` Clang offers faster compilation and better diagnostics: ```bash theme={null} # Install Clang sudo apt install clang # Configure to use Clang cmake -DCMAKE_C_COMPILER=clang \ -DCMAKE_CXX_COMPILER=clang++ \ -DCMAKE_BUILD_TYPE=Release \ .. make -j$(nproc) ``` ## Build System Options Traditional GNU Make build system: ```bash theme={null} cmake -DCMAKE_BUILD_TYPE=Release .. make -j$(nproc) ``` The `-j$(nproc)` flag enables parallel compilation using all CPU cores. Ninja is faster than Make for large projects: ```bash theme={null} # Install Ninja sudo apt install ninja-build # Configure for Ninja cmake -GNinja -DCMAKE_BUILD_TYPE=Release .. ninja ``` Ninja automatically detects and uses all available CPU cores. ## Installing Dependencies ### Required Dependencies ```bash theme={null} # Debian/Ubuntu - Minimal build sudo apt install build-essential cmake git # Debian/Ubuntu - With common features sudo apt install build-essential cmake git pkg-config \ libgtk-3-dev libavcodec-dev libavformat-dev \ libswscale-dev libv4l-dev libxvidcore-dev \ libx264-dev libjpeg-dev libpng-dev libtiff-dev \ gfortran openexr libatlas-base-dev python3-dev \ python3-numpy libtbb2 libtbb-dev libdc1394-dev ``` ### Optional Dependencies ```bash theme={null} # GTK+ 3 (recommended) sudo apt install libgtk-3-dev # Qt5 (alternative) sudo apt install qtbase5-dev qttools5-dev ``` ```bash theme={null} # FFmpeg libraries sudo apt install libavcodec-dev libavformat-dev libswscale-dev # GStreamer sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev # V4L (Video4Linux) sudo apt install libv4l-dev v4l-utils ``` ```bash theme={null} sudo apt install libjpeg-dev libpng-dev libtiff-dev \ libwebp-dev libopenexr-dev ``` ```bash theme={null} sudo apt install python3-dev python3-numpy python3-pip ``` ```bash theme={null} # Intel TBB (Threading Building Blocks) sudo apt install libtbb-dev # OpenMP (usually included with GCC) # Already available with gcc package ``` ## Cross-Compilation OpenCV provides toolchain files for cross-compilation to other architectures: ### ARM 32-bit (ARMv7) ```bash theme={null} # Install cross-compiler sudo apt install gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf # Configure with toolchain file cmake -DCMAKE_TOOLCHAIN_FILE=../platforms/linux/arm-gnueabi.toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ .. make -j$(nproc) ``` ### ARM 64-bit (AArch64) ```bash theme={null} # Install cross-compiler sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu # Configure with toolchain file cmake -DCMAKE_TOOLCHAIN_FILE=../platforms/linux/aarch64-gnu.toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ .. make -j$(nproc) ``` ### RISC-V 64-bit ```bash theme={null} # Install RISC-V toolchain sudo apt install gcc-riscv64-linux-gnu g++-riscv64-linux-gnu # Configure with toolchain file cmake -DCMAKE_TOOLCHAIN_FILE=../platforms/linux/riscv64-gcc.toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ .. make -j$(nproc) ``` Available toolchain files in `platforms/linux/`: * `arm-gnueabi.toolchain.cmake` - ARM v7 with hard float * `aarch64-gnu.toolchain.cmake` - ARM 64-bit * `riscv64-gcc.toolchain.cmake` - RISC-V 64-bit (GCC) * `riscv64-clang.toolchain.cmake` - RISC-V 64-bit (Clang) * `ppc64le-gnu.toolchain.cmake` - PowerPC 64-bit little-endian * `mips64r6el-gnu.toolchain.cmake` - MIPS 64-bit ## Build Configuration Options ### Common CMake Options ```bash theme={null} cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local \ -DBUILD_EXAMPLES=ON \ -DBUILD_TESTS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DWITH_OPENGL=ON \ -DWITH_TBB=ON \ -DWITH_GTK=ON \ .. ``` ### Minimal Build for Embedded Systems ```bash theme={null} cmake -DCMAKE_BUILD_TYPE=MinSizeRel \ -DBUILD_SHARED_LIBS=OFF \ -DBUILD_TESTS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DBUILD_EXAMPLES=OFF \ -DWITH_GTK=OFF \ -DWITH_QT=OFF \ -DWITH_OPENGL=OFF \ -DWITH_FFMPEG=OFF \ .. ``` ### GPU Acceleration ```bash theme={null} # Install CUDA Toolkit from NVIDIA # https://developer.nvidia.com/cuda-downloads cmake -DWITH_CUDA=ON \ -DCUDA_ARCH_BIN="6.0 6.1 7.0 7.5 8.0 8.6" \ -DCMAKE_BUILD_TYPE=Release \ .. ``` ```bash theme={null} # Install OpenCL headers and ICD loader sudo apt install opencl-headers ocl-icd-opencl-dev cmake -DWITH_OPENCL=ON \ -DCMAKE_BUILD_TYPE=Release \ .. ``` ```bash theme={null} # Install VA-API development files sudo apt install libva-dev cmake -DWITH_VA=ON \ -DCMAKE_BUILD_TYPE=Release \ .. ``` ## Installation ### System-Wide Installation System-wide installation requires root privileges and may conflict with distribution packages. Consider using a custom prefix instead. ```bash theme={null} # Default installation to /usr/local sudo make install sudo ldconfig # Files are installed to: # /usr/local/lib - libraries (.so files) # /usr/local/include/opencv4 - headers # /usr/local/bin - executables # /usr/local/share/opencv4 - data files # /usr/local/lib/cmake/opencv4 - CMake config ``` ### Custom Installation Directory ```bash theme={null} # Install to user directory cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local \ -DCMAKE_BUILD_TYPE=Release \ .. make -j$(nproc) make install # Update library path echo 'export LD_LIBRARY_PATH=$HOME/.local/lib:$LD_LIBRARY_PATH' >> ~/.bashrc source ~/.bashrc ``` ## Verification Verify your OpenCV installation: ```bash theme={null} # Check libraries ls build/lib/ # Check CMake package ls build/opencv*.cmake ``` ```python theme={null} python3 -c "import cv2; print(cv2.__version__)" ``` ```python theme={null} python3 -c "import cv2; print(cv2.getBuildInformation())" ``` ## Distribution-Specific Notes ```bash theme={null} # Install from distribution packages (older version) sudo apt install libopencv-dev python3-opencv # Or build from source for latest version sudo apt install build-essential cmake git pkg-config \ libgtk-3-dev libavcodec-dev libavformat-dev \ libswscale-dev ``` ```bash theme={null} # Install from distribution packages sudo dnf install opencv opencv-devel # Or build from source sudo dnf install gcc gcc-c++ cmake git gtk3-devel \ ffmpeg-devel python3-devel numpy ``` ```bash theme={null} # Install from distribution packages sudo pacman -S opencv vtk hdf5 glew # Or build from source sudo pacman -S base-devel cmake git gtk3 ffmpeg python-numpy ``` ## Troubleshooting Install development packages for missing dependencies: ```bash theme={null} # Check CMake output for missing packages # Install corresponding -dev or -devel packages sudo apt install lib-dev ``` Clean and rebuild: ```bash theme={null} rm -rf build/ mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=Release .. make -j$(nproc) ``` Ensure Python can find the cv2 module: ```bash theme={null} # Find where cv2.so was installed find /usr/local -name "cv2*.so" # Add to Python path if needed export PYTHONPATH=/usr/local/lib/python3.x/site-packages:$PYTHONPATH ``` ## Next Steps Explore all CMake configuration options Write your first OpenCV application Build for embedded systems Enable CUDA and OpenCL support # Building OpenCV on macOS Source: https://opencv-opencv.mintlify.app/platforms/macos Complete guide for building and installing OpenCV on macOS with support for Intel and Apple Silicon OpenCV provides native support for macOS on both Intel (x86\_64) and Apple Silicon (arm64) architectures. Build universal binaries or create XCFrameworks for multi-platform support. ## Quick Start Get OpenCV running on macOS in minutes: ```bash theme={null} xcode-select --install ``` Xcode 12.2 or later is required. The full Xcode app is recommended but not strictly necessary for basic builds. ```bash theme={null} /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` Via Homebrew: ```bash theme={null} brew install cmake ``` Or download from [cmake.org](https://cmake.org/download/) and install the .dmg package. ```bash theme={null} cd ~/ git clone https://github.com/opencv/opencv.git cd opencv mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=Release .. make -j$(sysctl -n hw.ncpu) sudo make install ``` ## System Requirements * **macOS**: 10.12 (Sierra) or later * **Xcode**: 12.2 or later * **CMake**: 3.19.0 or later (3.17+ for older Xcode) * **Python**: 3.8 or later (for Python bindings) macOS 12.3+ does not include Python 2.7. Install Python 3.x from [python.org](https://www.python.org/) or Homebrew. ## Installation Methods For the latest features and custom configuration: ```bash theme={null} # Clone repository git clone https://github.com/opencv/opencv.git cd opencv mkdir build && cd build # Configure cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local \ -DBUILD_EXAMPLES=ON \ .. # Build using all CPU cores make -j$(sysctl -n hw.ncpu) # Install sudo make install ``` For quick installation of stable releases: ```bash theme={null} # Install OpenCV brew install opencv # With contrib modules (unofficial tap) brew install opencv@4 ``` Homebrew packages may not include all optional features. Build from source for full control. For Python bindings only: ```bash theme={null} # Basic OpenCV pip install opencv-python # With contrib modules pip install opencv-contrib-python ``` ## Building for Apple Silicon OpenCV supports native Apple Silicon (M1/M2/M3) builds: ### Native ARM64 Build ```bash theme={null} # Build natively on Apple Silicon Mac cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_OSX_ARCHITECTURES=arm64 \ .. make -j$(sysctl -n hw.ncpu) ``` ### Universal Binary (x86\_64 + arm64) ```bash theme={null} # Build for both Intel and Apple Silicon cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64" \ .. make -j$(sysctl -n hw.ncpu) ``` Universal binaries work on both Intel and Apple Silicon Macs, but result in larger file sizes. ### Architecture-Specific Optimizations ```bash theme={null} # Apple Silicon - enable ARM NEON optimizations cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_OSX_ARCHITECTURES=arm64 \ -DENABLE_NEON=ON \ .. # Intel - enable AVX/AVX2 cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_OSX_ARCHITECTURES=x86_64 \ -DCPU_BASELINE=AVX2 \ .. ``` ## Building OpenCV Framework For use in Xcode projects, build OpenCV as a framework: ```bash theme={null} cd ~/ git clone https://github.com/opencv/opencv.git cd opencv/platforms/osx # Build framework for macOS python3 build_framework.py ~/opencv_build # Build universal binary framework python3 build_framework.py ~/opencv_build \ --macos_archs x86_64,arm64 ``` The script creates: * `~/opencv_build/opencv2.framework` - The framework * `~/opencv_build/build/` - Intermediate build files ### Framework Build Options ```bash theme={null} # With contrib modules python3 build_framework.py ~/opencv_build \ --opencv ~/opencv \ --contrib ~/opencv_contrib # Exclude specific modules python3 build_framework.py ~/opencv_build \ --without video --without objc # Enable non-free modules python3 build_framework.py ~/opencv_build \ --enable_nonfree # Build only for Apple Silicon python3 build_framework.py ~/opencv_build \ --macos_archs arm64 \ --build_only_specified_archs ``` ## Building XCFramework For multi-platform distribution (macOS + iOS + Catalyst): ```bash theme={null} cd opencv/platforms/apple python3 build_xcframework.py --out ~/opencv_xcframework ``` This builds OpenCV for: * **macOS**: x86\_64, arm64 * **iOS**: arm64, armv7 * **iOS Simulator**: x86\_64, arm64 * **Mac Catalyst**: x86\_64, arm64 The resulting `opencv2.xcframework` can be used across all Apple platforms. ### XCFramework Build Options ```bash theme={null} # Build only for macOS python3 build_xcframework.py --out ~/opencv_xcframework \ --macos_archs arm64,x86_64 \ --build_only_specified_archs # With contrib modules python3 build_xcframework.py --out ~/opencv_xcframework \ --contrib ~/opencv_contrib # Exclude modules to reduce size python3 build_xcframework.py --out ~/opencv_xcframework \ --without video --without objc ``` Building XCFramework can take 30-60 minutes as it compiles for 8 architectures across 4 platforms. Use `--build_only_specified_archs` to reduce build time. ## Dependencies and Optional Features ### Core Dependencies ```bash theme={null} # Install via Homebrew brew install cmake pkg-config # For Python support brew install python numpy ``` ### Optional Dependencies ```bash theme={null} brew install jpeg libpng libtiff webp openexr ``` ```bash theme={null} brew install ffmpeg # Enable in CMake cmake -DWITH_FFMPEG=ON .. ``` macOS uses native Cocoa framework by default (no additional dependencies). For Qt-based GUI: ```bash theme={null} brew install qt5 cmake -DWITH_QT=ON \ -DQt5_DIR=/usr/local/opt/qt5/lib/cmake/Qt5 \ .. ``` ```bash theme={null} # Intel TBB brew install tbb cmake -DWITH_TBB=ON .. ``` macOS includes OpenCL by default: ```bash theme={null} cmake -DWITH_OPENCL=ON .. ``` ## CMake Configuration Options ### Standard Build ```bash theme={null} cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local \ -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 \ -DBUILD_EXAMPLES=ON \ -DBUILD_TESTS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DWITH_TBB=ON \ -DWITH_OPENCL=ON \ .. ``` ### Python-Specific Configuration ```bash theme={null} cmake -DCMAKE_BUILD_TYPE=Release \ -DPYTHON3_EXECUTABLE=$(which python3) \ -DPYTHON3_INCLUDE_DIR=$(python3 -c "from sysconfig import get_paths; print(get_paths()['include'])") \ -DPYTHON3_NUMPY_INCLUDE_DIRS=$(python3 -c "import numpy; print(numpy.get_include())") \ .. ``` Python 2 support has been removed from recent OpenCV versions. Use Python 3.8 or later. ### Optimized Build for Apple Silicon ```bash theme={null} cmake -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_OSX_ARCHITECTURES=arm64 \ -DENABLE_NEON=ON \ -DWITH_TBB=ON \ -DWITH_OPENCL=ON \ -DBUILD_TESTS=OFF \ .. ``` ## Building with opencv\_contrib Include extra modules from opencv\_contrib: ```bash theme={null} # Clone both repositories cd ~/ git clone https://github.com/opencv/opencv.git git clone https://github.com/opencv/opencv_contrib.git # Build with contrib cd opencv mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=Release \ -DOPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \ .. make -j$(sysctl -n hw.ncpu) ``` ## Installation Locations ### Default Installation With `sudo make install`, files are placed in: ```bash theme={null} /usr/local/lib/ # Libraries (.dylib) /usr/local/include/opencv4/ # Headers /usr/local/bin/ # Executables /usr/local/share/opencv4/ # Data files /usr/local/lib/cmake/opencv4/ # CMake config /usr/local/lib/python3.x/site-packages/ # Python module ``` ### Custom Installation ```bash theme={null} # Install to user directory cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local \ -DCMAKE_BUILD_TYPE=Release \ .. make -j$(sysctl -n hw.ncpu) make install # Update paths export PATH=$HOME/.local/bin:$PATH export DYLD_LIBRARY_PATH=$HOME/.local/lib:$DYLD_LIBRARY_PATH export PKG_CONFIG_PATH=$HOME/.local/lib/pkgconfig:$PKG_CONFIG_PATH ``` ## Using OpenCV in Xcode Projects ### With CMake (Recommended) In your `CMakeLists.txt`: ```cmake theme={null} cmake_minimum_required(VERSION 3.17) project(MyProject) find_package(OpenCV REQUIRED) include_directories(${OpenCV_INCLUDE_DIRS}) add_executable(myapp main.cpp) target_link_libraries(myapp ${OpenCV_LIBS}) ``` Generate Xcode project: ```bash theme={null} cmake -G Xcode -DOpenCV_DIR=~/opencv/build .. ``` ### With OpenCV Framework 1. Drag `opencv2.framework` into your Xcode project 2. Add to **Target → Build Phases → Link Binary With Libraries** 3. Include in code: ```objectivec theme={null} #import ``` ## Verification Test your installation: Create `test.cpp`: ```cpp theme={null} #include #include int main() { std::cout << "OpenCV version: " << CV_VERSION << std::endl; return 0; } ``` Compile and run: ```bash theme={null} clang++ test.cpp -o test $(pkg-config --cflags --libs opencv4) ./test ``` ```bash theme={null} # Check version python3 -c "import cv2; print(cv2.__version__)" # Show build info python3 -c "import cv2; print(cv2.getBuildInformation())" ``` ```bash theme={null} # Verify framework structure ls -la ~/opencv_build/opencv2.framework/ # Check architectures lipo -info ~/opencv_build/opencv2.framework/opencv2 ``` ## Performance Optimization ### Use Accelerate Framework macOS's Accelerate framework provides optimized BLAS/LAPACK: ```bash theme={null} cmake -DWITH_LAPACK=ON \ -DLAPACK_LIBRARIES="-framework Accelerate" \ .. ``` ### Enable Multi-threading ```bash theme={null} cmake -DWITH_TBB=ON \ -DWITH_OPENMP=OFF \ # OpenMP not recommended on macOS .. ``` ## Troubleshooting Check installation path: ```bash theme={null} python3 -c "import sys; print(sys.path)" find /usr/local -name "cv2*.so" 2>/dev/null ``` Add to PYTHONPATH if needed: ```bash theme={null} export PYTHONPATH=/usr/local/lib/python3.x/site-packages:$PYTHONPATH ``` Update CMake: ```bash theme={null} brew upgrade cmake # Or download latest from cmake.org ``` Xcode 12.2+ requires CMake 3.19.0+ Ensure consistent architecture: ```bash theme={null} # Check what you're building for cmake -LA | grep CMAKE_OSX_ARCHITECTURES # Rebuild for specific architecture rm -rf build/ mkdir build && cd build cmake -DCMAKE_OSX_ARCHITECTURES=arm64 .. ``` Clean and retry: ```bash theme={null} rm -rf ~/opencv_build cd opencv/platforms/osx python3 build_framework.py ~/opencv_build ``` Check Xcode and CMake versions meet requirements. ## Next Steps Build OpenCV for iOS devices Write your first OpenCV application Use OpenCV with Python on macOS Build for all Apple platforms # Platform Support Overview Source: https://opencv-opencv.mintlify.app/platforms/overview OpenCV platform support matrix, system requirements, and architecture compatibility OpenCV is designed to be cross-platform and runs on a wide variety of operating systems and architectures. This page provides an overview of supported platforms and their requirements. ## Supported Platforms OpenCV supports the following major platforms: Native support for x86, x86\_64, ARM, and other architectures Full support via Visual Studio and MinGW toolchains Universal binaries for Intel and Apple Silicon AAR packages and NDK integration XCFramework with Objective-C and Swift support ## Architecture Support Matrix OpenCV supports a wide range of processor architectures across different platforms: | Platform | Architectures | Build System | | ----------- | ------------------------------------------------- | --------------------------- | | **Linux** | x86, x86\_64, ARM (v7, v8), RISC-V, PowerPC, MIPS | CMake + Make/Ninja | | **Windows** | x86, x86\_64, ARM64 | CMake + Visual Studio/Ninja | | **macOS** | x86\_64, arm64 (Apple Silicon) | CMake + Xcode | | **Android** | armeabi-v7a, arm64-v8a, x86, x86\_64 | CMake + Gradle/NDK | | **iOS** | armv7, arm64, i386, x86\_64 (simulator) | CMake + Xcode | ## Minimum Requirements ### Build Tools * GCC 5.x+ or Clang 3.4+ * CMake 3.5.1+ * Make or Ninja * Git (optional) * Visual Studio 2015+ (MSVC 14.0+) * CMake 3.5.1+ * Git (optional) * Xcode 12.2+ with Command Line Tools * CMake 3.19.0+ * Python 3.6+ * Android Studio or Android SDK/NDK * CMake 3.6+ * Python 3.6+ (for build scripts) * Gradle 6.0+ * Xcode 12.2+ with Command Line Tools * CMake 3.19.0+ * Python 3.6+ ### Runtime Requirements Minimum OS versions for running OpenCV applications: * **Linux**: Kernel 2.6.32+ (older kernels may work with reduced functionality) * **Windows**: Windows 7 SP1 or later * **macOS**: macOS 10.12 (Sierra) or later * **Android**: API Level 21 (Android 5.0 Lollipop) or later * **iOS**: iOS 9.0 or later ## Cross-Compilation Support OpenCV provides toolchain files for cross-compilation: ```bash theme={null} # ARM Linux cross-compilation example cmake -DCMAKE_TOOLCHAIN_FILE=platforms/linux/arm-gnueabi.toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release \ .. ``` ### Available Toolchains The `platforms/` directory contains toolchain files for: * **ARM**: `arm-gnueabi.toolchain.cmake`, `aarch64-gnu.toolchain.cmake` * **RISC-V**: `riscv64-gcc.toolchain.cmake`, `riscv64-clang.toolchain.cmake` * **MIPS**: `mips32r5el-gnu.toolchain.cmake`, `mips64r6el-gnu.toolchain.cmake` * **PowerPC**: `ppc64-gnu.toolchain.cmake`, `ppc64le-gnu.toolchain.cmake` ## Hardware Acceleration OpenCV leverages platform-specific acceleration technologies: | Platform | Acceleration Technologies | | ----------------- | ---------------------------------------- | | **All Platforms** | SSE2/3/4, AVX, AVX2, AVX-512, NEON | | **Linux** | OpenCL, CUDA, TBB, OpenMP, VA-API | | **Windows** | OpenCL, CUDA, TBB, OpenMP, DirectX | | **macOS** | OpenCL, TBB, Accelerate Framework | | **Android** | OpenCL, Vulkan, NNAPI, FastCV (Qualcomm) | | **iOS** | Accelerate Framework, Metal, Core ML | GPU acceleration (CUDA, OpenCL) requires compatible hardware and additional setup. Not all modules support all acceleration technologies. ## Binary Distributions Pre-built binaries are available for common configurations: * **Windows**: Installer packages for Visual Studio 2015-2022 (x86, x64) * **Android**: AAR packages via Maven Central * **iOS**: Pre-built frameworks available in releases * **Python**: `pip install opencv-python` for major platforms Building from source gives you more control over enabled features and optimizations for your specific hardware. ## Platform-Specific Considerations ### Mobile Platforms For iOS and Android: * Framework/AAR sizes can be large (\~50-100MB). Consider using modular builds to reduce size. * Some modules are not available on mobile (e.g., highgui video capture) * Use `--without` flag when building to exclude unnecessary modules ### Embedded Systems For resource-constrained embedded Linux systems: ```bash theme={null} # Minimal build for embedded systems cmake -DBUILD_SHARED_LIBS=OFF \ -DBUILD_TESTS=OFF \ -DBUILD_PERF_TESTS=OFF \ -DBUILD_EXAMPLES=OFF \ -DWITH_GTK=OFF \ -DWITH_QT=OFF \ .. ``` ## Getting Started Select your platform to view detailed installation and build instructions: ## Additional Resources * [General Installation Tutorial](https://docs.opencv.org/4.x/df/d65/tutorial_table_of_content_introduction.html) * [CMake Configuration Reference](https://docs.opencv.org/4.x/db/d05/tutorial_config_reference.html) * [GitHub Repository](https://github.com/opencv/opencv) * [Release Downloads](https://opencv.org/releases/) # Building OpenCV on Windows Source: https://opencv-opencv.mintlify.app/platforms/windows Complete guide for building and installing OpenCV on Windows using Visual Studio, MinGW, and other toolchains OpenCV provides full support for Windows platforms with Visual Studio, MinGW, and other compilers. Pre-built binaries are also available for quick setup. ## Quick Start with Pre-built Libraries The fastest way to get started with OpenCV on Windows: Download the Windows installer from the [OpenCV releases page](https://opencv.org/releases/): ```powershell theme={null} # Example: opencv-4.x.0-windows.exe ``` Pre-built packages include binaries for Visual Studio 2015-2022 (x86 and x64). Run the self-extracting archive. It will create a directory structure: ``` C:\opencv\ build\ x64\ # 64-bit binaries vc15\ # Visual Studio 2017 vc16\ # Visual Studio 2019 vc17\ # Visual Studio 2022 x86\ # 32-bit binaries sources\ # Source code ``` Add OpenCV to your system path: ```powershell theme={null} # Set OpenCV_DIR (adjust version as needed) setx OpenCV_DIR C:\opencv\build\x64\vc17 # Add bin directory to PATH setx PATH "%PATH%;%OpenCV_DIR%\bin" ``` Restart your terminal or IDE after setting environment variables. ## Building from Source For the latest features or custom configurations, build OpenCV from source. ### Prerequisites **Required:** * [Visual Studio 2015 or later](https://visualstudio.microsoft.com/) (Community Edition is free) * Install "Desktop development with C++" workload * [CMake 3.5.1 or later](https://cmake.org/download/) * [Git for Windows](https://git-scm.com/download/win) (optional but recommended) **Optional:** * [Python 3.6+](https://www.python.org/downloads/windows/) for Python bindings * [NumPy](https://numpy.org/) for Python interface **Required:** * [MinGW-w64](https://www.mingw-w64.org/downloads/) * [CMake 3.5.1 or later](https://cmake.org/download/) * [Git for Windows](https://git-scm.com/download/win) (optional) MinGW builds may have limited compatibility with some third-party libraries. ### Build Steps with Visual Studio Open Git Bash or Command Prompt: ```bash theme={null} cd C:\ git clone https://github.com/opencv/opencv.git cd opencv ``` Or download and extract a release archive. ```bash theme={null} mkdir build cd build ``` Launch CMake GUI: 1. Set **Source code** to: `C:/opencv` 2. Set **Build binaries** to: `C:/opencv/build` 3. Click **Configure** 4. Select your Visual Studio version and platform (x64 recommended) 5. Click **Finish** Enable "Grouped" view for easier navigation of CMake options. Key CMake options to consider: ```cmake theme={null} BUILD_EXAMPLES=ON # Build example applications BUILD_TESTS=OFF # Skip tests for faster build BUILD_PERF_TESTS=OFF # Skip performance tests BUILD_opencv_world=ON # Build single combined library BUILD_SHARED_LIBS=ON # Build DLLs (not static libs) WITH_CUDA=OFF # Enable if you have NVIDIA GPU WITH_TBB=ON # Enable Intel TBB ``` Click **Configure** again after changes. Once configuration completes without errors: 1. Click **Generate** 2. Click **Open Project** to launch Visual Studio In Visual Studio: 1. Select **Release** configuration (or Debug) 2. Right-click **ALL\_BUILD** project → **Build** 3. Wait for compilation (15-60 minutes depending on options) Build both **Release** and **Debug** configurations if you need both. Right-click **INSTALL** project → **Build** This copies files to the install directory (default: `C:/Program Files/opencv`). ### Build Steps with Command Line For automated builds or CI/CD pipelines: ```powershell theme={null} # Clone repository git clone https://github.com/opencv/opencv.git cd opencv mkdir build cd build # Configure cmake -G "Visual Studio 17 2022" -A x64 ^ -DCMAKE_BUILD_TYPE=Release ^ -DBUILD_EXAMPLES=ON ^ -DBUILD_opencv_world=ON ^ .. # Build Release cmake --build . --config Release --target ALL_BUILD -j 8 # Build Debug cmake --build . --config Debug --target ALL_BUILD -j 8 # Install cmake --build . --config Release --target INSTALL ``` ```powershell theme={null} cmake -G "Visual Studio 17 2022" -A x64 .. ``` ```powershell theme={null} cmake -G "Visual Studio 16 2019" -A x64 .. ``` ```powershell theme={null} cmake -G "Visual Studio 15 2017 Win64" .. ``` ```powershell theme={null} # From Visual Studio Developer Command Prompt cmake -G Ninja -DCMAKE_BUILD_TYPE=Release .. ninja ``` ## Using Git Bash for Automated Build A complete build script using Git Bash: ```bash theme={null} #!/bin/bash -e myRepo=$(pwd) CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 17 2022" # Clone repositories if [ ! -d "$myRepo/opencv" ]; then echo "Cloning opencv" git clone https://github.com/opencv/opencv.git else cd opencv && git pull --rebase && cd .. fi if [ ! -d "$myRepo/opencv_contrib" ]; then echo "Cloning opencv_contrib" git clone https://github.com/opencv/opencv_contrib.git else cd opencv_contrib && git pull --rebase && cd .. fi # Build mkdir -p build_opencv cd build_opencv CMAKE_OPTIONS=( -DBUILD_PERF_TESTS:BOOL=OFF -DBUILD_TESTS:BOOL=OFF -DBUILD_DOCS:BOOL=OFF -DWITH_CUDA:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DINSTALL_CREATE_DISTRIB=ON -DOPENCV_EXTRA_MODULES_PATH="$myRepo/opencv_contrib/modules" -DCMAKE_INSTALL_PREFIX="$myRepo/install/opencv" ) cmake "${CMAKE_GENERATOR_OPTIONS[@]}" "${CMAKE_OPTIONS[@]}" "$myRepo/opencv" # Build both configurations cmake --build . --config Debug cmake --build . --config Release # Install cmake --build . --target install --config Release cmake --build . --target install --config Debug ``` ## Building with opencv\_contrib To include extra modules from opencv\_contrib: ```powershell theme={null} # Clone opencv_contrib alongside opencv cd C:\ git clone https://github.com/opencv/opencv.git git clone https://github.com/opencv/opencv_contrib.git cd opencv\build # Configure with contrib modules cmake -G "Visual Studio 17 2022" -A x64 ^ -DOPENCV_EXTRA_MODULES_PATH=C:/opencv_contrib/modules ^ .. ``` ## Optional Dependencies ### Intel Threading Building Blocks (TBB) For improved parallel processing performance: 1. Download TBB from [Intel oneAPI](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html) 2. Extract to `C:\opencv\dep\tbb` 3. Add CMake option: `-DWITH_TBB=ON -DTBB_DIR=C:/opencv/dep/tbb` ### CUDA (NVIDIA GPU Acceleration) Download from [NVIDIA CUDA Downloads](https://developer.nvidia.com/cuda-downloads) Install with default options. ```powershell theme={null} cmake -G "Visual Studio 17 2022" -A x64 ^ -DWITH_CUDA=ON ^ -DCUDA_ARCH_BIN="6.0 6.1 7.0 7.5 8.0 8.6 8.9" ^ .. ``` Set `CUDA_ARCH_BIN` to match your GPU's compute capability. Build time increases significantly with CUDA enabled. ### Python Support ```powershell theme={null} # Install Python and NumPy pip install numpy # CMake will auto-detect Python # Or specify explicitly: cmake -DPYTHON3_EXECUTABLE="C:/Python311/python.exe" ^ -DPYTHON3_INCLUDE_DIR="C:/Python311/include" ^ -DPYTHON3_NUMPY_INCLUDE_DIRS="C:/Python311/Lib/site-packages/numpy/core/include" ^ .. ``` ## Build Configuration Options ### Recommended Settings for Development ```cmake theme={null} BUILD_EXAMPLES=ON # Sample applications BUILD_opencv_world=ON # Single library file BUILD_SHARED_LIBS=ON # DLL files ENABLE_SOLUTION_FOLDERS=ON # Organize VS projects BUILD_TESTS=OFF # Skip tests BUILD_PERF_TESTS=OFF # Skip perf tests WITH_TBB=ON # Parallel processing WITH_OPENGL=ON # OpenGL support ``` ### Minimal Build for Distribution ```cmake theme={null} BUILD_EXAMPLES=OFF BUILD_TESTS=OFF BUILD_PERF_TESTS=OFF BUILD_DOCS=OFF BUILD_opencv_apps=OFF BUILD_opencv_world=ON # Recommended for easier deployment INSTALL_CREATE_DISTRIB=ON ``` ## Setting Up Your Development Environment ### Using OpenCV in Visual Studio Projects ```powershell theme={null} setx OpenCV_DIR "C:\opencv\build\x64\vc17" ``` ```powershell theme={null} setx PATH "%PATH%;%OpenCV_DIR%\bin" ``` Or manually add `C:\opencv\build\x64\vc17\bin` to System PATH via Control Panel. In your project's CMakeLists.txt: ```cmake theme={null} find_package(OpenCV REQUIRED) include_directories(${OpenCV_INCLUDE_DIRS}) target_link_libraries(your_target ${OpenCV_LIBS}) ``` Or manually configure Include Directories and Library Directories in Visual Studio project properties. ## Verification Test your OpenCV installation: ```powershell theme={null} # Check Python binding python -c "import cv2; print(cv2.__version__)" # Run example application cd C:\opencv\build\bin\Release opencv_version.exe ``` ```powershell theme={null} # List built libraries Get-ChildItem C:\opencv\build\lib\Release # List executables Get-ChildItem C:\opencv\build\bin\Release\*.exe ``` ## Package Manager Installation ### vcpkg For dependency management: ```powershell theme={null} # Install vcpkg git clone https://github.com/Microsoft/vcpkg.git cd vcpkg .\bootstrap-vcpkg.bat # Install OpenCV .\vcpkg install opencv[contrib,cuda]:x64-windows # Integrate with Visual Studio .\vcpkg integrate install ``` ### Conan ```powershell theme={null} # Install Conan pip install conan # Install OpenCV conan install opencv/4.5.5@ ``` ## Troubleshooting * Ensure Visual Studio is installed with C++ tools * Use "Developer Command Prompt for VS" to run CMake * Specify generator explicitly: `-G "Visual Studio 17 2022"` Add OpenCV bin directory to PATH: ```powershell theme={null} setx PATH "%PATH%;C:\opencv\build\x64\vc17\bin" ``` Or copy DLL files next to your executable. * Verify NumPy is installed: `pip install numpy` * Check Python version matches (32/64-bit) * Verify cv2.pyd is in Python's site-packages * Try rebuilding with correct Python paths * Ensure CUDA Toolkit version matches VS version compatibility * Update GPU drivers * Reduce `CUDA_ARCH_BIN` to only your GPU's compute capability ## Next Steps Set up OpenCV in Visual Studio projects Explore all configuration options Write your first OpenCV application Use OpenCV with Python # Quickstart Guide Source: https://opencv-opencv.mintlify.app/quickstart Get started with OpenCV in minutes - from installation to your first working computer vision application ## Prerequisites Before starting, ensure you have OpenCV installed. See the [Installation Guide](/installation) if you haven't already set up OpenCV. ## Your First OpenCV Program Let's create a simple program to load and display an image. Create a new directory for your project: ```bash theme={null} mkdir opencv-quickstart cd opencv-quickstart ``` Create a file and load an image: Create `display_image.py`: ```python theme={null} import cv2 as cv import sys # Load an image img = cv.imread('image.jpg') # Check if image was loaded successfully if img is None: sys.exit("Could not read the image.") # Display the image cv.imshow("Display window", img) k = cv.waitKey(0) # Save if 's' key is pressed if k == ord("s"): cv.imwrite("output.png", img) ``` Make sure you have an image file named `image.jpg` in the same directory, or use `cv.samples.findFile("starry_night.jpg")` to use a built-in sample image. Create `display_image.cpp`: ```cpp theme={null} #include #include int main() { // Load an image cv::Mat img = cv::imread("image.jpg"); // Check if image was loaded successfully if (img.empty()) { std::cout << "Could not read the image" << std::endl; return 1; } // Display the image cv::imshow("Display window", img); int k = cv::waitKey(0); // Save if 's' key is pressed if (k == 's') { cv::imwrite("output.png", img); } return 0; } ``` Create `DisplayImage.java`: ```java theme={null} import org.opencv.core.Core; import org.opencv.core.Mat; import org.opencv.highgui.HighGui; import org.opencv.imgcodecs.Imgcodecs; public class DisplayImage { public static void main(String[] args) { // Load OpenCV native library System.loadLibrary(Core.NATIVE_LIBRARY_NAME); // Load an image Mat img = Imgcodecs.imread("image.jpg"); // Check if image was loaded if (img.empty()) { System.out.println("Could not read the image"); return; } // Display the image HighGui.imshow("Display window", img); HighGui.waitKey(0); } } ``` Execute your program: ```bash theme={null} python display_image.py ``` Compile and run: ```bash theme={null} # Using g++ g++ display_image.cpp -o display_image `pkg-config --cflags --libs opencv4` ./display_image # Using CMake (recommended) # Create CMakeLists.txt first cmake . make ./display_image ``` ```bash theme={null} javac -cp ".:/path/to/opencv-VERSION.jar" DisplayImage.java java -cp ".:/path/to/opencv-VERSION.jar" -Djava.library.path=/path/to/opencv/lib DisplayImage ``` A window will appear displaying your image. Press any key to close it, or press 's' to save the image. ## Basic Image Processing Now let's do some actual image processing: ```python theme={null} import cv2 as cv # Read the image img = cv.imread('image.jpg') # Convert to grayscale gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Apply Gaussian blur blurred = cv.GaussianBlur(gray, (5, 5), 0) # Detect edges using Canny edges = cv.Canny(blurred, 50, 150) # Display all results cv.imshow('Original', img) cv.imshow('Grayscale', gray) cv.imshow('Edges', edges) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include int main() { // Read the image cv::Mat img = cv::imread("image.jpg"); cv::Mat gray, blurred, edges; // Convert to grayscale cv::cvtColor(img, gray, cv::COLOR_BGR2GRAY); // Apply Gaussian blur cv::GaussianBlur(gray, blurred, cv::Size(5, 5), 0); // Detect edges using Canny cv::Canny(blurred, edges, 50, 150); // Display all results cv::imshow("Original", img); cv::imshow("Grayscale", gray); cv::imshow("Edges", edges); cv::waitKey(0); cv::destroyAllWindows(); return 0; } ``` ## Working with Video Process video from a file or webcam: ```python theme={null} import cv2 as cv # Open webcam (0 = default camera) cap = cv.VideoCapture(0) # Or open a video file # cap = cv.VideoCapture('video.mp4') while True: # Read frame ret, frame = cap.read() if not ret: break # Convert to grayscale gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) # Display cv.imshow('Webcam', gray) # Break on 'q' key if cv.waitKey(1) & 0xFF == ord('q'): break cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include int main() { // Open webcam cv::VideoCapture cap(0); // Or open a video file // cv::VideoCapture cap("video.mp4"); if (!cap.isOpened()) { std::cout << "Error opening video stream" << std::endl; return -1; } cv::Mat frame, gray; while (true) { // Read frame cap >> frame; if (frame.empty()) break; // Convert to grayscale cv::cvtColor(frame, gray, cv::COLOR_BGR2GRAY); // Display cv::imshow("Webcam", gray); // Break on 'q' key if (cv::waitKey(1) == 'q') break; } cap.release(); cv::destroyAllWindows(); return 0; } ``` ## Common Operations Here are some frequently used OpenCV operations: ### Reading and Writing ```python theme={null} # Read image img = cv.imread('input.jpg') # Save image cv.imwrite('output.jpg', img) # Read video cap = cv.VideoCapture('video.mp4') # Write video fourcc = cv.VideoWriter_fourcc(*'XVID') out = cv.VideoWriter('output.avi', fourcc, 20.0, (640, 480)) ``` ### Image Transformations ```python theme={null} # Resize resized = cv.resize(img, (640, 480)) # Rotate (h, w) = img.shape[:2] center = (w // 2, h // 2) M = cv.getRotationMatrix2D(center, 45, 1.0) rotated = cv.warpAffine(img, M, (w, h)) # Flip flipped = cv.flip(img, 1) # 1 = horizontal, 0 = vertical, -1 = both ``` ### Drawing ```python theme={null} # Draw line cv.line(img, (0, 0), (100, 100), (255, 0, 0), 2) # Draw circle cv.circle(img, (50, 50), 25, (0, 255, 0), -1) # Draw rectangle cv.rectangle(img, (10, 10), (100, 100), (0, 0, 255), 2) # Put text cv.putText(img, 'Hello OpenCV', (10, 30), cv.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2) ``` ## Next Steps Now that you've created your first OpenCV program, explore more features: Learn about filtering, transformations, and color spaces Detect objects, faces, and features in images Process video streams and track objects Explore the complete OpenCV API ## Troubleshooting Make sure OpenCV is installed: ```bash theme={null} pip install opencv-python ``` Verify installation: ```python theme={null} import cv2 as cv print(cv.__version__) ``` This means the image couldn't be loaded. Check: * The file path is correct * The image file exists * You have read permissions * The image format is supported For headless environments, use a different backend: ```bash theme={null} export QT_QPA_PLATFORM=offscreen ``` Or save images instead of displaying them. Try different camera indices: ```python theme={null} cap = cv.VideoCapture(0) # Try 0, 1, 2, etc. ``` On Linux, ensure you have camera permissions. ## Resources * [OpenCV Tutorials](/tutorials/image-operations) - Step-by-step guides * [API Documentation](/api/core/mat) - Complete API reference * [Examples](/examples/read-display) - Working code samples * [GitHub Repository](https://github.com/opencv/opencv) - Source code and issues # Camera Calibration Source: https://opencv-opencv.mintlify.app/tutorials/camera-calibration Learn how to calibrate cameras, compute intrinsic and extrinsic parameters, and remove lens distortion in OpenCV # Camera Calibration Learn how to calibrate cameras to correct lens distortion and obtain accurate 3D measurements from images. ## Why Camera Calibration? Camera calibration is essential for: * Removing lens distortion from images * Measuring real-world dimensions from images * 3D reconstruction and depth estimation * Augmented reality applications * Accurate object tracking and positioning ### Camera Parameters Internal camera properties: * **Focal length** (fx, fy): Distance from lens to sensor * **Principal point** (cx, cy): Image center offset * **Skew coefficient**: Axis skewness (usually 0) * **Distortion coefficients**: Radial and tangential distortion Represented as camera matrix K: ``` K = [fx 0 cx] [0 fy cy] [0 0 1] ``` Camera position and orientation in world space: * **Rotation matrix** (R): 3x3 matrix * **Translation vector** (t): 3x1 vector Transforms world coordinates to camera coordinates: ``` [X_cam] [R | t] [X_world] [Y_cam] = [--+--] [Y_world] [Z_cam] [0 | 1] [Z_world] ``` Lens distortion parameters: * **k1, k2, k3**: Radial distortion * **p1, p2**: Tangential distortion Distortion model: ``` x_distorted = x(1 + k1*r^2 + k2*r^4 + k3*r^6) + 2*p1*xy + p2*(r^2 + 2*x^2) y_distorted = y(1 + k1*r^2 + k2*r^4 + k3*r^6) + p1*(r^2 + 2*y^2) + 2*p2*xy ``` ## Calibration Pattern The most common calibration pattern is a chessboard: ### Creating a Chessboard Pattern Print a chessboard pattern with known square size (e.g., 25mm). Common sizes: * 9x6 inner corners (10x7 squares) * 8x6 inner corners (9x7 squares) Attach the pattern to a rigid, flat surface (cardboard, acrylic, etc.) Take 15-30 images of the pattern from different angles and distances Chessboard requirements: * High contrast between squares * Perfectly flat surface * No glare or reflections * Pattern fills 30-70% of image * Vary viewing angles (tilt, rotate, distance) ## Camera Calibration Process Based on OpenCV's calibrate.py sample: ### Single Camera Calibration ```python theme={null} import cv2 as cv import numpy as np from glob import glob # Chessboard dimensions (inner corners) pattern_size = (9, 6) square_size = 25.0 # millimeters # Prepare object points (0,0,0), (1,0,0), (2,0,0), ..., (8,5,0) pattern_points = np.zeros((np.prod(pattern_size), 3), np.float32) pattern_points[:, :2] = np.indices(pattern_size).T.reshape(-1, 2) pattern_points *= square_size # Arrays to store object points and image points obj_points = [] # 3D points in real world img_points = [] # 2D points in image plane # Load calibration images images = glob('calibration_images/*.jpg') print(f"Found {len(images)} images") for fname in images: print(f'Processing {fname}...') img = cv.imread(fname) gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Find chessboard corners found, corners = cv.findChessboardCorners(gray, pattern_size, None) if found: print(f' Corners found') # Refine corner locations criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) corners_refined = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria) # Store points obj_points.append(pattern_points) img_points.append(corners_refined) # Draw and display corners cv.drawChessboardCorners(img, pattern_size, corners_refined, found) cv.imshow('Chessboard', img) cv.waitKey(100) else: print(f' Pattern not found') cv.destroyAllWindows() # Calibrate camera print("\nCalibrating camera...") h, w = gray.shape[:2] ret, camera_matrix, dist_coeffs, rvecs, tvecs = cv.calibrateCamera( obj_points, img_points, (w, h), None, None ) # Print results print(f"\nCalibration successful!") print(f"RMS re-projection error: {ret:.4f}") print(f"\nCamera matrix:\n{camera_matrix}") print(f"\nDistortion coefficients:\n{dist_coeffs.ravel()}") # Save calibration np.savez('calibration.npz', camera_matrix=camera_matrix, dist_coeffs=dist_coeffs, rvecs=rvecs, tvecs=tvecs) print("\nCalibration saved to calibration.npz") ``` ```cpp theme={null} #include #include #include #include #include using namespace cv; using namespace std; int main() { // Chessboard dimensions Size pattern_size(9, 6); float square_size = 25.0f; // mm // Prepare object points vector pattern_points; for(int i = 0; i < pattern_size.height; i++) for(int j = 0; j < pattern_size.width; j++) pattern_points.push_back( Point3f(j*square_size, i*square_size, 0)); vector> object_points; vector> image_points; // Load images vector images; glob("calibration_images/*.jpg", images); cout << "Found " << images.size() << " images" << endl; Size img_size; for(const auto& fname : images) { cout << "Processing " << fname << "..."; Mat img = imread(fname); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); img_size = gray.size(); vector corners; bool found = findChessboardCorners(gray, pattern_size, corners); if(found) { cout << " corners found" << endl; // Refine corners TermCriteria criteria(TermCriteria::EPS + TermCriteria::MAX_ITER, 30, 0.001); cornerSubPix(gray, corners, Size(11, 11), Size(-1, -1), criteria); object_points.push_back(pattern_points); image_points.push_back(corners); // Draw corners drawChessboardCorners(img, pattern_size, corners, found); imshow("Chessboard", img); waitKey(100); } else { cout << " pattern not found" << endl; } } destroyAllWindows(); // Calibrate cout << "\nCalibrating camera..." << endl; Mat camera_matrix, dist_coeffs; vector rvecs, tvecs; double rms = calibrateCamera(object_points, image_points, img_size, camera_matrix, dist_coeffs, rvecs, tvecs); cout << "\nCalibration successful!" << endl; cout << "RMS error: " << rms << endl; cout << "\nCamera matrix:\n" << camera_matrix << endl; cout << "\nDistortion coefficients:\n" << dist_coeffs << endl; // Save calibration FileStorage fs("calibration.xml", FileStorage::WRITE); fs << "camera_matrix" << camera_matrix; fs << "dist_coeffs" << dist_coeffs; fs.release(); cout << "\nCalibration saved to calibration.xml" << endl; return 0; } ``` ## Undistorting Images ### Basic Undistortion ```python theme={null} import cv2 as cv import numpy as np # Load calibration calib = np.load('calibration.npz') camera_matrix = calib['camera_matrix'] dist_coeffs = calib['dist_coeffs'] # Load distorted image img = cv.imread('distorted.jpg') h, w = img.shape[:2] # Get optimal camera matrix new_camera_matrix, roi = cv.getOptimalNewCameraMatrix( camera_matrix, dist_coeffs, (w, h), 1, (w, h) ) # Undistort undistorted = cv.undistort(img, camera_matrix, dist_coeffs, None, new_camera_matrix) # Crop to ROI x, y, w, h = roi undistorted = undistorted[y:y+h, x:x+w] # Display cv.imshow('Original', img) cv.imshow('Undistorted', undistorted) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; int main() { // Load calibration FileStorage fs("calibration.xml", FileStorage::READ); Mat camera_matrix, dist_coeffs; fs["camera_matrix"] >> camera_matrix; fs["dist_coeffs"] >> dist_coeffs; fs.release(); // Load image Mat img = imread("distorted.jpg"); // Get optimal camera matrix Rect roi; Mat new_camera_matrix = getOptimalNewCameraMatrix( camera_matrix, dist_coeffs, img.size(), 1, img.size(), &roi ); // Undistort Mat undistorted; undistort(img, undistorted, camera_matrix, dist_coeffs, new_camera_matrix); // Crop to ROI undistorted = undistorted(roi); imshow("Original", img); imshow("Undistorted", undistorted); waitKey(0); return 0; } ``` ### Efficient Undistortion with Remapping For real-time video, precompute undistortion maps: ```python theme={null} import cv2 as cv import numpy as np # Load calibration calib = np.load('calibration.npz') camera_matrix = calib['camera_matrix'] dist_coeffs = calib['dist_coeffs'] # Open video cap = cv.VideoCapture(0) ret, frame = cap.read() h, w = frame.shape[:2] # Get optimal camera matrix new_camera_matrix, roi = cv.getOptimalNewCameraMatrix( camera_matrix, dist_coeffs, (w, h), 1, (w, h) ) # Precompute undistortion maps (only once) mapx, mapy = cv.initUndistortRectifyMap( camera_matrix, dist_coeffs, None, new_camera_matrix, (w, h), cv.CV_16SC2 ) # Process video while True: ret, frame = cap.read() if not ret: break # Fast undistortion using precomputed maps undistorted = cv.remap(frame, mapx, mapy, cv.INTER_LINEAR) cv.imshow('Undistorted Video', undistorted) if cv.waitKey(1) & 0xFF == ord('q'): break cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; int main() { FileStorage fs("calibration.xml", FileStorage::READ); Mat camera_matrix, dist_coeffs; fs["camera_matrix"] >> camera_matrix; fs["dist_coeffs"] >> dist_coeffs; VideoCapture cap(0); Mat frame; cap >> frame; // Precompute maps Mat new_camera_matrix = getOptimalNewCameraMatrix( camera_matrix, dist_coeffs, frame.size(), 1, frame.size() ); Mat mapx, mapy; initUndistortRectifyMap(camera_matrix, dist_coeffs, Mat(), new_camera_matrix, frame.size(), CV_16SC2, mapx, mapy); while(cap.read(frame)) { Mat undistorted; remap(frame, undistorted, mapx, mapy, INTER_LINEAR); imshow("Undistorted", undistorted); if(waitKey(1) == 'q') break; } return 0; } ``` ## Calibration Quality Assessment ```python theme={null} import cv2 as cv import numpy as np def evaluate_calibration(obj_points, img_points, rvecs, tvecs, camera_matrix, dist_coeffs): """Calculate reprojection errors for each image""" mean_error = 0 for i in range(len(obj_points)): # Project 3D points to image plane img_points2, _ = cv.projectPoints(obj_points[i], rvecs[i], tvecs[i], camera_matrix, dist_coeffs) # Calculate error error = cv.norm(img_points[i], img_points2, cv.NORM_L2) / len(img_points2) mean_error += error print(f"Image {i+1}: error = {error:.4f} pixels") mean_error /= len(obj_points) print(f"\nMean reprojection error: {mean_error:.4f} pixels") return mean_error # After calibration error = evaluate_calibration(obj_points, img_points, rvecs, tvecs, camera_matrix, dist_coeffs) if error < 0.5: print("Excellent calibration!") elif error < 1.0: print("Good calibration") else: print("Calibration may need improvement") ``` Calibration quality guidelines: * **RMS error \< 0.5**: Excellent * **RMS error \< 1.0**: Good * **RMS error > 1.0**: May need more images or better pattern detection Tips for better calibration: * Use 15-30 images minimum * Cover all areas of the image * Include tilted views (30-45 degrees) * Vary distances to pattern * Ensure sharp, well-lit images * Use higher resolution if possible ## Stereo Calibration Calibrate two cameras for stereo vision: ```python theme={null} import cv2 as cv import numpy as np # After detecting corners in both left and right images # obj_points, img_points_left, img_points_right are collected # Calibrate each camera individually first ret_left, mtx_left, dist_left, _, _ = cv.calibrateCamera( obj_points, img_points_left, img_size, None, None ) ret_right, mtx_right, dist_right, _, _ = cv.calibrateCamera( obj_points, img_points_right, img_size, None, None ) # Stereo calibration flags = cv.CALIB_FIX_INTRINSIC # Fix individual camera parameters ret, mtx_left, dist_left, mtx_right, dist_right, R, T, E, F = \ cv.stereoCalibrate( obj_points, img_points_left, img_points_right, mtx_left, dist_left, mtx_right, dist_right, img_size, criteria=(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 1e-6), flags=flags ) print(f"Stereo calibration RMS: {ret}") print(f"\nRotation matrix:\n{R}") print(f"\nTranslation vector:\n{T}") # Save stereo calibration np.savez('stereo_calibration.npz', mtx_left=mtx_left, dist_left=dist_left, mtx_right=mtx_right, dist_right=dist_right, R=R, T=T, E=E, F=F) ``` Common calibration mistakes: * Too few images (minimum 15 recommended) * Images too similar (vary angles and distances) * Motion blur or poor lighting * Chessboard not flat or warped * Pattern detection failures ignored * Not checking reprojection error ## Practical Applications After calibration, you can measure distances between points: ```python theme={null} # Get 2D image points point1 = (x1, y1) point2 = (x2, y2) # Convert to normalized coordinates # Then use triangulation or known depth ``` Use calibration for accurate AR overlay: ```python theme={null} # Detect marker # Estimate pose using solvePnP # Project 3D model onto image ``` Combine with stereo vision: ```python theme={null} # Stereo rectification # Disparity map computation # 3D point cloud generation ``` ## Next Steps * Apply calibration to [Video Processing](/tutorials/video-processing) * Use with [Deep Learning](/tutorials/deep-learning) for accurate 3D object detection * Explore stereo vision and depth estimation * Learn about pose estimation and AR applications # Deep Learning with OpenCV DNN Module Source: https://opencv-opencv.mintlify.app/tutorials/deep-learning Learn how to load and run neural networks including YOLO, SSD, and other deep learning models in OpenCV # Deep Learning with OpenCV DNN Module Learn how to use OpenCV's DNN (Deep Neural Networks) module to load and run pre-trained models for object detection, classification, and more. ## Introduction to OpenCV DNN OpenCV's DNN module allows you to: * Load models from TensorFlow, PyTorch, Caffe, ONNX, and Darknet * Run inference without installing deep learning frameworks * Deploy on CPU, GPU (CUDA), or OpenVINO backends * Use pre-trained models for various tasks ### Supported Frameworks * **ONNX** (.onnx) - Universal format, recommended * **TensorFlow** (.pb, .pbtxt) * **PyTorch** (via ONNX export) * **Caffe** (.caffemodel, .prototxt) * **Darknet** (.weights, .cfg) - YOLO models * **TensorFlow Lite** (.tflite) ## Loading and Running Models ### Basic Model Loading ```python theme={null} import cv2 as cv import numpy as np # Load a model (example: ONNX format) net = cv.dnn.readNet('model.onnx') # Or load specific formats: # net = cv.dnn.readNetFromTensorflow('model.pb', 'model.pbtxt') # net = cv.dnn.readNetFromCaffe('deploy.prototxt', 'model.caffemodel') # net = cv.dnn.readNetFromDarknet('yolov3.cfg', 'yolov3.weights') # Set computation backend and target net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # For GPU acceleration: # net.setPreferableBackend(cv.dnn.DNN_BACKEND_CUDA) # net.setPreferableTarget(cv.dnn.DNN_TARGET_CUDA) ``` ```cpp theme={null} #include #include using namespace cv; using namespace cv::dnn; int main() { // Load model Net net = readNet("model.onnx"); // Or specific formats: // Net net = readNetFromTensorflow("model.pb", "model.pbtxt"); // Net net = readNetFromCaffe("deploy.prototxt", "model.caffemodel"); // Net net = readNetFromDarknet("yolov3.cfg", "yolov3.weights"); // Set backend and target net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // For GPU: // net.setPreferableBackend(DNN_BACKEND_CUDA); // net.setPreferableTarget(DNN_TARGET_CUDA); return 0; } ``` ## YOLO Object Detection YOLO (You Only Look Once) is a popular real-time object detection system. ### YOLOv3/YOLOv4 Detection ```python theme={null} import cv2 as cv import numpy as np # Load YOLO network net = cv.dnn.readNetFromDarknet('yolov4.cfg', 'yolov4.weights') net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # Load class names with open('coco.names', 'r') as f: classes = [line.strip() for line in f.readlines()] # Load image img = cv.imread('street.jpg') height, width = img.shape[:2] # Create blob from image blob = cv.dnn.blobFromImage(img, 1/255.0, (416, 416), swapRB=True, crop=False) # Set input and run forward pass net.setInput(blob) # Get output layer names output_layers = net.getUnconnectedOutLayersNames() # Forward pass outputs = net.forward(output_layers) # Process detections boxes = [] confidences = [] class_ids = [] for output in outputs: for detection in output: scores = detection[5:] class_id = np.argmax(scores) confidence = scores[class_id] if confidence > 0.5: # Scale bounding box back to image size center_x = int(detection[0] * width) center_y = int(detection[1] * height) w = int(detection[2] * width) h = int(detection[3] * height) # Get top-left corner x = int(center_x - w / 2) y = int(center_y - h / 2) boxes.append([x, y, w, h]) confidences.append(float(confidence)) class_ids.append(class_id) # Apply Non-Maximum Suppression indices = cv.dnn.NMSBoxes(boxes, confidences, 0.5, 0.4) # Draw detections if len(indices) > 0: for i in indices.flatten(): x, y, w, h = boxes[i] label = f"{classes[class_ids[i]]}: {confidences[i]:.2f}" color = (0, 255, 0) cv.rectangle(img, (x, y), (x+w, y+h), color, 2) cv.putText(img, label, (x, y-10), cv.FONT_HERSHEY_SIMPLEX, 0.5, color, 2) cv.imshow('YOLO Detection', img) cv.waitKey(0) ``` ```cpp theme={null} #include #include #include using namespace cv; using namespace cv::dnn; using namespace std; int main() { // Load network Net net = readNetFromDarknet("yolov4.cfg", "yolov4.weights"); net.setPreferableBackend(DNN_BACKEND_OPENCV); net.setPreferableTarget(DNN_TARGET_CPU); // Load class names vector classes; ifstream ifs("coco.names"); string line; while(getline(ifs, line)) classes.push_back(line); // Load image Mat img = imread("street.jpg"); // Create blob Mat blob; blobFromImage(img, blob, 1/255.0, Size(416, 416), Scalar(), true, false); net.setInput(blob); // Get output layers vector outNames = net.getUnconnectedOutLayersNames(); vector outs; net.forward(outs, outNames); // Process detections vector classIds; vector confidences; vector boxes; for(size_t i = 0; i < outs.size(); ++i) { float* data = (float*)outs[i].data; for(int j = 0; j < outs[i].rows; ++j, data += outs[i].cols) { Mat scores = outs[i].row(j).colRange(5, outs[i].cols); Point classIdPoint; double confidence; minMaxLoc(scores, 0, &confidence, 0, &classIdPoint); if(confidence > 0.5) { int centerX = (int)(data[0] * img.cols); int centerY = (int)(data[1] * img.rows); int width = (int)(data[2] * img.cols); int height = (int)(data[3] * img.rows); int left = centerX - width / 2; int top = centerY - height / 2; classIds.push_back(classIdPoint.x); confidences.push_back((float)confidence); boxes.push_back(Rect(left, top, width, height)); } } } // NMS vector indices; NMSBoxes(boxes, confidences, 0.5, 0.4, indices); // Draw for(size_t i = 0; i < indices.size(); ++i) { int idx = indices[i]; Rect box = boxes[idx]; rectangle(img, box, Scalar(0, 255, 0), 2); string label = classes[classIds[idx]] + ": " + format("%.2f", confidences[idx]); putText(img, label, Point(box.x, box.y - 10), FONT_HERSHEY_SIMPLEX, 0.5, Scalar(0, 255, 0), 2); } imshow("YOLO Detection", img); waitKey(0); return 0; } ``` ### YOLOv8 with ONNX Modern YOLO versions export to ONNX format: ```python theme={null} import cv2 as cv import numpy as np # Load YOLOv8 model (ONNX format) net = cv.dnn.readNetFromONNX('yolov8n.onnx') net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # Load image img = cv.imread('image.jpg') original_height, original_width = img.shape[:2] # Preprocess input_size = 640 blob = cv.dnn.blobFromImage(img, 1/255.0, (input_size, input_size), swapRB=True, crop=False) # Run inference net.setInput(blob) output = net.forward() # YOLOv8 outputs shape: [1, 84, 8400] for COCO # Format: [x, y, w, h, class_scores...] output = output[0].transpose() # [8400, 84] # Process detections boxes = [] confidences = [] class_ids = [] img_height, img_width = img.shape[:2] x_scale = img_width / input_size y_scale = img_height / input_size for detection in output: # Extract box coordinates x, y, w, h = detection[:4] # Get class scores and find max class_scores = detection[4:] class_id = np.argmax(class_scores) confidence = class_scores[class_id] if confidence > 0.5: # Scale back to original image x = int((x - w/2) * x_scale) y = int((y - h/2) * y_scale) w = int(w * x_scale) h = int(h * y_scale) boxes.append([x, y, w, h]) confidences.append(float(confidence)) class_ids.append(class_id) # Apply NMS indices = cv.dnn.NMSBoxes(boxes, confidences, 0.5, 0.4) # Draw results for i in indices.flatten(): x, y, w, h = boxes[i] cv.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2) label = f"Class {class_ids[i]}: {confidences[i]:.2f}" cv.putText(img, label, (x, y-10), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) cv.imshow('YOLOv8 Detection', img) cv.waitKey(0) ``` ## SSD Object Detection SSD (Single Shot MultiBox Detector) for faster detection: ```python theme={null} import cv2 as cv import numpy as np # Load MobileNet-SSD model net = cv.dnn.readNetFromCaffe( 'MobileNetSSD_deploy.prototxt', 'MobileNetSSD_deploy.caffemodel' ) # COCO class names classes = ["background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"] # Load image img = cv.imread('image.jpg') height, width = img.shape[:2] # Prepare input blob = cv.dnn.blobFromImage(img, 0.007843, (300, 300), 127.5) # Run detection net.setInput(blob) detections = net.forward() # Process detections for i in range(detections.shape[2]): confidence = detections[0, 0, i, 2] if confidence > 0.5: class_id = int(detections[0, 0, i, 1]) # Get box coordinates box = detections[0, 0, i, 3:7] * np.array([width, height, width, height]) (x1, y1, x2, y2) = box.astype("int") # Draw detection label = f"{classes[class_id]}: {confidence:.2f}" cv.rectangle(img, (x1, y1), (x2, y2), (0, 255, 0), 2) cv.putText(img, label, (x1, y1-10), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) cv.imshow('SSD Detection', img) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; using namespace cv::dnn; int main() { Net net = readNetFromCaffe( "MobileNetSSD_deploy.prototxt", "MobileNetSSD_deploy.caffemodel" ); Mat img = imread("image.jpg"); Mat blob; blobFromImage(img, blob, 0.007843, Size(300, 300), Scalar(127.5, 127.5, 127.5)); net.setInput(blob); Mat detections = net.forward(); Mat detectionMat(detections.size[2], detections.size[3], CV_32F, detections.ptr()); for(int i = 0; i < detectionMat.rows; i++) { float confidence = detectionMat.at(i, 2); if(confidence > 0.5) { int x1 = detectionMat.at(i, 3) * img.cols; int y1 = detectionMat.at(i, 4) * img.rows; int x2 = detectionMat.at(i, 5) * img.cols; int y2 = detectionMat.at(i, 6) * img.rows; rectangle(img, Point(x1, y1), Point(x2, y2), Scalar(0, 255, 0), 2); } } imshow("SSD Detection", img); waitKey(0); return 0; } ``` ## Image Classification ```python theme={null} import cv2 as cv import numpy as np # Load ResNet model net = cv.dnn.readNetFromCaffe( 'ResNet-50-deploy.prototxt', 'ResNet-50-model.caffemodel' ) # Load ImageNet class labels with open('imagenet_classes.txt', 'r') as f: classes = [line.strip() for line in f.readlines()] # Load and preprocess image img = cv.imread('dog.jpg') # Create blob (ResNet expects 224x224 input) blob = cv.dnn.blobFromImage(img, 1.0, (224, 224), (104, 117, 123), swapRB=False, crop=False) # Run inference net.setInput(blob) predictions = net.forward() # Get top 5 predictions top5_indices = np.argsort(predictions[0])[::-1][:5] print("Top 5 predictions:") for i, idx in enumerate(top5_indices): label = classes[idx] confidence = predictions[0][idx] print(f"{i+1}. {label}: {confidence*100:.2f}%") # Display result top_label = classes[top5_indices[0]] cv.putText(img, top_label, (10, 30), cv.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) cv.imshow('Classification', img) cv.waitKey(0) ``` ## Face Detection with DNN Deep learning-based face detection (more accurate than Haar cascades): ```python theme={null} import cv2 as cv # Load face detection model net = cv.dnn.readNetFromCaffe( 'deploy.prototxt', 'res10_300x300_ssd_iter_140000.caffemodel' ) # Load image img = cv.imread('faces.jpg') height, width = img.shape[:2] # Preprocess blob = cv.dnn.blobFromImage(img, 1.0, (300, 300), (104.0, 177.0, 123.0)) # Detect faces net.setInput(blob) detections = net.forward() # Draw detections for i in range(detections.shape[2]): confidence = detections[0, 0, i, 2] if confidence > 0.5: box = detections[0, 0, i, 3:7] * np.array([width, height, width, height]) (x1, y1, x2, y2) = box.astype("int") # Draw box and confidence text = f"{confidence*100:.2f}%" cv.rectangle(img, (x1, y1), (x2, y2), (0, 255, 0), 2) cv.putText(img, text, (x1, y1-10), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) cv.imshow('Face Detection', img) cv.waitKey(0) ``` ## Video Processing with DNN ```python theme={null} import cv2 as cv import time # Load model net = cv.dnn.readNetFromCaffe( 'MobileNetSSD_deploy.prototxt', 'MobileNetSSD_deploy.caffemodel' ) # Open video cap = cv.VideoCapture('video.mp4') while True: ret, frame = cap.read() if not ret: break height, width = frame.shape[:2] # Prepare input blob = cv.dnn.blobFromImage(frame, 0.007843, (300, 300), 127.5) # Measure inference time start = time.time() net.setInput(blob) detections = net.forward() end = time.time() # Process detections for i in range(detections.shape[2]): confidence = detections[0, 0, i, 2] if confidence > 0.5: box = detections[0, 0, i, 3:7] * np.array([width, height, width, height]) (x1, y1, x2, y2) = box.astype("int") cv.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2) # Display FPS fps = 1 / (end - start) cv.putText(frame, f'FPS: {fps:.1f}', (10, 30), cv.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) cv.imshow('Detection', frame) if cv.waitKey(1) & 0xFF == ord('q'): break cap.release() cv.destroyAllWindows() ``` ## Performance Optimization ```python theme={null} import cv2 as cv net = cv.dnn.readNet('model.onnx') # CUDA backend (requires OpenCV built with CUDA) net.setPreferableBackend(cv.dnn.DNN_BACKEND_CUDA) net.setPreferableTarget(cv.dnn.DNN_TARGET_CUDA) # Or CUDA with FP16 (faster, slightly less accurate) net.setPreferableTarget(cv.dnn.DNN_TARGET_CUDA_FP16) ``` ```python theme={null} # Intel OpenVINO for optimized inference on Intel hardware net.setPreferableBackend(cv.dnn.DNN_BACKEND_INFERENCE_ENGINE) net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU) # Or use Intel GPU net.setPreferableTarget(cv.dnn.DNN_TARGET_OPENCL) ``` ```python theme={null} # Process multiple images at once images = [img1, img2, img3, img4] # Create batch blob blob = cv.dnn.blobFromImages(images, 1/255.0, (640, 640)) net.setInput(blob) outputs = net.forward() ``` Backend and target options: * `DNN_BACKEND_OPENCV` + `DNN_TARGET_CPU`: Default, works everywhere * `DNN_BACKEND_CUDA` + `DNN_TARGET_CUDA`: NVIDIA GPU acceleration * `DNN_BACKEND_INFERENCE_ENGINE` + `DNN_TARGET_CPU`: Intel OpenVINO * `DNN_TARGET_OPENCL`: OpenCL acceleration * `DNN_TARGET_CUDA_FP16`: Half-precision for faster inference Common issues: * Model input size must match the size used during training * Check if the model expects RGB or BGR input (use `swapRB` parameter) * Normalize input values correctly (typically 0-1 or mean subtraction) * Ensure OpenCV is built with the desired backend support ## Downloading Pre-trained Models OpenCV provides scripts to download common models: ```bash theme={null} # Download YOLOv3 python opencv/samples/dnn/download_models.py --name yolo # Download all models python opencv/samples/dnn/download_models.py --all ``` Common model sources: * [OpenCV Model Zoo](https://github.com/opencv/opencv_zoo) * [ONNX Model Zoo](https://github.com/onnx/models) * [TensorFlow Hub](https://tfhub.dev/) * [PyTorch Hub](https://pytorch.org/hub/) ## Next Steps * Explore the [OpenCV Model Zoo](https://github.com/opencv/opencv_zoo) for more pre-trained models * Learn about [Camera Calibration](/tutorials/camera-calibration) for 3D vision tasks * Combine with [Video Processing](/tutorials/video-processing) for real-time applications # Face Detection with Cascade Classifiers Source: https://opencv-opencv.mintlify.app/tutorials/face-detection Comprehensive guide to detecting faces and facial features using Haar cascade classifiers in OpenCV # Face Detection with Cascade Classifiers Learn how to detect faces, eyes, and other facial features in images and video using OpenCV's pre-trained Haar cascade classifiers. ## Introduction to Face Detection Face detection is one of the most common applications of computer vision. OpenCV provides robust pre-trained models that can detect faces in various conditions. ### Why Haar Cascades? * Pre-trained models available for immediate use * Fast enough for real-time detection * No GPU required * Works well for frontal faces * Lightweight and easy to deploy ## Basic Face Detection ### Single Face Detection ```python theme={null} import cv2 as cv # Load the cascade classifier face_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_frontalface_default.xml') # Read image img = cv.imread('face.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Detect faces faces = face_cascade.detectMultiScale( gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30) ) print(f"Found {len(faces)} face(s)") # Draw rectangle around each face for (x, y, w, h) in faces: cv.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2) cv.imshow('Face Detection', img) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { // Load cascade CascadeClassifier face_cascade; if(!face_cascade.load(samples::findFile( "haarcascades/haarcascade_frontalface_default.xml"))) { cout << "Error loading cascade" << endl; return -1; } // Read image Mat img = imread("face.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); // Detect faces vector faces; face_cascade.detectMultiScale(gray, faces, 1.1, 5, 0, Size(30, 30)); cout << "Found " << faces.size() << " face(s)" << endl; // Draw rectangles for(size_t i = 0; i < faces.size(); i++) { rectangle(img, faces[i], Scalar(255, 0, 0), 2); } imshow("Face Detection", img); waitKey(0); return 0; } ``` ## Complete Face and Eye Detection Based on OpenCV's facedetect.cpp sample: ```python theme={null} import cv2 as cv from video import create_capture from common import clock, draw_str def detect(img, cascade): """Detect faces or eyes using cascade classifier""" rects = cascade.detectMultiScale( img, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30), flags=cv.CASCADE_SCALE_IMAGE ) if len(rects) == 0: return [] # Convert to x1, y1, x2, y2 format rects[:,2:] += rects[:,:2] return rects def draw_rects(img, rects, color): """Draw rectangles on image""" for x1, y1, x2, y2 in rects: cv.rectangle(img, (x1, y1), (x2, y2), color, 2) def main(): import sys import getopt args, video_src = getopt.getopt(sys.argv[1:], '', ['cascade=', 'nested-cascade=']) try: video_src = video_src[0] except: video_src = 0 args = dict(args) cascade_fn = args.get('--cascade', 'haarcascades/haarcascade_frontalface_alt.xml') nested_fn = args.get('--nested-cascade', 'haarcascades/haarcascade_eye.xml') # Load cascades cascade = cv.CascadeClassifier(cv.samples.findFile(cascade_fn)) nested = cv.CascadeClassifier(cv.samples.findFile(nested_fn)) # Open camera or video cam = create_capture(video_src) while True: ret, img = cam.read() if not ret: break # Convert to grayscale and equalize histogram gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) gray = cv.equalizeHist(gray) # Measure detection time t = clock() # Detect faces rects = detect(gray, cascade) vis = img.copy() draw_rects(vis, rects, (0, 255, 0)) # Green rectangles for faces # Detect eyes within each face if not nested.empty(): for x1, y1, x2, y2 in rects: roi = gray[y1:y2, x1:x2] vis_roi = vis[y1:y2, x1:x2] subrects = detect(roi.copy(), nested) draw_rects(vis_roi, subrects, (255, 0, 0)) # Blue for eyes dt = clock() - t # Display detection time draw_str(vis, (20, 20), 'time: %.1f ms' % (dt*1000)) cv.imshow('Face Detection', vis) # Press ESC to exit if cv.waitKey(5) == 27: break print('Done') cv.destroyAllWindows() if __name__ == '__main__': main() ``` ```cpp theme={null} #include #include #include #include #include #include using namespace std; using namespace cv; void detectAndDraw(Mat& img, CascadeClassifier& cascade, CascadeClassifier& nestedCascade, double scale) { double t = 0; vector faces; Mat gray, smallImg; cvtColor(img, gray, COLOR_BGR2GRAY); double fx = 1 / scale; resize(gray, smallImg, Size(), fx, fx, INTER_LINEAR_EXACT); equalizeHist(smallImg, smallImg); t = (double)getTickCount(); cascade.detectMultiScale(smallImg, faces, 1.1, 2, CASCADE_SCALE_IMAGE, Size(30, 30)); t = (double)getTickCount() - t; printf("detection time = %g ms\n", t*1000/getTickFrequency()); for(size_t i = 0; i < faces.size(); i++) { Rect r = faces[i]; Mat smallImgROI; vector nestedObjects; Point center; Scalar color = Scalar(0, 255, 0); int radius; double aspect_ratio = (double)r.width/r.height; if(0.75 < aspect_ratio && aspect_ratio < 1.3) { center.x = cvRound((r.x + r.width*0.5)*scale); center.y = cvRound((r.y + r.height*0.5)*scale); radius = cvRound((r.width + r.height)*0.25*scale); circle(img, center, radius, color, 3, 8, 0); } else rectangle(img, Point(cvRound(r.x*scale), cvRound(r.y*scale)), Point(cvRound((r.x + r.width-1)*scale), cvRound((r.y + r.height-1)*scale)), color, 3, 8, 0); if(nestedCascade.empty()) continue; smallImgROI = smallImg(r); nestedCascade.detectMultiScale(smallImgROI, nestedObjects, 1.1, 2, CASCADE_SCALE_IMAGE, Size(30, 30)); for(size_t j = 0; j < nestedObjects.size(); j++) { Rect nr = nestedObjects[j]; center.x = cvRound((r.x + nr.x + nr.width*0.5)*scale); center.y = cvRound((r.y + nr.y + nr.height*0.5)*scale); radius = cvRound((nr.width + nr.height)*0.25*scale); circle(img, center, radius, color, 3, 8, 0); } } imshow("result", img); } int main(int argc, const char** argv) { VideoCapture capture; Mat frame; CascadeClassifier cascade, nestedCascade; double scale = 1; if(!cascade.load(samples::findFile( "haarcascades/haarcascade_frontalface_alt.xml"))) { cerr << "ERROR: Could not load classifier cascade" << endl; return -1; } if(!nestedCascade.load(samples::findFile( "haarcascades/haarcascade_eye_tree_eyeglasses.xml"))) cerr << "WARNING: Could not load classifier for nested objects" << endl; if(!capture.open(0)) { cout << "Capture from camera failed" << endl; return 1; } cout << "Video capturing started..." << endl; while(capture.read(frame)) { if(frame.empty()) break; Mat frame1 = frame.clone(); detectAndDraw(frame1, cascade, nestedCascade, scale); char c = (char)waitKey(10); if(c == 27 || c == 'q' || c == 'Q') break; } return 0; } ``` ## Smile Detection ```python theme={null} import cv2 as cv # Load cascades face_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_frontalface_default.xml') smile_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_smile.xml') cap = cv.VideoCapture(0) while True: ret, frame = cap.read() if not ret: break gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) # Detect faces faces = face_cascade.detectMultiScale(gray, 1.3, 5) for (x, y, w, h) in faces: cv.rectangle(frame, (x, y), (x+w, y+h), (255, 0, 0), 2) # Get face ROI for smile detection roi_gray = gray[y:y+h, x:x+w] roi_color = frame[y:y+h, x:x+w] # Detect smiles (only in lower half of face) smiles = smile_cascade.detectMultiScale( roi_gray[h//2:, :], # Lower half scaleFactor=1.8, minNeighbors=20, minSize=(25, 25) ) # Draw smile detection for (sx, sy, sw, sh) in smiles: cv.rectangle(roi_color, (sx, sy + h//2), (sx+sw, sy+sh + h//2), (0, 255, 0), 2) cv.putText(frame, 'Smiling!', (x, y-10), cv.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 2) cv.imshow('Smile Detection', frame) if cv.waitKey(1) & 0xFF == ord('q'): break cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { CascadeClassifier face_cascade, smile_cascade; face_cascade.load(samples::findFile( "haarcascades/haarcascade_frontalface_default.xml")); smile_cascade.load(samples::findFile( "haarcascades/haarcascade_smile.xml")); VideoCapture cap(0); Mat frame, gray; while(cap.read(frame)) { cvtColor(frame, gray, COLOR_BGR2GRAY); vector faces; face_cascade.detectMultiScale(gray, faces, 1.3, 5); for(size_t i = 0; i < faces.size(); i++) { Rect face = faces[i]; rectangle(frame, face, Scalar(255, 0, 0), 2); // Get face ROI Mat faceROI = gray(face); // Detect smile in lower half Rect lowerHalf(0, face.height/2, face.width, face.height/2); Mat smileROI = faceROI(lowerHalf); vector smiles; smile_cascade.detectMultiScale(smileROI, smiles, 1.8, 20, 0, Size(25, 25)); if(!smiles.empty()) { putText(frame, "Smiling!", Point(face.x, face.y - 10), FONT_HERSHEY_SIMPLEX, 0.9, Scalar(0, 255, 0), 2); } } imshow("Smile Detection", frame); if(waitKey(1) == 'q') break; } return 0; } ``` ## Profile Face Detection ```python theme={null} import cv2 as cv # Load both frontal and profile cascades frontal_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_frontalface_default.xml') profile_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_profileface.xml') img = cv.imread('group.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Detect frontal faces frontal_faces = frontal_cascade.detectMultiScale(gray, 1.1, 5) print(f"Frontal faces: {len(frontal_faces)}") # Detect profile faces (left) profile_faces = profile_cascade.detectMultiScale(gray, 1.1, 5) # Flip image to detect right-facing profiles gray_flipped = cv.flip(gray, 1) profile_faces_flipped = profile_cascade.detectMultiScale(gray_flipped, 1.1, 5) # Flip coordinates back width = img.shape[1] profile_faces_right = [(width - x - w, y, w, h) for (x, y, w, h) in profile_faces_flipped] print(f"Profile faces (left): {len(profile_faces)}") print(f"Profile faces (right): {len(profile_faces_right)}") # Draw all detections for (x, y, w, h) in frontal_faces: cv.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2) for (x, y, w, h) in profile_faces: cv.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 2) for (x, y, w, h) in profile_faces_right: cv.rectangle(img, (x, y), (x+w, y+h), (0, 0, 255), 2) cv.imshow('Face Detection (Green=Frontal, Blue=Left, Red=Right)', img) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { Mat img = imread("group.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); CascadeClassifier frontal_cascade, profile_cascade; frontal_cascade.load("haarcascade_frontalface_default.xml"); profile_cascade.load("haarcascade_profileface.xml"); vector frontal_faces, profile_faces, profile_right; // Detect frontal frontal_cascade.detectMultiScale(gray, frontal_faces, 1.1, 5); // Detect left profiles profile_cascade.detectMultiScale(gray, profile_faces, 1.1, 5); // Detect right profiles Mat gray_flipped; flip(gray, gray_flipped, 1); profile_cascade.detectMultiScale(gray_flipped, profile_right, 1.1, 5); // Flip coordinates back for(auto& r : profile_right) { r.x = img.cols - r.x - r.width; } // Draw detections for(auto& r : frontal_faces) rectangle(img, r, Scalar(0, 255, 0), 2); for(auto& r : profile_faces) rectangle(img, r, Scalar(255, 0, 0), 2); for(auto& r : profile_right) rectangle(img, r, Scalar(0, 0, 255), 2); imshow("Face Detection", img); waitKey(0); return 0; } ``` ## Improving Detection Accuracy ```python theme={null} # Convert to grayscale gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Equalize histogram for better contrast gray = cv.equalizeHist(gray) # Apply slight Gaussian blur to reduce noise gray = cv.GaussianBlur(gray, (3, 3), 0) ``` ```python theme={null} faces = face_cascade.detectMultiScale( gray, scaleFactor=1.1, # Smaller = more thorough, slower minNeighbors=5, # Higher = fewer false positives minSize=(30, 30), # Minimum face size maxSize=(300, 300), # Maximum face size flags=cv.CASCADE_SCALE_IMAGE ) ``` Try multiple scale factors and combine results: ```python theme={null} all_faces = [] for scale in [1.05, 1.1, 1.2, 1.3]: faces = face_cascade.detectMultiScale(gray, scale, 5) all_faces.extend(faces) # Remove duplicates using Non-Maximum Suppression # (implementation depends on your needs) ``` For video, track faces across frames: ```python theme={null} # Use object tracking to smooth detections # Only accept detections that appear in multiple consecutive frames ``` Best practices for face detection: * Always convert to grayscale first * Use histogram equalization for better contrast * Start with scaleFactor=1.1 and minNeighbors=5 * Adjust minSize based on expected face sizes * For video, resize frames for faster processing * Use nested detection (face → eyes) to verify results Limitations of Haar cascades: * Works best with frontal faces * Struggles with occlusions (sunglasses, masks, hands) * Sensitive to lighting conditions * Less accurate than deep learning methods * Can produce false positives For production applications requiring high accuracy, consider using deep learning-based face detection (see [Deep Learning tutorial](/tutorials/deep-learning)). ## Next Steps * Learn about [Deep Learning](/tutorials/deep-learning) face detection for better accuracy * Explore [Object Detection](/tutorials/object-detection) for detecting other objects * Try face recognition and facial landmarks detection # Feature Detection and Matching Source: https://opencv-opencv.mintlify.app/tutorials/feature-detection Learn how to detect corners, edges, blobs, and match features between images using OpenCV # Feature Detection and Matching Learn how to detect and match distinctive features in images, essential for tasks like image stitching, object recognition, and camera calibration. ## What are Features? Features are distinctive points or regions in an image that can be reliably detected across different views. Good features are: * Repeatable (can be found in different images of the same scene) * Distinctive (can be distinguished from nearby features) * Local (not affected by clutter or occlusion) * Efficient (fast to compute) ## Corner Detection ### Harris Corner Detector ```python theme={null} import cv2 as cv import numpy as np # Load image img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) gray = np.float32(gray) # Apply Harris corner detection dst = cv.cornerHarris(gray, blockSize=2, ksize=3, k=0.04) # Dilate to mark the corners dst = cv.dilate(dst, None) # Threshold for optimal value (adjust based on image) img[dst > 0.01 * dst.max()] = [0, 0, 255] cv.imshow('Harris Corners', img) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; int main() { Mat img = imread("image.jpg"); Mat gray, dst, dst_norm; cvtColor(img, gray, COLOR_BGR2GRAY); // Harris corner detection cornerHarris(gray, dst, 2, 3, 0.04); // Normalize normalize(dst, dst_norm, 0, 255, NORM_MINMAX); // Draw corners for(int i = 0; i < dst_norm.rows; i++) { for(int j = 0; j < dst_norm.cols; j++) { if((int)dst_norm.at(i,j) > 200) { circle(img, Point(j,i), 5, Scalar(0,0,255), 2); } } } imshow("Harris Corners", img); waitKey(0); return 0; } ``` ### Shi-Tomasi Corner Detector (Good Features to Track) ```python theme={null} import cv2 as cv import numpy as np img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Parameters maxCorners = 100 qualityLevel = 0.01 minDistance = 10 # Detect corners corners = cv.goodFeaturesToTrack(gray, maxCorners, qualityLevel, minDistance) corners = np.int0(corners) # Draw corners for corner in corners: x, y = corner.ravel() cv.circle(img, (x, y), 5, (0, 255, 0), -1) cv.imshow('Shi-Tomasi Corners', img) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; int main() { Mat img = imread("image.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); vector corners; int maxCorners = 100; double qualityLevel = 0.01; double minDistance = 10; goodFeaturesToTrack(gray, corners, maxCorners, qualityLevel, minDistance); for(size_t i = 0; i < corners.size(); i++) { circle(img, corners[i], 5, Scalar(0, 255, 0), -1); } imshow("Shi-Tomasi Corners", img); waitKey(0); return 0; } ``` ## Edge Detection ### Canny Edge Detector ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Apply Gaussian blur to reduce noise blurred = cv.GaussianBlur(gray, (5, 5), 0) # Canny edge detection # threshold1: lower threshold # threshold2: upper threshold edges = cv.Canny(blurred, threshold1=50, threshold2=150) cv.imshow('Original', gray) cv.imshow('Edges', edges) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; int main() { Mat img = imread("image.jpg"); Mat gray, blurred, edges; cvtColor(img, gray, COLOR_BGR2GRAY); GaussianBlur(gray, blurred, Size(5, 5), 0); Canny(blurred, edges, 50, 150); imshow("Original", gray); imshow("Edges", edges); waitKey(0); return 0; } ``` For Canny edge detection: * Use a 2:1 or 3:1 ratio between upper and lower thresholds * Lower threshold: detects weak edges * Upper threshold: detects strong edges * Edges are connected if they're above the lower threshold and connected to an edge above the upper threshold ## Feature Descriptors ### SIFT (Scale-Invariant Feature Transform) ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Create SIFT detector sift = cv.SIFT_create() # Detect keypoints and compute descriptors keypoints, descriptors = sift.detectAndCompute(gray, None) # Draw keypoints img_keypoints = cv.drawKeypoints(img, keypoints, None, flags=cv.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS) print(f"Number of keypoints: {len(keypoints)}") print(f"Descriptor shape: {descriptors.shape}") cv.imshow('SIFT Keypoints', img_keypoints) cv.waitKey(0) ``` ```cpp theme={null} #include #include #include using namespace cv; using namespace std; int main() { Mat img = imread("image.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); // Create SIFT detector Ptr sift = SIFT::create(); vector keypoints; Mat descriptors; sift->detectAndCompute(gray, Mat(), keypoints, descriptors); Mat img_keypoints; drawKeypoints(img, keypoints, img_keypoints, Scalar::all(-1), DrawMatchesFlags::DRAW_RICH_KEYPOINTS); cout << "Keypoints: " << keypoints.size() << endl; imshow("SIFT Keypoints", img_keypoints); waitKey(0); return 0; } ``` ### ORB (Oriented FAST and Rotated BRIEF) ORB is a fast alternative to SIFT and SURF: ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Create ORB detector orb = cv.ORB_create(nfeatures=400) # Detect and compute keypoints, descriptors = orb.detectAndCompute(gray, None) # Draw keypoints img_keypoints = cv.drawKeypoints(img, keypoints, None, color=(0, 255, 0)) print(f"Number of ORB keypoints: {len(keypoints)}") cv.imshow('ORB Keypoints', img_keypoints) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; int main() { Mat img = imread("image.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); Ptr orb = ORB::create(400); vector keypoints; Mat descriptors; orb->detectAndCompute(gray, Mat(), keypoints, descriptors); Mat img_keypoints; drawKeypoints(img, keypoints, img_keypoints, Scalar(0, 255, 0)); imshow("ORB Keypoints", img_keypoints); waitKey(0); return 0; } ``` ### AKAZE ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Create AKAZE detector akaze = cv.AKAZE_create() # Detect and compute keypoints, descriptors = akaze.detectAndCompute(gray, None) # Draw keypoints img_keypoints = cv.drawKeypoints(img, keypoints, None, flags=cv.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS) cv.imshow('AKAZE Keypoints', img_keypoints) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; int main() { Mat img = imread("image.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); Ptr akaze = AKAZE::create(); vector keypoints; Mat descriptors; akaze->detectAndCompute(gray, Mat(), keypoints, descriptors); Mat img_keypoints; drawKeypoints(img, keypoints, img_keypoints); imshow("AKAZE Keypoints", img_keypoints); waitKey(0); return 0; } ``` ## Feature Matching Based on OpenCV's find\_obj.py sample: ### Brute-Force Matcher ```python theme={null} import cv2 as cv import numpy as np # Load two images img1 = cv.imread('box.png', cv.IMREAD_GRAYSCALE) img2 = cv.imread('box_in_scene.png', cv.IMREAD_GRAYSCALE) # Create ORB detector orb = cv.ORB_create(400) # Detect and compute for both images kp1, desc1 = orb.detectAndCompute(img1, None) kp2, desc2 = orb.detectAndCompute(img2, None) # Create BFMatcher bf = cv.BFMatcher(cv.NORM_HAMMING, crossCheck=True) # Match descriptors matches = bf.match(desc1, desc2) # Sort matches by distance (best matches first) matches = sorted(matches, key=lambda x: x.distance) # Draw top 20 matches img_matches = cv.drawMatches(img1, kp1, img2, kp2, matches[:20], None, flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS) print(f"Number of matches: {len(matches)}") cv.imshow('Matches', img_matches) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { Mat img1 = imread("box.png", IMREAD_GRAYSCALE); Mat img2 = imread("box_in_scene.png", IMREAD_GRAYSCALE); Ptr orb = ORB::create(400); vector kp1, kp2; Mat desc1, desc2; orb->detectAndCompute(img1, Mat(), kp1, desc1); orb->detectAndCompute(img2, Mat(), kp2, desc2); BFMatcher bf(NORM_HAMMING, true); vector matches; bf.match(desc1, desc2, matches); Mat img_matches; drawMatches(img1, kp1, img2, kp2, matches, img_matches); imshow("Matches", img_matches); waitKey(0); return 0; } ``` ### FLANN-Based Matcher Faster for large datasets: ```python theme={null} import cv2 as cv import numpy as np img1 = cv.imread('box.png', cv.IMREAD_GRAYSCALE) img2 = cv.imread('box_in_scene.png', cv.IMREAD_GRAYSCALE) # Use SIFT for FLANN sift = cv.SIFT_create() kp1, desc1 = sift.detectAndCompute(img1, None) kp2, desc2 = sift.detectAndCompute(img2, None) # FLANN parameters FLANN_INDEX_KDTREE = 1 index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5) search_params = dict(checks=50) # Create FLANN matcher flann = cv.FlannBasedMatcher(index_params, search_params) # Find k=2 best matches for each descriptor matches = flann.knnMatch(desc1, desc2, k=2) # Apply ratio test (Lowe's ratio test) good_matches = [] for m, n in matches: if m.distance < 0.75 * n.distance: good_matches.append(m) print(f"Good matches: {len(good_matches)} / {len(matches)}") # Draw matches img_matches = cv.drawMatches(img1, kp1, img2, kp2, good_matches, None, flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS) cv.imshow('FLANN Matches', img_matches) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { Mat img1 = imread("box.png", IMREAD_GRAYSCALE); Mat img2 = imread("box_in_scene.png", IMREAD_GRAYSCALE); Ptr sift = SIFT::create(); vector kp1, kp2; Mat desc1, desc2; sift->detectAndCompute(img1, Mat(), kp1, desc1); sift->detectAndCompute(img2, Mat(), kp2, desc2); FlannBasedMatcher flann; vector> knn_matches; flann.knnMatch(desc1, desc2, knn_matches, 2); // Ratio test vector good_matches; for(size_t i = 0; i < knn_matches.size(); i++) { if(knn_matches[i][0].distance < 0.75 * knn_matches[i][1].distance) { good_matches.push_back(knn_matches[i][0]); } } Mat img_matches; drawMatches(img1, kp1, img2, kp2, good_matches, img_matches); imshow("FLANN Matches", img_matches); waitKey(0); return 0; } ``` ## Finding Homography Find the transformation between matched images: ```python theme={null} import cv2 as cv import numpy as np # After getting good matches (from previous example) # Extract location of good matches src_pts = np.float32([kp1[m.queryIdx].pt for m in good_matches]).reshape(-1, 1, 2) dst_pts = np.float32([kp2[m.trainIdx].pt for m in good_matches]).reshape(-1, 1, 2) # Find homography M, mask = cv.findHomography(src_pts, dst_pts, cv.RANSAC, 5.0) # Get dimensions of first image h, w = img1.shape # Define corners of first image pts = np.float32([[0, 0], [w, 0], [w, h], [0, h]]).reshape(-1, 1, 2) # Transform corners to second image dst = cv.perspectiveTransform(pts, M) # Draw bounding box in second image img2_color = cv.cvtColor(img2, cv.COLOR_GRAY2BGR) cv.polylines(img2_color, [np.int32(dst)], True, (0, 255, 0), 3) cv.imshow('Object Detection', img2_color) cv.waitKey(0) ``` ```cpp theme={null} #include #include #include using namespace cv; using namespace std; // After getting good_matches vector src_pts, dst_pts; for(size_t i = 0; i < good_matches.size(); i++) { src_pts.push_back(kp1[good_matches[i].queryIdx].pt); dst_pts.push_back(kp2[good_matches[i].trainIdx].pt); } Mat H = findHomography(src_pts, dst_pts, RANSAC, 5.0); // Transform corners vector corners(4); corners[0] = Point2f(0, 0); corners[1] = Point2f(img1.cols, 0); corners[2] = Point2f(img1.cols, img1.rows); corners[3] = Point2f(0, img1.rows); vector scene_corners(4); perspectiveTransform(corners, scene_corners, H); // Draw box Mat img2_color; cvtColor(img2, img2_color, COLOR_GRAY2BGR); line(img2_color, scene_corners[0], scene_corners[1], Scalar(0, 255, 0), 3); line(img2_color, scene_corners[1], scene_corners[2], Scalar(0, 255, 0), 3); line(img2_color, scene_corners[2], scene_corners[3], Scalar(0, 255, 0), 3); line(img2_color, scene_corners[3], scene_corners[0], Scalar(0, 255, 0), 3); ``` ## Blob Detection ```python theme={null} import cv2 as cv import numpy as np img = cv.imread('blobs.jpg', cv.IMREAD_GRAYSCALE) # Setup SimpleBlobDetector parameters params = cv.SimpleBlobDetector_Params() # Filter by area params.filterByArea = True params.minArea = 100 # Filter by circularity params.filterByCircularity = True params.minCircularity = 0.1 # Filter by convexity params.filterByConvexity = True params.minConvexity = 0.5 # Filter by inertia params.filterByInertia = True params.minInertiaRatio = 0.01 # Create detector detector = cv.SimpleBlobDetector_create(params) # Detect blobs keypoints = detector.detect(img) # Draw detected blobs img_with_keypoints = cv.drawKeypoints(img, keypoints, None, (0, 0, 255), cv.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS) print(f"Number of blobs: {len(keypoints)}") cv.imshow('Blobs', img_with_keypoints) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; int main() { Mat img = imread("blobs.jpg", IMREAD_GRAYSCALE); SimpleBlobDetector::Params params; params.filterByArea = true; params.minArea = 100; params.filterByCircularity = true; params.minCircularity = 0.1; params.filterByConvexity = true; params.minConvexity = 0.5; params.filterByInertia = true; params.minInertiaRatio = 0.01; Ptr detector = SimpleBlobDetector::create(params); vector keypoints; detector->detect(img, keypoints); Mat img_with_keypoints; drawKeypoints(img, keypoints, img_with_keypoints, Scalar(0, 0, 255), DrawMatchesFlags::DRAW_RICH_KEYPOINTS); imshow("Blobs", img_with_keypoints); waitKey(0); return 0; } ``` Feature detector comparison: * **SIFT**: Most robust, patented (free since 2020), slower * **SURF**: Fast, patented, good for real-time * **ORB**: Free, fast, good alternative to SIFT/SURF * **AKAZE**: Free, fast, works well with planar scenes * **BRISK**: Free, very fast, binary descriptor When matching features, always use the ratio test (Lowe's ratio test) to filter out ambiguous matches and reduce false positives. ## Next Steps * Use features for [Object Detection](/tutorials/object-detection) * Apply to [Camera Calibration](/tutorials/camera-calibration) * Explore [Image Stitching and Panoramas](/tutorials/image-stitching) # Image Operations Source: https://opencv-opencv.mintlify.app/tutorials/image-operations Learn basic image operations including resize, crop, rotate, flip, and filters in OpenCV # Image Operations Learn how to perform basic image operations in OpenCV, including resizing, cropping, rotating, flipping, and applying various filters to images. ## Reading and Displaying Images Before performing any operations, you need to load an image: ```python theme={null} import cv2 as cv import numpy as np # Read an image img = cv.imread('image.jpg') # Read as grayscale gray_img = cv.imread('image.jpg', cv.IMREAD_GRAYSCALE) # Display the image cv.imshow('Image', img) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; int main() { // Read an image Mat img = imread("image.jpg"); // Read as grayscale Mat gray_img = imread("image.jpg", IMREAD_GRAYSCALE); // Display the image imshow("Image", img); waitKey(0); destroyAllWindows(); return 0; } ``` ## Resizing Images Resize images to specific dimensions or by a scale factor: ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # Resize to specific dimensions resized = cv.resize(img, (640, 480)) # Resize by scale factor scale = 0.5 width = int(img.shape[1] * scale) height = int(img.shape[0] * scale) resized_scale = cv.resize(img, (width, height), interpolation=cv.INTER_LINEAR) # Maintain aspect ratio - scale by factor fx, fy = 0.5, 0.5 resized_aspect = cv.resize(img, None, fx=fx, fy=fy, interpolation=cv.INTER_AREA) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat resized, resized_scale; // Resize to specific dimensions resize(img, resized, Size(640, 480)); // Resize by scale factor double scale = 0.5; resize(img, resized_scale, Size(), scale, scale, INTER_LINEAR); ``` Interpolation methods: * `INTER_LINEAR`: Bilinear interpolation (default) * `INTER_AREA`: Best for shrinking images * `INTER_CUBIC`: Bicubic interpolation for enlarging * `INTER_LANCZOS4`: Lanczos interpolation over 8x8 neighborhood ## Cropping Images Crop images using array slicing: ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # Crop using slicing [y1:y2, x1:x2] cropped = img[100:400, 200:500] # Crop a region of interest (ROI) x, y, w, h = 100, 50, 300, 200 roi = img[y:y+h, x:x+w] cv.imshow('Cropped', cropped) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); // Define rectangle for ROI Rect roi(100, 50, 300, 200); // x, y, width, height Mat cropped = img(roi); imshow("Cropped", cropped); waitKey(0); ``` ## Rotating Images ### Simple 90-degree Rotations ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # Rotate 90 degrees clockwise rotated_90 = cv.rotate(img, cv.ROTATE_90_CLOCKWISE) # Rotate 90 degrees counter-clockwise rotated_90_ccw = cv.rotate(img, cv.ROTATE_90_COUNTERCLOCKWISE) # Rotate 180 degrees rotated_180 = cv.rotate(img, cv.ROTATE_180) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat rotated; // Rotate 90 degrees clockwise rotate(img, rotated, ROTATE_90_CLOCKWISE); // Rotate 90 degrees counter-clockwise rotate(img, rotated, ROTATE_90_COUNTERCLOCKWISE); // Rotate 180 degrees rotate(img, rotated, ROTATE_180); ``` ### Arbitrary Angle Rotation ```python theme={null} import cv2 as cv import numpy as np img = cv.imread('image.jpg') height, width = img.shape[:2] # Get rotation matrix for 45 degrees center = (width // 2, height // 2) angle = 45 scale = 1.0 rotation_matrix = cv.getRotationMatrix2D(center, angle, scale) # Apply rotation rotated = cv.warpAffine(img, rotation_matrix, (width, height)) cv.imshow('Rotated', rotated) cv.waitKey(0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat rotated; Point2f center(img.cols / 2.0, img.rows / 2.0); double angle = 45.0; double scale = 1.0; Mat rotation_matrix = getRotationMatrix2D(center, angle, scale); warpAffine(img, rotated, rotation_matrix, img.size()); ``` ## Flipping Images ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # Flip horizontally (left-right) flipped_h = cv.flip(img, 1) # Flip vertically (top-bottom) flipped_v = cv.flip(img, 0) # Flip both directions flipped_both = cv.flip(img, -1) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat flipped; // Flip horizontally flip(img, flipped, 1); // Flip vertically flip(img, flipped, 0); // Flip both flip(img, flipped, -1); ``` ## Image Filtering and Smoothing ### Gaussian Blur Remove noise and detail from images: ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # Apply Gaussian blur # Kernel size must be odd: (3,3), (5,5), (7,7), etc. blurred = cv.GaussianBlur(img, (5, 5), 0) # Larger kernel = more blur blurred_more = cv.GaussianBlur(img, (15, 15), 0) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat blurred; // Apply Gaussian blur GaussianBlur(img, blurred, Size(5, 5), 0); // Larger kernel for more blur GaussianBlur(img, blurred, Size(15, 15), 0); ``` ### Median Blur Excellent for removing salt-and-pepper noise: ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # Apply median blur (kernel size must be odd) median = cv.medianBlur(img, 5) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat median; medianBlur(img, median, 5); ``` ### Bilateral Filter Smooths images while preserving edges: ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') # d: diameter of pixel neighborhood # sigmaColor: filter in color space # sigmaSpace: filter in coordinate space bilateral = cv.bilateralFilter(img, d=9, sigmaColor=75, sigmaSpace=75) ``` ```cpp theme={null} #include using namespace cv; Mat img = imread("image.jpg"); Mat bilateral; bilateralFilter(img, bilateral, 9, 75, 75); ``` ## Complete Example: Image Processing Pipeline ```python theme={null} import cv2 as cv import numpy as np # Load image img = cv.imread('input.jpg') if img is None: print('Error loading image') exit() # Resize to a standard size img = cv.resize(img, (800, 600)) # Apply Gaussian blur to reduce noise blurred = cv.GaussianBlur(img, (5, 5), 0) # Crop region of interest h, w = blurred.shape[:2] roi = blurred[h//4:3*h//4, w//4:3*w//4] # Rotate the ROI center = (roi.shape[1]//2, roi.shape[0]//2) matrix = cv.getRotationMatrix2D(center, 15, 1.0) rotated = cv.warpAffine(roi, matrix, (roi.shape[1], roi.shape[0])) # Display results cv.imshow('Original', img) cv.imshow('Processed', rotated) cv.waitKey(0) cv.destroyAllWindows() # Save result cv.imwrite('output.jpg', rotated) ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { // Load image Mat img = imread("input.jpg"); if (img.empty()) { cout << "Error loading image" << endl; return -1; } // Resize to standard size resize(img, img, Size(800, 600)); // Apply Gaussian blur Mat blurred; GaussianBlur(img, blurred, Size(5, 5), 0); // Crop region of interest int h = blurred.rows, w = blurred.cols; Rect roi_rect(w/4, h/4, w/2, h/2); Mat roi = blurred(roi_rect); // Rotate the ROI Point2f center(roi.cols/2.0, roi.rows/2.0); Mat matrix = getRotationMatrix2D(center, 15, 1.0); Mat rotated; warpAffine(roi, rotated, matrix, roi.size()); // Display results imshow("Original", img); imshow("Processed", rotated); waitKey(0); // Save result imwrite("output.jpg", rotated); return 0; } ``` When working with images, always check if the image was loaded successfully before performing operations. Use `if img is None:` in Python or `if (img.empty())` in C++. ## Additional Operations Convert between different color spaces: ```python theme={null} # BGR to Grayscale gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # BGR to HSV hsv = cv.cvtColor(img, cv.COLOR_BGR2HSV) # BGR to RGB rgb = cv.cvtColor(img, cv.COLOR_BGR2RGB) ``` Create binary images: ```python theme={null} gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Simple threshold ret, thresh = cv.threshold(gray, 127, 255, cv.THRESH_BINARY) # Adaptive threshold adaptive = cv.adaptiveThreshold(gray, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY, 11, 2) ``` Be careful with image data types when performing operations. OpenCV typically uses `uint8` (0-255) for display, but intermediate calculations may require `float32` or `float64` to avoid overflow. ## Next Steps * Learn about [Video Processing](/tutorials/video-processing) for working with video streams * Explore [Feature Detection](/tutorials/feature-detection) for finding keypoints in images * Master [Object Detection](/tutorials/object-detection) to identify objects in images # Object Detection Source: https://opencv-opencv.mintlify.app/tutorials/object-detection Learn how to detect objects using Haar cascades and HOG detectors in OpenCV # Object Detection Learn how to detect objects in images and video using classical computer vision techniques including Haar cascades and Histogram of Oriented Gradients (HOG) detectors. ## Haar Cascade Classifiers Haar cascades are machine learning-based classifiers trained to detect specific objects. OpenCV comes with pre-trained models for faces, eyes, pedestrians, and more. ### Loading Cascade Classifiers ```python theme={null} import cv2 as cv # Load pre-trained cascade classifier face_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_frontalface_default.xml') eye_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_eye.xml') # Alternative: load from file path # face_cascade = cv.CascadeClassifier('haarcascade_frontalface_alt.xml') # Check if cascade loaded successfully if face_cascade.empty(): print('Error loading cascade classifier') exit() ``` ```cpp theme={null} #include #include using namespace cv; int main() { CascadeClassifier face_cascade; CascadeClassifier eye_cascade; // Load cascades if(!face_cascade.load(samples::findFile( "haarcascades/haarcascade_frontalface_default.xml"))) { cout << "Error loading face cascade" << endl; return -1; } if(!eye_cascade.load(samples::findFile( "haarcascades/haarcascade_eye.xml"))) { cout << "Error loading eye cascade" << endl; return -1; } return 0; } ``` ### Basic Object Detection ```python theme={null} import cv2 as cv # Load image img = cv.imread('group_photo.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Load cascade face_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_frontalface_default.xml') # Detect faces faces = face_cascade.detectMultiScale( gray, scaleFactor=1.1, # How much image size is reduced at each scale minNeighbors=5, # How many neighbors each candidate should have minSize=(30, 30), # Minimum object size flags=cv.CASCADE_SCALE_IMAGE ) print(f"Found {len(faces)} faces") # Draw rectangles around detected faces for (x, y, w, h) in faces: cv.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2) cv.imshow('Face Detection', img) cv.waitKey(0) ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { Mat img = imread("group_photo.jpg"); Mat gray; cvtColor(img, gray, COLOR_BGR2GRAY); CascadeClassifier face_cascade; face_cascade.load(samples::findFile( "haarcascades/haarcascade_frontalface_default.xml")); vector faces; face_cascade.detectMultiScale(gray, faces, 1.1, 5, 0, Size(30, 30)); cout << "Found " << faces.size() << " faces" << endl; for(size_t i = 0; i < faces.size(); i++) { rectangle(img, faces[i], Scalar(0, 255, 0), 2); } imshow("Face Detection", img); waitKey(0); return 0; } ``` ### Nested Detection (Faces and Eyes) Based on OpenCV's facedetect.py sample: ```python theme={null} import cv2 as cv def detect(img, cascade): """Detect objects using cascade classifier""" rects = cascade.detectMultiScale(img, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30), flags=cv.CASCADE_SCALE_IMAGE) if len(rects) == 0: return [] rects[:,2:] += rects[:,:2] # Convert to (x1, y1, x2, y2) return rects def draw_rects(img, rects, color): """Draw rectangles on image""" for x1, y1, x2, y2 in rects: cv.rectangle(img, (x1, y1), (x2, y2), color, 2) # Load cascades face_cascade = cv.CascadeClassifier(cv.samples.findFile( 'haarcascades/haarcascade_frontalface_alt.xml')) eye_cascade = cv.CascadeClassifier(cv.samples.findFile( 'haarcascades/haarcascade_eye.xml')) # Load and process image img = cv.imread('face.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) gray = cv.equalizeHist(gray) # Improve contrast # Detect faces faces = detect(gray, face_cascade) vis = img.copy() draw_rects(vis, faces, (0, 255, 0)) # Green for faces # Detect eyes within each face if not eye_cascade.empty(): for x1, y1, x2, y2 in faces: roi = gray[y1:y2, x1:x2] vis_roi = vis[y1:y2, x1:x2] eyes = detect(roi.copy(), eye_cascade) draw_rects(vis_roi, eyes, (255, 0, 0)) # Blue for eyes cv.imshow('Face and Eye Detection', vis) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; void detectAndDraw(Mat& img, CascadeClassifier& face_cascade, CascadeClassifier& eye_cascade) { Mat gray, smallImg; cvtColor(img, gray, COLOR_BGR2GRAY); double fx = 1.0 / 1.3; resize(gray, smallImg, Size(), fx, fx, INTER_LINEAR_EXACT); equalizeHist(smallImg, smallImg); vector faces; face_cascade.detectMultiScale(smallImg, faces, 1.1, 2, CASCADE_SCALE_IMAGE, Size(30, 30)); for(size_t i = 0; i < faces.size(); i++) { Rect r = faces[i]; Scalar color = Scalar(0, 255, 0); // Draw face rectangle Point center(cvRound((r.x + r.width*0.5)*1.3), cvRound((r.y + r.height*0.5)*1.3)); int radius = cvRound((r.width + r.height)*0.25*1.3); circle(img, center, radius, color, 3); // Detect eyes within face if(!eye_cascade.empty()) { Mat smallImgROI = smallImg(r); vector eyes; eye_cascade.detectMultiScale(smallImgROI, eyes, 1.1, 2, CASCADE_SCALE_IMAGE, Size(30, 30)); for(size_t j = 0; j < eyes.size(); j++) { Rect er = eyes[j]; Point eye_center(cvRound((r.x + er.x + er.width*0.5)*1.3), cvRound((r.y + er.y + er.height*0.5)*1.3)); int eye_radius = cvRound((er.width + er.height)*0.25*1.3); circle(img, eye_center, eye_radius, color, 3); } } } imshow("Detection", img); } ``` Key parameters for `detectMultiScale()`: * **scaleFactor**: How much the image size is reduced at each scale (1.1 = 10% reduction). Smaller values are more thorough but slower. * **minNeighbors**: How many neighbors each candidate rectangle should retain. Higher values result in fewer but more accurate detections. * **minSize**: Minimum object size. Objects smaller than this are ignored. ## HOG (Histogram of Oriented Gradients) Detector HOG descriptors are excellent for pedestrian detection. ### People Detection with HOG Based on OpenCV's peopledetect.py sample: ```python theme={null} import cv2 as cv def inside(r, q): """Check if rectangle r is inside rectangle q""" rx, ry, rw, rh = r qx, qy, qw, qh = q return rx > qx and ry > qy and rx + rw < qx + qw and ry + rh < qy + qh def draw_detections(img, rects, thickness=1): """Draw detection rectangles""" for x, y, w, h in rects: # HOG detector returns slightly larger rectangles # so we shrink them a bit pad_w, pad_h = int(0.15*w), int(0.05*h) cv.rectangle(img, (x+pad_w, y+pad_h), (x+w-pad_w, y+h-pad_h), (0, 255, 0), thickness) # Load image img = cv.imread('people.jpg') # Create HOG descriptor hog = cv.HOGDescriptor() # Set default people detector hog.setSVMDetector(cv.HOGDescriptor_getDefaultPeopleDetector()) # Detect people found, weights = hog.detectMultiScale(img, winStride=(8, 8), padding=(32, 32), scale=1.05) # Filter overlapping detections found_filtered = [] for ri, r in enumerate(found): for qi, q in enumerate(found): if ri != qi and inside(r, q): break else: found_filtered.append(r) print(f"Found {len(found_filtered)} people (from {len(found)} detections)") # Draw all detections draw_detections(img, found) # Highlight filtered detections draw_detections(img, found_filtered, 3) cv.imshow('People Detection', img) cv.waitKey(0) cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; class Detector { private: HOGDescriptor hog; public: Detector() { hog.setSVMDetector(HOGDescriptor::getDefaultPeopleDetector()); } vector detect(Mat& img) { vector found; hog.detectMultiScale(img, found, 0, Size(8,8), Size(), 1.05, 2, false); return found; } void adjustRect(Rect& r) { // Shrink rectangles slightly for better visualization r.x += cvRound(r.width*0.1); r.width = cvRound(r.width*0.8); r.y += cvRound(r.height*0.07); r.height = cvRound(r.height*0.8); } }; int main() { Mat img = imread("people.jpg"); if(img.empty()) { cout << "Error loading image" << endl; return -1; } Detector detector; vector found = detector.detect(img); cout << "Found " << found.size() << " people" << endl; for(size_t i = 0; i < found.size(); i++) { Rect r = found[i]; detector.adjustRect(r); rectangle(img, r.tl(), r.br(), Scalar(0, 255, 0), 2); } imshow("People Detection", img); waitKey(0); return 0; } ``` ### Real-time Detection on Video ```python theme={null} import cv2 as cv import time # Initialize HOG detector hog = cv.HOGDescriptor() hog.setSVMDetector(cv.HOGDescriptor_getDefaultPeopleDetector()) # Open video or camera cap = cv.VideoCapture(0) # or 'video.mp4' while True: ret, frame = cap.read() if not ret: break # Resize for faster processing frame = cv.resize(frame, (640, 480)) # Measure detection time start_time = time.time() # Detect people found, weights = hog.detectMultiScale(frame, winStride=(8, 8), padding=(8, 8), scale=1.05) elapsed_time = time.time() - start_time fps = 1.0 / elapsed_time # Draw detections for (x, y, w, h) in found: cv.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2) # Display FPS and count cv.putText(frame, f'People: {len(found)}', (10, 30), cv.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) cv.putText(frame, f'FPS: {fps:.1f}', (10, 70), cv.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) cv.imshow('HOG People Detection', frame) if cv.waitKey(1) & 0xFF == ord('q'): break cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { VideoCapture cap(0); if(!cap.isOpened()) return -1; HOGDescriptor hog; hog.setSVMDetector(HOGDescriptor::getDefaultPeopleDetector()); Mat frame; while(cap.read(frame)) { resize(frame, frame, Size(640, 480)); int64 t = getTickCount(); vector found; hog.detectMultiScale(frame, found, 0, Size(8,8), Size(), 1.05, 2, false); t = getTickCount() - t; double fps = getTickFrequency() / t; // Draw detections for(size_t i = 0; i < found.size(); i++) { rectangle(frame, found[i], Scalar(0, 255, 0), 2); } // Display info putText(frame, format("People: %d", found.size()), Point(10, 30), FONT_HERSHEY_SIMPLEX, 1, Scalar(0, 255, 0), 2); putText(frame, format("FPS: %.1f", fps), Point(10, 70), FONT_HERSHEY_SIMPLEX, 1, Scalar(0, 255, 0), 2); imshow("HOG People Detection", frame); if(waitKey(1) == 'q') break; } return 0; } ``` ## Available Pre-trained Cascades OpenCV includes many pre-trained cascade classifiers: * `haarcascade_frontalface_default.xml` - General frontal face detection * `haarcascade_frontalface_alt.xml` - Alternative frontal face * `haarcascade_frontalface_alt2.xml` - Another alternative * `haarcascade_profileface.xml` - Profile (side) faces * `lbpcascade_frontalface.xml` - LBP-based face detection (faster) * `haarcascade_eye.xml` - General eye detection * `haarcascade_eye_tree_eyeglasses.xml` - Eyes with glasses * `haarcascade_lefteye_2splits.xml` - Left eye * `haarcascade_righteye_2splits.xml` - Right eye * `haarcascade_fullbody.xml` - Full body detection * `haarcascade_upperbody.xml` - Upper body * `haarcascade_lowerbody.xml` - Lower body * `haarcascade_smile.xml` - Smile detection * `haarcascade_frontalcatface.xml` - Cat face detection * `haarcascade_frontalcatface_extended.xml` - Extended cat face * `haarcascade_licence_plate_rus_16stages.xml` - Russian license plates ## Custom Cascade Training You can train custom cascade classifiers for specific objects: Gather positive samples (images containing the object) and negative samples (images without the object). Create text files listing the locations of positive samples and paths to negative samples. Use `opencv_createsamples` to generate training samples from your positive images. Use `opencv_traincascade` to train the classifier. This can take hours or days depending on data size. Test the classifier and collect more samples if needed to improve accuracy. Training custom cascades requires: * Hundreds to thousands of positive samples * Even more negative samples * Significant computation time (can take days) * Careful parameter tuning For most modern applications, consider using deep learning-based detection instead. ## Performance Optimization ```python theme={null} import cv2 as cv img = cv.imread('image.jpg') gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY) # Resize for faster detection scale = 0.5 small = cv.resize(gray, None, fx=scale, fy=scale) face_cascade = cv.CascadeClassifier(cv.data.haarcascades + 'haarcascade_frontalface_default.xml') # Detect on smaller image faces = face_cascade.detectMultiScale(small, 1.1, 5) # Scale coordinates back to original size faces = [[int(x/scale), int(y/scale), int(w/scale), int(h/scale)] for (x, y, w, h) in faces] # Draw on original image for (x, y, w, h) in faces: cv.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2) ``` ```cpp theme={null} Mat img = imread("image.jpg"); Mat gray, small; cvtColor(img, gray, COLOR_BGR2GRAY); double scale = 0.5; resize(gray, small, Size(), scale, scale); CascadeClassifier face_cascade; face_cascade.load("haarcascade_frontalface_default.xml"); vector faces; face_cascade.detectMultiScale(small, faces, 1.1, 5); // Scale back to original size for(size_t i = 0; i < faces.size(); i++) { faces[i].x /= scale; faces[i].y /= scale; faces[i].width /= scale; faces[i].height /= scale; rectangle(img, faces[i], Scalar(0, 255, 0), 2); } ``` Performance tips: * Process at lower resolution (0.5x or 0.25x scale) * Use histogram equalization on grayscale images * Adjust `scaleFactor` (larger = faster but less accurate) * Increase `minNeighbors` to reduce false positives * Set appropriate `minSize` to skip small detections ## Next Steps * Learn [Face Detection](/tutorials/face-detection) for specialized face detection techniques * Explore [Deep Learning](/tutorials/deep-learning) for more accurate modern detection methods * Try [Video Processing](/tutorials/video-processing) to apply detection to video streams # Video Processing Source: https://opencv-opencv.mintlify.app/tutorials/video-processing Learn how to read, write, and process video files and camera streams frame-by-frame in OpenCV # Video Processing Learn how to capture video from files and cameras, process frames in real-time, and write processed video to disk. ## Video Capture Basics ### Capturing from Camera ```python theme={null} import cv2 as cv # Create VideoCapture object for default camera (0) cap = cv.VideoCapture(0) # Check if camera opened successfully if not cap.isOpened(): print("Error: Cannot open camera") exit() # Read and display frames while True: ret, frame = cap.read() if not ret: print("Can't receive frame. Exiting...") break cv.imshow('Camera', frame) # Press 'q' to quit if cv.waitKey(1) & 0xFF == ord('q'): break # Release resources cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { // Create VideoCapture object VideoCapture cap(0); if (!cap.isOpened()) { cout << "Error: Cannot open camera" << endl; return -1; } Mat frame; while (true) { cap >> frame; if (frame.empty()) { cout << "Can't receive frame" << endl; break; } imshow("Camera", frame); // Press ESC to quit if (waitKey(1) == 27) break; } cap.release(); destroyAllWindows(); return 0; } ``` Camera indices start at 0. If you have multiple cameras: * 0: Default camera (usually built-in webcam) * 1, 2, 3...: Additional cameras ### Reading Video Files ```python theme={null} import cv2 as cv # Open video file cap = cv.VideoCapture('video.mp4') if not cap.isOpened(): print("Error: Cannot open video file") exit() # Get video properties fps = cap.get(cv.CAP_PROP_FPS) width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT)) frame_count = int(cap.get(cv.CAP_PROP_FRAME_COUNT)) print(f"FPS: {fps}, Size: {width}x{height}, Frames: {frame_count}") while True: ret, frame = cap.read() if not ret: break cv.imshow('Video', frame) # Wait time to match video FPS (ESC to exit) if cv.waitKey(int(1000/fps)) & 0xFF == 27: break cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; int main() { VideoCapture cap("video.mp4"); if (!cap.isOpened()) { cout << "Error: Cannot open video file" << endl; return -1; } // Get video properties double fps = cap.get(CAP_PROP_FPS); int width = cap.get(CAP_PROP_FRAME_WIDTH); int height = cap.get(CAP_PROP_FRAME_HEIGHT); int frame_count = cap.get(CAP_PROP_FRAME_COUNT); cout << "FPS: " << fps << ", Size: " << width << "x" << height << ", Frames: " << frame_count << endl; Mat frame; while (cap.read(frame)) { imshow("Video", frame); if (waitKey(1000/fps) == 27) break; } return 0; } ``` ## Writing Video Files ### Basic Video Writer ```python theme={null} import cv2 as cv # Open camera cap = cv.VideoCapture(0) # Get video properties from source width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT)) fps = 20.0 # Define codec and create VideoWriter fourcc = cv.VideoWriter_fourcc(*'mp4v') out = cv.VideoWriter('output.mp4', fourcc, fps, (width, height)) if not out.isOpened(): print("Error: Cannot open video writer") exit() while True: ret, frame = cap.read() if not ret: break # Write frame to output video out.write(frame) cv.imshow('Recording', frame) if cv.waitKey(1) & 0xFF == ord('q'): break cap.release() out.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; int main() { VideoCapture cap(0); if (!cap.isOpened()) return -1; int width = cap.get(CAP_PROP_FRAME_WIDTH); int height = cap.get(CAP_PROP_FRAME_HEIGHT); double fps = 20.0; // Define codec and create VideoWriter int fourcc = VideoWriter::fourcc('m','p','4','v'); VideoWriter out("output.mp4", fourcc, fps, Size(width, height)); if (!out.isOpened()) return -1; Mat frame; while (cap.read(frame)) { out.write(frame); imshow("Recording", frame); if (waitKey(1) == 'q') break; } return 0; } ``` Common video codecs (FourCC codes): * `'mp4v'`: MPEG-4 (good compatibility) * `'XVID'`: Xvid codec * `'H264'` or `'X264'`: H.264 (best compression) * `'MJPG'`: Motion JPEG (larger files, faster encoding) ## Frame-by-Frame Processing ### Edge Detection on Video Based on OpenCV's edge.py sample: ```python theme={null} import cv2 as cv import numpy as np def nothing(x): pass # Create window with trackbars cv.namedWindow('edge') cv.createTrackbar('threshold1', 'edge', 2000, 5000, nothing) cv.createTrackbar('threshold2', 'edge', 4000, 5000, nothing) cap = cv.VideoCapture(0) while True: ret, frame = cap.read() if not ret: break # Convert to grayscale gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY) # Get threshold values from trackbars thrs1 = cv.getTrackbarPos('threshold1', 'edge') thrs2 = cv.getTrackbarPos('threshold2', 'edge') # Apply Canny edge detection edges = cv.Canny(gray, thrs1, thrs2, apertureSize=5) # Create visualization vis = frame.copy() vis = np.uint8(vis / 2.0) # Darken original vis[edges != 0] = (0, 255, 0) # Highlight edges in green cv.imshow('edge', vis) if cv.waitKey(5) & 0xFF == 27: break cap.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include using namespace cv; int main() { VideoCapture cap(0); if (!cap.isOpened()) return -1; namedWindow("edge"); int threshold1 = 2000, threshold2 = 4000; createTrackbar("threshold1", "edge", &threshold1, 5000); createTrackbar("threshold2", "edge", &threshold2, 5000); Mat frame, gray, edges, vis; while (cap.read(frame)) { cvtColor(frame, gray, COLOR_BGR2GRAY); // Apply Canny edge detection Canny(gray, edges, threshold1, threshold2, 5); // Create visualization frame.copyTo(vis); vis = vis / 2; // Darken vis.setTo(Scalar(0, 255, 0), edges); // Highlight edges imshow("edge", vis); if (waitKey(5) == 27) break; } return 0; } ``` ### Video Processing Pipeline Complete example with multiple processing steps: ```python theme={null} import cv2 as cv import numpy as np def process_frame(frame): """Apply multiple processing steps to a frame""" # Resize for faster processing frame = cv.resize(frame, (640, 480)) # Apply Gaussian blur blurred = cv.GaussianBlur(frame, (5, 5), 0) # Convert to HSV for better color detection hsv = cv.cvtColor(blurred, cv.COLOR_BGR2HSV) # Define color range (example: detect blue objects) lower_blue = np.array([100, 50, 50]) upper_blue = np.array([130, 255, 255]) # Create mask mask = cv.inRange(hsv, lower_blue, upper_blue) # Apply morphological operations kernel = np.ones((5, 5), np.uint8) mask = cv.morphologyEx(mask, cv.MORPH_CLOSE, kernel) mask = cv.morphologyEx(mask, cv.MORPH_OPEN, kernel) # Apply mask to original frame result = cv.bitwise_and(frame, frame, mask=mask) return result, mask # Main processing loop cap = cv.VideoCapture('input.mp4') # Setup video writer width = int(cap.get(cv.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv.CAP_PROP_FRAME_HEIGHT)) fps = cap.get(cv.CAP_PROP_FPS) fourcc = cv.VideoWriter_fourcc(*'mp4v') out = cv.VideoWriter('processed.mp4', fourcc, fps, (640, 480)) frame_count = 0 while True: ret, frame = cap.read() if not ret: break # Process frame processed, mask = process_frame(frame) # Write to output out.write(processed) # Display cv.imshow('Original', cv.resize(frame, (640, 480))) cv.imshow('Processed', processed) cv.imshow('Mask', mask) frame_count += 1 if frame_count % 30 == 0: print(f"Processed {frame_count} frames") if cv.waitKey(1) & 0xFF == ord('q'): break print(f"Total frames processed: {frame_count}") cap.release() out.release() cv.destroyAllWindows() ``` ```cpp theme={null} #include #include using namespace cv; using namespace std; Mat processFrame(Mat& frame) { Mat blurred, hsv, mask, result; // Resize for faster processing resize(frame, frame, Size(640, 480)); // Apply Gaussian blur GaussianBlur(frame, blurred, Size(5, 5), 0); // Convert to HSV cvtColor(blurred, hsv, COLOR_BGR2HSV); // Detect blue objects Scalar lower_blue(100, 50, 50); Scalar upper_blue(130, 255, 255); inRange(hsv, lower_blue, upper_blue, mask); // Morphological operations Mat kernel = getStructuringElement(MORPH_RECT, Size(5, 5)); morphologyEx(mask, mask, MORPH_CLOSE, kernel); morphologyEx(mask, mask, MORPH_OPEN, kernel); // Apply mask bitwise_and(frame, frame, result, mask); return result; } int main() { VideoCapture cap("input.mp4"); if (!cap.isOpened()) return -1; double fps = cap.get(CAP_PROP_FPS); int fourcc = VideoWriter::fourcc('m','p','4','v'); VideoWriter out("processed.mp4", fourcc, fps, Size(640, 480)); Mat frame; int frame_count = 0; while (cap.read(frame)) { Mat processed = processFrame(frame); out.write(processed); imshow("Original", frame); imshow("Processed", processed); frame_count++; if (frame_count % 30 == 0) cout << "Processed " << frame_count << " frames" << endl; if (waitKey(1) == 'q') break; } cout << "Total frames: " << frame_count << endl; return 0; } ``` ## Advanced Video Capture ```python theme={null} cap = cv.VideoCapture(0) # Set resolution cap.set(cv.CAP_PROP_FRAME_WIDTH, 1280) cap.set(cv.CAP_PROP_FRAME_HEIGHT, 720) # Set FPS cap.set(cv.CAP_PROP_FPS, 30) # Set brightness, contrast, etc. cap.set(cv.CAP_PROP_BRIGHTNESS, 0.5) cap.set(cv.CAP_PROP_CONTRAST, 0.5) ``` ```python theme={null} cap = cv.VideoCapture('video.mp4') # Jump to frame 100 cap.set(cv.CAP_PROP_POS_FRAMES, 100) ret, frame = cap.read() # Get current frame number current_frame = cap.get(cv.CAP_PROP_POS_FRAMES) ``` For better performance with slow cameras: ```python theme={null} import cv2 as cv from threading import Thread from queue import Queue class VideoCapture: def __init__(self, src): self.cap = cv.VideoCapture(src) self.q = Queue(maxsize=3) self.stopped = False def start(self): Thread(target=self.update, daemon=True).start() return self def update(self): while not self.stopped: if not self.q.full(): ret, frame = self.cap.read() if not ret: self.stopped = True return self.q.put(frame) def read(self): return self.q.get() def stop(self): self.stopped = True self.cap.release() # Usage cap = VideoCapture(0).start() while True: frame = cap.read() cv.imshow('Frame', frame) if cv.waitKey(1) & 0xFF == ord('q'): break cap.stop() ``` When writing videos, ensure the frame size matches the size specified in VideoWriter. Mismatched sizes will result in errors or corrupted output. ## Performance Tips Process at lower resolution for real-time applications, then upscale if needed. For non-real-time processing, you don't need to match the original video FPS. H.264 provides best compression but slower encoding. MJPEG is faster but larger files. Always call `cap.release()` and `out.release()` when done to free resources. ## Next Steps * Apply [Face Detection](/tutorials/face-detection) to video streams * Learn [Object Detection](/tutorials/object-detection) for tracking objects in video * Explore [Deep Learning](/tutorials/deep-learning) for advanced video analysis