Computer vision / OpenCV / Kurento / Case study
Keeping AR annotations anchored in live video
Keeping annotations attached to moving objects in a live video stream requires more than storing screen coordinates. This case study examines a tracking solution built with OpenCV and integrated into Kurento, using SIFT feature matching to preserve annotation positions as the camera or objects move.
The visual model
Annotation geometry vs. tracking region
The solid shape is the original annotation. The dashed region provides more texture for tracking; it does not enlarge the annotation.
01 / Reference selection
Input: shape type and points. Select tracking context around the original annotation.
02 / Current frame
Output: project the original geometry only when matching and geometric checks support the attachment.
Illustrative planar motion, not captured footage or measured tracking output. A valid transform moves the original mark; insufficient confidence hides it.
- Media sourceIncoming WebRTC
- Frame processingOpenCV-based Kurento media filter
- Filtered streamOutgoing WebRTC
Geometric validation
Accepted and rejected correspondence patterns
Geometric checks pass
The supporting points fit a consistent transformation.
Candidate rejected
The proposed correspondences provide insufficient agreement for the assumed acceptance criteria.
Schematic examples, not tracker output. An accepted geometric model may still follow background features; surface selection needs a separate check.
Camera movement makes fixed screen coordinates unsuitable for remote-assistance annotations. I separated the geometry drawn by the user from the image region used for tracking, allowing a small mark to use additional surrounding texture without changing the original geometry.
This case study is a high-level reconstruction of professional work. Diagrams are illustrative. Employer source, client footage, internal configuration, and production metrics are excluded. Homography validation, reference management, selection revisions, and recovery policies below explain the reconstructed design; they are not a verified record of every mechanism shipped.
01 / Annotation and tracking regions
Separating annotation geometry from tracking regions
The implemented workflow accepted a shape type and its defining points. Small rectangles and short lines could identify something useful to a technician while containing too little distinctive texture for feature matching. Expanding the feature-detection region addressed that problem while preserving the annotation coordinates.
The reconstruction also makes coordinate mapping explicit: convert displayed-video positions into decoded-frame coordinates, accounting for scaling, cropping, letterboxing, and mirroring. The captured reference image and selection must represent the same moment. Extract features from clean pixels before drawing an overlay.
A larger region can include a separately moving background. Its features may dominate even when the annotation refers to a foreground object. Keep the region inside the image bounds and assess whether the added texture belongs to the intended surface; more area alone cannot establish reliable attachment.
02 / Matching and geometric validation
Feature matching and geometric validation
SIFT detects keypoints and computes descriptors that tolerate changes in scale and orientation, with limited tolerance for viewpoint and appearance changes. For conventional SIFT descriptors, nearest-neighbor matching uses L2, or Euclidean, distance. Lowe's ratio test compares the best and second-best candidate distances to reject ambiguous correspondences.
In the reconstructed pipeline, RANSAC estimates a homography from matched coordinates. A homography models views of a planar surface, or a scene under pure camera rotation; it does not generally explain non-rigid deformation or depth-dependent parallax. Estimation needs at least four non-degenerate correspondences. Rendering also requires sufficient inlier support, low reprojection error, useful spatial coverage, and a numerically valid, plausible transformation.
A geometrically consistent model can still follow the wrong surface. Background inliers may support an excellent homography while moving the annotation away from the intended object. Confidence thresholds cannot resolve that semantic error. Once a model is accepted, project the original coordinates through it: construct and transform all rectangle corners, or sample curved outlines rather than merely moving their centers.
03 / Media processing and recovery
Media integration, processing cost, and tracking recovery
The tracking feature was integrated into a Kurento media-processing workflow. In the reconstruction, an OpenCV-based Kurento media filter reads an unannotated frame, estimates the transformation, and draws the accepted projection before forwarding the processed video.
The original workflow displayed annotations when matching support was sufficient. The reconstructed recovery policy suppresses an unreliable overlay and attempts reacquisition against reference information. A fresh result must pass validation before display resumes. Selection revisions are a proposed synchronization mechanism here: results for an older annotation cannot replace a newer selection.
Feature extraction, descriptor matching, and geometric estimation consume CPU time in the video path. Full detection on every frame may increase latency or create a backlog. Constrained search regions, less frequent detection, and optical flow between detections are alternatives to evaluate, not claimed production optimizations. Reducing work can also delay recovery or miss rapid motion; processing frequency needs representative footage and latency measurements.
System flow
The reconstructed tracking pipeline
- 01
Receive video
Read a clean decoded frame from the incoming media path.
- 02
Select reference context
Capture the shape points and a suitable feature-detection region.
- 03
Estimate motion
Match SIFT descriptors and validate the estimated geometry.
- 04
Render the annotation
Project the original shape when the current result is accepted.
Decision record
Design decisions and tradeoffs
↔ Scroll horizontally to compare approaches and tradeoffs.
| Decision | Approach | Tradeoff |
|---|---|---|
| Separate geometry and tracking context | Expand the feature region while preserving the user's drawing. | Additional features can belong to unrelated surfaces. |
| Validate correspondence and geometry | Use descriptor filtering and robust estimation in the reconstruction. | Adds computation; geometric consistency does not prove correct surface selection. |
| Control work in the video path | Evaluate detection frequency, constrained search, and lighter intermediate tracking. | Lower CPU demand can reduce recovery speed or robustness during rapid motion. |
| Define reference and selection ownership | Reconstruction: fixed-reference matching and revision-bound selection updates. | Fixed references can fail after appearance changes; keyframe updates introduce drift and state-management risks. |
Failure cases & validation
Failure scenarios to validate
Proposed validation scenarios and expected behavior. These are design checks, not reported production test results.
A small rectangle contains too little texture.
Expand suitable same-surface context without enlarging the annotation.
Background features dominate the matches.
Reconsider the tracking region; high inlier support alone does not establish correct attachment.
Occlusion or feature loss interrupts tracking.
Continue only with sufficient support; otherwise hide the overlay until a fresh result passes validation.
Perspective changes distort the selected surface.
Project the original geometry and reject invalid or implausible results.
A selection changes during frame processing.
In the revision-based reconstruction, discard results belonging to the earlier selection.
Reflection
Engineering lessons
- 01
Separate annotation coordinates from feature-detection bounds so small selections can use surrounding texture.
- 02
Evaluate whether matches represent the intended surface as well as whether they support a geometric model.
- 03
Balance tracking reliability against processing latency, and define what users see when a result cannot be trusted.
Evidence & further reading