Stitching video frames into one under-vehicle image with OpenCV

An under-vehicle scanning system watches a car drive over a camera in the road and has to produce one long, usable photograph of the undercarriage. Security checkpoints use these. The camera sees a narrow moving slice of a large object, and the output has to be a single coherent image a human can inspect.

This one is in active development, so read it as a record of a problem being solved rather than a finished product. The interesting part is why the obvious approach fails, because the reason generalises well past cameras.

The obvious approach, and why it fails

The intuitive model: the car moves at a steady speed, so each video frame shows the next slice. Crop a narrow strip from the centre of every frame, stack the strips in order, and you have the whole underside.

I built that first, in Node with FFmpeg doing frame extraction:

frame 1        frame 2        frame 3
┌───┬───┬───┐  ┌───┬───┬───┐  ┌───┬───┬───┐
│   │▓▓▓│   │  │   │▓▓▓│   │  │   │▓▓▓│   │   crop the centre strip (~50px)
└───┴───┴───┘  └───┴───┴───┘  └───┴───┴───┘
      │              │              │
      └──────────────┴──────────────┘
                     v
            ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  paste in motion order

It produces an image. The image is wrong, in six distinct ways:

Every one of those traces back to one assumption: that displacement between consecutive frames is constant and known. It is not. The driver accelerates, the camera has a lens, and the underside is not a flat plane at a fixed distance.

The honest reframing: this is not an image concatenation problem with a cosmetic seam issue. It is a geometric alignment problem, and blending is the last step, not the fix.

What the pipeline had to become

  video
    │
    ├─ 1. extract frames with timestamps        (ffmpeg / ffprobe)
    ├─ 2. undistort using camera calibration    (removes lens curvature)
    ├─ 3. detect features per frame             (ORB or SIFT)
    ├─ 4. match descriptors between frames      (BFMatcher)
    ├─ 5. filter matches                        (ratio test)
    ├─ 6. estimate homography                   (RANSAC)
    ├─ 7. reject frames with weak geometry      (too few inliers)
    ├─ 8. warp into a shared coordinate system  (warpPerspective)
    ├─ 9. blend overlaps                        (feather / multi-band)
    └─ 10. crop to presentation format
    │
    v
  one undercarriage image

Step 6 is the one that replaces the broken assumption. Instead of guessing how far the vehicle moved, each frame pair tells me its own transform, computed from features both frames can see. Speed changes stop mattering, because nothing depends on speed being constant.

RANSAC matters as much as the homography does. Feature matching on a dirty undercarriage produces plenty of wrong matches: repeated bolts, symmetric panels, reflections that move independently of the car. Fitting a transform to all matches gives a transform corrupted by the bad ones. RANSAC fits to the largest self-consistent subset and reports how many matches agreed, which doubles as a confidence signal I can threshold on.

The hardest decision

Frame decimation. A vehicle passing slowly generates hundreds of frames with huge overlap, and processing all of them is slow and adds accumulated error. A fast vehicle generates frames with barely any overlap and needs all of them.

Deciding by fixed interval or by timestamp reintroduces the constant-speed assumption through the back door. Deciding by content does not: keep a frame only when feature similarity against the last kept frame drops below a threshold. Then a stationary car contributes almost nothing and a fast one contributes everything, without either case being special-cased.

The threshold I started from was around 50 matched features. That number is not transferable. It depends on camera resolution, lighting, how textured the undercarriage is, and vehicle speed, so it is a calibration knob that needs tuning per installation, not a constant to copy.

The lesson that generalises

Better blending can hide mild exposure differences. It cannot repair an incorrect homography.

I spent real time on blending methods - linear feathering, exposure compensation, pyramid and multi-band blending - while the underlying alignment was still wrong. Every one of them made the seams less obvious and the image no more correct. Smoothing over a structural error makes it harder to see, which is worse than the visible seam that at least told me something was broken.

This shows up everywhere outside computer vision. Retries that paper over an unreliable call, a cache that hides a slow query, a rounded-up dashboard that smooths a real spike. If the layer underneath is producing wrong output, polishing the layer above buys presentation, not correctness.

The multi-camera phase

One camera cannot see the full width of a vehicle. The four-camera design chains pairwise transforms into one coordinate system:

  cam 1 ──H12──> cam 2 ──H23──> cam 3 ──H34──> cam 4

Each pairwise homography is estimated once during installation and saved to config, since the cameras are bolted down and their relationship does not change per vehicle. What does need watching is accumulated error: a small misalignment in H12 propagates through the composition, so error at camera 4 is the sum of three estimates, not one. That gets validated against a set of frames rather than assumed, and each camera gets its own lens calibration because they are not identical units.

Where it stands

The feature-based pipeline is implemented and the four-camera geometry is in validation. I am not going to claim a finished production result here, because it does not have one yet. What is settled is the diagnosis: strip concatenation cannot work, the problem is alignment before it is appearance, and content-based frame selection is the right way to handle variable speed.

Technologies: Node.js, Python, OpenCV, FFmpeg, FFprobe, ORB, SIFT, BFMatcher, RANSAC, homography estimation, multi-band blending, Blender for rig simulation.