MemOIR: Driving Video Generation from View-Space Blocks World with Object-Level Memory

Rebuttal Video Results

Videos in each block play in sync; use the tabs to switch between scenes.

1. Input-matched comparison on Lyft L5 (zero-shot)

1.1 Comparison with matched inputs

All methods receive the same real initial frame, except MemOIR (No Initial Frame), which generates from the blocks-world condition alone; 15.4 s generations (185 frames at 12 Hz).

Frame rate. Lyft L5 provides camera frames and 3D box annotations at 5 Hz. For all Lyft L5 videos on this page, we interpolate each instance's 3D box trajectory between its annotated keyframes to obtain smooth 12 Hz conditions, and every method generates 12 Hz video from them. The real video is shown at its native 5 Hz, with each frame held to match 12 Hz playback.

1.2 Quality over time with full-chunk fine-tuning

MemOIR fine-tuned on 25-frame clips and on longer 61-frame chunks, with matched frame counts.

2. Consistency experiments

2.1 Forward–reverse revisit consistency

Lyft L5 (Scenes 1 and 4) and Argoverse 2 (Scenes 2–3). The first chunk drives forward; the second replays the ego and instance trajectories in reverse, so every instance is revisited. In Scenes 1–3, revisited instances are outlined in blue on the way out and in orange on the way back.

2.2 Pedestrian identity after re-entering the frame across the chunk boundary (WalkingTours, Wan2.1-VACE)

The chunk boundary is placed while the target person is out of view; markers on the timeline show the chunk boundary and the return. While fully visible, the target person is outlined in blue in chunk 1 and in orange in chunk 2.

2.3 Face identity after a turn-away across the chunk boundary (CelebV-HQ, Wan2.1-VACE)

The chunk boundary is placed mid-turn, when only the profile or the back of the head is visible.

3. Robustness experiments

3.1 Noisy box conditioning

One target vehicle per scene is perturbed in the conditioning canvas by location jitter (σ = 0.3 m per axis) or box dropout (30% of frames).

3.2 Identity switch within a chunk

The ego vehicle swerves until the target vehicle has almost left the frame; when it comes back, its box carries a new instance ID within the same chunk. The control keeps the original ID. While almost fully visible, the target is outlined in blue with its original ID and in orange with the new ID.

4. Blocks-world construction from a single image

4.1 Single-image to Blocks-world Construction Pipeline (No GT utilized)

Step-by-step construction of the blocks world from a single image and its camera intrinsics; off-the-shelf models only, without annotations, maps or LiDAR.

4.2 LLM-agent refinement

Closed-loop refinement of the automatically constructed blocks world (Scene 1 of 4.1) by an LLM agent that calls the same off-the-shelf tools; it then extends the world beyond the field of view of the image.

Generation from the initial image along a drive through the refined blocks world.