Object removal models have improved faster than the metrics used to judge them. Diffusion erasers now reconstruct shadows, reflections and occluded structure convincingly, yet PSNR, SSIM, LPIPS, ReMOVE and CFD frequently rank their outputs the wrong way. The root cause is structural: erasure is an ill-posed, one-to-many task, so no single ground truth exists to compare against. A team from MiLM Plus, Xiaomi Inc. has released PROVE (Perceptual RemOVal cohErence), accepted at ACM MM 2026, to close that gap. PROVE pairs two perception-aligned metrics — RC-S for spatial coherence and RC-T for temporal consistency — with PROVE-Bench, a two-tier real-world video benchmark. Both metrics score the edited region locally, using sliding-window Maximum Mean Discrepancy over DINOv2 features, and neither requires a reference video.
Is it deployable?
Yes — as an evaluation harness, not a product feature. PROVE ships as an Apache 2.0 PyTorch repo with one CLI entry point (run_prove_metrics.py). It needs Python 3.10+, PyTorch 2.6+, Transformers 4.51+, and DINOv2-giant weights. Masks are mandatory; white pixels mark the removed object.
- Company level: Any team with one GPU and a mask pipeline. RC-S runs at 134.6 ms/frame on a single RTX 4090, so nightly CI gates are practical for startups; enterprises with large editing catalogs gain most, since no paired ground truth is needed.
- Industries: Smartphone gallery and camera apps, short-video editing, e-commerce catalog cleanup, advertising and stock media, film/VFX post-production, real estate imagery, and privacy redaction in mapping archives.
- Applications: Model bake-offs, CI gates on RC-S/RC-T, tuning inference steps or quantization without a reference video, filtering training data, and shaping reward models.
- Who buys in: CV researchers and applied AI/ML engineers on editing teams first; MLOps engineers wiring the score into CI second; product managers running vendor comparisons third.
- Where it does not fit: Real-time on-device scoring, and side effects such as large shadows or reflections extending beyond the cropped evaluation region.
Three documented failure modes
Object removal is ill-posed and one-to-many: many restorations are plausible for the same hole, so no unique ground truth exists.
- Full-reference bias: PSNR, SSIM and LPIPS assume point-to-point correspondence, so they reward copy-paste over genuine erasure. Residual shadows occupy few pixels and incur almost no penalty. Worse, cutting diffusion inference steps improves PSNR and SSIM while visual quality collapses — regression to the mean.
- No-reference blind spots: On ROSE-Bench, the authors progressively blur the masked region. Neither ReMOVE nor CFD degrades; both eventually surpass their unblurred baselines. CFD’s SAM-based hallucination term also misfires: a legitimately restored bicycle seat is flagged as a hallucination, scoring worse than the unedited input.
- Temporal insensitivity: Temporal Consistency and Temporal Flickering operate on full-frame features. Since the edited area is a small fraction of the frame, injected Random Drop and Random Replace corruptions on DAVIS barely move them, and sometimes move them the wrong way.
RC-S and RC-T
Both share one idea: local distribution matching in deep feature space instead of global aggregation.
RC-S (spatial): Connected-component analysis splits the mask into independent targets. Each bounding box is expanded by one third of its side length, the crop goes to DINOv2, and the mask is downsampled to feature resolution. A w×w window slides across the feature map, computing squared Maximum Mean Discrepancy with a Gaussian RBF kernel between masked and local background features. Scores are averaged per target, then across targets.
RC-T (temporal): Adjacent frames are cropped jointly under the union of their masks to avoid misalignment, then MMD is computed only inside the intersection — the region restored in both frames. An ablation shows that removing the crop makes RC-T blind to injected corruption.
Results
Against human rankings from 20 participants aggregated by Borda count, RC-S reaches 0.59 average Kendall’s τ and 0.66 Spearman’s ρ, versus 0.26/0.29 for ReMOVE and 0.16/0.18 for CFD. It ranks first on five of six benchmarks. Background-only FR variants correlate negatively on most datasets. On RORD-Val, RC-S prefers the clean image over blurred and region-swapped variants in 100% of cases; ReMOVE manages 60.06% and CFD 49.27% under blur. RC-T responds monotonically to rising corruption where TC and TF do not.
Ablations isolate each contribution: DINOv2 (0.59 avg τ) beats DINOv3 (0.51) and SAM (0.44); dropping the sliding window costs 0.11; swapping MMD for cosine costs 0.07. RC-S is also the cheapest spatial metric tested, 13.7× faster than CFD.
PROVE-Bench
PROVE-M holds 80 paired videos: tripod-captured input and target-free footage shot within two minutes, SAM3 masks refined frame by frame, three-stage quality control, then Ken Burns-style motion augmentation applied synchronously to the triplet. Each clip is 81 frames at 1080p. PROVE-H adds 100 hard videos without ground truth — crowds, flowing water, flames, textured terrain, multi-puddle reflections, fast motion — using unrefined SAM3 masks on purpose.
On the public leaderboard, SVOR (1.3B) leads combined RC-S at 0.5197; EffectErase leads RC-T at 0.2525.
‘;
}else{
var w=Math.max(0,(v+0.3)/1.1*100);
h+=”;
}
}
$(‘bars’).innerHTML=h;rz();
}
/* ———- resize ———- */
function rz(){
try{
var el=document.getElementById(‘pv’);
var hgt=el.offsetHeight+40;
parent.postMessage({mtpProveHeight:hgt},’*’);
}catch(e){}
}
window.addEventListener(‘resize’,function(){moveWin();rz()});
drawBlur();moveWin();drawT();drawBars();
setTimeout(function(){moveWin();rz()},260);
setTimeout(rz,900);
})();
