Make the box bigger than the thing
The box does not move. Play the whole clip and size the box to cover everywhere the face or object goes, with a margin. For a number plate on a parked car a tight box is fine; for a person who shifts in their seat, cover the whole area they move through.