Imperial College researchers adapt a vision-language model to predict vehicle collisions and explain why

Researchers at Imperial College London have adapted a general-purpose vision-language model (VLM) to warn of imminent vehicle collisions from video and to state the reasoning behind each warning. The work, by Ruici Zhang, Yuxiang Feng, Jose Escribano and Mohammed Quddus of the Centre for Transport Engineering and Modelling, appeared in Communications in Transportation Research on 30 September and was publicised in a news release from Tsinghua University Press on 6 November.

How the model works

The team started from VideoLLaMA2 and fine-tuned it with LoRA on short driving clips, half of them recorded just before a collision and half showing normal driving. According to the paper, the pre-collision clips come from the MM-AU dataset and the normal ones from CCD, with separate 3,000-clip datasets for prediction horizons of one, two and three seconds.

Rather than asking for a yes-or-no answer directly, the researchers use a two-step chain-of-thought prompt: the model first describes the scene and identifies possible sources of risk, then decides whether a crash is likely. The output therefore pairs a prediction with a written explanation.

Reported results

Averaged across the three horizons, the customised model reached 84.7% accuracy, 83.6% precision and 86.8% recall, the release says. The paper compares this with conventional deep-learning baselines, the strongest of which, a Swin Transformer, scored 81.3% accuracy; the unmodified VideoLLaMA2 managed 67.1%. Accuracy fell as the horizon lengthened, from about 87% at one second to about 82% at three seconds.

Because an 8.4-billion-parameter model is slow, the authors distilled it into Qwen2-VL-2B by aligning the student's hidden states with the teacher's. The smaller model averaged 82.4% accuracy while cutting latency from 2.78 seconds to 1.20 seconds per clip, measured on a workstation with two Nvidia RTX A6000 GPUs.

Limits and next steps

The authors acknowledge several constraints. The explanations are the model's own justifications; the team did not examine its internal reasoning. No speed, acceleration or position data were used, which the paper attributes to a lack of public collision datasets containing them, and the fine-tuned model was not re-tested on general benchmarks. Latency was measured only on that workstation, not on in-vehicle hardware, and even the distilled model's 1.2-second figure is close to the shortest prediction horizon tested. The release suggests future work could add vehicle kinematics and probe how VLMs reason internally. The paper lists Imperial President's PhD Scholarships as funding.