MIT's UQ-DAAAM lets robots flag and correct uncertain memories of objects

A robot that maps a building may record a wall-mounted defibrillator as a fire extinguisher and then fail to find it when it matters. Researchers at MIT say they have a way for robots to notice such doubtful memories and repair them. Their system, UQ-DAAAM, was described by MIT News on November 5 and is the subject of a paper to be presented at the NeurIPS conference.

How the system works

UQ-DAAAM builds on DAAAM, an earlier system from the same group that uses a vision-language model to write captions for objects placed in a 3D map. Partial views, distance, motion blur or poor lighting can lead to wrong descriptions, and the original system had no means of revisiting them.

The new version computes an uncertainty score for every object. According to the paper, it combines two signals: how confident the model is in the words it generated, and how consistent the captions of the same object are when seen from different angles. If the score exceeds a threshold, the system selects a small number of additional frames likely to reduce the doubt, using a view-quality heuristic. A large language model then merges the captions into a single description, favouring the newer and clearer ones while keeping unresolved alternatives. The authors say they proved the frame selection can be done without re-running the vision-language model, which keeps the cost low enough for a moving robot.

Reported results

With a budget of two extra captions per object, the method resolved uncertainty for 92.3% of flagged objects in the paper's tests, compared with 34.2% for quality-based refinement alone and 5.6% for random choices. On a question-answering benchmark about what objects are, where they are and when they were last seen, accuracy rose to 0.761, against 0.711 for the original DAAAM. MIT describes this as a 5% gain in description accuracy over the state of the art.

Limits and next steps

The figures come from the authors' own experiments on recorded robot data and have not been independently replicated. About one in thirteen flagged objects remained unresolved, and the paper's guarantees are probabilistic, resting on an assumption that better views tend to reduce uncertainty. The team, led by associate professor Luca Carlone with graduate students Harry Zhang and Nicolas Gorlo, plans to extend the work to mobile manipulators.