THE ESSENTIALS IN 30 SECONDS

  • Ai2 presents ToyBlocks, published on November 6, as the first of what it calls FoundationWorlds: controlled simulated domains with task generation, demonstrations and metrics.
  • The initial dataset holds 85,336 long-horizon demonstrations, about 1,600 hours of robot execution and 144 million frames, generated in Isaac Lab.
  • A Molmo 2-based planner combined with a ManiFlow policy succeeded in 15 of 30 real-world episodes on a Franka FR3, which Ai2 says needs more trials to confirm.

The Allen Institute for AI (Ai2) released ToyBlocks on November 6, an open simulation framework in which a robot arm pushes, flips, rotates and stacks blocks on a table. The goal is not a more capable robot but a controlled setting where researchers can change one factor at a time, such as training data, planning or low-level control, and see which one causes a policy to fail.

A "FoundationWorld" for one domain

Ai2 describes ToyBlocks as the first of a series it calls FoundationWorlds: simulated environments for a specific robotics domain, each bundled with automatic task generation, demonstration generation and metrics that track robot behaviour over time. Tasks range from single skills to long-horizon problems such as spelling three-letter words with lettered blocks.

The first release contains 85,336 long-horizon demonstrations, each chaining roughly six to twelve skills, for close to 1,600 hours of robot execution and 144 million frames. According to Ai2, skills are first trained in Isaac Lab with reinforcement learning and motion planning, then a task-and-motion planner sequences them, running hundreds of plans in parallel on one GPU at up to 24 times real time. The code is on GitHub under the Apache 2.0 licence, with some submodules and assets under their own terms.

Two models and real-robot checks

Ai2 also published two models. A planner fine-tuned from Molmo2-4B, about 4.5 billion parameters according to its model card, chooses the next skill and target block from a camera image. A ManiFlow-based policy then executes the commanded skill.

On a Franka FR3 arm with a Robotiq gripper across 30 scene configurations, the ManiFlow skills reached 81.1% success, close to 80.0% in simulation. The combined planner and policy succeeded in 15 of 30 real episodes, against 55.5% in simulation; Ai2 notes that more trials are needed for statistical confidence.

Early findings and open questions

The lab reports that skill descriptions written by an agent raised success on a search-and-spell task from 24% to 63%, above the 36% obtained with expert-written ones. It adds that several popular vision-language-action models generalised poorly in early spatial tests. In world-model experiments, adding 100 generated rollouts lifted one policy's success in an unseen region from 19% to 78%.

These results come from Ai2's own blog; the GitHub page lists a technical report as coming soon, so the methods have not yet been documented in a paper. The Molmo-based planner's card also notes that some third-party training data behind its base model is restricted to non-commercial research.