Open Source RoboticsRobots, software & learning

Scale Up Strategically

Software

Instruction-factor bias evaluation and demonstration collection for robot manipulation policies in a ManiSkill task. [1] [2]

Installation & first result →Official websiteSource codeGetting started

For
Robot policy researchers evaluating compositional instruction following. [1]
Hardware
Simulated tabletop manipulation task [1]
Source rights
Some licenses are unresolved. Review licenses
Unknowns
operating system not fully established. Review gaps

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
not fully assessed for software, documentation · Details by part

Requirements · Research details and dates

On this page

Overview

This CoRL 2026 project supplies a concrete diagnostic task for a common policy failure: a model can appear to follow an instruction while relying on one correlated feature, such as color, and ignoring the requested action. The released repository includes a ManiSkill 3 fork with VerbObjectColor-v1, demonstration collection and evaluation code for Factor Dominance Rate and Hierarchy. It is a distinct evaluation workflow, not another general ManiSkill profile. [1] [2] [3]

Is this for you?

Workflows
  • Collect conflict demonstrations
  • Fine-tune/serve a policy
  • Label rollouts and calculate factor dominance

[1]

Known limitations
  • No root repository-wide license identified.
  • Requires policy serving and VLM judging in separate environments.
  • Simulation results are not physical robot results.

[1] [2] [4]

What you need

Languages / runtimes
  • Python

[1]

Dependencies
  • ManiSkill 3 fork
  • mplib
  • GR00T or OpenPI for documented policy recipes
  • VLM judge

[1]

Host computer

Linux required for motion planning; policy training needs separate GPU environment.

[1]

Compute for the workload
  • Linux/mplib and separate policy training environment; exact GPU minimum not established.

[1]

Installation & first result

Open the upstream getting-started guide [1]

  • Install the ManiSkill fork plus collection/evaluation requirements on Linux.
  • Collect factor-conflict demonstrations, serve a selected policy and run evaluation clients.
  • Compute FDR/FDH from rollout labels.

[1] [2] [3]

Installation

README gives editable ManiSkill installation and separate collection/evaluation requirements.

[1]

Hardware, models & data

Supported hardware
  • Simulated tabletop manipulation task; no physical robot route established in this review.

[1]

Inputs and outputs

Factor-conflict demonstrations and policy rollouts to factor labels and FDR/FDH measures.

[1]

Algorithms and models
  • Factor Dominance Rate
  • Factor Dominance Hierarchy

[1]

Evaluation

FDR/FDH comparison of instruction factors in held-out conflicts; no result reproduced here.

[1]

Project notes

The documented route

The README starts with installing the fork and the collection and evaluation requirements. Motion planning needs Linux and mplib; policy training and serving run in separate GR00T or OpenPI environments, with evaluation clients connecting over the network. Training examples pair instruction factors that always co-occur; held-out scenes then force the policy to choose between them. The default video-labeling route uses Gemini-2.5-Flash and a GEMINI_API_KEY; the documented --env-dir route instead computes factor dominance from simulator success counts. Neither route was run here. [1] [2] [3]

For each factor pair, positive or negative FDR indicates which factor dominated the labeled rollouts; FDH ranks factors across pairs. These measures diagnose a specific shortcut in instruction following rather than providing a general policy-success score. [1] [2]

The root repository has no identified repository-wide license in this review. Its maniskill/ fork has an Apache-2.0 license file, which does not settle rights for the root evaluation and collection code, assets or data. This entry is therefore source-available with unresolved reuse terms at that scope. No installation, rollout, fine-tuning, VLM judging or metric calculation was performed here. [1] [4]

What we checked

Source research only. No commercial relationship or hands-on testing. No upstream media copied to the public site.

Not yet established

  • Requirements: operating system

Updates

Upstream last updated
Not established
Latest release
Not established
Profile updated
Nov 7, 2026
Sources last checked
Nov 7, 2026
Next review
Dec 7, 2026
Reviewed revision
Commit snapshot b5b32418bd070b2ce55f3fcde21d742306331ce0; no matching GitHub release found

Source files and licenses

Released robot-specific diagnostic task and implementation, distinct from the ManiSkill base; root reuse rights are unresolved.

LayerAvailability / license
softwareavailable · License not fully assessed
Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [2]
documentationavailable · License not fully assessed
Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [1]

No separate artifact selected for this scoped entry.

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.7.0-draft.5

Change history