Scale Up Strategically
SoftwareInstruction-factor bias evaluation and demonstration collection for robot manipulation policies in a ManiSkill task. [1] [2]
Installation & first result →Official websiteSource codeGetting started
- For
- Robot policy researchers evaluating compositional instruction following. [1]
- Hardware
- Simulated tabletop manipulation task [1]
- Source rights
- Some licenses are unresolved. Review licenses
- Unknowns
- operating system not fully established. Review gaps
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- not fully assessed for software, documentation · Details by part
Overview
This CoRL 2026 project supplies a concrete diagnostic task for a common policy failure: a model can appear to follow an instruction while relying on one correlated feature, such as color, and ignoring the requested action. The released repository includes a ManiSkill 3 fork with VerbObjectColor-v1, demonstration collection and evaluation code for Factor Dominance Rate and Hierarchy. It is a distinct evaluation workflow, not another general ManiSkill profile. [1] [2] [3]
Is this for you?
- Workflows
- Collect conflict demonstrations
- Fine-tune/serve a policy
- Label rollouts and calculate factor dominance
What you need
- Languages / runtimes
- Python
- Dependencies
- ManiSkill 3 fork
- mplib
- GR00T or OpenPI for documented policy recipes
- VLM judge
- Host computer
Linux required for motion planning; policy training needs separate GPU environment.
- Compute for the workload
- Linux/mplib and separate policy training environment; exact GPU minimum not established.
Installation & first result
Open the upstream getting-started guide [1]
- Install the ManiSkill fork plus collection/evaluation requirements on Linux.
- Collect factor-conflict demonstrations, serve a selected policy and run evaluation clients.
- Compute FDR/FDH from rollout labels.
- Installation
README gives editable ManiSkill installation and separate collection/evaluation requirements.
Hardware, models & data
- Supported hardware
- Simulated tabletop manipulation task; no physical robot route established in this review.
- Inputs and outputs
Factor-conflict demonstrations and policy rollouts to factor labels and FDR/FDH measures.
- Algorithms and models
- Factor Dominance Rate
- Factor Dominance Hierarchy
- Evaluation
FDR/FDH comparison of instruction factors in held-out conflicts; no result reproduced here.
Project notes
The documented route
The README starts with installing the fork and the collection and evaluation requirements. Motion planning needs Linux and mplib; policy training and serving run in separate GR00T or OpenPI environments, with evaluation clients connecting over the network. Training examples pair instruction factors that always co-occur; held-out scenes then force the policy to choose between them. The default video-labeling route uses Gemini-2.5-Flash and a GEMINI_API_KEY; the documented --env-dir route instead computes factor dominance from simulator success counts. Neither route was run here. [1] [2] [3]
For each factor pair, positive or negative FDR indicates which factor dominated the labeled rollouts; FDH ranks factors across pairs. These measures diagnose a specific shortcut in instruction following rather than providing a general policy-success score. [1] [2]
The root repository has no identified repository-wide license in this review. Its maniskill/ fork has an Apache-2.0 license file, which does not settle rights for the root evaluation and collection code, assets or data. This entry is therefore source-available with unresolved reuse terms at that scope. No installation, rollout, fine-tuning, VLM judging or metric calculation was performed here. [1] [4]
What we checked
Source research only. No commercial relationship or hands-on testing. No upstream media copied to the public site.
Not yet established
- Requirements: operating system
Updates
- Upstream last updated
- Not established
- Latest release
- Not established
- Profile updated
- Nov 7, 2026
- Sources last checked
- Nov 7, 2026
- Next review
- Dec 7, 2026
- Reviewed revision
- Commit snapshot b5b32418bd070b2ce55f3fcde21d742306331ce0; no matching GitHub release found
Source files and licenses
Released robot-specific diagnostic task and implementation, distinct from the ManiSkill base; root reuse rights are unresolved.
| Layer | Availability / license |
|---|---|
| software | available · License not fully assessed Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [2] |
| documentation | available · License not fully assessed Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [1] |
No separate artifact selected for this scoped entry.
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 Benchmark workflow
Revision, date & where cited
yqi19/CoRL2026-Scale-Up-Strategically contributors · github.com · revision
b5b3241· README.md at · Accessed Nov 7, 2026
Used for: Overview, Is this for you?, What you need, Installation & first result and 3 more - S002 Evaluation implementation guide
Revision, date & where cited
yqi19/CoRL2026-Scale-Up-Strategically contributors · github.com · revision
b5b3241· evaluation/README.md at · Accessed Nov 7, 2026
Used for: Overview, Is this for you?, Installation & first result, The documented route and 1 more - S003 Demonstration collection guide
Revision, date & where cited
yqi19/CoRL2026-Scale-Up-Strategically contributors · github.com · revision
b5b3241· data_collection/README.md at · Accessed Nov 7, 2026
Used for: Overview, Installation & first result, The documented route - S004 Fork subdirectory license
Revision, date & where cited
yqi19/CoRL2026-Scale-Up-Strategically contributors · github.com · revision
b5b3241· maniskill/LICENSE at · Accessed Nov 7, 2026
Used for: Is this for you?, The documented route
Correction guidance · Discussion is not enabled · Profile format 0.7.0-draft.5