NVIDIA has released Cosmos3-H-Dreams, a real-time generative world model for robotic surgery that simulates clinical gallbladder operations and responds live to a surgeon’s controls. The model checkpoint and companion code were published on October 20 on GitHub and Hugging Face, in an NVIDIA-authored announcement that describes it, to the company’s knowledge, as the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.
The release moves NVIDIA’s Cosmos world-model line from tabletop experiments into real operating-theatre footage. Cosmos3-H-Dreams is fine-tuned on recordings from CMR Surgical’s Versius robot and is offered as a research simulator for surgical robotics teams, not as a validated system for patient care.
How Cosmos3-H-Dreams Generates A Surgical Scene From Robot Actions
The model is an action-conditioned world model: it learns visual dynamics directly from synchronised video and robot kinematics rather than from an explicit model of scene geometry or tissue deformation. It builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model trained on the Open-H-Embodiment dataset, and specialises a bidirectional teacher on curated cholecystectomy video and kinematics. Cosmos3-H-Dreams itself has about 15.17 billion parameters and uses a Mixture-of-Transformers architecture that pairs a Qwen3-VL-8B-Instruct reasoning tower with a latent-diffusion generation expert, operating on Wan2.2 VAE latents with action embedding layers for kinematic conditioning.
Its action interface uses 28 dimensions per time step, with 14 dimensions per arm covering anchor-relative translation, 6D rotation, jaw position, electrosurgical energy activation and thumbstick inputs, extended from 20 dimensions for the Versius embodiment. Training used 832 x 480 video with actions aligned at 10 Hz. The teacher was fine-tuned first on 24 future frames for 2,000 iterations and then on a 72-frame horizon for 4,400 iterations, before distillation converted it into a block-causal student through causal teacher forcing and Self-Forcing with distribution-matching distillation. The released student generates video autoregressively with four denoising steps per latent frame, and a second, separately trained checkpoint uses two steps for faster inference at lower visual quality. Streaming runs through FlashDreams, NVIDIA’s streaming-inference library, with an adapted FadeMem module that consolidates the growing key/value cache during long rollouts.
Cosmos3-H-Dreams Training Data, Frame Rates And Licensing
Fine-tuning used proprietary CMR Surgical Versius clinical recordings drawn from 4,792 episodes totalling roughly 17 million frames at 60 Hz and 1080p, subsampled to 10 Hz, with surgeons teleoperating the robot and action labels derived automatically from robot telemetry. Sample selection kept only windows in which exactly two instruments were visible and engaged, with a left Cadiere grasper and a right monopolar hook as the canonical pairing. The training set comprised 64 recordings and about 526 unique minutes of video, with recordings split at episode level and about 2 percent held out for validation.
On performance, the model card reports 30.0 frames per second at 832 x 480 with four diffusion steps and 43.2 FPS with two steps on an RTX PRO 6000 Blackwell Workstation Edition, 20.2 and 31.2 FPS on the Max-Q edition, and 16.9 FPS on DGX Spark at a reduced 512 x 288 resolution. Real-time interactive serving was validated on a single RTX PRO 6000 and on DGX Spark, and the model also runs on a Hopper H100. The model is released under the NVIDIA Open Model License for commercial or non-commercial use, the companion GitHub repository is Apache-2.0, and the code supports replaying recorded actions to MP4 output as well as live control from a WebXR-compatible headset with two tracked controllers. NVIDIA also published a step-by-step recipe for teacher fine-tuning and Self-Forcing distillation so developers can adapt the system to other datasets and embodiments. The release follows NVIDIA’s wider open-model push for agents and PCs covered in NVIDIA’s Nemotron 3.5 Super VL and Bolt open models, and sits alongside its data-centre partnerships such as the Germany and NVIDIA industrial AI partnership.
Why It Matters
Cosmos3-H-Dreams matters because surgical robotics has a data problem that text and image AI never faced: policies cannot be trained or evaluated on real patients, and physical simulators struggle to reproduce how tissue stretches, smoke obscures the view and reflections shift during a procedure. A generative simulator trained on 4,792 real clinical episodes gives researchers a way to evaluate surgical robotics policies in simulation, explore teleoperated world models interactively and generate synthetic training data without touching a patient. NVIDIA is explicit about the limits: visual realism is a starting point for validation, not proof of biomechanical accuracy, and a plausible rollout that ignores control commands cannot serve as an interactive environment. Its stated research directions — modelling the full cholecystectomy procedure with task boundaries and success criteria, connecting generative and numerical simulation, evaluating with surgeons and developing autonomous surgical policies in a closed loop — show where the company thinks world models go next, in the same way its GPU and CUDA platform carried earlier AI waves from research into production.
Background
Cosmos is NVIDIA’s world foundation model family for physical AI, built to predict how environments evolve so robots and autonomous systems can be trained and tested in simulation. Cosmos3-H-Dreams succeeds Cosmos-H-Dreams, an earlier model built on Cosmos-Predict2.5 that targeted tabletop suturing on the dVRK research robot; the new release is the family’s step from laboratory rigs to clinical scenes recorded on CMR Surgical’s Versius system, which is used in real hospitals.
Source: Unite.AI reporting NVIDIA’s official Hugging Face announcement and model card (Oct 20, 2026)
0 Comments