1/ Real2Sim and Sim2Real are well-known paradigms in robotics🤖. Is there another paradigm where sim and the real world are more tightly interlinked? We introduce Real-IS-Sim: a framework featuring an always-in-the-loop, correctable simulator for behavior cloning policies.

Grateful to @NVIDIA for featuring our work on their Tech Blog: Physically Embodied Gaussians, an interpretable real time world model for #robotics using Warp and #GaussianSplatting. @krshnrana @nikoSuenderhauf @QUTRobotics.

🙏 Supported by @rai_inst .#NeuralRendering #PhysicsSim #Robotics #ComputerVision #GaussianSplatting.

🚀 Embodied Gaussians is now opensource! 🔥. 📂 GitHub: ✨ Key Features:. 1️⃣ Gaussian Splatting (GSplat) + Physics (Warp) 💥.2️⃣ Parallelizable environments! ⚡.3️⃣ Render virtual cameras in parallel 📸. 🔜 Coming Soon (hopefully):. 🌀 Deformables!

6/ Videos and paper are available at and

5/ This project was completed during my internship at @rai_inst (formerly Boston Dynamics AI). Special thanks to my team—Lingfeng Sun, @krshnrana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, and Laura Herlant for their support during this project.

4/🧠 With an always-in-the-loop sim acting as a mediator, policies can access privileged state—enabling training on any representation that can be derived from the simulator, including object poses and virtual renders. On PushT, a virtual gripper camera outperforms alternatives.

3/⚡ Real-Is-Sim enables seamless switching between real world deployment and offline evaluation - allowing rapid, parallel evaluation of imitation learning policies.

2/ 🔁The always-in-the-loop simulator ( is active during data collection, policy evaluation, and deployment—maintaining a real-time digital twin that unifies simulation and reality into a single, coherent interface for policy development.

5/ More videos and details are available at the project page ( and in the paper (. A special thanks to @krshnrana, Feras Dayoub, @nikoSuenderhauf, and @QUTRobotics for their support during this project.

4/ Particles collide with each other and are affected by external forces and kinematic constraints. The particles transport their linked Gaussians along with them. The system runs in realtime @ 30Hz with only 3 cameras. It works on both deformables and rigid objects.

3/ The representation is kept synchronized with reality by applying "visual forces" on the particles which are derived from the photometric error between the rendered representation and images from the workspace.

2/ We introduce a dual "Gaussian-Particle" representation which connects splattable 3D Gaussians (the visual state) to particles that can be acted upon by a physics simulator (the physical state). This fusion allows robots to both render and simulate the world.

1/ What is a good representation of the physical world for a robot🤖? We believe it is one that is: initialized easily, 3D, physically constrained, correctable with new observations (in realtime), and capable of being forward simulated (faster than realtime).

RT @krshnrana: 📌How can we scale imitation learning to long-horizon, multi-object tasks & generalise (spatial & intra-category) from only 1….