AI Research Breakthroughs from NVIDIA Research (Hosted by Karoly of Two Minute Papers) | NVIDIA GTC YouTube video by NVIDIA Developer

Wasn't at GTC but a sneak peak of some of my internship work at NVIDIA was! (sim2real cable routing presented by my manager Yashraj)
www.youtube.com/watch?v=d5wm...

some points of comparisons could include e.g. manager based configs (mjlab/isaac lab) vs direct style (maniskill/robosuite/also isaac lab), wrappers vs no gym wrappers, features for simplifying parallelization, visualization tooling and more

would people be interested in a blog series on "simulation infrastructure" where I look at popular robotics sim tools (e.g. mjlab, isaac...) and explain the pros and cons of their design decisions. The series would cover what I believe to be key pillars of good, usable, sim code

I will be in SF from November 10-17 🌉

If you work on something interesting and want to meet up let me know! Or if there’s a fun event i’d love some invites 😃

I’m also looking to visit robotics companies (especially startups!), if you have time lmk if I can visit!

1/1 neurips submission accepted! Will share more details later, covers an often overlooked aspect of on policy RL training when scaling to large scale parallelized environments

Rumors going around are that NeurIPS PCs had to reject ~400 papers because of venue constraints that were in the accept pile.

Hard things like this happen but the PCs and conference should explicitly tell all the authors who had this happen to them. Or create something like accept w/o presentation.

my partner just hopped online, very talented with a great eye for design. She’s also behind all the beautiful graphics in the Lux AI Challenge that helped us gain 100s of competitors! She’s on the job market right now so DM her if you are looking for a designer/marketer/PM!

what’s your next blog going to be?

i’ve been also meaning to pick it up again but writers block lol

It works best if the object meshes you use for diff. rendering have a corresponding segmentation mask. If there are occlusions then the default option of using SAM2 to generate segmentations will perform a little worse. That being said even if the mask has some gaps the optimization can work

Big thanks to Linghao Chen (author of EasyHEC) for helping out. I simply made some examples and simplified some code for reproducibility/readability.

Opensourcing a tool to calibrate camera extrinsics painlessly in a minute, no checkerboards! It's based on EasyHEC, using diff. rendering to optimize extrinsics given object meshes+poses. Crazy that even a piece of paper works too.

Code:
github.com/StoneT2000/s...
(paper example in next post)

[Feature] Faster IK based controllers using levenberg-marqurdt and align pd ee pos controllers in gpu/cpu sims by StoneT2000 · Pull Request #1213 · haosulab/ManiSkill Cleaned up version of #1148 to close #955. Updates the original code to permit various other pd ee controller options with the new solver. LM solver is now the default. Users can modify the IK sol...

Big thanks to Jeremy Morgan (PhD @ USC) for helping massively accelerate the IK solvers in ManiSkill! Open source really helps grow this project (and i get to learn tons of new things like IK optimization)
github.com/haosulab/Man...

ManiSkill3 simple example, Mujoco Playground, and the DextrahRGB paper all do very similar things.

I still think transfer techniques are useful, i’d think of these sim2real attempts as just explorations in the large scale visual DR space with parallel rendering. Both could be used together probably

ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI Simulation has enabled unprecedented compute-scalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks and lack features crit...

see section D of maniskill3 paper: arxiv.org/abs/2410.00425

tldr: Green screen background, segment out robot and cube in sim and only render that. Large scale visual domain randomization (robot, cube textures, camera parameters)

very simple trick that is fairly reproducible for simple tasks

might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code

Combining sim to real with cheap hardware like the so-100 really is making cutting edge robot learning so accessible and its wonderful

That doesn’t seem to make sense, you don’t need a RGBD camera, phone works fine. Happy to address it more if details can be shared on github if needed

what bug did you encounter with OpenCVCameraConfig?

For papers that aren’t ready / i think should be rejected due to many weaknesses, i often struggle to put anything in strengths and tend to just say some general thing. I only then elaborate more detailed (i hope) in the weaknesses section

My latest post: The American DeepSeek Project

Build fully open models in the US in the next two years to enable a flourishing, global scientific AI ecosystem to balance China's surge in open-source and an alternative to building products ontop of leading closed models.
buff.ly/kvJQE3I

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works

Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

Excited to announce that I will be interning at @nvidia research this summer on robotics/embodied AI! I’ll be in seattle for the summer, let me know if you want to meet up and chat! 🦾

June 25 EEB 248 @ 3pm - Towards Embodiment Scaling Laws in Robot Locomotion by @BoAi0110

June 25 OHE 122 @ 11am - ImVR: Immersive VR Teleoperation System for General Purpose by Yulin Liu

Rest of my lab has various presentations at RSS2025, please check out their awesome work!

June 25 EEB 248 @ 10am - Hardware Optimization for In-Hand Rotation by K. Fay

~1 hour give or take maybe 20 mins (depends how close your camera is technically), and only 8gb of GPU memory!

someone even reproduced our sim2real tutorial with lerobot during the lerobot hackathon with a google colab gpu and a macbook for robot deployment

The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors.

At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle

I’ll be at #RSS2025 from June 21 to June 23!

I’ll be giving a presentation on ManiSkill3 on June 21, 5:30 PM

We will also have two live demo sessions, on June 21, 12:30-2:00PM and 6:30-8:00PM. Swing by to see live demos of zero shot RGB sim2real, cool sim demos, and VR teleop!

Foundations of Computer Vision The print version was published by

Our computer vision textbook is now available for free online here:
visionbook.mit.edu

We are working on adding some interactive components like search and (beta) integration with LLMs.

Hope this is useful and feel free to submit Github issues to help us improve the text!

Email so far ahead is interesting. Is this just substack emails?

the amount of cope on the other site is crazy