After finishing my papers for my PhD, I spent some time exploring new directions. I ended up working on Diffusion Language Models with @haoyuhe.bsky.social (he made it work 🚀), @yongcao.bsky.social, @andreasgeiger.bsky.social.

I learned a lot of new things and I am very excited about the results. 🥳

GitHub - autonomousvision/CaRL: [ArXiv 2025] CaRL: Learning Scalable Planning Policies with Simple Rewards [ArXiv 2025] CaRL: Learning Scalable Planning Policies with Simple Rewards - autonomousvision/CaRL

We have released the code for our work, CaRL: Learning Scalable Planning Policies with Simple Rewards.

The repository contains the first public code base for training RL agents with the CARLA leaderboard 2.0 and nuPlan.

github.com/autonomousvi...

📢Excited to present our poster "SimLingo" tomorrow at #CVPR2025. Drop by to talk about vision-language-action models, language-action grounding, or anything else :)

📍Saturday, 10:30 - 12:30 Poster #130

Project page: www.katrinrenz.de/simlingo/

Your personalized CVPR 25 @cvprconference.bsky.social conference programs are now available for you!
www.scholar-inbox.com/conference/c...

🚗 Pseudo-simulation combines the efficiency of open-loop and robustness of closed-loop evaluation. It uses real data + 3D Gaussian Splatting synthetic views to assess error recovery, achieving strong correlation with closed-loop simulations while requiring 6x less compute. arxiv.org/abs/2506.04218

📢 We have a PR[AI]RIE PhD position opening at Inria Paris co-advised with R. de Charette & @tuanhungvu.bsky.social
[please distribute]
💡Topic: Physics-Grounded Vision Foundation Models
⏳Application deadline: 20 May 2025
🗓️ Start date: Fall 2025
📝Detailed description: linked below

Hi Sebastian, could you also add me?:)

Thanks to my great collaborators: Long Chen, Elahe Arani and Oleg Sinavski

And thanks to Wayve for the great time during my internship and all the support.

[CVPR25, Spotl.] SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment YouTube video by Katrin Renz

Check out the paper or full video for more details.

Full Video: www.youtube.com/watch?v=Mpbn...

⛳️We introduce a DREAMING flag with which the model can differentiate between driving mode, where only safe instructions are executed, and dreaming mode, where the actions for all instructions are predicted.

💭Action Dreaming: A safe way to test Language-Action alignment. We test not only expert behaviour but a wide variety of possible actions (e.g., speed changes, driving towards a specific object, lane change manoeuvres).

🫱🏻‍🫲🏽 Language-Action Alignment: On normal driving datasets, the action can often be inferred from the visual cue alone. Our new dataset includes multiple different actions for each sample, together with the language instruction. This forces the model to listen to the instruction.

🥇State-of-the-art: SimLingo is the first VLA model on the CARLA Leaderboard, achieving state-of-the-art driving performance on multiple benchmarks.

📣 Excited to share our #CVPR2025 Spotlight paper and my internship project @wayve: SimLingo.
A Vision-Language-Action (VLA) model that achieves state-of-the-art driving performance with language capabilities.

Code: github.com/RenzKa/simli...
Paper: arxiv.org/abs/2503.09594

📢 New paper CVPR 25!
Can meshes capture fuzzy geometry? Volumetric Surfaces uses adaptive textured shells to model hair, fur without the splatting / volume overhead. It’s fast, looks great, and runs in real time even on budget phones.
🔗 autonomousvision.github.io/volsurfs/
📄 arxiv.org/pdf/2409.02482

In my first research project I was super excited about getting any stars on GitHub. Now having a project with 1k stars feels unreal🤯 wouldn’t have been possible without the tremendous effort of @chonghaosima.bsky.social during the main project and afterwards with the challenge 🙏🏼

🆕 The CARLA Route Generator is a new Python application that provides a GUI for creating and editing routes, as well as defining scenarios within the CARLA simulator. It can also be used in conjunction with CARLA Leaderboard 2.0!
github.com/autonomousvi...

Come join us!

We have just released a new tool to create custom routes and insert scenarios for the CARLA Leaderboard 2.0. The tool was written by our great research assistant Jens. 🥳

Github: github.com/autonomousvi...

#CARLA #AutonomousDriving

LeRobot goes to driving school: World’s largest open-source self-driving dataset We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Learning to Drive (L2D): the most exciting dataset release of the year by @hf.co & @yaak-ai.bsky.social
- 5K hours of driving data from 3 cameras
- lots of other synchronized data: GPU, IMU, CAN, actions, task descriptions
- 90TB of data
- LeRobot data formatting
huggingface.co/blog/lerobot...