I should clarify that I don’t mean current events but rather projects, ideas, things people are building, new curiosities.

Where are you getting your news from these days?

Are there any good robot agent papers beyond DynaMem I should catch up on? I’d like to reimplement the RoBart VLM agent.

Thoughts on AgileX NERO vs. YAM? One is 7dof and one is 6dof. Specifically, for UMI type data collection. I would imagine 7dof performs better but AgileX’s resources for NERO are pretty meager (not even a MuJoCo model).

Clausewitz Code.

Would anyone be interested in purchasing a barely used AgileX Piper?

Signs of life! The YAM arm model is stable enough to grip cubes under real-time motion control via an iPhone. I think the "swimminess" here is partly due to MuJoCo's terrible render latency and partly due to the modeled motors. iPhone -> IK pipeline needs work, too.

I think the problem has to do with the arm's "jerkiness". Starting any motion as a result of actuation creates an impulse that shakes the arm. Visible here when I divide the the sweep into several equally spaced points and set the motor position targets for each one.

AgileX Piper with Robotiq Gripper Cube Grasp - Cube slips out of gripper during arm motion AgileX Piper with Robotiq Gripper Cube Grasp - Cube slips out of gripper during arm motion - 2f85.xml

Gist of my MuJoCo XMLs: gist.github.com/trzy/0b5d22e...

Tried a pre-modeled Robotiq gripper in MuJoCo this week thanks to a helpful suggestion here but still can't grip. Files linked in comments. Is there a different simulator I should be using? Surely this task is possible in real-time, in simulation? Happy to share code if anyone wants to take a look:)

Final results of Robotiq: not working :( The cube just comes loose way too quickly. I didn't tweak any parameters and am using a 500Hz timestep. Friction of 0.5 on the cube. Thinking of trying IsaacSim but I think its performance will be a problem.

I tried the Robotiq gripper. Getting better results! However, it mysteriously messes up my IK. This hard-coded trajectory works for both grippers but the arm won't work outside this plane of operation, so I can't drive it with my iPhone. Will need to debug and find a better IK solver.

This is amazing and I hope we see a PC port. If I had time I'd do it myself. Would be especially nice to get track editors and online multiplayer.

Getting around to it this weekend finally. I will try this Robotiq gripper and see if it works! The suggestions in the documentation didn't work. I think the core problem is the oscillation/vibration, but playing with gains, forces, solvers, didn't help.

I feel like I've tried everything. Reducing gain, reducing force, enabling multiccd as per the documentation's suggestions, implicit and implicitfast. Nothing helps. Super frustrating. It's just two cubes gripping a third cube lol. I need to look at how other grippers are modeled.

Thanks! I'm going to give this a try tomorrow!

They're defined as below. I tried setting the range so that it closes to just a little less than the width of the cube but it doesn't seem to make any difference.

I tried some of these but nothing quite seems to work. I think the arm itself is moving too much? Reducing the max force on the grippers helps a bit but it’s super unstable.

It’s definitely perceptible! I didn’t measure it but you can sort of see it by looking at when the direction key indicator lights up in the GUI vs motion occurs in the stream.

Any MuJoCo experts out there able to help me understand why this keeps happening? The yellow boxes are the actual collision meshes. I am moving the arm in real-time using an iPhone. I just can't maintain a stable grip. Can share more details!

GitHub - trzy/RoBart at webrtc-teleop An iPhone-based LLM-controlled autonomous robot. Contribute to trzy/RoBart development by creating an account on GitHub.

Source code here: github.com/trzy/RoBart/...

RoBart now features WebRTC teleop. Native async Swift on the iOS side and a browser-based control app to run on PCs. Source code available (see thread).

Great minds think alike!

Gripper teleoperation using optical detection of fiducial markers.

RoBart, my iPhone-based and LLM-powered autonomous robot, is now open source.

github.com/trzy/RoBart

Health care is a space I’m interested in so I put together this demo of RoBart performing contactless vital sign measurement and a cognitive assessment. Would love to chat with folks in the health care space. 👋👨‍⚕️👩‍⚕️

Using LLMs and ARKit for robot vision: How RoBart sees YouTube video by BartTrzynadlowski

How RoBart sees. How to use an LLM to identify specific points in an image and ARKit to create those points.

youtube.com/shorts/68Blq...

Detecting UMI gripper state visually with mounted iPhone.

I’m building stuff :)

Nevada. 🌕👽✨

Nevada. 🌕👽✨