A New Way to Make Robots Understand Natural Language

Join this channel to get access to perks:
https://www.youtube.com/channel/UCQs0lwV6E4p7LQaGJ6fgy5Q/join

Here we take a look at natural language processing to control a Reachy Mini robot. We use Whisper for speech to text, then run the text through a decision model cascade. This determines the meaning of the prompt and selects an appropriate motion clip to play on the robot.

In the video, we are using a Jetson AGX Thor, but this can also run on Jetson AGX Orin and Jetson Orin NX using fully local models, such as Liquid AI d1-3B shown here. You can also use a hosted decision model, called a System One model, like Jev for this type of application.

For a Jetson Orin Nano, you will likely need to run a hosted model like Jev, as the decision models can be cause quite a bit of memory pressure.

We're using Whisper in the video for speech to text and wakeup word recognition.

Jetson AGX Thor Developer Kit: https://link.amazon/B05CCIhDS
Jetson AGX Orin Developer Kit: https://link.amazon/B04OUe0rv


00:00 Intro
01:04 Hardware
01:40 Prompt to Action 1
02:45 Prompt to Action 2
04:45 Prompt to Action 3
06:36 Unedited Sequences



As an Amazon Associate I earn from qualifying purchases.
Visit the JetsonHacks storefront on Amazon: https://www.amazon.com/shop/jetsonhacks

Visit the website at https://jetsonhacks.com
Sign up for the newsletter! https://newsletter.jetsonhacks.com
Github accounts: https://github.com/jetsonhacks
https://github.com/jetsonhacksnano
Twitter: http://twitter.com/jetsonhacks

Some of these links here are affiliate links. As an Amazon Associate I earn from qualifying purchases at no extra cost to you.
#NVIDIAJetson #Robotics #ReachyMini #EdgeAI #JetsonHacks #Python #NVIDIARobotics