j5 – Augmented Reality Robot
The goal of this project was to create an immersive robotic tele-presence experience. J5 uses a combination of systems to achieve this. The robot streams stereoscopic video over the wireless network into a Google Cardboard iOS app. The robot can be driven via socket.io communication over a web server. The camera also pans and tilts using the device orientation angles sent via a socket.io client to the web server. To control the robot, the user opens the iOS app and accompanying web app. The iOS app is placed inside a Google Cardboard unit with the stereoscopic video stream playing in each eye. The user can now visualize the robot’s perspective. By looking up, the robot cameras look up. By looking to the left and/or right, the robot cameras pan to the left and right. Using the web app, the user can control the robot driving controls by holding down the ‘Forward’, ‘Backward’, ‘Left’ and ‘Right’ buttons.
Two raspberry pi’s host video servers while only one also acts as control server for the device orientation and driving commands. The control server is a node.js server that uses socket.io to listen for socket messages. Once the messages are received, commands are issues over serial to the Arduino or command line. The Arduino handles incoming integers over serial and outputs commands to the physical outputs that control DC motors. Command line commands control the pan/tilt servos directly from the Raspberry Pi.