Earlier this year at Viam, we launched our TypeScript SDK. I’m a big fan of working in a React, so I was excited to try creating a basic web app that displays data from a live connection with a real robot in the web programming stack I’m most familiar with.
Specifically, I wanted to try extending our React / TypeScript teleoperation repo demo to integrate a 3rd-party data visualization library built on top of both Tailwind CSS and Tremor, designing the user interface specifically as a dashboard.
Below, I will share an overview of what I built, the development stack used, and details on core parts of the implementation. All of the code is available here on GitHub: @viam-labs/ts-dash.
Project overview
My project consisted of two primary tasks: getting the data we’re interested in, and then displaying that data in an effective format.
Below, I’ll share an overview of how I:
- Retrieved the data for the application by connecting to the robot
- Created clients to relevant components
- Queried the state or sensor readings from those clients
- Visualized the returned data
Project setup
This project uses the following tools:
- React (front-end development framework)
- Tailwind CSS (CSS styling library)
- Viam TypeScript SDK (TypeScript interface for interacting with robots running viam-server)
- Zustand (React state management)
- Tremor (React data visualization library)
- Pigeon Maps (React mapping library)
We’re not using any backend framework (such as Express), since for now I plan to just operate this project locally on my own computer.
In order to install the above libraries and frameworks, I did the following:
- I used the Create React App doc instructions to set up basic React app scaffolding with TypeScript support, and added Tailwind CSS to the project by following their “Install Tailwind CSS with Create React App” guide.
- I installed the Viam Typescript SDK, Zustand, Tremor, and Pigeon Maps with the following commands:
Robot connection
After creating the project scaffolding and installing required libraries, we needed to implement a means of connecting to and querying data from our robot. To do so, we created two files: client.ts and state.ts.
The client.ts file directly uses the Viam TypeScript SDK to set up connection with our robot and acts as a factory for robot component and sensor clients.
The state.ts file manages the state of these connections within our React app and exposes them globally using Zustand (a popular React state management library), letting us connect or disconnect from the robot, get sensor readings, handle media streams, and keep track of the connection status.
You can see the full source code in the repository, but in short we created a getRobotClient function in client.ts that returned a RobotClient instance created via the Viam TypeScript SDK’s createRobotClient method. We then called this getRobotClient function from state.ts, and stored the returned RobotClient in our Zustand store, which made it accessible throughout the rest of our app.
We followed this same pattern for sensors. For example, to query GPS sensor data, we created a getGPSMovementSensorClient function in client.ts that returned a MovementSensorClient instance for the specified sensor name passed-in, and the state.ts file stores and exposes this client for use.
Robot data queries
Once we established clients to our robot components in state.ts, we could easily access them throughout the React app and access sensor readings by calling the appropriate method on the sensor client.
For example, to query live GPS data from my robot, I called getPosition on the gpsMovementSensorClient that we instantiated and exposed in client.ts and state.ts.
We could access camera feeds by calling the useStream function we defined in state.ts and passing the returned MediaStream to the VideoStream component.
Data display
Once we had access to data from the robot, we needed to create appropriate components for displaying that type of data. I set this project up with Tremor, as it provides great out of the box components for creating dashboard interfaces (even though we barely use Tremor’s charting abilities in this current project, it’s still useful for basic UI components such as <Card/>, <Title />, and <Metric />, and we’re well situated for future extensibility to support further data visualization).
Camera Feeds
Because Viam’s TypeScript SDK enables us to retrieve MediaStreams directly from the robot, we could pass these streams into native HTML <video/> elements in order to display our robot’s camera feeds. The VideoStream component encapsulates this logic and accepts the MediaStream as a prop.
GPS Data
In order to render robot GPS data, I used the pigeon-maps library to render our longitude and latitude on an interactive map. The GPSLocation component encapsulates this logic and accepts the robots’ GPS position as props.
Working with ROS
Because you can integrate a ROS running robot with Viam, this approach for building a custom dashboard can be applicable for any ROS robot.
In about 15 minutes, I forked the starter TypeScript dashboard we just created and configured it to work with the Yahboom Transbot ROS robot we integrated with Viam in a previous blog post. As you can see, we accessed both a live camera feed and IMU sensor readings via the Viam TypeScript SDK on a Yahboom Transbot running ROS:
Accessing sensor readings on this ROS robot is the same as any other sensor component on Viam. For example, with this IMU sensor component on the Yahboom ROS Transbot, we call the `getAngularVelocity` method available on MovementSensorClients:
You can see the full code for this Yahboom Transbot extended version here: yahboom-tsdash.
Conclusion
A goal of Viam is to make it easier for software engineers to build in the world of robotics. One of the ways we do that is by creating and maintaining APIs in modern, popular languages such as TypeScript. An exciting aspect of working on this project has been that this project is a great example of the value of creating more bridges between the world of robotics and the world of software.
Because Viam exposes my robot in TypeScript, it’s easier to leverage the same powerful tools I use for building web apps that are available in the JavaScript open source ecosystem to build tooling for robots.
All the code discussed in this blog post is available on our Viam Labs GitHub here:
TypeScript dashboard: https://github.com/viam-labs/tsdash
Yahboom rover fork: https://github.com/viam-labs/yahboom-tsdash/tree/main/src
