The world is moving fast and if you look around, robots have successfully entered our everyday life with many more expected in the not too distant future.
This rise in popularity and the broadening application of robots also affects customer expectations, particularly around usability and human machine interfaces (HMI). For robotics companies and manufacturers, that means prioritizing customers' preferred ways for interfacing with robots is more critical than ever.
Why building mobile interfaces for robots is increasingly important
We've all gotten used to using smartphones and tablets, and since each of us is a potential customer, installing apps on our preferred devices is more convenient and sustainable than receiving a new screen per robot.
For a robot manufacturer, that means leveraging a bring-your-own-device approach lowers the cost of a robotic system for the end customer.
But why don't more robot manufacturers do this? One big reason is that popular robotics stacks such as ROS (Robot Operating System) don’t have mobile and cloud enabled connectivity built in.
That makes building an HMI to a mobile or web application time-consuming and complex, especially given the various technologies and knowledge areas already required to create HMIs. And, the fact that there are multiple platforms and devices in use by consumers makes this challenge even more complex.
When building an HMI, the aim is to get fast iteration cycles while serving both major mobile platforms (iOS and Android), inclusive of varied screen sizes, and utilizing engineers with existing mobile development experience.
These factors affect a business’s ability to grow its business while maintaining healthy margins. Critically, it is important to be on one common platform that can accomplish this for future robots, existing robots in the field, and robots in development.
With Viam, you can start the mobile journey today on existing robots—whether ROS-based, non-ROS-based (proprietary or other) and combinations thereof—including robots already in use by customers.
How we make building mobile interfaces for robots easier
I'll start by explaining how the Viam robotics platform bridges the gap between the hardware and software layers on a robot, helping you build innovative human machine interfaces faster while reducing time wasted.
Robotics applications are often developed in C++ or Python. Given the nature of those languages, developers often need to design, develop, and test an API layer to which you can connect from your client/controller application or HMI.
Viam is built as a distributed system, using protocols and standardized APIs for easy integration across wide varieties of platforms. All communications between components on the robot, including across the internet, use gRPC / protocol buffers to streamline integration.
gRPC and protocol buffers dramatically simplify cross-platform communication and allow Viam to efficiently provide a wide variety of different SDKs for all the popular programming languages.
What you can build with the Viam Flutter SDK
In this blog post, I'm focusing more on the combination of mobile devices such as Android or iOS with robots and so will go into more detail about using the Viam Flutter SDK.
Flutter allows you to build an application once while being able to run it across smartphones and tablets using iOS or Android to Linux, Mac, and Windows PCs. It was originally developed by Google and has become popular for building a wide variety of mobile and cross-platform apps.
An example Flutter app for remotely controlling a robot
I am far from being an experienced Flutter developer, and the first thing I tried to do was remote control because that is something I think is a really common requirement and fun to do from my smartphone. I was really pleasantly surprised by how easy this was, particularly since I’m new to Flutter.
(If you are new to Flutter like me, I highly recommend looking into one of the many available Get Started resources for Flutter. These resources guide you all the way through from setting up your development environment to writing your first application, and help you grasp the concepts and structures with ease.)
When you take a look at the structure of my little sample application, you'll quickly figure out that it is much in line with the Get Started resource linked above, and my guess is that you'll feel comfortable with the code very quickly.
The app is kept simple for introductory purposes and contains a login widget as shown on the screenshot to the left and a remote control widget on the right.
How the Viam Flutter SDK makes it easy to connect to your robot
To get started with the Viam Flutter SDK, all you have to do is to create a new Flutter application and add the Viam Flutter SDK.
Through the SDK, you can then easily instantiate a robot object and establish a connection to the physical robot from pretty much wherever you are.
Viam uses gRPC over WebRTC. Why WebRTC, you might ask? Well, establishing connectivity across network boundaries is difficult and potentially requires reconfiguration of firewalls and routers, which is very cumbersome if ever possible.
Using WebRTC allows us to leverage signaling servers, which mediate connection attempts between devices spread across multiple networks, such as a robot connected to a private Wi-Fi network and a mobile phone connected through GSM. It's basically the same approach that modern collaboration tools such as Slack, Google Meet, Microsoft Teams, and many more use to establish direct connectivity between video conferencing clients.
Using the Viam APIs to implement robotics use cases
Once your mobile Flutter app is connected to the robot, you get idiomatic access to all of the APIs, and from there you can start implementing your specific use cases with ease—such as sending commands or retrieving telemetry data and status information or video streams from cameras.
Obviously you can use all of these APIs and design/build your own widgets for very different scenarios. If you want to move fast however, the Viam Flutter SDK also provides out-of-the-box widgets for different components such as sensors, the board, and even a remote control widget, which is what I used to create something usable within hours. It's also how you can create a minimum viable product first, and then figure out if this is what your customers want before putting more effort into it.
Can you use the Viam Flutter SDK with a ROS stack? Yes!
One of the most frequently asked questions is if this approach is also feasible with ROS 1 or ROS 2 powered robots—and we are happy to say, of course! Many companies have put tremendous amount of work into their ROS stack and don’t plan to replace it anytime soon for good reasons.
As illustrated in the architecture diagram above, you can easily integrate ROS with the Viam RDK through the ROS integration described here for ROS 1 (an integration for ROS 2 will follow shortly).
Once your ROS stack is connected to the Viam RDK, you can then use all our SDKs such as the Flutter SDK described in this article.
Additionally you have the option of using Viam's built-in cloud services for fleet management, data collection, ML model training, remote control, and more.
Try it out: Viam & Flutter
Get hands-on with creating a mobile or cross-platform Flutter app using Viam by building the Flutter app we described in this article! Just create a Viam account and use our provided Flutter demo app code to get started.
For the mobile app lovers, we recommend also taking a look at the brand new Viam mobile app—now available in the Android and Apple app stores.
You can also join the Viam Community Discord to ask our team questions and discussing mobile interfaces for robots with others in the robotics space.
Want to learn more about using Viam with ROS to build robots faster and more flexibly? Check out these resources:
- Get data from your ROS robot into the cloud in just 5 minutes using Viam
- Integrate Viam with a ROS 1 enabled robot to collect data and train an machine learning model
