Data dive: navigating sensor data for actionable insights

The job of data analysts sifting through massive volumes of sensor data to find insights is hard enough, let alone while dealing with specialized tools and daunting processes to access that data. They often need help with various issues like data overload, the complexity of data sources and their integration, and the hurdles around real-time analysis. 

Let's look at the issues in depth:

• Data overload: As data volumes grow exponentially, analysts often find themselves overwhelmed by the sheer amount of information they need to process. This overload can make it challenging to extract meaningful insights efficiently.
• Complexity of data sources: Data comes in various formats and from disparate sources, including structured and unstructured data, sensor data, image data from cameras, and data from IoT devices. Integrating and analyzing these diverse data sources can be complex and time-consuming.
• Hurdles around real-time data analysis: In today's fast-paced business environment, there is a demand for real-time insights to make timely decisions. However, performing real-time data analysis requires robust infrastructure and advanced analytics capabilities.

Fortunately, Viam's data management platform empowers analysts with flexible query tools to unlock valuable insights. 

Data in action: laundry machine use case

Let’s give a real-life use case and steps to fulfill the project: you have a laundry machine and want to do predictive analysis and maintenance. You want to know when the machine is operating correctly, when it needs to be repaired, or more about the machine's performance in general. Using Viam, you can capture real-time and historical data and forecast maintenance needs. 

In addition, you may want to know when your washing session is done and get related alerts. You may have a camera checking the amount of time left for the session to be over or an accelerometer/gyro to assess the changes in the machine's movement. We can use the gyroscope as an example. 

The first step would be wiring the sensor, then creating a machine in the Viam app, and adding the sensor. 

This is how the configuration of the gyro would look like in the app: 

Now, it’s time to add the data management service to your machine and configure the data capture and cloud sync. 

After syncing tabular data to the Viam app, you can view the live sensor data through the control panel and, under the Data tab, the Sensors sub-tab. What is sensor data? Here, from the sensor, you get information about angular velocity, linear acceleration, and temperature in Celsius. 

Now that we have done sensor data collection and observed it in the app, it’s time to perform a query against that data using SQL or MQL. 

Query for laundry machine use case

With Viam, you can:

• Query tabular data in the Viam app: You can run SQL or MQL queries against your synced tabular data from the Query subtab under the Data tab in the Viam app.
• Query tabular data directly from a compatible client: You can directly query tabular data from an MQL-compatible client, such as Mongosh.

Let’s query tabular data in the Viam app. Navigate to the Query tab and select the query mode you prefer. 

Here are two examples of SQL and MQL queries. By running these, you get all of the readings from the washer-movement component (the movement sensor) we configured above in the app. The query limits the readings to the last five. 

SQL command with the corresponding response in the app: 

SELECT * FROM readings WHERE component_name='washer-movement' LIMIT 5

‍

MQL command with the corresponding response in the app:‍

[{"$match":{"component_name":"washer-movement"}},{"$limit":5}]

In the SELECT section, you can specify whether you want to see only the data itself and not when it is requested, received, the parameters, org information, etc. 

In SQL:‍

SELECT data FROM readings WHERE component_name='washer-movement' LIMIT 1

In MQL: 

[{"$match":{"component_name":"washer-movement"}}, {"$project": { "data.readings": 1}},{"$limit":1}] 

You can get more and more granular if you keep adding details about which query you want to get. For example, if you don’t care about angular_velocity and linear_ acceleration and only want to get the latest temperature data, you can run the command:

In SQL: 

SELECT count(*) as temperature_celsius FROM readings WHERE component_name='washer-movement' LIMIT 1

Or in MQL: 
‍

[{"$match":{"component_name":"washer-movement"}}, {"$project": { "data.readings.temperature_celsius": 1}},{"$limit":1}]

If you want to query the data directly from a compatible client, you can configure the query through the MongoDB Atlas Data Federation instance, where your machine’s synced data is stored. Once you have synced data to the Viam app and configured a database user, you can directly query that data from an MQL-compatible database client, such as the mongosh shell, MongoDB Compass, or one of many third-party tools. 

After query: What now? How does Viam help?

After querying, you can engage in several subsequent activities to derive insights and drive decision-making, such as: 

• Cleaning and preprocessing the data to address any inconsistencies and errors. In the app, for example, you can use the query language to filter out incorrect or invalid data. You can also perform aggregations on the data directly using MQL or SQL (for example, calculating an average).
• Doing exploratory data analysis, such as visualizing the data and identifying patterns, trends, outliers, and relationships between variables, to gain a deeper understanding. For instance, you can use a MongoDB plugin for Grafana to produce visualizations and see anomalies.
• Doing statistical analysis to quantify relationships within the data, test hypotheses, and validate assumptions.
• Building predictive models to forecast future trends, identify anomalies, or classify data into categories.
• Generating insights for implications and recommendations based on the analyzed data. 

Let’s remember the three issues data analysts face: data overload, complexity of data sources, and hurdles around real-time data analysis. As we’ve seen through the blog post, the Viam platform and data solutions offer a comprehensive solution tailored to address these:

• Simplified data querying: Viam provides a user-friendly interface for querying sensor data using SQL or MQL. By streamlining the querying process, Viam empowers analysts to extract insights efficiently, overcoming the hurdle of data overload.
• Integration of diverse data sources: Viam seamlessly integrates with various data sources, such as data from sensors and IoT devices, and offers a unified API for accessing them. This integration simplifies the process of analyzing diverse datasets, overcoming the hurdle of complex data sources.
• Real-time data analysis: Viam enables analysts to access and explore data in near real-time, as it is synced from live machines. This is important since data analysts leverage historical and real-time data to forecast equipment failures, recommend proactive maintenance schedules, minimize downtime, and save energy and cost. 

Now that you've explored data analysts' challenges and how Viam addresses these hurdles, it's time to take action. 

To apply the concepts from this blog:

• You can create an air quality monitor with a fleet of sensors, where you learn how to use the Viam TypeScript SDK to query sensor data and create a custom dashboard.
• You can check out how to boost office energy efficiency through smart HVAC monitoring and automation, where you learn how to write scripts to query the data service for temperature readings using MQL.
• You can visualize data with Grafana or Tableau to build interactive dashboards that let users dig into specific data points, apply filters, and see real-time updates.

Don't let complexity hold you back—join the Viam community today and revolutionize your approach to data analysis by streamlining your data collection and retrieval pipelines! 

‍

Technical content review by: Nick Hehr,Vijay Vuyyuru and Tahiya Salam