Using AI to reduce home energy usage

Last year, we installed solar at our home, hoping to greatly reduce our carbon footprint and reliance on fossil fuels. We also added a level 2 charger for our plug-in hybrid car and a heat-pump hot water heater large enough to service the 4 people in our household (some of whom choose to take showers longer than I thought were humanly possible).

Can we optimize our home energy usage?

Overall, we are happy with the results—we are on track this year to likely break even on our consumption versus production, and we have an app that makes it possible to monitor our home’s energy at any time. However, one thing I’ve noticed is that our overall electricity consumption has gone up. Are we introducing bad habits because we feel that we have “free electricity”? What if we want to offset more of our carbon footprint by adding heat pumps to heat our home—will our solar production even come close? With these questions in mind, I decided I needed to collect more data to determine how we might optimize our home energy usage.

I’ve noticed that there seems to be a clear correlation between how many people are home and how much energy is being used, but I want to back that observation with data. To do so, I decided to use AI to determine when people are leaving or returning home to track the current count of people in the household.

A simple and inexpensive hardware setup

To do this inexpensively, I acquired an ESP32-WROVER microcontroller with a camera, which can be purchased for less than $10. To collect images remotely, I installed viam-micro-server, then placed the camera facing the door of my house. I was able to start collecting images with the Viam app, and also did the same with the Viam Python SDK.

It was getting dark and the images were low resolution, but as a human I could tell if the images contained someone going in or out. 

Would AI be up to the task?

Deploying AI to help

In order to run AI against images collected from the ESP32, I decided to use viam-server running elsewhere to easily plug in ML models and other integrations from the Viam Registry. I could have run this on my personal laptop, but wanting to make sure the setup would always be up and running, I decided to run it on a virtual machine in the cloud instead. The Viam platform automatically provides a secured and authenticated network connection, so leveraging the ESP32 from the cloud would not require a complex security setup or custom network stack.

I deployed a $4/month Droplet from DigitalOcean, then set up a new Viam machine and followed the instructions to install viam-server on the Droplet. I linked the ESP32 with a secure connection to it as a “remote part”, which allows me to interface with both the ESP32 and any resources managed by viam-server in the cloud through the same secured SDK interface.

Next, I set up an ML model service and a computer vision service, enabling an ML model to be used with images from our configured ESP32 camera.

While popular ML models like the EfficientDet detector (trained on the COCO dataset) can detect people, I was not able to find a pre-trained model that would tell me if someone was coming in the door or going out the door. I considered using Viam to train my own custom model, but I knew that might require a good deal of training data to be collected and labeled. I realized that I could combine both the EffcientDet detector (which was doing a good job of detecting when a person is seen) with vision language model capabilities from ChatGPT, which could in fact identify when a person was coming or going (even in poor lighting conditions). This combination would allow me to call ChatGPT only after a person is detected, avoiding excessive charges from it.

Leveraging the ChatGPT vision module along with images from the ESP32 and Viam vision service boils down to a few lines of code using the Viam Python SDK:

machine = await connect()

detector = VisionClient.from_robot(machine, "coco-detector")
chatgpt_vision = VisionClient.from_robot(machine, "gpt-vision")
cam = Camera.from_robot(machine, "remote-1:cam")
    
img = await cam.get_image()

detections = await detector.get_detections(img)
for d in detections:
	if d.confidence > 50 and d.class_name = "Person":
		response = chatgpt.get_classifications(img, 1, extra={"question": "if there is a person coming in reply with the word 'in'. If there is a person going out reply with the word 'out'. Otherwise reply with the word 'none'"})

Great! I can now use this configuration and program logic to keep track of how many people are currently in the house.

Correlating AI findings with energy usage data

Next, I want to correlate the people count with the current energy usage in my house. I have a SolarEdge inverter, allowing me to get real-time readings with the SolarEdge Monitoring API, which can be used in our Viam project by installing the solaredge-monitor module from the registry.

While not everyone has a Solaredge inverter, there are a number of solutions that allow you to monitor your energy usage including Kasa, Emporia, and more. Some have integrations already on the Viam Registry, or you can create a new module on the registry for your own and others' usage.

Now that I have access to data around energy usage and home occupancy, I can start collecting that data and automatically storing it securely in the cloud with Viam’s Data Management. From there, I can run queries on that data, import it into other tools, or run custom ML training scripts on it for deeper insights.

Early insights and next steps

I plan to spend more time adding more data collection points and automation for a more in-depth analysis. For example, I could integrate sensors to determine where lights or heat are being left on with nobody present. My hot water heater also has an API—perhaps I can correlate those long showers with our energy usage! But in the meantime, I exported some data I collected on the first day in this format:

[
{"datetime": "2024-09-25 00:00:00", "avg_usage_kw": 0.55, "people_home": 4}
{"datetime": "2024-09-25 01:00:00", "avg_usage_kw": 0.35, "people_home": 4}
{"datetime": "2024-09-25 02:00:00", "avg_usage_kw": 0.35, "people_home": 4}
{"datetime": "2024-09-25 03:00:00", "avg_usage_kw": 0.32, "people_home": 4}
{"datetime": "2024-09-25 04:00:00", "avg_usage_kw": 0.31, "people_home": 4}
{"datetime": "2024-09-25 05:00:00", "avg_usage_kw": 0.35, "people_home": 4}
{"datetime": "2024-09-25 06:00:00", "avg_usage_kw": 1.75, "people_home": 4}
{"datetime": "2024-09-25 07:00:00", "avg_usage_kw": 1.85, "people_home": 4}
{"datetime": "2024-09-25 08:00:00", "avg_usage_kw": 0.75, "people_home": 2}
{"datetime": "2024-09-25 09:00:00", "avg_usage_kw": 1.75, "people_home": 2}
{"datetime": "2024-09-25 10:00:00", "avg_usage_kw": 1.75, "people_home": 2}
{"datetime": "2024-09-25 11:00:00", "avg_usage_kw": 1.05, "people_home": 2}
{"datetime": "2024-09-25 12:00:00", "avg_usage_kw": 1.75, "people_home": 2}
{"datetime": "2024-09-25 13:00:00", "avg_usage_kw": 1.25, "people_home": 2}
{"datetime": "2024-09-25 14:00:00", "avg_usage_kw": 0.75, "people_home": 2}
{"datetime": "2024-09-25 15:00:00", "avg_usage_kw": 0.75, "people_home": 2}
{"datetime": "2024-09-25 16:00:00", "avg_usage_kw": 2.75, "people_home": 3}
{"datetime": "2024-09-25 17:00:00", "avg_usage_kw": 3.75, "people_home": 4}
{"datetime": "2024-09-25 18:00:00", "avg_usage_kw": 2.75, "people_home": 4}
{"datetime": "2024-09-25 19:00:00", "avg_usage_kw": 2.75, "people_home": 4}
{"datetime": "2024-09-25 20:00:00", "avg_usage_kw": 2.05, "people_home": 4}
{"datetime": "2024-09-25 21:00:00", "avg_usage_kw": 1.75, "people_home": 4}
{"datetime": "2024-09-25 22:00:00", "avg_usage_kw": 1.75, "people_home": 4}
{"datetime": "2024-09-25 23:00:00", "avg_usage_kw": 1.15, "people_home": 4}
]

I then asked ChatGPT for some insights with the prompt: “This is energy usage in a home for a given day, what insights can you tell me?”. Part of the summary it returned:

From the energy usage data for September 25, 2024, several insights can be drawn:

Key Time Insights:
	6:00 AM to 7:00 AM: Significant energy increase, possibly due to morning routines.
  
	5:00 PM: Highest energy usage of the day (3.75 kW) when everyone is home,likely related to evening activities.
  
	Overnight: Consistent low energy usage, with minimal fluctuations, suggesting stable baseline consumption.
  
This data could help in identifying periods where energy-saving measures can be applied, such as the use of energy-efficient appliances or adjusting routines to shift some energy consumption away from peak hours.

I am constantly amazed at how AI is rapidly developing and being applied to assist us in real life. Having accessible tools that allow us to integrate data from hardware, sensors, external APIs, and beyond and then use AI to help us interpret that data opens up so many possibilities for innovation. How will you leverage these newer technologies to improve your lifestyle while also reducing your impact on the environment?

Get started using the Viam platform today for free, and let us know what you build!