Picture the scene: you're at a Halloween party, surrounded by friends and family. You put hours of effort into your Dracula costume over the past few weeks and you look like you belong in a Transylvanian castle rather than NYC. Your friend Joe on the other hand decided not to wear a costume…not cool. Makes you wonder: what if there was a smart machine that could see who was dressed up enough to get Halloween candy?
That’s why we created the Treat-or-Trickster Smart Machine. Press its button and receive jelly beans—delightfully flavored for those in costume, and mischievously flavored for those without. Watch from afar as Joe gets a surprise taste of soap or a rotten egg!
How we built this
The Halloween spirit model
To determine whether or not someone was worthy of the better-flavored jelly beans, we first needed to create a machine learning model, specifically an image classification model, that identified individuals with and without costumes.
Here’s how we built this model:
- Scraping Pinterest for images surrounding costumes and downloading them to our computer.
- Using the directory function in Viam’s Data Management Service to easily upload these images into the Data tab.
- Training these images directly within the app using Viam’s ML Model Service. Halloween-based images were given the “halloween” tag. Those without this tag were categorized as “not in the halloween spirit.”
- Creating a model around these tags, configuring a transform camera for visualization, and deploying the model onto our smart machine.
With Viam’s built-in components and services, this entire process was extremely straightforward, taking about 30 minutes total.
The candy dispensing mechanism
Building the candy-dispensing mechanism for our project posed a unique challenge. Our initial design featured a 3D printed funnel to hold the jelly beans, using a rotating platform attached to a servo motor to release them by leveraging gravity. However, this design often resulted in jammed jelly beans.
Our improved design incorporated a screw within the release chamber, agitating the jelly beans to prevent blockages. Although still not foolproof, it significantly outperforms the original design.
.png)
Bridging the gap between the candy-dispensing mechanism and our Halloween spirit model, we incorporated an array of hardware. Our smart machine was composed of the Halloween spirit model, a camera, dual motors, and a Raspberry Pi, all neatly encased within a large 3D-printed pumpkin.
The process initiates with a press of the prominent red button, which activates one of our GPIO pins to a high state. By monitoring this pin status, we can control the dispensing of the jelly beans.
For mobility and dispensing, we connected both motors to the mechanism, controlled by an L298N motor driver. This integration enables our Raspberry Pi to communicate with the motors, directing them to dispense jelly beans only when the pin state is high.
Putting it all together
With all the smart machine components working separately, we used Viam’s Python SDK and the convenient code samples to wire it all up.
We first get all the components including the model, the motors, and the board. Then in a loop, we detect whether the button has been pushed. For example, if the pin is high, the device will capture an image and power the appropriate motor to dispense candy.
See this in action through the code below.
The end product and future work
As you can see, the smart machine appropriately detected Katherine as being not in the Halloween spirit and dispensed the appropriate jelly beans. This means it’s ready for patrolling any and all future Halloween parties we attend.
But as you know, building and refining a machine is an iterative process, so we’ll be continuing to enhance this device until next October in the following ways:
- Model Training: To make our ML model more accurate, we’ll train it on a broader range of images, especially ones taken within the final party setting.
- Dispenser Design: While our design has been upgraded, it still could use some work. We’ll continue iterating the screw-dispensing mechanism to ensure smoother functionality.
- Pre-Built Fragment: To enable others to quickly and easily build their own smart candy-dispensing device, we’re looking to build a fragment that simplifies the entire process.
- Python Code Module: We’re also looking to build a module for our Python code so users can quickly integrate this into their build using our Modular Registry.
Our takeaways
Diving into this build was fun, but not without its challenges! Here’s some of our tips for creating a smart machine.
- Lean on your collaborators for help: Think about what each of you bring to the table and delegate accordingly. In our instance, Kurt champions electrical engineering while Katherine’s collaboration with the Data & ML team makes her a whiz in Viam’s ML Model Service.
- Use your tools wisely: Viam's platform streamlined the ML model training process. With its user-friendly interface, we uploaded and labeled images without any code, making potential future refinements to model accuracy a breeze.
- Tap into online resources available: With platforms like Viam, dive into their documentation for step-by-step guidance, and check to see if there’s a community to connect and collaborate with like-minded enthusiasts.
What ideas do you have for a fun, seasonal smart machine? Share with us and other builders in our Discord community.
