The last round of user research for my project with Mozilla’s Connected Devices team threw up a ton of useful ideas and insights. We shuffled them around on a gazillion post-its and narrowed in eventually on a theme of communication - specifically simple, non-intrusive/non-interrupting ambient messages. We saw a recurring need for ways to say “I’m still here”, “I’m OK”, “I’m thinking of you”. I was reminded of the Goodnight Lamp - one of the first really nice IoT products I remember seeing.
We wanted to validate our thoughts, and dig a little deeper into this area, so we came up with another study, this time using a simple functional prototype. Not so much a product prototype, more a prop and a way to move away from the abstract and focus in on actual reactions when interacting with a thing. In this post I’ll go into some detail on what we built, how we went about it and what we learnt.
A couple of weeks ago I wanted to look into Amazon’s IoT service and conceived a slightly less-dry “hello world” project I could build as a vehicle for this research. You have probably seen the “Useless Box” concept before - its simply a box with a switch on it. When you flip the switch, a flap opens in the box and some kind of finger comes out and switches it back. I wanted to build that, but IoT-enable it, using the MQTT broker to listen for the state change in the switch, and notify the servo-listener to kick into action and put the world back to rights.
For the SmartHome project, we’ve taken a step back to better understand the problem space and come at solutions based on research and evidence. As a team we spent last week interviewing people in their homes, with questions on a theme of freedom and independence. We choose early teens (12-15yrs) and post-retirement folks as a demographic that might have useful insights into this topic.
As a software engineer this has been an interesting process. There’s a temptation to jump into any new project with both feet and start hacking on code; starting with simple successes and working iteratively to add features and meet requirements. And we’ve done some of that in order to get familiar with what we expect will be our prototyping platforms - the raspberry pi and ESP8266. But the more we looked at the problem the more it became clear that we weren’t yet sure what technical questions the project would ask, let alone how to solve them. In the meantime, our team was trying to figure out a better vision for the smart home that would align with Mozilla’s values and potential solutions in a broad and confusing product space. None of us were in our comfort zone, so we made a determination to put roles aside and roll up sleeves and muck in. We’ve posted craigslist ads, we’ve had Skype calls, house visits, coffee shop rendezvous; interviewed, transcribed and now begun to process input from 15 different people.
It turns out that a curious mind, a knack for spotting patterns, analysing outcomes for the motivations and circumstances that produced them - these are the stock-in-trade of any software engineer - and they are skills that work just as well in user research and exploratory product definition as they do in software development. And perhaps more important than that, before we are engineers, we are people. We have families, jobs, aspirations, frustrations and concerns. We were young and hope to grow old. Talking to people this last week has been a great reminder that it is people that solve problems, not code.
With no keyboard or pointer inputs, ensuring the Smart Mirror can be restarted and booted up entirely automatically was high on my priority list. Once installed, I can’t startx or click on any icons; it needs to bring up all the backend services and the dashboard to leave it in a working state without any user intervention. That lead me down a merry path and was (for me) the trickiest part of this project.
Here’s the moving parts:
The kernel and OS itself, with networking and other key systems
The display and window manager - the subsystems that allow me to put my dashboard up on the screen
The mosquitto message broker
The gpio listeners
The web server
The browser, which should load up my dashboard URL
I went through lots of helpful posts and projects on creating linux kiosks to figure out potential approaches. While the mirror isn’t really a kiosk - a kiosk usually has keyboard/pointer/touch user input - its a reasonable match up. After a few false starts trying Firefox/Iceweasel and Chromium kiosk options, I settled on the approach outlined in this Dashing-Pi page. This eschews the LXDE desktop environment entirely and uses nodm and the matchbox window manager to boot into the browser with the minumum of unecessary fluff inbetween.
Orchestrating startup is a bit fiddly even so. First, nodm is configured to startup with/as the ‘pi’ user. The rest of the graphics/display related startup is then in a script copied to /home/pi/.xsession, which starts up the matchbox display manager, and the Uzbl browser to load the dashboard. For the backend pieces, Raspian uses the init.d system, so we install scripts in /etc/init.d/ to start up mosquitto, pm2 (which manages the node.js server(s)) and scripts to relay GPIO events as MQTT messages for the rest of the system.
That done, I can plug the thing in and in just a minute or so it brings up the dashboard on screen and responds to sensor events. The Uzbl browser is a wrapper around WebKit. It supports commands via a socket, which means that once up, I can ssh to the rPi and remotely refresh the page, navigate to other URLs and so on which has proved valuable during development as I have none of the traditional inputs (e.g ctrl+r on the keyboard) to accomplish this otherwise.
One of the goals of this project is to get my hands dirty with integrating data and events coming from hardware (sensors attached to the raspberry pi or functions of the pi itself) and content in the browser. The raspberry pi ecosystem leans heavily on python, so while I was already using node.js for the web server, I didn’t want to get tied into a system which limited my implementation choices for publishing or consuming data. I wanted lightweight, language-agnostic messaging.
My first thought was to simply use the filesystem: writing to some files and watching for changes. That seemed fraught with potential problems and reinventing too much of the wheel. So I went shopping for messaging solutions.
After looking into ZeroQ and ZeroRPC, I settled on Mosquitto and MQTT. The Mosquitto project is a message broker (think central hub for dispatching messages to subscribers) that implements the MQTT protocol. As well as the broker (available as a raspbian-ready package) it also has command-line clients for publishing and subscribing which proved useful. There are MQTT implementations for both python and node.js (as well as a bunch of other languages.) The pub-sub paradigm turns out to be a great fit here. My “hello world” was to use the RPi.GPIO python module to watch for rising-edge button events on the GPIO pin I had jumpered a momentary button to, and publish a message over MQTT when the button is pressed. Logged into the raspberry pi, I run:
$ mosquitto_sub -t sensors/buttonup
.. to see those messages from the broker. In node.js land, the mqtt module provides the same functionality. We connect to the host/port the broker is on, subscribe to some topics and register callbacks for when messages arrive.
Finally, to bring the browser into the equation, I ended up making an adapter to subscribe to MQTT topics, and use socket.io to relay to the dashboard page. There is some (new) websocket support in mosquitto, and probably other more direct ways to accomplish this, but this works for now. As a proof of concept, I handle ‘gpio/button’ events from socket.io, and dispatch a ‘hardbuttonup’ event on the window object. A listener for this event toggles a class to flash the screen blue. This opens up a lot of opportunities as I have flexibility at every stage: the button is a stand-in for any sensor or input device that can toggle a GPIO pin high/low. The MQTT message produced can be caught by any program running on this device, or even anywhere on the network potentially. Turning this message into a DOM event enables a seamless tie-in so you can use existing frameworks to respond to the event.
Its a bit subtle, but in the video I’m clicking the button on the breadboard, and the display is flashing blue.
There are a few key requirements and moving parts to this project that I wanted to explore early on. I began at the end - the UI I wanted to show on the mirror. I initially tried out Atlasboard but quickly found it did more than I needed in some areas, and fought me to do what I wanted in others. To KISS, a ground-up implementation was going to be easier in the long run.
To get started I created a static page with a simple dashboard/widget system. It had a fixed pre-defined number of slots - a 3x3 grid of rows and columns - making layout and addressing really easiy with CSS and getElementById calls. This gave me a straight-forward way to get stuff on the screen. The widgets were a rudimentary base class implementing a init/update/render (and optionally poll to update and re-render) lifecycle.
On the server-side I made an “socket-board” express app to do reverse proxying, and keep things like API keys out of the client-side code. The name alludes to my eventual goal to hook it up to websockets for sending data between the client and data and events on device. All my feed/api requests from the browser go through the express app, and it sends on their response. To that end I created a basic config library and a request-forwarding middleware.
At this point, the dashboard runs anywhere with node.js and a modern browser.
Between the initial implementation and this screenshot, I’ve added a widget for the local IP address, which is particularly useful during development.
With the winding down of the Firefox OS / smartphone project, and a new interest in IoT and connected devices, I thought it would be fun and instructive to go out and build something connected. I settled on the “smart mirror” concept as a nifty thing that would be actually useful at home, and fun to make. I first saw this via hackaday featuring a Magic Mirror project. Its a doable project and should drop me in the middle of some more or less unfamiliar territory: getting hardware talking to software, lots of linux hackery, sensor inputs and gpio, and orchestrating moving parts up and down the stack. It should also be a good test platform to build on, to explore different technologies and interactions.
So, I have in mind a dashboard of sorts, displayed on a monitor/tv behind a two-way mirror. It should collect together some local and remote data of a sort that might be useful to know before you head out into the world: date/time, weather, appointments, actual (vs forecast) conditions and so on. As there wont be any keyboard/mouse/touch inputs, any direct user interactions (not sure yet what they would be if anything) would need to be hands-off - maybe voice input? some kind of presence or simply proximity detection to activate it?
I’m keeping notes as I go about this and I’ll post a series of updates with progress, dead-ends and observations. Some decisions I’ve already made - I’ll start out running this on a Raspberry Pi, the dashboard will be an HTML page in a fullscreen browser - others I’ll figure out along the way. In the past I’ve found that there is no substitute for actually making something to understand what the nature of the challenges will be, and its usually not what you expect. So, I expect to fall into all the traps, trip on all the hurdles and generally fumble my way though. Sounds like fun :)
While its fresh in my head, I jotted down some notes on some minor progress I made with a simple project I’m working on - my own take on the LED candle. I want to end up with a compact, battery powered/rechargable unit that will sit and flicker candle-ishly. Maybe I’ll write up the project in full, but here’s how I’ve finally got up and running programming the attiny85 microcontroller that is the brains of the thing:
USBAsp programmer, to flash the chip. It provides 3 pairs of pins and little jumpers to set:
“power” - i.e use the power provided via the programmer’s (the 5V via USB from my pc) to power the chip while we program it
“slow” - use the jumper for a slower clock speed (i.e 1mhz - which is how they ship IIRC) I had to remove it to program successfully once I’d set the fuse to run at 8mhz
“service” - is to do with programming the usbasp itself I think?
So the slightly fiddly but 100% reliable method I’m using is this:
insert the attiny85 into a socket on a the board I prepared with the 10pin headers wired up to the 8 pin socket
use 10pin ISP ribbon cable to connect usbasp to socket board. Eventually I’ll figure out how to do this in-circuit, but for now, moving the chip from the socket to my breadboard once programmed isnt so bad
Open the Boards Manager and it downloads that and its entries will now show up in Tools > Board menu.
Pick attiny from the boards submenu
Pick attiny85 from the processor submenu
Pick Clock 8Mhz (internal) from the clock submenu. Dont pick external clock unless you’ve got the right oscillator to hand otherwise you’ll render it unusable until you do
Flash the code to the chip with “upload” in arduino IDE.
In the code, bear in mind different clock speed from the bigger & faster chip the arduino uses, and calibrate delays/timing as necessary. In order to debug, I’m using SoftwareSerial library. The attiny85 must be running at 8+mhz for this to work. Basically wherever code says Serial., you do mySerial..
Connect physical pin 2 on attiny to rx on arduino board
Connect pin 3 on attiny to tx on arduino board
Connect ground pin 4 on attiny to arduino’s ground.
Connect vcc pin 8 on attiny to arduino’s 5v out.
Connect USB from arduino to pc, so we can monitor attiny’s Serial output in arduino’s serial monitor
Bear in mind v. limited space on attinys. Pick appropriate types for variables (e.g. use byte instead of int when you know the value will by <= 255.) Arduino’s String library and lots of its string functions & operators not available. Probably you could include it, but memory is at a premium. Mostly I was just using this for debugging anyhow, so once I figured out that my debug code was my problem, it was easy to work around.
Once code is working:
Remove the Serial stuff (i used preprocessor define/ifdefs to make it easy to flip all that stuff off).
Remove the rx/tx connection.
Wire up intended Vcc/ground (battery or whatever) to finally cut the cord and be free.
I found this article stub in my drafts folder, dated back to 2014. I had apparently intended to write about the cancelling of the Firefox Metro (Firefox for Windows 8) project, but never put pen to paper. 18 months later, I cant think anything better to say than “Alas, Metro.”
Backstory: I was thermoforming some more bits for my bottle light project and picked a clear plastic food container out of the recyling bin. I popped it into my jig and in the toaster oven at 250F for 5 minutes - which is normally enough to hit the glass transition point where it takes the new form. This one didn’t. I jacked up the heat a bit, and again. Even at over 300F it really wanted to spring back to its original form. It struck me that this was a solution for a different problem I’ve been mulling over. I want to create the little joule thief circuit I’m using for the bottle lights directly onto the cylinder structure. Or as near as possible. When I apply foil traces to the plastic and solder components to them, the heat badly melts and distorts that substrate. I had some limited success applying foil traces to heavy duty greenhouse tape and soldering components to that. But getting the tape off my work surface and transferring to the plastic structure bends it too much and the tape lifts in places and the solder joints are put under a lot of stress.
The new plastic is a polypropylene, and the container was marked as microwave and top-shelf dishwasher safe. As a quick experiment I put down copper traces to hook up a SMD resistor and LED. It worked great. I flux where the solder needs to flow on the tape, add a dab of solder, place the component and re-melt the solder to flow onto the contacts. I’m using a normal iron with chisel bit that I would use for through-hole soldering and there was no distortion. The quick video shows the result.
As I’m wanting to make each component - the battery holder and switch, the LED driver (joule thief), the LED string - modular, this should work out nicely. I can pre-shape a piece of the heat-safe plastic (at a higher temperature), apply copper foil traces and solder up the circuit (I think manually, I don’t expect reflow techniques to be practical here) and join it to the main cylinder structure with double-sided tape or a rivet/eyelet or some other adhesive.