I just finished up a fun but exhausting weekend at the Bay Area Maker Faire, where after the mafan and technical headaches associated with showing Scratch-n-Sniff last fall, I decided to show AL-Gorithm, my favorite electricity-less project (and an Editor’s Choice winner!). For this installment, I substituted wood for metal as a backing material, placed a tap light behind the text for added drama, and substituted an actual file cabinet armature for the iron dowels I hung the piece on last time.
As always, I had a number of really good conversations about the abstractions that underlie computing, including one specifically about analog computers with Dwight Elvey of the Computer History Museum, who actually had an analog computer set up right across from my table. One woman compared my hanging pages to a slide rule, which I really liked, while it reminded several people of the skewer-like physical sorting mechanism from the days of punch cards.
The zen drudgery of mechanical repetition—as I learned while creating this piece—often leads to insight, and the repetition of presenting this project to hundreds of people whose technical understanding ranged from “Unix? I thought they weren’t allowed to castrate boys any longer” to knowing chuckles “Yes, I’m familiar with grep” is no exception.
I tried a number of narratives to explain why I thought it worthwhile to painstakingly cut out hundreds of paper strips. Experiencing text as a computer does resonated only with coders, who were definitely in the minority of my visitors. Forgetting how to read, while zingy, added too little substance. In the end, the explanation that clicked for both the technically oriented and the overwhelmed families that composed the majority of my audience was the power tools analogy: if all you’ve ever used is a power saw, you’re never mindful of the difficulties and characteristics of sawing, nor can you fully appreciate the materials you’re cutting. Likewise, if you only ever work with digital text, you can’t fully appreciate its texture or what it means to alter, filter, cut, or rearrange it.
Connections are most interesting when they are broken, when they’re forbidden, when they are unintended. Secret liaisons make good stories, short-circuits end in fires, ruptured pipelines induce panic. This is the great appeal of the mashup—possibly even the power of cinema (wasn’t it Eisenstein who wrote about cutting and the mental jumps the mind makes?)—the juxtaposition of disparate elements that our minds nonetheless connect. That is pretty much how my favorite parts of my brain work, making strange and unexpected connections with unpredictable outcomes. It’s also the way humor works, uncovering the unexpected connection to spark a laugh.
Everything I know about interaction design I learned by making a scratch-n-sniff television
My favorite thing about my Scratch-n-Sniff TV is the conversations it spawns. I showed it recently at Maker Faire NY, and as at previous showings at ITP and at Greylock Arts, reactions were divided. About 70% of people were totally incredulous until they tried it, and then were delighted and had to find out how it worked. Of the remaining 30%, half looked at it suspiciously and rebuffed invitations to try it and the other half tried to predict how it worked before using it and then complained that the smells weren’t “accurate.” All of these reactions reveal an underlying attitude towards technology and its possibilities: the first, marvel—the what will they think of next effect; the second, suspicion—this has got to be a trick; the third, which shares elements of the second, a need to establish that we control technology—not the other way around.
Smell is subjective, it’s ephemeral, and it’s not binary. What smells like citrus to one person smells like air freshener to another; smells can’t be turned on and off, they waft, so getting people to believe that their actions resulted in equal and opposite smell reactions required some clever sleight of nose. First of all, I gave people clear visual cues. When you scratch a picture of chocolate, you’re much more likely to interpret the resulting smell as chocolate. I also made the screen respond to being scratched by fading, just as scratch-n-sniff stickers do after vigorous scratching. This tie-in to a direct physical analogue was key, as people were much more likely to smell the screen where they’d scratched it and the one-to-one correspondence between action and reaction primed people to smell. A couple of times I ran out of scents, and several people still swore they’d smelled scents that simply weren’t there!
HOW IT WORKS
I found that the transistor-based model of the Glade Flameless Candle automatic air freshener would fire once approximately every two seconds if powered for 500 milliseconds (as opposed to the earlier version that relies on a resonant circuit that requires ten seconds before firing), so I hooked up its battery terminals to an Arduino, and voila! Controllable atomization of non-oil based scents!
Trying to create an effective scent disperser from scratch is madness. One of the benefits of piggybacking on all of Glade’s hard work is that it’s easy to fill the provided smell canisters with other scents. I got most of mine from the nice folks at Demeter.
I aligned the scent dispensers under a touchscreen sending touch coordinates to the Arduino via Processing sketch. Thanks to the hydrostatic properties of the fine particle mist, when emitted, it flows up the screen and across it, sticking to it until the scent evaporates a few seconds later.
Last summer I worked in Tokyo at a division of TBS, where I was asked to develop a prototype for an English listening comprehension game for Japanese kids. I spent a month conceiving the game, laying it out, developing the code, and art directing the incomparable Nina Widartawan-Spenceley, who created the characters and animated the grotesque death sequences.
Once again, Flash Player in Firefox is a bit screwy. Mozilla, what’s going on?
You need Flash Player 10 to run this bad boy because I’m using its sound generating capabilities. Oh, and if you don’t have a webcam and a mic, you’re out of luck, sorry. Also, for some reason this isn’t working with the latest version of Firefox.
I finally ported the Sensorship sketch I wrote in Processing to Flash so that you too can enjoy the marvels of real-time censorship. It’s not quite as slick as its Processing predecessor, but it works on the web, and there’s no arguing with that.
There are a number of ports of OpenCV to ActionScript based on Ohtsuka Masakazu’s original port (“Marilena“) and also a native library called Deface. I ended up using one of the former, Mario Klingemann’s slightly modded Marilena, not because I have a particular preference but because I’m lazy and he very generously included an example that used the webcam.
After making the necessary adjustments to the face detecting code to draw the black bar and flip it and the image to make them mirror reflections (it always weirds me out to wave with my left hand only to see myself waving back with my right), I used the new SampleDataEvent that Adobe has helpfully documented along with insights gleaned from Jeff Swartz’s helpful tutorial on generating sounds dynamically to generate what can only be described as a horrifically annoying beep any time the microphone’s input passes a certain threshold.
Google Map IP geolocation with two sticks and a bit of twine!
X marks your spot
I found myself in a bit of sticky situation a couple of days ago. I wanted to map site visitors’ approximate location using Maxmind’s amazing free IP address geolocation database. I used the same database for one of my paywalls so I thought it would be a fifteen-minute kind of affair: getting site visitors’ IP address using and looking it up in the Geolitecity database, then encoding the results into a Google Maps url that would display the map on the site.
As anyone who’s worked with static Google Maps knows, you pass in a bunch of parameters (latitude, longitude, size, markers) on a url string and Google returns a map. The problem for me was that even if I managed to pass the requisite latitude and longitude coordinates to the page, the link itself would not end with .png, .gif, or .jpg, which meant that WordPress would not display it as an image.
I will spare you the cursing and hair pulling that ensued and instead delight you with the multi-part solution. I wrote a PHP script that takes a visitor’s IP address and looks up its latitude and longitude in the Maxmind database, formulates a syntactically correct Google Maps url request, and uses a header to redirect the browser there. In order to convince WordPress that I was adding an image, I used mod_rewrite in an .htaccess file to redirect requests for “here.jpg” to my PHP script, and voila! Success!
Eliza sits at her desk in her office. She completes ordinary office tasks—she checks her email, she drinks her coffee, she gets up to go photocopy something or talk to a colleague, and once in a while she checks out the New York Times. Little does she know, she’s being livestreamed to the whole world over the web. If someone calls, she picks up. Sometimes she recognizes the caller, sometimes she does not, and sometimes the connection is so bad that she hangs up and calls back.
Eliza lives on a screen in an eddy of a high-trafficked area, say an out-of-the-way elevator lobby in a busy building. A user sees her and after a couple of minutes, his curiosity gets the best of him and he succumbs to the flashing invitation and calls. To his surprise, after a couple of rings Eliza picks up. Phone conversations are ritualized in the first place and the added awkwardness of voyeurism and conversing with a stranger create the ideal situation for Eliza’s black-belt phone jujitsu which with minimal effort wrests control of the conversation from her interlocutor. It’s a bit like a good dancer foxtrotting and waltzing an overwhelmed novice around the floor.
The prototype is rough, but it works, though because of Flash’s arcane and draconian cross-domain security measures, I can only run it locally through the Flash IDE or stream from my machine using a personal broadcasting service like ustream or livestream (in order for it to work properly on the web, I’d have to host all the components I enumerate below on a single box, something I have neither the hardware nor the inclination to do). The main problem is that I’m making XML socket connections from Flash; if I used a PHP intermediary, I could probably get it working, but again, the whole inclination thing is missing and the thing is already mindbogglingly complicated. Maybe at some point in the future. The following video demonstration will have to do in the meantime.
SO HOW DOES IT WORK?
Warning: this is not for the faint of heart.
Eliza has a ton of moving parts:
The Asterisk script: A simple program that answers phone calls and hands them to a PHP script, which connects via socket to the main SWF.
Various PHP scripts: One to handle connections from Asterisk, one to reset connections from Asterisk after a call ends, and one to initiate callbacks when required.
A simple Java socket server: Adapted from Dan Shiffman’s example, this program runs in the background on the Asterisk server, waiting for connections (phone calls). When a call comes in, it connects it and broadcasts call events (new call, hangup, button press, etc) to the PHP scripts and the main SWF and allows them to talk to each other.
The main SWF: This is the brains of the operation. It loads the movies of Eliza and controls the logic of their looping as well as the logic of the audio (via socket connection back to PHP and then to Asterisk via AGI).
The looping movie files (not completely smooth in this prototype, notice the moving phone and the changing light conditions!): These live in the same directory as the main SWF, which streams them as needed (for a web deployment, they’d probably have to be pre-loaded and played back).
The sound files: These live on the Asterisk box (completely separate from the movies) and are played back over the phone, not the web.
NEXT STEPS UPDATE: I’m presenting Eliza at Astricon in DC in October, so I should have some interesting observations to report soon. There are several things I’d really like to do. First, I’d like to actually get this working somewhere where I can observe lots of unsuspecting folks interacting with Eliza. I never really got to see someone who didn’t know the backstory calling in, partly because I was exhausted from thesis when I had the chance to show it and partly because there were lingering bugs I had not yet located that occasionally caused the whole thing to stop working—there are so many things on so many separate machines that can go wrong, it took quite a while to troubleshoot. A larger sample of reactions would allow me to rework the conversations so that they’re more disorientingly convincing—better pause timing, more realistically intoned, and taking into account repeat callers’ stratagem’s to see if Eliza is real. I could then reshoot the video so it is completely seamless. That would require monitors, good lighting, laser levels, an OCD continuity editor, and several days of shooting.
If you know of an easy way to overcome the cross-domain headaches, leave me a comment! If you want to fund such an undertaking, please do get in touch! Otherwise, enjoy the video above.
I currently lead a small team of engineers and designers at Google that explores new interaction modalities with computers. We were responsible for Google Cardboard and Google Tone among other things. Before that, I was in graduate school in New York. And before that I lived and worked in Shanghai for 6 years.