Last summer I worked in Tokyo at a division of TBS, where I was asked to develop a prototype for an English listening comprehension game for Japanese kids. I spent a month conceiving the game, laying it out, developing the code, and art directing the incomparable Nina Widartawan-Spenceley, who created the characters and animated the grotesque death sequences.
Once again, Flash Player in Firefox is a bit screwy. Mozilla, what’s going on?
You need Flash Player 10 to run this bad boy because I’m using its sound generating capabilities. Oh, and if you don’t have a webcam and a mic, you’re out of luck, sorry. Also, for some reason this isn’t working with the latest version of Firefox.
I finally ported the Sensorship sketch I wrote in Processing to Flash so that you too can enjoy the marvels of real-time censorship. It’s not quite as slick as its Processing predecessor, but it works on the web, and there’s no arguing with that.
There are a number of ports of OpenCV to ActionScript based on Ohtsuka Masakazu’s original port (“Marilena“) and also a native library called Deface. I ended up using one of the former, Mario Klingemann’s slightly modded Marilena, not because I have a particular preference but because I’m lazy and he very generously included an example that used the webcam.
After making the necessary adjustments to the face detecting code to draw the black bar and flip it and the image to make them mirror reflections (it always weirds me out to wave with my left hand only to see myself waving back with my right), I used the new SampleDataEvent that Adobe has helpfully documented along with insights gleaned from Jeff Swartz’s helpful tutorial on generating sounds dynamically to generate what can only be described as a horrifically annoying beep any time the microphone’s input passes a certain threshold.
My former ITP classmate Dimitris Makris has a way with words. He weaves them together into rich tapestries of thought before tugging at a single idea and unraveling them into a meta-mountain of meaning qua meaning—you get the idea. To commemorate our graduation from ITP, I had a way with his words too. It goes a little like this:
alex | Audio, Post-ITP | Comments Off on Makrimix (Dimitris Redux) | #
July 16th, 2010
Google Map IP geolocation with two sticks and a bit of twine!
X marks your spot
I found myself in a bit of sticky situation a couple of days ago. I wanted to map site visitors’ approximate location using Maxmind’s amazing free IP address geolocation database. I used the same database for one of my paywalls so I thought it would be a fifteen-minute kind of affair: getting site visitors’ IP address using and looking it up in the Geolitecity database, then encoding the results into a Google Maps url that would display the map on the site.
As anyone who’s worked with static Google Maps knows, you pass in a bunch of parameters (latitude, longitude, size, markers) on a url string and Google returns a map. The problem for me was that even if I managed to pass the requisite latitude and longitude coordinates to the page, the link itself would not end with .png, .gif, or .jpg, which meant that WordPress would not display it as an image.
I will spare you the cursing and hair pulling that ensued and instead delight you with the multi-part solution. I wrote a PHP script that takes a visitor’s IP address and looks up its latitude and longitude in the Maxmind database, formulates a syntactically correct Google Maps url request, and uses a header to redirect the browser there. In order to convince WordPress that I was adding an image, I used mod_rewrite in an .htaccess file to redirect requests for “here.jpg” to my PHP script, and voila! Success!
Eliza sits at her desk in her office. She completes ordinary office tasks—she checks her email, she drinks her coffee, she gets up to go photocopy something or talk to a colleague, and once in a while she checks out the New York Times. Little does she know, she’s being livestreamed to the whole world over the web. If someone calls, she picks up. Sometimes she recognizes the caller, sometimes she does not, and sometimes the connection is so bad that she hangs up and calls back.
Eliza lives on a screen in an eddy of a high-trafficked area, say an out-of-the-way elevator lobby in a busy building. A user sees her and after a couple of minutes, his curiosity gets the best of him and he succumbs to the flashing invitation and calls. To his surprise, after a couple of rings Eliza picks up. Phone conversations are ritualized in the first place and the added awkwardness of voyeurism and conversing with a stranger create the ideal situation for Eliza’s black-belt phone jujitsu which with minimal effort wrests control of the conversation from her interlocutor. It’s a bit like a good dancer foxtrotting and waltzing an overwhelmed novice around the floor.
The prototype is rough, but it works, though because of Flash’s arcane and draconian cross-domain security measures, I can only run it locally through the Flash IDE or stream from my machine using a personal broadcasting service like ustream or livestream (in order for it to work properly on the web, I’d have to host all the components I enumerate below on a single box, something I have neither the hardware nor the inclination to do). The main problem is that I’m making XML socket connections from Flash; if I used a PHP intermediary, I could probably get it working, but again, the whole inclination thing is missing and the thing is already mindbogglingly complicated. Maybe at some point in the future. The following video demonstration will have to do in the meantime.
SO HOW DOES IT WORK?
Warning: this is not for the faint of heart.
Eliza has a ton of moving parts:
The Asterisk script: A simple program that answers phone calls and hands them to a PHP script, which connects via socket to the main SWF.
Various PHP scripts: One to handle connections from Asterisk, one to reset connections from Asterisk after a call ends, and one to initiate callbacks when required.
A simple Java socket server: Adapted from Dan Shiffman’s example, this program runs in the background on the Asterisk server, waiting for connections (phone calls). When a call comes in, it connects it and broadcasts call events (new call, hangup, button press, etc) to the PHP scripts and the main SWF and allows them to talk to each other.
The main SWF: This is the brains of the operation. It loads the movies of Eliza and controls the logic of their looping as well as the logic of the audio (via socket connection back to PHP and then to Asterisk via AGI).
The looping movie files (not completely smooth in this prototype, notice the moving phone and the changing light conditions!): These live in the same directory as the main SWF, which streams them as needed (for a web deployment, they’d probably have to be pre-loaded and played back).
The sound files: These live on the Asterisk box (completely separate from the movies) and are played back over the phone, not the web.
NEXT STEPS UPDATE: I’m presenting Eliza at Astricon in DC in October, so I should have some interesting observations to report soon. There are several things I’d really like to do. First, I’d like to actually get this working somewhere where I can observe lots of unsuspecting folks interacting with Eliza. I never really got to see someone who didn’t know the backstory calling in, partly because I was exhausted from thesis when I had the chance to show it and partly because there were lingering bugs I had not yet located that occasionally caused the whole thing to stop working—there are so many things on so many separate machines that can go wrong, it took quite a while to troubleshoot. A larger sample of reactions would allow me to rework the conversations so that they’re more disorientingly convincing—better pause timing, more realistically intoned, and taking into account repeat callers’ stratagem’s to see if Eliza is real. I could then reshoot the video so it is completely seamless. That would require monitors, good lighting, laser levels, an OCD continuity editor, and several days of shooting.
If you know of an easy way to overcome the cross-domain headaches, leave me a comment! If you want to fund such an undertaking, please do get in touch! Otherwise, enjoy the video above.
I currently lead a small team of engineers and designers at Google that explores new interaction modalities with computers. We were responsible for Google Cardboard and Google Tone among other things. Before that, I was in graduate school in New York. And before that I lived and worked in Shanghai for 6 years.