I think that despite all his calls for “out of the box” thinking, Scott Jenson’s thinking is as bounded as that he decries. I agree, apps suck, and yes, I love the idea of browser as operating system, but I also think the idea of phones themselves as interfaces sucks. They are the apps of the physical world. We won’t need a Google for ranking the sensor-enabled objects around us because they exist in three-dimensional space just as we do. The whole point of physical computing is to eliminate screens as go-betweens.

To use his (kind of lame) example, if I want to interact with my stereo, I shouldn’t have to go to my phone. He just got done telling me how much it sucks that there needs to be an app for that and then he tells me I can tap through a list of objects around me on my phone to interact with them. How about I look at my stereo? Or I talk to it? Or I point at it? Or I think about it?

What we’re seeing is the dying of a computing metaphor. We have always had to go to computers and speak to them in their language. At first, hundreds of us flocked to massive computers and spoke to them in punchcard, an entirely human-unintelligible language. Then a revolution: one man, one computer. The graphical user interface, handmaiden to this revolution, allowed us to speak to the computer in a way we could comprehend, though it still required us to learn how to manipulate its appendages to accomplish the tasks we wanted performed. Now we’re in a world where each person has multiple, increasingly tactile computers. And as processor speeds grow and prices drop, it seems likely that the computer to people ratio will continue to increase.

The desktop metaphor, with its graphically nested menus and multiple windows, won’t survive. It didn’t translate well onto the pocket-sized screens of smartphones, and Siri is the first of peal of its death knell. Siri eliminates the physical analog of a desktop/button pad altogether and replaces it with a schema-less model where I can use a computer without learning anything about how it works.

Couple that with the increasing physical awareness and falling cost of networked devices equipped with cameras and sensors, and what you end up with is not a small computer we can carry with us to interact with the world around us but a giant computer which we inhabit, and which treats us and what we do as input.

What’s tricky about this is imagining the output. With each jump in computing, the new modes did not replace the old modes. They overlapped a bit, but mostly they expanded the possibilities of computing and the number of computable operations. No one programming on the command line imagined that a computer would one day be great for editing films. The command line is still very much in use today as it is still the best method of doing many things, but the GUI has greatly expanded the computable universe. Likewise, while it’s relatively easy to imagine the region where a physical user interface (PUI?) intersects the GUI (advancing slides in a keynote presentation without a remote, for instance), it’s much harder to imagine those tasks we’ve never even thought of as within the reach of computability.

And that’s what I’m really interested in, the film editing scenarios. Context and object awareness won’t require phones to rank nearby objects as we’ll be able to interact with them with minimal or no perceptible interfaces. We’ve slowly watched consumerization turn sophisticated operating systems into shiny idiot-proof button pads. There’s no reason to believe the trend won’t continue spreading into the backend, turning programming itself into a consumer behavior. At Google we’re obsessed with machine learning, but it seems to me the future may be its converse—human teaching. If people can tell their computers exactly what they want without having to learn C or Java, then they can start to ignore their computers entirely.

That’s the ultimate goal: invisible computing. After all, how often do you think about how you’re light switch works when you go turn on the lights in a dark room?

ADVANCE! is a sneaky little game. It’s both a craftily created study of systemic biases in corporate settings masquerading as a game and also as a kind of time-release pill to confront players with evidence of their own biases which they might otherwise be able to plausibly deny. If someone in a study setting asks you to sort through a group of resumes based on the candidates’ appropriateness for a particular job, you will probably make every effort to appear equanimous, even if in a real-life situation you would rarely chose a female candidate over a male candidate. But people love games, they love figuring out the rules and winning, so if you create a game which rewards behaviors that in a non-game setting might be considered uncouth, you can in theory short-circuit political correctness and self-censorship—provided you make the gameplay compelling enough.

And that’s where I came in. I created the visual feel for the game—an info-graphicky, isometric layout with a consistent information panel to the right modeled on a simplified CRM platform—and helped to distill the actual gameplay so that each element and each interaction reinforced the theme while also adrenalizing the game’s fun-ness and making it simple to make complicated judgements based on multiple data points quickly. The final result is a game that looks great and boasts some really nifty mechanics.

Players run a job agency responsible for staffing a faceless but multi-ethnic corporation in a boring high-rise office building. As applicants enter the job queue, players can look for appropriate job openings within the company. Each job requires certain minimum qualifications. If the player doesn’t fill a job quickly, the company will hire an NPC internally.

As the game proceeds, the costs of running the business escalate, so it becomes increasingly important to place candidates quickly into jobs in which they’re happy. As characters are promoted, more floors are added to the building, so finding job openings requires moving floor to floor, which uses valuable time. The ultimate goal is that players become so focused on staying afloat that they don’t notice the subtle biases that are randomly attributed to the client company at the beginning of each game. In one game, instance, the client company may promote men more than women and show a distinct preference for Asian candidates. In one version of the game, identifying the bias correctly during gameplay results in a giant cash settlement; in the other version, there is no mechanism for addressing the bias.

Much my work developing the interface was iterative simplification—removing unnecessary or irrelevant complications while maintaining the game’s overall information-dense statistical feel, and ensuring players’ ability easily and intuitively make multi-dimensional decisions such as comparing a candidate’s qualifications with job requirements while previewing that candidate’s relationship with his/her potential colleagues. I played the skinless prototype—a grid of geometric shapes—which I found totally compelling; I can’t wait to play the finished product!

I just finished up a fun but exhausting weekend at the Bay Area Maker Faire, where after the mafan and technical headaches associated with showing Scratch-n-Sniff last fall, I decided to show AL-Gorithm, my favorite electricity-less project (and an Editor’s Choice winner!). For this installment, I substituted wood for metal as a backing material, placed a tap light behind the text for added drama, and substituted an actual file cabinet armature for the iron dowels I hung the piece on last time.

As always, I had a number of really good conversations about the abstractions that underlie computing, including one specifically about analog computers with Dwight Elvey of the Computer History Museum, who actually had an analog computer set up right across from my table. One woman compared my hanging pages to a slide rule, which I really liked, while it reminded several people of the skewer-like physical sorting mechanism from the days of punch cards.

The zen drudgery of mechanical repetition—as I learned while creating this piece—often leads to insight, and the repetition of presenting this project to hundreds of people whose technical understanding ranged from “Unix? I thought they weren’t allowed to castrate boys any longer” to knowing chuckles “Yes, I’m familiar with grep” is no exception.

I tried a number of narratives to explain why I thought it worthwhile to painstakingly cut out hundreds of paper strips. Experiencing text as a computer does resonated only with coders, who were definitely in the minority of my visitors. Forgetting how to read, while zingy, added too little substance. In the end, the explanation that clicked for both the technically oriented and the overwhelmed families that composed the majority of my audience was the power tools analogy: if all you’ve ever used is a power saw, you’re never mindful of the difficulties and characteristics of sawing, nor can you fully appreciate the materials you’re cutting. Likewise, if you only ever work with digital text, you can’t fully appreciate its texture or what it means to alter, filter, cut, or rearrange it.

My fellow presenters spoke about hacking up a windshield display using a pico projector and a mobile phone, replacing the expensive and proprietary handheld devices used by large enterprises to track inventory and maintenance requests with already ubiquitous mobile handsets—basically, “why isn’t there an app for that?”—, and using web sockets with telephony platforms like Tropo to create persistent connections (for games and the like). There was a lot of code and engineering speak, peppered with good-natured technical objections from the audience.

I spoke about the importance of playing, of doing things that seem totally useless but fun in the interest of stumbling upon new ideas that might not be so useless. I showed my perennial favorite, Generative Social Networking, the ever-popular Botanicalls, the ill-fated Popularity Dialer, and the soon-to-be-huge Megaphone alongside my Eliza project and my more recent forays into accented speech synthesis (clicking will initiate a free Skype call, don’t be alarmed!).

If I’d had a bit more time, I would also have shown Sebastian Buys’s amazingly Rube Goldberg-y World of Warcraft phone-in project, which is tragically not documented anywhere. Because Blizzard doesn’t include any hooks for third-party developers in its code, Sebastian captured screenshots of the game, used OCR to turn the image of the chat box at the bottom of the screen into machine-readable text, and then fed it into an Asterisk script so that a remote user can call in and have his guild’s chatter read to him by a robot over the phone even when he’s not close to his computer. Amazing.

I might also have shown my knock-knock jokes (again, Skype call, again free).

I was a little disappointed I didn’t get any objections from the audience, but I did get a couple of very nice emails!

I’ve been experimenting recently with the hosted Asterisk at Tropo.com, and I have to say, it’s the best API I’ve ever played with, especially after spending months wrangling an Asterisk server. They’ve abstracted away all the eccentricities of Asterisk and created wrappers for Ruby, JavaScript, PHP, and a couple of other languages.

And speaking of other languages, they’ve also included and easy-wrapped a bunch of cool text-to-speech and voice recognition modules for a number of languages. When I saw “Jorge” the Castillian, I had an idea: can a computer voice have an accent? I read a piece in the Times or on some feed that I can’t track down recently that argued that English-language learners have an easier time learning from teachers who share their accent. It makes sense.

I remember an American friend of mine’s mother in Madrid who could not understand why Spaniards kept on thinking she was saying seis (six) when she was saying tres (three). The reason, I explained, was that she was pronouncing tres (which is pronounced like “press” in English) as “trays” which is exactly how seis sounds.

I tell this story as a way of explaining how I arrived at my ESL answering machine here (clicking the link will call a bizarre number in Skype, don’t worry, it’s free).

Getting this to work required some reverse phonetic hacking. Here are a couple of examples, see if you can guess the language:

“Jelo. Mai nem is Inigo Montoya. Llu kild mai fáder, pripeer tu dai.”

“Chateau Haut-Brion 1959, magnifisainte waillene, Aille love Frinch waillene, layke aille love ze Frinch leinguaje. aille ave simpelte everi leinguaje. Frinch ise maille favorite. Fantastic leinguaje, especially tu coeurse wits. Nom de Dieu de putain bordel de merde de saloperies de connards d’enculis de ta meire. Yu si? itte ise layke waille pine yoeur asse wits silk. Aille love itte!”

I’ll be posting a bunch more little phone experiments soon, so check back, you hear!

This post is password protected. To view it please enter your password below:

Password:

Astricon in DC a couple of weeks ago was my first trade show as an exhibitor, and I had a fabulous time. John Todd, Digium’s Asterisk Open Source Community Director invited me to attend and show off Eliza, my video chatterbot. The conference took place at the gargantuan Gaylord National Resort and Convention Center in the altogether bizarre and otherworldly National Harbor development on the banks of the Potomac.

My table was in the little open-source corner of the hall, tucked between some very fancy commercial exhibitors and the constantly rotating cornucopia of caffeinated beverages and high-calorie snacks. Eliza was set up between Astlinux, a custom Linux distribution centered around Asterisk, and the rowdy Atlanta Asterisk Users Group. I was also within spitting distance of the OpenBTS project (roll your own GSM cell tower), of which I’m a big fan, and Areski Belaid, a developer with a finger in numerous telephony pies, including Star2Billing, which essentially allows anyone to become a long-distance phone company. Really interesting stuff.

The most surprising thing about the whole experience, other than the incredible amounts of cookies and sweets, was the communityness of the Asterisk community. Everyone seemed to know everyone, most people over a certain age were way into ham radio, there was nary a GUI in sight, and everyone seemed genuinely interested in everyone else’s projects, including mine.

I spoke for nearly an hour to Tim Panton from PhoneFromHere, a company that integrates voice and chat services into existing websites so businesses can interact directly with their customers over the web. He suggested I cut Flash out of Eliza by using HTTP Live Streaming, which also made me realize that I might also be able to ditch the socket server and use HTML5 web sockets!

Mark Spencer, the boffin responsible for Asterisk, stopped by and seemed genuinely pleased to see that a couple of years on, ITPers are still playing with his baby, making it contort in unexpected ways.

The folks at LumenVox (speech recognition) and GM Voices (speech synthesis and lightning-turnaround voice recording) generously offered to help robustify Eliza for her next iteration.

Also enthusiastic were Jason Goecke and Ben Klang, who are the principal movers behind the Ruby Adhearsion framework which reskins Asterisk in a slick modern web way and also involved with Tropo, by far the best cloud-hosted Asterisk service I’ve seen—write scripts in a variety of languages, host them yourself or on their servers and debug them through a web interface, take advantage of the built-in speech recognition system, seamlessly integrate with AGI, and best of all it’s all free for development, pay only when you’re looking to cash in! They turned me onto this interactive phone/video piece, which got me thinking.

ELIZA 2.0

For her next iteration, Eliza’s going to be on the web, hopefully in gloriously standards-compliant HTML5. Instead of canned conversations, she’ll rely on silence detection and Markov chains to generate much more dynamic conversations. The GM Voices people told me that they often record vocabularies—phrases in a variety of intonations so that you can do text to speech with real voices rather than those slightly Scandinavian sounding canned computer voices. I’ll be posting my progress soon.

Part php and part spit, Jumblr is a website that makes unexpected connections by jumbling links, either within a site or between two sites. It works by scraping a site, parsing its links and storing them in an array using XPath, shuffling the array, and then returning them via preg_replace(). Right now it breaks when sites use relative links because they end up pointing to my domain. I’ve been to lazy to revisit constructing conditional regular expressions, but once I do, I’ll be able to fix that problem. I’m also going to develop a Javascript link interceptor so that the randomization persists with each link click.

My favorite thing about my Scratch-n-Sniff TV is the conversations it spawns. I showed it recently at Maker Faire NY, and as at previous showings at ITP and at Greylock Arts, reactions were divided. About 70% of people were totally incredulous until they tried it, and then were delighted and had to find out how it worked. Of the remaining 30%, half looked at it suspiciously and rebuffed invitations to try it and the other half tried to predict how it worked before using it and then complained that the smells weren’t “accurate.” All of these reactions reveal an underlying attitude towards technology and its possibilities: the first, marvel—the what will they think of next effect; the second, suspicion—this has got to be a trick; the third, which shares elements of the second, a need to establish that we control technology—not the other way around.

Smell is subjective, it’s ephemeral, and it’s not binary. What smells like citrus to one person smells like air freshener to another; smells can’t be turned on and off, they waft, so getting people to believe that their actions resulted in equal and opposite smell reactions required some clever sleight of nose. First of all, I gave people clear visual cues. When you scratch a picture of chocolate, you’re much more likely to interpret the resulting smell as chocolate. I also made the screen respond to being scratched by fading, just as scratch-n-sniff stickers do after vigorous scratching. This tie-in to a direct physical analogue was key, as people were much more likely to smell the screen where they’d scratched it and the one-to-one correspondence between action and reaction primed people to smell. A couple of times I ran out of scents, and several people still swore they’d smelled scents that simply weren’t there!

HOW IT WORKS

1. I found that the transistor-based model of the Glade Flameless Candle automatic air freshener would fire once approximately every two seconds if powered for 500 milliseconds (as opposed to the earlier version that relies on a resonant circuit that requires ten seconds before firing), so I hooked up its battery terminals to an Arduino, and voila! Controllable atomization of non-oil based scents!

2. Trying to create an effective scent disperser from scratch is madness. One of the benefits of piggybacking on all of Glade’s hard work is that it’s easy to fill the provided smell canisters with other scents. I got most of mine from the nice folks at Demeter.

3. I aligned the scent dispensers under a touchscreen sending touch coordinates to the Arduino via Processing sketch. Thanks to the hydrostatic properties of the fine particle mist, when emitted, it flows up the screen and across it, sticking to it until the scent evaporates a few seconds later.