The iSmell: every five years, like clockwork

Ophone


It’s that time again. Yup, computer scent peripheral. This incarnation requires users to stick their noses right up against the disturbingly organic end of a little scent stick for a personalized whiff “symphony.” The interaction, while bizarre, is definitely an improvement over more pervasively ambient approaches, but imagining someone sniffing at a plastic cylinder while absorbed in a screen makes me think of the people I sit next to on the bus who have the off-putting habit of smelling their fingers while they read. That or one of those old Vicks eucalyptus inhalers.

Timer reset. Sigh.

I hate Tom Cruise




I gave a five-minute Ignite talk about gesture interfaces at Foo Camp a couple of weeks ago. It was an attempt to distill my thinking about the difficulties and opportunities of mid-air gesturing as input into a single serving. I wrote about 7000 words and whittled them down to 700 over the course of a week. Fun, but five minutes can contain only so much hatred, and I have oh so much more.

Some thoughts on where computing is headed

This is largely a response to this video informed by this other video:

I think that despite all his calls for “out of the box” thinking, Scott Jenson’s thinking is as bounded as that he decries. I agree, apps suck, and yes, I love the idea of browser as operating system, but I also think the idea of phones themselves as interfaces sucks. They are the apps of the physical world. We won’t need a Google for ranking the sensor-enabled objects around us because they exist in three-dimensional space just as we do. The whole point of physical computing is to eliminate screens as go-betweens.

To use his (kind of lame) example, if I want to interact with my stereo, I shouldn’t have to go to my phone. He just got done telling me how much it sucks that there needs to be an app for that and then he tells me I can tap through a list of objects around me on my phone to interact with them. How about I look at my stereo? Or I talk to it? Or I point at it? Or I think about it?

What we’re seeing is the dying of a computing metaphor. We have always had to go to computers and speak to them in their language. At first, hundreds of us flocked to massive computers and spoke to them in punchcard, an entirely human-unintelligible language. Then a revolution: one man, one computer. The graphical user interface, handmaiden to this revolution, allowed us to speak to the computer in a way we could comprehend, though it still required us to learn how to manipulate its appendages to accomplish the tasks we wanted performed. Now we’re in a world where each person has multiple, increasingly tactile computers. And as processor speeds grow and prices drop, it seems likely that the computer to people ratio will continue to increase.

The desktop metaphor, with its graphically nested menus and multiple windows, won’t survive. It didn’t translate well onto the pocket-sized screens of smartphones, and Siri is the first of peal of its death knell. Siri eliminates the physical analog of a desktop/button pad altogether and replaces it with a schema-less model where I can use a computer without learning anything about how it works.

Couple that with the increasing physical awareness and falling cost of networked devices equipped with cameras and sensors, and what you end up with is not a small computer we can carry with us to interact with the world around us but a giant computer which we inhabit, and which treats us and what we do as input.

What’s tricky about this is imagining the output. With each jump in computing, the new modes did not replace the old modes. They overlapped a bit, but mostly they expanded the possibilities of computing and the number of computable operations. No one programming on the command line imagined that a computer would one day be great for editing films. The command line is still very much in use today as it is still the best method of doing many things, but the GUI has greatly expanded the computable universe. Likewise, while it’s relatively easy to imagine the region where a physical user interface (PUI?) intersects the GUI (advancing slides in a keynote presentation without a remote, for instance), it’s much harder to imagine those tasks we’ve never even thought of as within the reach of computability.

Computing Paradigms Bubble Chart

And that’s what I’m really interested in, the film editing scenarios. Context and object awareness won’t require phones to rank nearby objects as we’ll be able to interact with them with minimal or no perceptible interfaces. We’ve slowly watched consumerization turn sophisticated operating systems into shiny idiot-proof button pads. There’s no reason to believe the trend won’t continue spreading into the backend, turning programming itself into a consumer behavior. At Google we’re obsessed with machine learning, but it seems to me the future may be its converse—human teaching. If people can tell their computers exactly what they want without having to learn C or Java, then they can start to ignore their computers entirely.

That’s the ultimate goal: invisible computing. After all, how often do you think about how you’re light switch works when you go turn on the lights in a dark room?

ADVANCE! — a first-person recruiter

Before moving out to the West Coast, I worked for several months with the irrepressibly delightful Jessica Hammer on refining the mechanics and creating the visual style for ADVANCE!, a Diner Dash-ish game at the center of her doctoral dissertation. Jessica has an extremely sophisticated understanding of games and game dynamics, so naturally the game she envisioned was much more than a resource distribution clickfest.

Advance! screenshot

ADVANCE! is a sneaky little game. It’s both a craftily created study of systemic biases in corporate settings masquerading as a game and also as a kind of time-release pill to confront players with evidence of their own biases which they might otherwise be able to plausibly deny. If someone in a study setting asks you to sort through a group of resumes based on the candidates’ appropriateness for a particular job, you will probably make every effort to appear equanimous, even if in a real-life situation you would rarely chose a female candidate over a male candidate. But people love games, they love figuring out the rules and winning, so if you create a game which rewards behaviors that in a non-game setting might be considered uncouth, you can in theory short-circuit political correctness and self-censorship—provided you make the gameplay compelling enough.

And that’s where I came in. I created the visual feel for the game—an info-graphicky, isometric layout with a consistent information panel to the right modeled on a simplified CRM platform—and helped to distill the actual gameplay so that each element and each interaction reinforced the theme while also adrenalizing the game’s fun-ness and making it simple to make complicated judgements based on multiple data points quickly. The final result is a game that looks great and boasts some really nifty mechanics.

Players run a job agency responsible for staffing a faceless but multi-ethnic corporation in a boring high-rise office building. As applicants enter the job queue, players can look for appropriate job openings within the company. Each job requires certain minimum qualifications. If the player doesn’t fill a job quickly, the company will hire an NPC internally.

Advance! screenshot

Each job also comes with its own politics, symbolized on the game board by hearts and skulls beneath the colleagues that come with a particular job. The higher the ratio of hearts to skulls, the more likely a candidate will thrive in a position and become eligible for promotion.

Advance! screenshot

Promotions (and demotions) happen automatically when a job opens on a floor above a certain character’s current floor and his/her qualifications have increased sufficiently. Alternately, the player may choose to train characters to manually enhance their skills, though this can prove very expensive.

phpEfJh5j

The abstract notion of a score is replaced with a running tally of the job agency’s bank account balance. Running the business costs money, so this balance creeps downward throughout the game. Successfully placing job candidates results in cash bonuses, and the player receives a percentage of their salary as long as they’re employed. Training candidates improves their job prospects as well as the bounty a player receives for placing them, but as a character becomes more experienced, the cost of training grows exponentially.

As the game proceeds, the costs of running the business escalate, so it becomes increasingly important to place candidates quickly into jobs in which they’re happy. As characters are promoted, more floors are added to the building, so finding job openings requires moving floor to floor, which uses valuable time. The ultimate goal is that players become so focused on staying afloat that they don’t notice the subtle biases that are randomly attributed to the client company at the beginning of each game. In one game, instance, the client company may promote men more than women and show a distinct preference for Asian candidates. In one version of the game, identifying the bias correctly during gameplay results in a giant cash settlement; in the other version, there is no mechanism for addressing the bias.

Much my work developing the interface was iterative simplification—removing unnecessary or irrelevant complications while maintaining the game’s overall information-dense statistical feel, and ensuring players’ ability easily and intuitively make multi-dimensional decisions such as comparing a candidate’s qualifications with job requirements while previewing that candidate’s relationship with his/her potential colleagues. I played the skinless prototype—a grid of geometric shapes—which I found totally compelling; I can’t wait to play the finished product!

AL-Gorithm at Maker Faire

Editor's Choice 2011

I just finished up a fun but exhausting weekend at the Bay Area Maker Faire, where after the mafan and technical headaches associated with showing Scratch-n-Sniff last fall, I decided to show AL-Gorithm, my favorite electricity-less project (and an Editor’s Choice winner!). For this installment, I substituted wood for metal as a backing material, placed a tap light behind the text for added drama, and substituted an actual file cabinet armature for the iron dowels I hung the piece on last time.

As always, I had a number of really good conversations about the abstractions that underlie computing, including one specifically about analog computers with Dwight Elvey of the Computer History Museum, who actually had an analog computer set up right across from my table. One woman compared my hanging pages to a slide rule, which I really liked, while it reminded several people of the skewer-like physical sorting mechanism from the days of punch cards.

Algorithm Close up

The zen drudgery of mechanical repetition—as I learned while creating this piece—often leads to insight, and the repetition of presenting this project to hundreds of people whose technical understanding ranged from “Unix? I thought they weren’t allowed to castrate boys any longer” to knowing chuckles “Yes, I’m familiar with grep” is no exception.

I tried a number of narratives to explain why I thought it worthwhile to painstakingly cut out hundreds of paper strips. Experiencing text as a computer does resonated only with coders, who were definitely in the minority of my visitors. Forgetting how to read, while zingy, added too little substance. In the end, the explanation that clicked for both the technically oriented and the overwhelmed families that composed the majority of my audience was the power tools analogy: if all you’ve ever used is a power saw, you’re never mindful of the difficulties and characteristics of sawing, nor can you fully appreciate the materials you’re cutting. Likewise, if you only ever work with digital text, you can’t fully appreciate its texture or what it means to alter, filter, cut, or rearrange it.

I Told You So: New York Times Paywall has Arrived

Finally: the Times announces that it will finally institute the subscription model it first announced last year. Maybe my thesis will get some press! Check out some alternative paywalls while we wait to see whether people will cough up the cash, old-school print journalism will be saved, and I’ll finally get that TED invite I’ve been waiting for.

Make it Ridiculous (Till it’s Awesome)

I’m finally getting around to writing up my presentation at the “Telephony is Sexy” edition of the SF Telephony Meetup on December 16th. I wish I’d spoken to Clay Shirky before I went out to California, as I have him to thank for the apt title of this post, a phrase he overheard used recently to describe the modus operandi of ITP, which over the last two years I’ve largely co-opted as my own.

My fellow presenters spoke about hacking up a windshield display using a pico projector and a mobile phone, replacing the expensive and proprietary handheld devices used by large enterprises to track inventory and maintenance requests with already ubiquitous mobile handsets—basically, “why isn’t there an app for that?”—, and using web sockets with telephony platforms like Tropo to create persistent connections (for games and the like). There was a lot of code and engineering speak, peppered with good-natured technical objections from the audience.

I spoke about the importance of playing, of doing things that seem totally useless but fun in the interest of stumbling upon new ideas that might not be so useless. I showed my perennial favorite, Generative Social Networking, the ever-popular Botanicalls, the ill-fated Popularity Dialer, and the soon-to-be-huge Megaphone alongside my Eliza project and my more recent forays into accented speech synthesis (clicking will initiate a free call from the browser, don’t be alarmed, and make sure to authorize the mic when the browser asks).

If I’d had a bit more time, I would also have shown Sebastian Buys’s amazingly Rube Goldberg-y World of Warcraft phone-in project, which is tragically not documented anywhere. Because Blizzard doesn’t include any hooks for third-party developers in its code, Sebastian captured screenshots of the game, used OCR to turn the image of the chat box at the bottom of the screen into machine-readable text, and then fed it into an Asterisk script so that a remote user can call in and have his guild’s chatter read to him by a robot over the phone even when he’s not close to his computer. Amazing.

I might also have shown my knock-knock jokes (again, browser call, again free).

I was a little disappointed I didn’t get any objections from the audience, but I did get a couple of very nice emails!

Text-to-English-as-a-Second-Language

I don't speak English - But I promise not to laugh at your Spanish.

I’ve been experimenting recently with the hosted Asterisk at Tropo.com, and I have to say, it’s the best API I’ve ever played with, especially after spending months wrangling an Asterisk server. They’ve abstracted away all the eccentricities of Asterisk and created wrappers for Ruby, JavaScript, PHP, and a couple of other languages.

And speaking of other languages, they’ve also included and easy-wrapped a bunch of cool text-to-speech and voice recognition modules for a number of languages. When I saw “Jorge” the Castillian, I had an idea: can a computer voice have an accent? I read a piece in the Times or on some feed that I can’t track down recently that argued that English-language learners have an easier time learning from teachers who share their accent. It makes sense.

I remember an American friend of mine’s mother in Madrid who could not understand why Spaniards kept on thinking she was saying seis (six) when she was saying tres (three). The reason, I explained, was that she was pronouncing tres (which is pronounced like “press” in English) as “trays” which is exactly how seis sounds.

I tell this story as a way of explaining how I arrived at my ESL answering machine. (Clicking the button below will call the app directly from the browser, I’m no longer paying for Skype and phone numbers for any of these phone apps).

Make sure you allow the browser to use your mic after you click the button.

Getting this to work required some reverse phonetic hacking. Here are a couple of examples, see if you can guess the language:

“Jelo. Mai nem is Inigo Montoya. Llu kild mai fáder, pripeer tu dai.”

“Chateau Haut-Brion 1959, magnifisainte waillene, Aille love Frinch waillene, layke aille love ze Frinch leinguaje. aille ave simpelte everi leinguaje. Frinch ise maille favorite. Fantastic leinguaje, especially tu coeurse wits. Nom de Dieu de putain bordel de merde de saloperies de connards d’enculis de ta meire. Yu si? itte ise layke waille pine yoeur asse wits silk. Aille love itte!”

I’ll be posting a bunch more little phone experiments soon, so check back, you hear!

« Previous Entries