Archive for the 'UX' Category

Some thoughts on where computing is headed

This is largely a response to this video informed by this other video:

I think that despite all his calls for “out of the box” thinking, Scott Jenson’s thinking is as bounded as that he decries. I agree, apps suck, and yes, I love the idea of browser as operating system, but I also think the idea of phones themselves as interfaces sucks. They are the apps of the physical world. We won’t need a Google for ranking the sensor-enabled objects around us because they exist in three-dimensional space just as we do. The whole point of physical computing is to eliminate screens as go-betweens.

To use his (kind of lame) example, if I want to interact with my stereo, I shouldn’t have to go to my phone. He just got done telling me how much it sucks that there needs to be an app for that and then he tells me I can tap through a list of objects around me on my phone to interact with them. How about I look at my stereo? Or I talk to it? Or I point at it? Or I think about it?

What we’re seeing is the dying of a computing metaphor. We have always had to go to computers and speak to them in their language. At first, hundreds of us flocked to massive computers and spoke to them in punchcard, an entirely human-unintelligible language. Then a revolution: one man, one computer. The graphical user interface, handmaiden to this revolution, allowed us to speak to the computer in a way we could comprehend, though it still required us to learn how to manipulate its appendages to accomplish the tasks we wanted performed. Now we’re in a world where each person has multiple, increasingly tactile computers. And as processor speeds grow and prices drop, it seems likely that the computer to people ratio will continue to increase.

The desktop metaphor, with its graphically nested menus and multiple windows, won’t survive. It didn’t translate well onto the pocket-sized screens of smartphones, and Siri is the first of peal of its death knell. Siri eliminates the physical analog of a desktop/button pad altogether and replaces it with a schema-less model where I can use a computer without learning anything about how it works.

Couple that with the increasing physical awareness and falling cost of networked devices equipped with cameras and sensors, and what you end up with is not a small computer we can carry with us to interact with the world around us but a giant computer which we inhabit, and which treats us and what we do as input.

What’s tricky about this is imagining the output. With each jump in computing, the new modes did not replace the old modes. They overlapped a bit, but mostly they expanded the possibilities of computing and the number of computable operations. No one programming on the command line imagined that a computer would one day be great for editing films. The command line is still very much in use today as it is still the best method of doing many things, but the GUI has greatly expanded the computable universe. Likewise, while it’s relatively easy to imagine the region where a physical user interface (PUI?) intersects the GUI (advancing slides in a keynote presentation without a remote, for instance), it’s much harder to imagine those tasks we’ve never even thought of as within the reach of computability.

Computing Paradigms Bubble Chart

And that’s what I’m really interested in, the film editing scenarios. Context and object awareness won’t require phones to rank nearby objects as we’ll be able to interact with them with minimal or no perceptible interfaces. We’ve slowly watched consumerization turn sophisticated operating systems into shiny idiot-proof button pads. There’s no reason to believe the trend won’t continue spreading into the backend, turning programming itself into a consumer behavior. At Google we’re obsessed with machine learning, but it seems to me the future may be its converse—human teaching. If people can tell their computers exactly what they want without having to learn C or Java, then they can start to ignore their computers entirely.

That’s the ultimate goal: invisible computing. After all, how often do you think about how you’re light switch works when you go turn on the lights in a dark room?

ADVANCE! — a first-person recruiter

Before moving out to the West Coast, I worked for several months with the irrepressibly delightful Jessica Hammer on refining the mechanics and creating the visual style for ADVANCE!, a Diner Dash-ish game at the center of her doctoral dissertation. Jessica has an extremely sophisticated understanding of games and game dynamics, so naturally the game she envisioned was much more than a resource distribution clickfest.

Advance! screenshot

ADVANCE! is a sneaky little game. It’s both a craftily created study of systemic biases in corporate settings masquerading as a game and also as a kind of time-release pill to confront players with evidence of their own biases which they might otherwise be able to plausibly deny. If someone in a study setting asks you to sort through a group of resumes based on the candidates’ appropriateness for a particular job, you will probably make every effort to appear equanimous, even if in a real-life situation you would rarely chose a female candidate over a male candidate. But people love games, they love figuring out the rules and winning, so if you create a game which rewards behaviors that in a non-game setting might be considered uncouth, you can in theory short-circuit political correctness and self-censorship—provided you make the gameplay compelling enough.

And that’s where I came in. I created the visual feel for the game—an info-graphicky, isometric layout with a consistent information panel to the right modeled on a simplified CRM platform—and helped to distill the actual gameplay so that each element and each interaction reinforced the theme while also adrenalizing the game’s fun-ness and making it simple to make complicated judgements based on multiple data points quickly. The final result is a game that looks great and boasts some really nifty mechanics.

Players run a job agency responsible for staffing a faceless but multi-ethnic corporation in a boring high-rise office building. As applicants enter the job queue, players can look for appropriate job openings within the company. Each job requires certain minimum qualifications. If the player doesn’t fill a job quickly, the company will hire an NPC internally.

Advance! screenshot

Each job also comes with its own politics, symbolized on the game board by hearts and skulls beneath the colleagues that come with a particular job. The higher the ratio of hearts to skulls, the more likely a candidate will thrive in a position and become eligible for promotion.

Advance! screenshot

Promotions (and demotions) happen automatically when a job opens on a floor above a certain character’s current floor and his/her qualifications have increased sufficiently. Alternately, the player may choose to train characters to manually enhance their skills, though this can prove very expensive.


The abstract notion of a score is replaced with a running tally of the job agency’s bank account balance. Running the business costs money, so this balance creeps downward throughout the game. Successfully placing job candidates results in cash bonuses, and the player receives a percentage of their salary as long as they’re employed. Training candidates improves their job prospects as well as the bounty a player receives for placing them, but as a character becomes more experienced, the cost of training grows exponentially.

As the game proceeds, the costs of running the business escalate, so it becomes increasingly important to place candidates quickly into jobs in which they’re happy. As characters are promoted, more floors are added to the building, so finding job openings requires moving floor to floor, which uses valuable time. The ultimate goal is that players become so focused on staying afloat that they don’t notice the subtle biases that are randomly attributed to the client company at the beginning of each game. In one game, instance, the client company may promote men more than women and show a distinct preference for Asian candidates. In one version of the game, identifying the bias correctly during gameplay results in a giant cash settlement; in the other version, there is no mechanism for addressing the bias.

Much my work developing the interface was iterative simplification—removing unnecessary or irrelevant complications while maintaining the game’s overall information-dense statistical feel, and ensuring players’ ability easily and intuitively make multi-dimensional decisions such as comparing a candidate’s qualifications with job requirements while previewing that candidate’s relationship with his/her potential colleagues. I played the skinless prototype—a grid of geometric shapes—which I found totally compelling; I can’t wait to play the finished product!