Interview with Nora Young on CBC Radio One, Spark

May 28, 2009

Originally hosted at http://www.cbc.ca/spark/2010/01/full-interview-mark-coleran-on-fantasy-user-interfaces/

Hi, I'm Nora Young, the host of Spark on CBC Radio 1. This is my full interview with Mark Coleran. Mark is an Ottawa-based user interface designer, but it's one part of his work that's attracting a lot of attention. He designs fantasy user interfaces for movies. He's created the whiz-bang computer interface environments of such movies as Mission Impossible 3, Mr. and Mrs. Smith, The Island, and The Bourne Identity. A shorter version of this interview will air on the January 31st episode of Spark, but right now, here's my full conversation with Mark Coleran.

Hi, Mark.

Hello, Nora.

So I guess it had never really occurred to me that this might be work that someone would do designing fantasy user interfaces. First of all, what do you mean by fantasy user interfaces?

What we mean by that is that anything that is created to go in film and television that is not really a real interface. There's good reasons for them not having real interfaces, so we have to make them up for several different reasons.

What are some of those reasons?

Well, the first one is that a normal computer isn't actually very interesting to look at. Visually, it doesn't do much. You do what you're going to do on it, but it doesn't really tell you anything, doesn't do anything very much, so you have to ramp it up and speed it all up and make it look a lot sexier just to tell people the story of what they're actually doing with that. Of course, the second reason is that we have to create fantasy ones because to use real ones would actually require the approval of whoever owns those interfaces. So whether it be Apple or Microsoft or people like that, they'd love to see their stuff in films, but they also want a little bit of control over it. So in some of the senses, it's very good to do that, but then they don't like people doing anything bad with those computers or anything bad happening to them, like crashing or things like that.

Because that never happens in real life [laughs]. I think we've all had those kind of wow cool moments when we've been watching, especially science fiction or action movies. What's the key to designing these interfaces well?

There's several factors involved in it, and the primary thing is you have to tell a story where it is there for a purpose. That purpose may be to build up the background to make it look like they're doing something sophisticated. You may be explaining something on there that might have been abstract concepts, which could be too wordy or too difficult to just explain from person to person in the film. And so we use them for those kind of purposes. And they do add, they can look quite spectacular when you do that. But we also do try and keep a lot of design cues, at least I personally try to keep a lot of design cues on its feet on the ground. So it looks like it could be something plausible. So we do try that.

As they say in storytelling "Show, don't tell." We've all seen those terrible, painful movies and television shows where somebody goes into a long, technical, boring of what's going on, and there's nothing visually stimulating about what you're looking at at all.

Like a Dan Brown film. It really is important because you only have a short amount of time to tell that story. And everything in any story has to keep moving along. Do you spend two or three minutes explaining something, or do you have two or three seconds to quickly show it? Generally the latter works a lot better.

Are there classic science fiction or action films with user interfaces that you personally really love and admire?

I have a couple that really stand out for me, and for very different reasons. My favorite personally is actually "Dr. Strangelove". You use screens in different ways. Sometimes it's just set dressing. Sometimes it is as a key element to telling a story. But there's a third application which is usually to link scenes together when you have two different things going on in two different locations.

The War Room in "Dr. Strangelove" had probably the best screen display they ever saw in a film. The large world maps showing what is going on in other places and connecting the people in that room with the stuff going on on the plane. That was the first example of a good screen I saw in a movie. I'll always remember that.

Other things are "2001: A Space Odyssey" and "War Games", movies like that, real classics. Even today, even though they're so basic now, they still work very well because they told that story. Or they really added texture to the background and the scenes.

I guess I'd never really thought about the narrative power of using the screen to advance the story or to connect different parts of the story together.

It is a powerful tool. Not everybody uses it well. There are a lot of cliches that get employed and it's a feedback loop, unfortunately, because people use a lot of prior reference when they're deciding what they want to do with it. And it's not always good reference. So we unfortunately have to battle with some of those cliches.

What do you mean by prior reference?

Screen design in itself is a new art, even though it's been around for a long time, it's building up bigger and bigger and it's getting used. In traditional film craft, it's not one of the main crafts that are used. It's not camera, it's not prop making, it's not electrical and all these different areas that come together to make a movie. It's still outside of that. So it doesn't even really get thought about very much. When people are putting all this stuff together, it's not even a big part of the thinking process to the story in the film. Some people do it a lot better than others, but generally it can be left to the last moment. And because of that lack of thought, sometimes it's easier to go back to another example in a previous film and say, okay, we want it like this one. And the head is like, yeah, that's how we want it and that's what we want. So you do have to deal with that.

I don't know if you've seen the "Let's Enhance" video, this montage of all the TV shows and movies where people tell the computer expert to enhance images as though we have this magical power to limitlessly enhance and magnify images - or CSIs, another example where computers can do absolutely anything. Are there other magical computer capabilities that you see in film that just make your head spin a little bit?

Well, there's quite a few. I mean, I was upset about the Enhance video because they beat me to it. I actually have collected a lot of that stuff. It was on the cars and I thought, damn, they did it. But yeah, there's a few like that.

The classic is the Access Denied thing. It's a real struggle to tell people what I actually do in these things. And they look at me blankly and then I'll say, well, you know, when they're typing away and it says Access Denied, oh, yeah, okay, I get it now. And that is one of those great cliches. The other one is the dueling keyboards where you're going to have two people, one's trying to hack in, the other one's trying to lock them out. And so they'll be battling on screen like that. Unfortunately, the other one as well is that you can somehow mysteriously make a computer blow up by doing something to it on screen. So just by typing in a few keywords, you can make smoke and sparks come out the back of a machine. And unfortunately, people are still doing that one. But I blame Star Trek for that one.

We've all seen older movies where the technology is featured, but it's become dated. You watch the movie where the main character is supposed to be flaunting a cutting edge cell phone that's the size of a brick or something. How do you keep things from ending up looking dated?

Well, the simple thing is you can't. It's just not possible. Things move along, and the fact is, you can't really guess where they're going to go. And it doesn't really matter, because the way you will design a lot of this stuff, and you'll design these almost operating systems in the film, it has to be contextual into the film, and it has to be in that place and in that time. If something is a realistic screen, whether it's a thriller or whatever, you have to set it in a point in time. You use visual references from that time. And it's not really a huge issue. And the fact is, they don't really change that much.

It's one part of my job has been to do futurism for these things as well. And the one thing I was astounded by when I really started doing the research is that things don't change very much. They change very slowly. I'll ask people questions about how old you think the ideas behind your computer and the way it works or the windows and the icons, and they'll be "I don't know, well, Windows or Apple came out with this two years ago". But it's 25 years old now as a public release. It's older than that. It's 30 years old in development. And even over that period of time, it hasn't changed significantly. It's more colorful. It's more slick. But it still uses the same principles.

We try and bear that in mind when we design the stuff. So just make it look slick.

I can think of some examples of interfaces that have been really influential or really stood out. I mean, lots of people will reference, say, "Minority Report" where Tom Cruise's character manipulates things with his hands. Do you think it's possible that fantasy interfaces that are really iconic could influence what actually ends up getting designed in the real world?

Absolutely. But there's actually more to that story than there appears. There's a separation as well between the interaction idea and the design that goes on screen. So what you had with "Minority Report" is there was a guy consulted on that called John Underkoffler from the Media Lab at MIT. His brief was to create a plausible gesture system to do this, so Tom Cruise could do a performance that looked like it would be a genuine thing. And then of course, the artists have that to work with and then create this tableau behind them that mirrors the way they're interacting with.

Now, of course, people like Mr. Underkoffler doing that kind of work and coming from MIT, it's things that they're already thinking about. And even from personally, from the stuff that I've done, I scour the net and I scour publications and research papers, and I look at what people are doing in university labs and in corporate labs. The corporate labs are very free and easy on sharing a lot of this stuff, their future ideas and visions. You can take a lot of that stuff and take it from this concept stage, and some of these very rough and ready prototypes and make it look like something that works. Get it up on screen and everybody goes, oh, that's cool. And then down the road, some of that stuff actually gets realized - but of course, the problem is we get the credit for it [laughs], which is really unfair because we're actually taking ideas that other people have done and just making them look like they work.

So it's more of a feedback loop.

Yeah. It's a very circular thing. And then of course, you find out later that some people are inspired by what you do in the film and the circle keeps going. But people like Mr. John Underkoffler are actually... He has his own company now that is creating a system that works like Minority Report. They're actually realizing and making these things.

After seeing "Minority Report", I went to Gestural Computing Lab at the University of Toronto. And they were doing really amazing things that seemed very similar, except the prototypes they were working with were made out of things like ski gloves and stuff. It was really not as high end or elegant at all. But certainly at the conceptual idea, they were working with the same kinds of ideas.

Yeah. It's the idea and realizing that. It can be very bound on engineering initially. But it takes a long, long stage between the ideas and realizing that as a public thing.

You also do interface design more generally. What's the difference between what you design for the real world and these fantasy interfaces?

A huge, huge difference. For the film, your primary thing is either to make it look good as a piece of set dressing or to tell a little piece of the story. With real interface design, the issues are vastly, vastly greater. You have to do something that works.

The brief for something in the film is it works if it tells the story. We get a lot of stick for making stuff that looks unrealistic, but we fulfilled the brief when we told the story. With the real stuff, it has to work. It has to work day in, day out. There's a lot, lot more thought goes into that. As a good example, I think for "The Island", they must have done close to 200 screens for the entire film over a period of about four to five months. So I thought I'm doing one interface for a company here in Ottawa and they shouldn't take too long. I'm still at it two and a half years later [laughs].

There's a lot more thought and work and analysis has to go into the real stuff.

What are some of the trends that you see emerging in terms of real interface design?

As far as trends go, it's hard to say really. From what I've been seeing and looking at occurring is that it's not really so much the interface design is what you can do with those interfaces. A huge thing that's happened recently is the idea of using touch rather than some kind of device like a pointer or control to manipulate stuff, so people are actually being able to manipulate stuff directly. And that is a trend that's not going to go away. It has some fantastic applications, and from what we see is happening, that's only going to get bigger and bigger.

So the trend there is how do we use natural gestures? Even in gaming, they're looking at removing hardware devices as the control. You become the controller itself. There's lots of interesting stuff happening in that way.

If you're thinking about having a more of a natural user interface, what does that mean for the kind of stuff that you do?

Stuff still has to be represented on the actual screen, and you still have to have a point of interaction. Even if you are not touching a device, or you're touching it directly, you still have an interaction point. You still have to have an interface that works around that, that can respond to that. It's a new challenge, and it's actually quite an exciting challenge to do.

Mark has been great hearing about your work. Thanks so much.

Thank you very much.