The universe of “Black Mirror” continues to expand with each new episode, adding more layers and nuance to how technology of today can evolve in the near future. From the very beginning, the show was focusing much less on the technology itself, but rather on how it can change the fabric of our everyday interactions from the micro level of a single individual to the macro level of the society at large. And yet, the presence of technology in the universe of “Black Mirror” can not be denied, even through the most fleeting glimpses at the outer manifestation of that technology – glass surfaces, or screens.

Continuing the ongoing series of interviews on fantasy user interfaces, it’s my pleasure to welcome Simon Russell. In this interview he talks about his work on audio geometry experimentation and music visualization for concert stages, the symbiotic relationship between tools and imagination, the difficulty of creating something truly new and the drive to best serve the storyline with screen graphics. In between and around, we talk about Simon’s work on the screens of “Black Mirror”, from the corporate technology of “Hated in the Nation” to the futuristic graphics in “USS Callister” to the soft round shape of the coaching device in “Hang the DJ”.

Screens of “USS Callister” episode of “Black Mirror”, season 4.

Kirill: Please tell us about yourself and your path so far.

Simon: I did a degree in visual communication and moving image design at Ravensbourne, and then started working in the motion graphics industry. My first job was at the Cartoon Network, doing lots of kids stuff. Then I did lots of shiny R&B adverts for a company that was in the music business and then for a startup that basically stopped quite quickly, I’ve been doing freelancing in the last eight years or so.

My direction changed somewhat when I started 3D. I found it stimulating and challenging in a way I hadn’t found 2D work. Then I began to bring particles particles and simulations into the work and something really clicked. And that’s where I’m sitting at the moment – somewhere in between VFX and motion graphics.

Recently I’ve been doing music visualizations for concerts and projection mapping, and that brings me back to my college days. I did projects on Kandinsky when I was 15, and I loved the idea of visualizing music even then. And now many years later I’m coming back to it. It’s oddly circular.

Kirill: It’s quite interesting to see the hardware advances in that area and how much they are enabling in the last decade or so. You go to a concert or watch award shows, and it’s amazing to see all those screens in different shapes and sizes everywhere. And it didn’t even feel a gradual process. All of a sudden, these gigantic screens were everywhere.

Simon: I’ve been interested in music visualization for so long. I’ve went away from it and now I’m getting paid to do it on such a big scale. I did visuals for the Shawn Mendes world tour visuals, and the screens were insane. It’s the hardware and the playback that make it possible. It’s really exciting.

My motivation is to see it as pure experimental design. Everyone puts their own spin on it, and people see it on these futuristic screens. Aside from “Black Mirror” and live event work, I’ve been doing audio geometry experiments on my site. I’m getting some work from that, and it’s driving the jobs I’m doing. It’s nice and surprising that it’s working out like that [laughs]. It’s not often that things fall nicely into place like that. Maybe I’ve been doing it for so long that eventually it just clicks.

Kirill: We’re talking about number of screens, each with its own shape and size which is usually quite huge so that it can be seen from the back of that space. When you sit down to first think about it, what’s your approach to visualizing it? Do you do it on paper, or in some kind of a digital environment?

Simon: You start thinking about the idea, about what it is you’re trying to get across. For that, it doesn’t matter what is the shape of the box and how you are trying to draw it. It’s the same process. You get your concepts, you sketch, you make little experiments to prototype it in 3D. A screen is just a 2D surface, and it doesn’t take a huge leap of imagination to do it.

But the project I’m working on at the moment is this tunnel with 42 projectors super bright projectors. It’s going to be really long and really bright. And we’re using the whole tunnel, roof and all. The playout system we’re using can preview the setup in VR so you can really get a sense of the space and what you’ll be seeing. It’s amazing to see these particles waves flowing in time to the music, flowing down the tunnel. If it’s even close to that in real life it’ll be very powerful.

Visuals for Shawn Mendes Illuminate live tour, courtesy of Simon Russell.

I also worked with another client on a project where we visualise the space in Unreal engine to really get a sense of it. It can be used as a communication tool to show such spaces to clients, like film directors. Designers that work on more technical things know how it’s going to look and feel, but sometimes you need to lead people. So if you can put them into that world, it’s a very practical use of VR. Everyone is scrambling around VR at the moment, but nobody knows what it is going to end up being. I believe that this particular approach is going to be genuinely useful.

Kirill: Do tools matter as much as your imagination? The tools at your disposal continue evolving, but what good are those tools if, as a designer, you don’t have the right idea to work off of?

Simon: It’s a symbiotic relationship. It’s a fair point that you can have the highest-end computer, but it’s useless if you don’t know how to use it and you don’t have any imagination. On the other hand, you can have crazy ideas and no means to achieve them.

As ever, the truth is somewhere in the middle. This is where Painting Practice are so strong. They stand in the middle of that place. You have post-production houses which are very technical even when they have their design departments. But it’s hard to do simulations of what is physically plausible and still be loose and creative. You need to be in the brain-space to think about it in a purely creative way. It doesn’t always go hand-in-hand with those giant post places. This is where Painting Practice fits in. They lead conceptually, but also know how to follow it through and push those ideas down. It’s about defining those clear, beautiful, emotive ideas, about really creating a powerful concept but being able to lead that through all the hurdles and challenges of a huge technical production and still keep that original essence.

Visuals for Shawn Mendes Illuminate live tour, courtesy of Simon Russell.

It’s probably been said a thousand times but you can have these multi-million dollar superhero productions and it can be dry at times. It is soulless because there are no ideas behind it, no heart. It’s all about the effects. Marshall McLuhan said that the medium is the message, and that’s the case here. Because they can do these huge effects driven spectaculars they do. The medium becomes the product. Computers are really good at doing certain things, and you’re sort of led by that and by what everybody else is doing. If you look at Cinema 4D motion graphics you get a lot of mography, rigid body plexus style effects because they’re super easy to do. It’s the path of least resistance. You work with the tools, and thereafter they shape you. You create technology and afterwards it shapes you.

There’s a lot of wastage in creating something new and genuinely fresh, it’s hard and time-consuming. You can just copy what has been done before and stand on the shoulders of those giants. Or you can start by building up on a concept from first principles, and it’s really slow.

When I started doing music visualizations, I got into Houdini because it can handle audio as well. It takes such a long time even for the simplest idea. You start with a concept, which is the best way to create something original. But it’s slow, and if you’re working for a client, they don’t necessarily care.

Animations for Quartet for the End of Time, courtesy of Simon Russell.

Kirill: Speaking about imagination and ideas, do you worry about your field being displaced by automated tools and processes in the near future? There are some experimentations around machine learning that try to come up with the answer to the question “what good design is” and taking it over from there. I’m not quite sure how feasible it is to replicate that creative spark that starts the whole process in a designer’s head.

Simon: It’s hugely difficult to predict what AI and machine learning are going to bring. I don’t think anyone knows really.

I like to think about the rising water analogy. The easiest things will get picked up by the machines, and you see it everywhere as things get automated. But the higher-level, more conceptual things will probably take much more time to be caught up with. Nobody can say how high the watermark can rise in theory. Some futurists like Ray Kurzweil think that it’s going to be completely automated. But I agree more with people who say that we don’t know where the bar will be. It’s going to be interesting to find that out.

And longer-term it’s probably going to be a partnership with machines. It will be a two-way communications, learning to work with each other. But no one can say where the barrier is going to be.

I do worry about it. You see it in other industries. You see animation and post-production work going from USA and Europe to developing countries. The knowledge is there. The ability to learn from the Internet is there. The desire for people to use technology is there. It’s very much a changing landscape.

It’s about positioning yourself where you want to be and where you think is a sensible place for it.

Animations for Quartet for the End of Time, courtesy of Simon Russell.

Kirill: So maybe the balance between your imagination and your tools then shifts towards the former. That becomes your differentiator.

Simon: If you’ve demonstrated that you can conceptually grasp it, organize it and lead that vision, you’ll be fine.

Kirill: I’ve been watching the episodes of “Black Mirror” that you’ve worked on, and while there are a couple of lower-level, terminal interfaces, the rest are closer to those music visualizations that you’ve been talking about earlier. You visualize certain patterns to highlight what is going on in the story at that point, sort of a graphical summary of it.

Simon: Screen graphics are a strange beast. It is 100% about telling the story, and you see that in any good feature or episodic production. There’s lot of thought and background detail in it, even though hardly any of it is seen explicitly. There’s logic and depth to it, and that’s what makes it feel real.

In “Hated in the Nation” I’ve worked a lot on the branding of Granular. I spent a couple of weeks just on the branding side. Every episode of “Black Mirror” is essentially a film production. They have different directors and different crew. In some way, the depth of it feels excessive, but it’s needed to create that reality. It helps to tell the story. It helps to bring emotion and reality to it.

Logo exploration for “Hated in the Nation” episode of “Black Mirror”, season 3.

People can see what goes into it, even if it’s on a subconscious level. If I really enjoy doing it, and I put all those details into it, it all adds up. It’s like color. People are not aware of what you’re using, but it adds up to a deeply emotional effect. The end result is quite concentrated. You put in a lot of raw material, and some of it doesn’t go anywhere. That’s how you create quality. It was eye opening to see the attention to detail and work that goes into Black Mirror from the team at Painting Practice, well the whole production crew. The world becomes very real. Not least for the people who have worked on every episode since the early days because they understand in a much more complete sense. I was only brought in for a few episodes so it was really helpful to be able to tap into people’s knowledge of the whole Black Mirror world. The whole thing exists in its own world with its own consistent internal logic.

Kirill: And as you said, a lot of it is in background as set decoration. They need to blend into that environment without calling attention to themselves.

Simon: Kind of… Painting Practice were clear that every screen and every idea I did on “Black Mirror” had its own internal logic. But some GUI elements require more conceptual depth than others. Like the screen graphics I worked on for the end of “USS Callister” that are generally futuristic texture and not so much a part of that story. The brief there was just to create a beautiful intricate sprawl of technological doodads! But generally you’re trying to put as much logic and thought into it. You think about how it works to get something interesting. Even though 95% of the people are never going to pick up on it, it’s still there.

Console screen graphics for “Hated in the Nation” episode of “Black Mirror”, season 3.

Kirill: You designed the seven screens for the company’s control room on “Hated in the Nation”. How did it work for you, designing for an environment that you don’t necessarily have in your office setup?

Simon: It comes from the script, with the art department generally having a plan of every room and every screen, and you working with the list of what the screen deliverables need to be. Morgan Kennedy designed the final control room and I worked with Erica, the director (James Hawes) and the Painting Practice team to really flesh out each beat. It was really quite complex as it had to fit the logic of the technology, work as a storytelling device and work in terms of playback because it was all in-camera.

Revolver AV did the playback. They brought a pig PC that plugged into all the screens, and I think they had Mac Minis that were plugged separately into each individual screen. Some of the content was running as a background loop. I created somewhere around 8 or 10 Quicktime videos, some looping and some being cued when a button was pressed. You needed to make sure that there were no jumps between the videos, and that the overlays when they started typing would come at the right place on the screen. It was all in-camera, worked out ahead of time.

Console screen graphics for “Hated in the Nation” episode of “Black Mirror”, season 3.

Kirill: Both detective Blue and Granular’s project lead Rasmus are portrayed as technical people, and their preferred interaction is via terminal. Those interactions usually get scrutinized quite closely by the more “hardcore” members of the audience. What kind of research did you do on portraying those interactions?

Simon: Everything is researched, and I love doing that. The more you internalize it, the more you feel it. If I had it my way, everything would be done as realistically as possible [laughs]. But you have to work stylistically.

You research as much as possible, and after that people need to agree on concepts and ideas. Then you throw it all away and start from scratch again. You want to be free. You cannot be constrained by research. I tried to be authentic with the code. But at the end of the day, most of it is just screen dressing.

Kirill: I look at music visualizations, and it’s this explosion of colors, shapes, gradients and so much going on. But then in the world of screen graphics there seems to be this self-reinforcing vocabulary that red is bad, and blue / green are good. Do you see that as just another constraint?

Simon: Every project and brief has its constraints. I like that aesthetic, and I’m quite happy to work within that. I like the rawness and the beauty of the numbers within it. It’s the same thing – another brief to explore.

Screens of “USS Callister” episode of “Black Mirror”, season 4.

Kirill: Returning to the “USS Callister” episode in season 4 of “Black Mirror” and Daly’s home computer, the interfaces there are very highly tuned to what he needs. It felt quite stripped-down to the task he has at hand, almost terminal-like. But yet it’s quite powerful as he scans the DNA samples and kick off the entire process of creating her digital clone.

Simon: To a certain degree, there’s almost a house-style to everything in “Black Mirror”. There’s cleanness in that near-future world which isn’t too far away from what we have. When I looked at the DNA compiler, I spoke with Dan May about some ideas. Afterwards I put out 5-6 ideas with little motion graphics animations to see what the feeling is. Then we go back and forth, refining it as more people get involved from the art department.

Quite often, conceptually they have a more or less clear idea of what they want. It’s taking that and the leeway that you get from them to play with it. It’s like any design process. You iterate and listen to feedback.

Screens of “USS Callister” episode of “Black Mirror”, season 4.

Kirill: Do you ever dream about coming up with a design that is so final and pure that nobody can come up with any suggestions to improve upon?

Simon: In the military they say that no plan that ever survives the contact with the enemy. And that’s quite similar to our field. I have had projects where they love it from the first sight and you go with it. Sometimes it’s that easy.

That’s what separates personal design work and creating a solution that fits a very specific set of needs. There’s also complexity and flexibility that keeps on evolving collaboratively throughout. And even for your own personal work, it’s such a complex beast that it’s hard to be pure about it. It takes a huge amount of work to bring something into the world. And if something is perfect, maybe it’s not really that good. Maybe it needs to be pushed.

I was listening to an interview with Christine McVie of Fleetwood Mac, and she talked the ‘Songbird’, and said that it just came to her. Almost out the ether. You hear that a lot from musicians that something comes almost fully formed to them. In a way, I have ideas like that, but the process of giving birth to it is messy. The more people get involved, the more complex that Venn diagram becomes.

Screen graphics for “Nosedive” episode of “Black Mirror”, season 3.

Kirill: That’s what is the most impressive for me when I see how many people come together to work on a feature film or on a season of an episodic show, and how much needs to happen to make this one story to come to my screens.

Simon: It’s impressive even working within it, and it’s amazing that anything gets made [laughs]. You can see how hard it is to make something good.

Kirill: Going to the end of “USS Callister”, you did these more modern looking graphics for the bridge. As they get into what sounds like a multi-player gaming universe, did you look at AAA games and what kinds of interfaces they have?

Simon: It was about creating the feel of a sophisticated and intricate video game. There is an accepted language for that, and it was a lot of fun to do that. It was about building up wireframes, motion graphics animations and a large widget set. Then it went over for another pass to John Wilkinson who made the graphics for the main story line on that bridge, and then it was prepared for the playback. So here again I had the nice bit [laughs] to create lots of intricate sparkly little things, which is fun.

Screens of “USS Callister” episode of “Black Mirror”, season 4.

You asked earlier about coming from the world of color, composition and sound that are a part of the sensual aesthetic of music visualization. What is nice is that you can apply exactly the same principles of color, composition, contrast and type. It’s superficially different, but essentially you’re doing the same thing. You’re using the same traditional artistic techniques.

You see that with Painting Practice. They have so many people doing matte paintings and conceptual sketches by hand. It feels like an art studio rather than a more technical one with computers. They have that, but so much of what they do comes from painting and drawing.

Kirill: This is what I loved when I talked with Joel and Dan – the blend of different techniques, digital and physical. It’s about finding the right tool for the job. And when I as a viewer see the final frame, as long as it’s consistent, it doesn’t matter if it was a matte painting or something created in a digital environment.

Simon: They are completely agnostic as long as it works. It doesn’t matter, and it shouldn’t matter. It’s not about the tools. It is about the idea.

This is what storytelling is. You’re making this little sketch and the lines are all perfect. And then you create a really detailed painting, and the challenge of making a film is carrying that through with hundreds of people and all the complexity, while still keeping the original movement of the line with all those layers on top of it. It’s quite impressive what Painting Practice manages to do.

Screen graphics for the coach assistant device in “Hang the DJ” episode of “Black Mirror”, season 4.

Kirill: Was it much different designing for the circular form factor of the little assistant device in “Hang the DJ” episode?

Simon: It was quite fun. I designed the shape of it as well, doing a lot of pebble sketches. That actually challenges your brain more than having seven screens all around. The circular shape was challenging, and I did a lot of concepts of sound waves and other interactions.

Kirill: When people ask what you do for a living, what do you say?

Simon: It depends on the person, but generally I say that I do animation. Then if they want, they can ask more questions. I don’t start with 3D animation or motion graphics. That’s just technical.

Map screen graphics for “Hated in the Nation” episode of “Black Mirror”, season 3.

Kirill: For the dramatic effect “Black Mirror” takes the potential of a near-future technology and pushes it into both extreme positive and extreme negative. Stepping away from this show, what are your thoughts on our symbiotic relationship with technology in our daily lives?

Simon: It’s a huge question. There’s a book that I’m reading at the moment called “Sapiens“. It’s the history of humanity, technology and science, and I really recommend it.

To narrow the question a bit, seeing what the Internet is doing to the world is very interesting, as I remember the time before it. You see how ingrained social and digital media are, and how the information is becoming more of a driving force within the world economy. You see huge transitions. You see profound macro trends, and sometimes it’s hard to do that when you’re on the inside.

I think we have profound changes going on, and there’s a trend at the moment about how dangerous social media can be, and how much of a time drain it is. It is habitual. It is compulsive. And that’s even before genetic engineering, robotics and machine learning take over to stop us from talking to each other completely.

I’m definitely more on the side of “Black Mirror”, I guess. It’s a stereotype, but positivity tends to be more of an American style of thinking, and the more British style is reflected in “Black Mirror”. I’m more on the dystopian side, but you can’t give up the hope.

Kirill: But then if you look at your personal career so far, it has been largely enabled by advances in technology, including the Internet.

Simon: That’s not to say that I wouldn’t be happy creating, let’s say, wood etchings or whatever. It is remarkable and wonderful that I can create some of the things that I do. If Kandinsky could see the tools that we have now, he would be amazed. You can literally do what he was talking about, which is purely concept. It is utterly incredible and mind-blowing. It’s a complex multi-faceted picture. It’s really such a big question [laughs].

One of the saddest things that came with the march of progress is the annihilation of the environment, and that will be one of the long-lasting marks.

Holographic map screen graphics for “Hated in the Nation” episode of “Black Mirror”, season 3.

And here I’d like to thank Simon Russell for taking the time out of his busy schedule to talk with me about his work on “Black Mirror”, and for sharing the supporting images. “Black Mirror” is playing on Netflix as we speak. And if you’re interested to read additional interviews about the wonderful world of screen graphics and user interfaces for film and TV, click here for more.