The computational gaze

Image: Tim Ellis. Flickr

I’ve written and spoken before about what I call mechanomorphism — a word that I developed to describe the concept of machine intelligence as a companion species. This framing of AI is distinct from anthropomorphism, where we try (and inevitably fail) to make machines approximate human behavior. Instead, I envision a future where we appreciate computers for the ways in which they’re innately “other”.

Another way to put it is that I’m fascinated by the computational gaze — how machines see, know, and articulate the world in a totally alien manner. I’ve been talking a lot with my boss, John Maeda, about computational literacy and how to help people understand foundational concepts of computing. But computational literacy posits the machine as a tool (which it often is!). The computational gaze, on the other hand, suggests the machine as a collaborator or companion intelligence.

Collaborating with machine intelligence means being able to leverage that particular, idiosyncratic way of seeing and incorporate it into creative processes. This is why we universally love the “I trained a neural net on [x] and here’s what it came up with” memes. It has this delightful “almost-but-not-quite-ness” to it that lets us delight in the strangeness of that unfamiliar gaze, but also can help us see hidden patterns and truths in our human artifacts.

The increasing accessibility of tools for working with machine learning means that I’m seeing more examples of artists, writers and others treating the machine as collaborator — working with the computational gaze to create work that is beautiful, funny, and strange. Here are some folks who are doing particularly interesting work in this arena:

Visual feedback loops

In the visual arts, Ronan Barrot and Robbie Barrat have a show in Paris where they collaborate with a GAN to paint skulls. “It’s about having a neural network in a feedback loop with a painter, influencing each other’s work repeatedly — and the infinitude of generative systems.

Mario Klingemann has also been playing with GANs in his “Neural Glitch” series:

“Neural Glitch” is a technique I started exploring in April 2018 in which I manipulate fully trained GANs by randomly altering, deleting or exchanging their trained weights. Due to the complex structure of the neural architectures the glitches introduced this way occur on texture as well as on semantic levels which causes the models to misinterpret the input data in interesting ways, some of which could be interpreted as glimpses of autonomous creativity. 

—Mario Klingemann
Mario Klingemann, Neural Glitch

Writing with machines

Alison Parrish does wonderful creative writing work in collaboration with generative systems. Some of her highlighted work is here, and many projects have open-source code or tutorials. Here’s an example of Alison’s Semantic Similarity Chatbot, which she describes as “uncannily faithful to whatever source material you give it while still being amusingly bizarre”.

Alison Parrish, Semantic Similarity Chatbot

I also often come back to Robin Sloan’s “Writing with the Machine” project from a couple of years ago, where he trained an RNN on a corpus of old sci-fi stories and used it to auto-suggest sentence completions in his text editor.

Robin Sloan, Writing with the Machine

Enjoying the weirdness

From a more playful perspective, I particularly love the work that Janelle Shane has been doing, documented on her site AI Weirdness:

I train neural networks, a type of machine learning algorithm, to write unintentional humor as they struggle to imitate human datasets. Well, I intend the humor. The neural networks are just doing their best to understand what’s going on. 

— Janelle Shane

Here’s her illustration of some of the cookies her neural net came up with when trained on cookie recipes:

Janelle Shane’s neural net-generated cookies

Machines cheat in bizarre ways

One of my favorite things is seeing how machine learning systems will find bizarre ways to “cheat” in order to fulfill the goals that are set for them. Recently, there was a lot of discussion around this AI that steganographically encoded invisible data into maps in order to achieve the stated goal of recreating aerial imagery from said map. There’s also a fantastic Google sheet that describes all the ways various AI systems have found unexpected and strange workarounds!

Indolent cannibals

In an artificial life simulation where survival required energy but giving birth had no energy cost, one species evolved a sedentary lifestyle that consisted mostly of mating in order to produce new children which could be eaten (or used as mates to produce more edible children).

The literal computational gaze

This last piece is not about machines as collaborators, but is still one of my favorite pieces in that it so powerfully evokes the sense of the machine’s alien gaze. This is from 2012, and is a video by Timo Arnall called Robot Readable World.

I find this kind of work delightful and meaty, and I hope to see more of it. As soon as I learned to code, I started making generative things — fake ad generators, chatbots, etc. I loved making work that, even though I had shaped it, continued to surprise me. I felt warmth and curiosity towards my strange mechanical collaborators. In a moment where the computational gaze is being used in so many exploitative and questionable ways, I hope that there is also space for work that allows us to explore all that is delightful and creative about our computational companions.

Coding as Creativity

Maze-like output of 10 Print, a classical "code poem".
From @10print_bot on Twitter. For more information, see

An executive recently confided in me that he was surprised to learn that developers cared what problems they were working on. “I thought they just cared about writing code, not what it was for,” he said, in what seemed like newfound respect.

While this might seem like an isolated opinion, many technology leaders direct their developers in ways that support this statement, even if they wouldn’t put it so starkly. Developers are often seen as interchangeable parts who can go from working on internal tools to email templates to site reliability to API development without a second thought. Further, the tools and processes we use strengthen this view; tickets are often assigned to teams who align on a certain skill, not a given product or problem set, and organizations are often delineated by understanding of a portion of a “stack” (front-end, back-end, APIs, etc.)

These distinctions make some logical sense (I’ve recently been guilty of organizing teams by these assumptions) but to foster true creativity, new techniques will be required.

More than any other technique (and I’ll write about others in the coming weeks) the one that must come first is to realize that development is not about writing code, but rather using code to solve problems. Too often tech teams are given tasks without context or goals, cutting off an avenue for innovation and creativity. With clear communication and trust, we can unlock the creativity of our tech teams by treating tech as a strategic partner and not a service bureau.

This begins with sharing strategy, goals, measurements, and reasoning with technologists, including everyone from the CTO to individual developers. Recent history is flush with examples of developers and engineers creating new lines of business within their organizations; Amazon Prime is likely the most successful of these. These contributions were only possible because these developers understood their company’s goals and could apply their unique skills to those problems.

Unleashing this creativity requires clear, candid communication. Only by sharing hopes and fears honestly will every member of your staff be in a position to contribute with all of their skills.

Once they’ve been properly briefed, developers tasked with solving a given problem should work together across boundaries of expertise. Whether you call them teams, squads, pods, whatever, people working together to solve problems will yield solutions that are both more creative and implemented more quickly. The tight feedback loop of design and interface development, or API development and information display, for example, creates this effect. If at all possible, this should include participation from design, product development, and even editorial and marketing, if appropriate.

Treating technologists as service practitioners will guarantee you get exactly what you ask for. Including technology in the early stages of defining problems and opportunities will mean getting solutions far more creative, efficient, and sustainable than you could have imagined.

Design process for the messy in-between

I tweeted this last week, and figured I should put my keyboard where my mouth is and take a stab at talking about design process for the real world. First, a caveat: I do think it’s valuable to frame ideal processes so that we know what we’re aspiring to. But often writing about design process has an all-or-nothing tone to it: It makes you feel that if you’re not doing it the “right” way, then you’re not doing good work, and won’t end up with a good product.

So first: there’s no one “right” way to do things. But there are a set of approaches that are generally good practice for user experience and product design: things like talking to your users, making sure to do divergent exploration, getting feedback, and continual iteration. However, it’s rare that I see a designer in a situation where they can execute a design process exactly as they would like to.

Instead, we all end up working in the messy in-between — a place where we need to make trade offs in our process due to real-world constraints. Those constraints tend to be things like:

  • Limited time: Deadlines won’t always accommodate a perfect process.
  • Skeptical stakeholders: People with authority over the project may not believe in the value of a thorough design process and see it as something that slows down the project or adds to cost.
  • The way things have been done before: If you’re trying to grow a design practice in an organization that hasn’t had a strong design or product culture, change doesn’t happen overnight. 
  • Personnel constraints: Sometimes you don’t have enough people or the right people to execute on all the pieces of the design process thoroughly.
  • Budget: This one is self-explanatory 🙂
  • And much more…

So, given those constraints, how do you decide where to cut corners and where to push for more? What’s a good design process for your design process? 

In my experience, here are a few rubrics for making these decisions:

1. Know your strengths and focus resources on your weaknesses.

What are your core abilities as an individual or a team? If you’re really familiar with your intended users, perhaps you don’t need to go as deep on user research, and instead you focus intensively on design exploration. On the other hand, if you have strong UX/UI design experience and instincts on your team, you might be able to spend less time exploring and iterating and more time talking to users.

This piece of the puzzle requires the ability to accurately self-assess. Be honest with yourself about your strengths and weaknesses, and design your process to support you where you need it most. If you have deep experience in one area, don’t be afraid to trust your instincts.

It can feel like sacrilege to say “we don’t need [x] because we’re really good at [y]”, but remember that ideal design processes are designed to check you — to make sure you’re considering options and needs that you might not immediately think of. Deep experience and skill can also help provide some of those checks and balances.

2. Learn to identify the immovable objects

In looking at your constraints, know which ones are fixed and which ones can’t be budged. This is a bit easier with things like budget, time, and people — for example, if you don’t have a budget for extensive user research, it’s clear you will have to work with some guerrilla research tools and approaches. But it’s more challenging to know which cultural pieces are immovable.

For example, you may have a stakeholder who just doesn’t buy the value of a strong design process. Most designers will find themselves in this position at some point, especially if you’re working in-house. Know when not to waste your time on unwinnable arguments. In those situations, there are one of two paths forward. One is that you can find small ways to inject better process and show how those approaches led to better outcomes. Seeing tangible proof of the utility of a good design process can lead to more investment and trust in that process for future projects. The second path is — unfortunately — that some stakeholders just won’t be convinced and it will prove to be a serious constraint on your ability to do deep design work.

It takes time to figure out which situation you are in, but in either case, knowing how fixed your constraints are helps you identify where to focus your efforts.

3. What has to be perfect now and what can be fixed later?

As designers, it’s always crucial to understand the overall product and business strategy for the experiences we’re designing. One of the reasons for this is that it can help to prioritize where to focus resources in our “messy in-between” processes. What features or users are most critical to the success of the product?

Constraints mean that we almost always have to pick things that aren’t going to get as much love and attention as we would ideally like. Can a feature be removed for launch, or is there a scaled-down MVP of that feature that will suffice for now? Which user group has to have their needs deeply met for success? Can other groups’ needs come later? It’s hard not to want everything to be perfect, but knowing what truly has to be perfect can help in focusing limited resources on the right things.

These are by no means exhaustive, but they are a few key rubrics that I frequently use. Most importantly, I hope that we can all share more about how we navigate design in situations that rarely meet the platonic ideal. In doing so, I believe we can alleviate a lot of the guilt and impostor syndrome that seems to be common amongst designers who are worried that they aren’t “doing it right”. Let’s embrace the imperfections of design process in real organizations and projects, and share tools for creating the best work within the constraints of those situations.

Before you make a thing

For his course on Technology & Society, Jentery Sayers has created a document entitled “Before you make a thing” that is a fantastic overview of how to critically approach designing and making with technology. The guide is divided into three sections: Theories and Concepts, Practices, and Prototyping Techniques. Here are a few of my favorite bits:

Examine the “default settings” of technologies; doing so asks for whom, by whom, and under what assumptions they are designed, and who they may exclude and enable. All projects have intended audiences, even if those intentions are not always conscious or deliberate.

Remember that data are produced, not given or captured; doing so emphasizes how this becomes that, or how data is structured, collected, and expressed for interpretation. 

Conjecture with affordances; doing so demonstrates how design is relational. It happens between people, environments, and things; it’s not just a quality or property of objects.

Make a useless or disinterested version of your project; doing so may underscore the creative and critical dimensions of technology and society. After all, not all technologies must increase productivity or efficiency. Consider the roles of technologies in art, theory, and storytelling. 

There’s a wealth of great guidance for both craft and thinking here, along with links to source materials for more in-depth study — go and read the whole thing!