Synthetic Loopholes

A rock, a river, a building, a score. The Game, when it was still referred to as one, started out simply enough. It was a means of killing time in the third global pandemic in under a decade, when lockdown cycles seemed to merge into one infinitely long stretch of time spent indoors. This was time spent largely alone, loitering inside one’s own personal biosphere: a climate-conditioned safe house removed from all the chaos Out There—raging wildfires, unsurvivable ozone levels, multispecies extinction. A bubble inside which to project the world’s reinvention. A time spent dreaming up alternatives.

In hindsight, I realized the Game’s instant success could be chalked up to the proximity it had to other tools and technological processes that became household staples in earlier pandemics. Ranging in complexity, these spanned from daily digital puzzles, language games, and text-to-image generators to virtual social spaces, AR filters, and motion capture devices for broadcasting one’s physical activity. But above all, the Game bore the closest affinity to the wildly popular “Digital Earth” mindfulness apps, originally billed as self-guided psychodynamic therapy exercises. As satellite-based nature livestreams keyed on yet undamaged parts of the planet, they were designed to assist their cooped-up subscribers better process complex bouts of climate grief during extended periods of lockdown.

Many of the patients I interviewed when researching for the Game spoke of a kind of mind-body disassociation that arose from the sensory withdrawal of Inside Space. One woman emphasized a deep depression caused by the inability to feel the sun on her face in early spring, the crunch of leaves underfoot come autumn, and the cool whip of winter wind biting exposed skin. The body sought out what it could no longer sense, but the brain couldn’t stomach that absence. It was as if the reduction of the senses also shut down the sense of self, negating one’s existence in the world.

In designing the Game, I was asked to replicate some of these scenarios—with the unspoken subscript being to bring some of these feelings back to life. The commissioning body had read up on the basics of game mechanics; they understood the value of conflict and cooperation, a compelling theme and storyline, and the importance of aesthetics and rewards. In a sense, the look and feel of the Game wouldn’t appear out of place in a line-up of popular Extended Play (EP) titles also released that year—virtual worlds made accessible through expensive multisensory body suits. But with the Game’s intended social impact, the brief—and the world it represented—got a lot more complicated.

A recent social psychology paper receiving a great deal of attention at the time championed the idea of “synthetic sensing”—or the process of inducing real feelings through artificial environments. It didn’t matter if the ecological context that stimulated these emotional states was completely fabricated, built inside a game engine and uploaded to the Game’s cloud. In fact, this simulation was preferable; it was better to have little to no reminder of the state of the world Out There. But the pleasant feeling that overtakes the body while observing an 8K palm tree model wafting in gentle southern Californian golden hour light, or the sublime pin-prick feeling triggered by witnessing a pod of simulated whales erupting from a foggy, pixelated rendering of the Pacific Ocean—those emotions are completely genuine. What could these rich emotional states achieve beyond the context of their synthetic environments?

I was instructed to build a world that was oriented around the idea of a shared future. A remarkably outmoded social politics in this day and age, it felt equivalent to referencing the “common land” concept in medieval Europe. But by feeling more together, the Game’s commissioners believed, a shared consensus mechanism, absent from the world in recent living memory, would soon follow. They thought that this abundance of feeling in gamespace could trigger what they called a “synthetic loophole”: a portal into a world of shared affect that had long since been abandoned in the world Out There. I was curious about this idea. I was also out of a job since the latest pandemic began, and had no counter-theory of my own. So I set to work.

History remembers the future; the future is always reinventing the past.

I started working on the Game the year that the world’s population was supposed to climax. Despite the long-settled permacrisis, a state which had been forecasted by climate scientists and economists in the early 2020s, we had managed to clear that figure well ahead of schedule. But it seemed like the more humans that populated our planet, the less invested in its future they became. Perhaps this shouldn’t have come as a surprise, seeing as the burden of imagining the world differently was continually reassigned to the newest arrivals. As the more damning impacts of the climate crisis stretched out to a global omnipresence, people grew further apart, not closer together. Could it be any different in the artificial sensing space of the Game?

Set in a rendered landscape modeled on early computer desktop backgrounds, the Rock level integrates a full moon ritual with the coordination of player’s decisions.

I designed the Rock level first. It was the perfect beginner experience—minimum user effort with maximum reward. Intended to coax even the least likely player into the magic circle of the Game and seed the desire to complete all four levels, the Rock level begins in a picturesque open field: verdant, grassy, green. It’s a place that feels intimately familiar, yet nobody alive on Earth has ever experienced it firsthand. For design inspiration, I scraped the depths of the commissioner’s digital media archives, uncovering a boon of naturalistic desktop backgrounds from computer operating systems dating back to the 1990s. Flattened, ambient, and hyperreal, I wanted to conjure the sense of a space that was forever just out of reach.

The player enters this world just as the sun sets. The first few pinpricks of stars flicker high up in the sky; bleeding backwards from orange, to grey, to midnight blue, it’s hugged tight by the mountains and their thick underbelly of fog. A breeze kicks up in the mountains, rustling its way through the tall grass. If the player purchased the latest model of the Extended Play bodysuit, they will feel this motion as a pleasurable tingle, beginning at the top of their skull and fanning out through the base of their spine. A warm-up synthetic feeler.

Like the rest of the Game, the Rock level requires basic strategy, with an emphasis on the social dynamics of a shared goal. Players must cooperate as they fan out through the tall grass in search of special rocks—gnarled, glittering black chunks of compacted silicon. They don’t know what they’re supposed to be looking for; the task’s instructions are kept intentionally vague, in order to heighten the players’ emotional engagement within the Game. By copying the actions of other players or intuitively feeling out the situation, players will end up grabbing one of the silicon rocks and bringing it back to where they began—a shared clearing with concentric circles scrawled deep into the damp dirt. Ten rocks are placed inside ten rings. Then, they wait.

During the wait, Game time will accelerate; it’s not unusual for ten days to pass in under an hour. When the full moon rises over the mountains on the horizon, the rocks will begin to hum. What begins as a faint tune eventually ramps up into a full-blown song that echoes across the field of scattered silicon. The player’s view pans upwards and out toward the sky. Around them, the trampled grass—a living memory of their shared motions in search of the stones—gradually takes the shape of a crop circle. The pattern slowly loads into view through an upward-tracking camera, rigged to infinity. It hovers above them, propellers lightly buzzing, reminiscent of a hummingbird. Close your eyes. Can you see it?

The River level involves a synthetic Jump from a human to a nonhuman body, measured in multispecies memory and the aberrant flows of water.

Long ago the river used to flood this place. Every five or six years, choked with rainwater barreling down from the mountains, it would break its banks, sweeping all traces of human civilization out to the sea. But the river was not flooding; it was returning. Against the linear march of human progress, water is a shapeshifter; it remembers everything.

The River level is more complex. It requires what I call a synthetic Jump: a move from the player’s physical body in the world into the simulated body of the river. This process is challenging, and can only happen organically through the player’s own tactics; in the Game, there are no cheat codes.

To successfully complete the Jump, the player must undergo two state changes. The first—from physical body to virtual body—is second nature; we’ve all been doing it for years. The second is much harder: the player Jumps from the body of a human to that of a nonhuman, the River. To do this, they must understand that their character and the River are not two separate entities, but rather that they are the same body. When the Jump is completed, there’s no obvious shift in the game mechanics or environment; it’s the same old River. But the player will feel the change through their Extended Play suit: decentralized, it ripples through this shared body, unearthing new sensations at each watery mouth, elbow, and foot.

This ecosystemic body is more than the senses; it is an unraveling current of feeling. Its alluvial flows mark the rungs of nonlinear time we think of as experience. Like tracing one’s fingers absentmindedly across the rings of a fallen tree, it is deep time rendered tactile in death. As you breathe in and out, life flows through the circuit board of your nervous system, replenishing the swimming pool of cells that unities your body with the ecological body of the Earth. The water returns, and the body remembers.

Across the state changes of a weirded climate, people talk about revolutions as if the only thing to do with the past is to break up with it. A hard severance. This is, of course, impossible—and no less so in the simulated environment of the Game as in the world Out There. The trick is reorienting the map. Making people see differently what’s already there.

Early on in my work as a game designer, well before building the Game, I learned about the importance of failure. Not in the sense of making a bad game that nobody played, but how failure functions in the game world; what other worlds it opens up. Historically, there were endless arcade games like Tetris or PacMan in which you can’t win, so you just learn to lose more slowly. The inverse, of course, are classic RPGs, where even after all the quests have been completed and all the badges have been earned, you’re left to roam that open-world environment which has shrunken from kingdom to container throughout the duration of gameplay. No closure, no satisfaction; a space where winning gradually starts to feel like its own special kind of losing.

As CPUs and graphics cards grew more advanced, glitches opened up another caveat that was failure-adjacent. I mean the edge-of-world gamespace you’d occasionally tumble into by accident—falling forever until you switched off the overheating console. What would a game that came to terms with its own failure look like? Would the gameplay experience feel like how a Borges story reads? An infinite library, where the universe is contained within the volume of a book; a perfectly impossible system that comes to life through the act of reading it?

By deconstructing a building in the third level, players collaboratively uncover the productive potentials of failure and degrowth.

The Building level opens onto the edge of a lake. The sky is bruise purple, the landscape is ash red; a gigantic hollow sun, rendered in iridescent blue scales, rests half-sunken on the horizon, outlining the silhouette of a magnificent tower. And it’s almost complete! You, the player, realize that it’s up to you to top it out. But, there’s a problem. Every other player in the game is told the opposite: the foundations are all wrong, the structure lacks integrity, the style is already outdated. It must be demolished and started anew.

When you place a brick, a dialog box pops up, informing you that a different brick in the structure has mysteriously disappeared. Every time you place another brick, twice as many bricks as the previous amount are instantly dislodged. 2, 4, 8, 16, 32, 64, 128… soon, the building starts to look like an ancient ruin, its Swiss cheese facade accumulating a steady permutation of losses. In the low score charts, your name flickers in bold lights before your eyes.

In game theory, outcomes are considered to be in Nash equilibrium when knowledge of the other players’ strategies would not lead any player to change their own strategy. You place a brick, and lose again. But the mirror between gaming and reality can sometimes cloud over, becoming a vast, inscrutable void. You remove a brick, and the two surrounding it instantly disappear. Peering through the hollow structure, you catch a shadow of movement on the other side. In the real world, outside of gamespace’s unique rule book, sociological studies show time and again that cooperation is commonly chosen in groups—even when the betrayal of others could lead to a better individual reward. The hindering of progress suddenly feels like an open door, an invitation. It’s human nature to act together; it just takes the synthetic environment of a game world to act as a reminder. You pull more bricks, and the faces of the other players begin to emerge. The building shudders, distorted, and the tower and level go down—together.

The fourth level of the Game focuses on the creation of shared songs between players and nonhuman life on Earth.

Seen through the eyes of a machine trained on Anthropocentric belief systems, any vision of the future is a frankensteined regurgitation of the past. The world that players thought they were waking up to was a world that had already existed. Far from auguring any sense of prophetic culture, our technological advances lodged us in a recurring feedback loop. How can the cycle be broken? By getting away from our human-centric mythology and entering into an Earth-centric one.

When I made the Game, I wanted to get away from the totalizing authority of language. As I strapped myself into an EP suit to play a prototype of its first three levels, I realized the answer lay within the synthetic sensing brief. By shedding language and its neurotic impulse to make meaning, players could tap into a deeper understanding of the self and planet—somewhere beyond the Cartesian split of body and mind and the Anthropocentric divide of human and nonhuman. To my surprise, as I worked towards the completion of the Game, these two goals—my own and its commissioner’s—became increasingly intertwined. Sound seemed like most collective sense, and music the most intentional form of listening. So it was obvious that the Game’s final level should involve the collaborative creation of a score. Within the synthetic environment of the Game, this score could foster not just a deeper connection among players, but become a bridge between human and nonhuman worlds.

One day, a drought sees the mountains go up in flames; the next, a flood wipes out the last chunk of the old town. How to make sense of a world that makes a mockery of historical precedents and a posteriori knowledge systems? It was clear we needed new models. Out There, ecologists pioneered an AI-based data visualization approach known as State-Space Modelling (SSM) to map the increasingly aberrant trajectories of a weird new climate. As a type of model that yields to measurement errors across ecological time as it grapples with the messiness of live data, SSMs felt like an important metaphor for better understanding the conditions of the world we live in now, something that stretches beyond the paralyzing doomer mentality of climate forecasting.

In the stale recycled air of my own biosphere, I’d spend countless hours scrolling through online SSM collections, finding an odd sense of satisfaction in their wild, frenzied lines. The superimposition of sea lion migration patterns on the Pacific Coast on top of Colorado River water levels from the last two centuries; the movements of the two remaining Mountain Gorillas across high-altitude forests in Uganda transposed onto the growing surface area of synthetic banana plantations cropping up around the country’s western plains; an Arctic Tern’s flight pattern mapped onto the development of 3D-printed bamboo infrastructure in Hong Kong. There was a mysticism to this data interpolation, a suggestive interconnection that humans couldn’t yet decipher. To me, they hinted at a kind of lingua ignota, an unknown language that stretched beyond the proxies of human imagination itself.

As a nerdy tribute to these beautiful, impenetrable diagrams and their hidden logic systems, I decided I’d run some of them through a sonification script to turn this ecological data into music, engineering my own multi-dimensional state-space model in the process. Players would, of course, be free to improvise in collaboratively creating their scores, but the musical backbone guiding the Game’s final level would come from the Earth itself. Real time data, energy, and movement of the planet’s nonhuman life were recorded from the world Out There, then communally synthesized into song.

The Game’s AI-powered engine enables players to create unique musical scores from live ecological data and sounds encoded from the world Out There.

Unlike music, humans once wielded language exclusively; they used the technology as a cognitive infrastructure, modeling the world around them and reinforcing their primacy over other lifeforms as they went. But through the rapid advances of AI-powered natural language processing models, language became sculpted more by machines than their human counterparts. Eventually, language became dislodged from the question of consciousness. The narrative of human exceptionalism that it towed around became a moot point; language was as synthetic as the artificial intelligence built to model it.

During this time, humans turned increasingly toward non-linguistic tools of communication, like music, gesture, and movement. They solicited the senses to reanimate what language closed down a hundred thousand years ago: altered states that transcend the vague approximations clawed at by words. In this era, the wordless transfer of complex emotional states and dreams through real-time gameplay became second nature. By listening to these climate songs composed by an interspecies AI, people became better communicators with the second body of the Earth. A shared ecological intelligence, co-authored by human, nonhuman, and machine, gave way to a new mythology for the planet.

Through the synthetic sensing environment of the Game, the breakdown of language leads to the collaborative creation of a new shared world.

Thanks to the multispecies musical apparatus of its fourth level, the Game really took off. A powerful feedback loop emerged between the songs generated in the Game and the score of nonhuman life on Earth; the interspecies modeling of gameplay slowly diffused into reality. Out There, nobody was sure who was doing the mapping and the modeling anymore—human or otherwise—and it wasn’t long before those distinctions started to break down, too. As the world Out There and the world of the Game morphed into mirror images of each other, the climactic conditions that separated human and nonhuman worlds lessened in their extremity. People began stepping out of their biospheres: tentatively at first, their exposure to the world limited by their EP suits, and then uninhibited, discovering a ready wayfinder in the sensory communication with rocks, rivers, wind patterns, and other ecological agents that they had built up in gameplay. The synthetic feedback loop of the Game gradually cultivated a new way of being and sensing with the world; Inside Space and Out There became one.

Shifting baseline syndrome describes a gradual change in how a system is evaluated, usually cross-checked against previous reference points. However, this slow gradation of change often registers as a non-event; short-term memory occludes the large-scale shifts made visible across deep time. A river, seemingly on course for survival, dries up over the years. A building, continuously redesigned to withstand a future threat, is never completed. A World, reduced to a word, gradually adopts processes of alternative meaning-making in order to transcend itself, language.

By tapping into the sound of the world, the Game’s players unlocked an ecological consciousness that could be likened to deep time. The cause-and-effect model of recent history became obviously aberrant, something that an ecosystemic awareness could course correct towards a mutually survivable future. The artificial sensing space of the Game created a synthetic loophole between players and nonhumans, between a virtual environment and the Earth, auguring a new mythology in the process. I’d like to think of it as an answer, but in reality, this loophole opened up still more questions: meaning-making is always a two-way street, a dance between storytellers and their audiences, like movements in an improvised score. To what new shared worlds does this song lead? It’s impossible to say, but easy to listen.