Why surrealist art is a perfect application for augmented reality

Long before face filters and dancing hot dogs could alter the way we see the world and ourselves, there was already a group of people experimenting with the notion of the “real”: surrealist artists. 

In recent years, a number of museums, artists, and institutions have created experiences that combine art with digital supplements, augmented reality, or virtual reality. One such project comes from the San Francisco Museum of Modern Art (SFMOMA), as part of its late career retrospective of surrealist René Magritte: The Fifth Season. 

SFMOMA commissioned the creative agency frog design for the project, asking them for an interactive technology integration that would build upon, not simply mirror, the artist’s work. 

Frog came back with the “Magritte Interpretive Gallery”: The final room of the exhibit, it’s an augmented reality gallery that allows visitors to interact with digital interpretations of some of the Magritte work featured, such as the iconic High Society. Stand-alone windows contain depth- and motion-sensing cameras, and screens in place of window panes integrate the images of viewers into Magritte landscapes in ways that don’t conform to ordinary rules like perspective, distance, and time. Unlike many other AR and VR integrations, there’s no smartphone or VR headset required. The windows are the screens through which visitors can glimpse the altered worlds.

It’s perhaps one of the museum x AR trend’s best, and most intoxicating, executions.

Few artists directly interrogated expectations about reality and art more than the 20th-century artist René Magritte. Magritte is famous for a hyper-realistic style of painting that he applies to esoteric concepts like the difference between an object and the representation of an object (The Treachery of Images), obfuscation and desire (The Son of Man), or our rigid and easily disturbed perception of reality (The Dominion of Light). “I suppose you can call me a surrealist. But one should really say I am concerned with realism … the real with the mystery that is in the real,” Magritte said.

SFMOMA and frog wanted to combine AR tools with Magritte’s philosophy and his art to build upon these themes and go beyond just producing digital iterations of the original work.

“We didn’t want to do a one-for-one replication,” Charles Yust, frog’s principal design technologist, told Mashable. “They’re all interpretations, and the paintings were a jumping-off point.”

As a result, the gallery incorporates the visuals from many of the paintings with an interactive space that furthers Magritte’s themes by letting museum goers inhabit an altered reality. 

“Once you get through all of the chapters of his late career, finally you land in the interpretive gallery,” Oonie Chase, frog’s executive creative director, said. “There, you get to play with these ideas of identity, perception, paradoxes, and visual puns that he is so well known for.”

Looking through a window into the woods of Le Blanc Seing (rough translation: “Free Rein”), you see your reflection chopped up among the trees as if you’re both walking through the woods, and a part of the trees themselves. Peering into the clouds of High Society makes your reflection appear in a far-off window; in another window, looking into a blue sky, only your features appear, recalling Magritte’s Sheherazade. The gallery aims to surprise and prompt visitors to think about why we gaze into a mirror when we always expect to see the same thing. By inverting reality, it shows us how the laws of nature that we so take for granted are themselves extraordinary.

“We wanted people to walk out with a little bit of an uncertainty about the world they’re walking into,” Chase said. “To explore the idea of what is real, and the question of what you give up when you just accept reality as it is.”

To inspire that surprise and curiosity, making the technology non-intrusive was key. Yust’s team had to engineer the room very specifically so the cameras captured just the people standing in front of it, and no more. They specifically decided against using any external technology that visitors themselves would have to handle, like headsets or smartphone apps. And they aimed to make the technology function as seamlessly as possible.

“The overarching theme and way that we were guided was just to make this technology as invisible as possible,” Yust said.

SFMOMA and frog has provided Mashable with a video to show how frog used advanced depth-sensing cameras and motion-tracking technology to create the immersive but technologically seamless environment. You can see the interactive gallery and the tech that powers it in action below.

One recent visitor of the exhibit described the Interpretive Gallery to me as the “cherry on top” of the exhibit. It inserted energy and vivacity, adding an element of delight to the experience. SFMOMA is finding through visitor surveys that the experience is deepening people’s relationship with the art, too.

“These kinds of wordless, playful experiences can have tremendous impact,” Chad Coerver, SFMOMA’s chief content officer, said. “It somehow serves to cement the experiences and bring them closer to the artwork in ways that words can’t.”

Art was and remains a fertile playground for AR and VR as it developed into viable consumer tech. The world of Van Gogh came to life in an immersive VR environment called The Night Cafe. Leading fine art auctioneer Sotheby’s turned Dali and Magritte paintings into VR landscapes that visitors could explore with headsets. In a February 2017 survey of the VR/AR x art landscape, the New York Times declared, and then asked: “Virtual Reality Has Arrived in the Art World. Now What?“

At least part of the answer resides in exhibitions like SFMOMA and frog’s Interpretive Gallery. It’s essential that museums and artists incorporate this technology for more than just the sake of technology itself. These applications have to consider the artists and themes of that artist, and find a way to create something new — not just a digital, virtual replica.

SFMOMA’s Coerver and the team at frog agreed that the Magritte exhibition made perfect sense for an AR integration. Primarily because the capabilities of the technology and the themes of the artist were so aligned.

“Magritte was fascinated by the intersection of reality, perception and imagination,” SFMOMA’s Coerver said. “His paintings are puzzles in which the boundaries between the three are deliberately blurred — inviting us to question what we assume to be true and to be attentive to the mysteries that abound even in daily life. In this way, his art could be called ‘mixed reality’ 50 years before its digital counterpart came into existence.”

Magritte and contemporaries like Salvador Dalí and Pablo Picasso may have had traditional tools like paint and canvas at their disposal, but they were able to create representations of altered realities so affecting that they shocked the world, and changed art forever. Now, tools like AR and VR can extend the idea of altering reality to be more tangible and interactive — to expand the scope of the art beyond the canvas. 

“We said working through this that if he were working today, he would be playing with these kinds of technologies,” Chase said. “At this part of his career, his paintings were augmenting reality.”

AR gives museums a chance to tackle contemporary questions through the lens of the artists they’re showcasing, too. For example, Chase said that frog specifically designed the exhibit to make it difficult to take a selfie. That’s because Magritte was interested in themes of identity, reflection, and obfuscation — and a museum is a place that, especially in recent years, people are particularly focused on capturing their likenesses alongside art. 

“In many of Magritte’s works at this part of his career, he was obstructing identity,” Chase said. “So in the forest, we slice you up, it’s hard to see your full face. In some of the back zones, we have time shifted, so you don’t see yourself unless you move very fast. Another, you walk in front of it, and the camera’s at your back. You have to work very hard to get a view of your own face.”

Despite all the technological innovations, an effective AR installation can show how universal themes explored decades ago remain to this day. And again, in the Interpretive Gallery, surrealism finds a phantom, futuristic limb in AR.

“Magritte’s sensibility and subject matter make his art perfectly suited to exploration in mixed reality platforms,” Coerver said. “But even more interestingly, his art presages the profound philosophical issues raised by VR, AR, and advanced imaging technologies. Where does reality end and imagination begin? What does truth mean anymore in a world where media is routinely manipulated? Are we heading toward a next phase of existence in which IRL, VR, and AR are no longer meaningful distinctions? I think Magritte would be delighted to see where all of this leads.”

Yes, he would — and so are we.

You can see René Magritte: The Fifth Season at SFMOMA from May 19–October 28, 2018.