Marina Abramović performance from augmented reality studio Tin Drum to premiere on Magic Leap

Marina Abramović and Tin Drum’s Todd
Eckert.
Tin Drum

• Technologists think that augmented reality is the next
  big thing that could replace smartphones, TVs, and all the
  screens in your life.
• Alumni from Magic Leap, a hyped augmented reality
  startup, are now using the technology to create music and art
  pieces.
• Their first work stars Marina Abramović and will
  premiere simultaneously to 50 people in London in
  February. 

Marina Abramović, the renowned performance artist, is going to be
the star of a piece premiering February 18 at the Serpentine
Theater, in London.

But she’s not going to be there in person — or, at least, she’s
not performing live.

Instead, she’s going to be performing in the lenses of 50
augmented reality headsets simultaneously, all of whom will be
wearing AR hardware, including the Magic Leap One, a hyped pair
of smartglasses that launched earlier this year. 

“Up until now it has been impossible to understand what the
artist does and the purpose of the event is unless you were in
the room at the time of the event,” Todd Eckert, founder of
Tin Drum, which is producing the piece, told Business
Insider. 

“As a performance artist, my work is about energy, so the work
we’ve been doing here, what is the chance that I can really keep
energy that can radiate to the public, and you can feel
it,” Abramović said in a video starring her 3D avatar. 

“This is a really promising media,” Abramović continued.

What is augmented reality? 

Magic Leap
One.

Magic Leap

Augmented reality is a technology that, if you listen to
investors in Silicon Valley, might one day be able to replace the
smartphone and every other screen in your life by showing
advanced computer graphics seamlessly mixed with the real
world. 

But Tin Drum, a studio partially staffed by Magic Leap alumni and
other technologists, is betting that before augmented reality
becomes an everyday technology, it’s going to become a new tool
for artists and directors to create works that can’t be displayed
on a traditional, flat screen.

With this most recent piece, “you have the ability to feel what
performance is like, not as an artifact of something that already
happened, but the energy of what’s happening now as if you are
actually there at the time of the event,” Eckert explained. 

Eckert first started thinking about augmented reality and its
challenges when he worked for Magic Leap as its director of
content development. There, he reached out to a variety of
musicians and artists to get their ideas on how to transfer a
live performance into a wearable augmented reality experience.

So, after he left Magic Leap, he decided to start a production
studio focused on augmented reality projects — and the Abramović
piece is Tin Drum’s first public exhibition.

There are lots of challenges when recording a person in three
dimensions, often called “volumetric capture,” he explained. It
usually requires an array of cameras, as many as 32, as well as a
lot of care and time to put the images together in a way that can
be placed into a real-world environment. 

Other challenges include how colors are represented in augmented
reality — red looks very different than, say, black, inside a
headset that’s based around displaying graphics inside
transparent lenses. And each different augmented reality headset,
like Hololens or ODG, also requires tweaks. 

“The positioning of cameras and the specific lighting attributes
during the time of capture is very specific based upon each
project,” Eckert explained. To create
the Abramović piece, Tin Drum teamed up with 4DViews, a
volumetric capture company in France.

Marina
Abramović recording sound with Tin Drum
employees.
Tin
Drum

I was able to see a short preview of the Abramović piece on a
Magic Leap One, although I was nowhere near London. Inside the
smartglasses, you could see the artist, and although there were
technological limitations including the headset’s field-of-view,
cutting off parts of the virtual people in frame, I felt her
“presence” in the room with me — but I was just in a conference
room in New York. 

But simply capturing a 3D image of an artist is only the first
step. There’s lots of post-production that’s needed in order to
take the raw footage and turn it into a graphic that can be
integrated with the real world inside a pair of augmented reality
glasses. Then Tin Drum’s specialists, who are technical and
proficient in software like Unity, tweak the footage in Tin
Drum offices until it’s perfect.

Tin Drum wants to produce other high-quality mixed-reality
experiences, and Eckert said the company is close to raising
venture capital so it can continue to develop technologies for
volumetric capture.

Todd Eckert, Alysha
Naples, and Abramović. Producing a volumetric image often happens
in a green room, so the background can be removed from the
3D-image.
Tin
Drump

After capturing video of a subject in a green screen room, the
background can be removed, and the video can be placed in the
real world, like this: 

Going forward, Eckert believes that in a new medium like
augmented reality, it’s going to be more important to experiment
with completely new ways of thinking about media, as opposed to
shoehorning older modes of thinking onto new technologies. 

“I think people have a tendency of trying to use what they know
and in order to kind of embrace whatever they perceive as new,”
Eckert said. “This is recognizing that the technology can do
things beyond anything else that’s ever existed.”