SXSW is perhaps one of the most wide-reaching festivals in the world as far as the scope of coverage available to experience. With its roots in music, the festival has expanded beyond to showcase innovations in education, film, and just about every facet of media imaginable. For those fascinated with the rising opportunities of XR and its modalities, SXSW has a lot to offer and some truly unique experiences for attendees.
Modalities of XR
XR, or extended reality refers to the use of technology, usually digital, to extend, mirror, or enhance an experience. It’s an expansive term that covers a lot of ground and applications. Your phone’s real-time GPS tracking and guidance function is a great example of practical XR. So is a 360° digital tour of a campus or hospital. XR divides into two major categories, virtual reality, and augmented reality. Some argue that there are more modalities, but as the universe of XR expands, the lines between any distinct categorization blur.
Virtual Reality
The most familiar form of XR is virtual reality or VR. VR is the use of a completely simulated, digital reality. The first development of VR began in the 1970s and the technology has been famously used for flight training in simulators. Star Trek: TNG brought the concept to Trekkie pop culture by introducing the Holodeck, a fully simulated environment for television characters to live out fantasies. In the 1990s, VR became a viable and exciting field in video gaming as CGI gained ground in sophistication. The 1992 sci-fi-thriller The Lawnmower Man brought the concept of VR to the big screen.
Now VR is synonymous with simulation headsets. The technology company, Oculus VR Inc., dominated the industry and led the way in affordable headsets for home VR play. In 2014, Meta, then Facebook, purchased Oculus and continues to expand the repertoire of games and experiences to living rooms across the world.
The most obvious feature of VR is your ability to have a 360° field of vision while watching a movie or playing a game. In other words, you can see what is going on above, behind, and to any side of you. Thus, gaming takes on a new level of immersion, such as watching a film.
SXSW VR Experience: Behind the Dish
Behind The Dish is a three-part docuseries about women chefs who have thrived despite personal adversity and rampant sexism in the professional culinary world. Three 10-minute movies were captured on 360° film and act as an intimate biopic of three different chefs.
VR studio TARGO aimed to create a 3D perspective of food as prepared by each chef. The footage of the dishes was shot in an extreme close-up. The effect when watching in the VR headset, provided by Meta is stunning. Of the three documentaries, I was able to watch Yumi Chiba’s very intimate and beautiful story. As one of the very few female Sushi Masters, Yumi not only overcomes being a woman in her field but also survives breast cancer as she continues to perfect her craft. We follow Yumi to her fish market as she selects live fish to prepare and serve each day. Then we’re off to the wasabi fields, where she hand-picks the freshest wasabi to offer with her delectable sushi.
One of Master Sushi Chef Yumi Chiba’s culinary creations from “Behind The Dish”
The other chefs featured in the project include 6-Michelin Star French Chef Helène Darrose and African American Soul Food Chef Deborah VanTrece. Each of these three superstar chefs and their cuisine gets fantastic treatment in Behind The Dish.
Directed and written by Jiabao Li, Once A Glacier is a form of VR conservationist poetry. The experience begins in an animated waiting room of sorts. You can move about the room using your handpiece by pointing to an area. Click correctly and you zoom toward your destination. The handpiece was tricky for me to use and some of the details in this area were skipped. The point of this “room” is the set the stage for the main experience of the installation. There were educational stopping points to read and learn, priming the viewer for the main event.
Once the central journey of Once A Glacier activates, you find yourself in a beautiful blue-and-white world, as young woman kayaks among glaciers. Soft soothing sounds surround you as you glide along. A poem is recited into your head and some of the writing is scrawled into the sky in white flowing lines.
“Once A Glacier” features immersive animation
Decades pass in the story and we are again in the company of the same woman, now aged. She is lamenting the melting of her beloved glaciers. As the experience draws to a close, you are beckoned to pick up a virtual watering can to add water to a small remnant of a once majestic glacier. This represents your commitment to joining the conservation movement to preserve the natural beauty of the glaciers.
Extremely abstract and trippy, Once A Glacier is a good example of how VR can be more of an art piece than a straightforward immersive film. Fully animated, ethereal, and strange Once a Glacier is as different a VR film as can be to Behind The Dish. As such, these two examples of VR exhibit just how vast the range of possibilities can be in this field.
Augmented Reality
The second form of XR is augmented reality or AR. The main difference is that AR adds more sensory experience. For example, Mission: SPACE in Epcot is an AR experience because the ride creates a 2.5 G-force on your body during the launch portion of the experience. The physical manipulation of your body during the ride adds to the visuals on the screen and augments your reality. Creating the G-force sensation is an example of somatosensory stimulation, where your body perceives a change to its status, albeit a false one.
AR can also incorporate touch by having different items in the environment to feel. This is called haptic stimulus. Taste and smell are also options in AR dependent on the design and purpose of the experience.
SXSW AR Game: Time Investigators
Developed by Picture This Productions Time Investigators is a multi-sensory, mystery game. It incorporates scent, sound, animation, and actors through an interactive environment using a smartphone as a guide. You play as an investigator for his Royal Highness Henry VIII assigned to uncover the reason for the sinking of his ship, the Mary Rose in 1545. Using research based on actual historical evidence, the game leads you through a trail of clues including interviews with members of the crew. It even features sketches of actual crew members based on their skeletal remains.
A sketch based on skeletal remains from the Mary Rose that serves as a clue to the mystery in “Time Investigators”
As you follow each clue, your phone prompts you to scan various sections of the game area to move the story along. It progresses much like an escape room, where each clue pushes you to the next step. The player wears a backpack that dispenses special blends of scented liquid. As the game app senses your progression to various environments in the game, it releases the appropriate scent. For example, when you encounter sailors drinking, you smell the fragrance of beer.
Educational and thought-provoking, Mary Rose’s demise is unveiled bit by bit. The game finishes with a bird’s eye view of animated footage of her sinking. The possibilities here for amazing educational experiences are clear in this project, directed by Charlotte Mikkelborg. Time Investigators is an immersive history lesson that becomes unforgettable through the interactive nature of solving a mystery. This is a brilliant concept and one that makes textbook history something vital and fascinating.
The room of clues for “Time Investigators”
SXSW AR Experience: Symbiosis
Strange and haunting, Symbiosis is the most elaborate experience from the four covered XR Exhibits. Designed by Polymorf, it represents the ultimate augmented reality experience as it encompasses every sensory option.
In the storyline, it’s two-hundred years into the future, and the world as we know has changed due to climate change. A new chapter in biological diversity has flourished from artificially enhanced evolution. The participants experience a short journey as a polymorphic organism. Each creature is a genetic hybrid of two or more animals. There are six uniquely different organisms to choose from and each person has a different storyline that converges with each other in the conclusion.
The future landscape of Earth in “Symbiosis”
In addition to the traditional headset and earphones, six different body suits emulate the anatomical features of each of the organisms. If your assigned organism doesn’t have arms, for example, the suit binds your arms to your body. Air compressors fill various balloons in the suit to mimic the movements of each creature.
These suits release fragrances as you encounter different objects and environments in the experience. Food is served directly to your mouth when your creature finds an edible item in the virtual world. Adding to the sophistication of Symbiosis, is the release of oxytocin hormone in the form of a vapor, bringing a hormonal influence to the experience. Oxytocin creates feelings of love or affection. The creators use this feature to trigger good feelings when interacting with Symbiosis.
The Camilla suit for “Symbiosis”
When offered a choice of creatures, I chose Camilla, a Monarch-orchid-human hybrid creature. We removed our shoes and a technician assisted each of us into our body suits. My suit included two poles which I held in my hands to emulate antennae. We could move about as we wished. However, the exhibit space was small and air compression tubes were scattered around the ground. My technician warned me not to swing my “antennae” too much. The suit applied pressure around my legs and thighs to simulate flight positions during the experience.
In the Camilla version of Symbiosis, a female narrator speaks in the first person as Camilla. She describes the intimate journey of transitioning into a genetic hybrid as you fly through the world. You fly past various creatures, including humanoids with dragonfly wings that seem to be part of small tribal societies. At several stops, the Camilla narrator mentions wanting to drink or eat something. Unfortunately, the edible items were not available for us, so I was unable to experience this aspect of the project.
The Camilla Monarch-Human-Orchid creature in “Symbiosis”
Although the digital images were a little glitchy in some parts, the overall effect was amazing and truly bizarre. Camilla moves towards the story’s objective, which is to commune with the 5 other creature characters at the end of her flight. This was a little confusing as I didn’t quite understand what was happening at the end.
To fully understand Symbiosis, it’s probably a good idea to try out the different creatures. Due to time restraints and a long line of people waiting to suit up, this was not an option. It’s probably safe to assume that these technical issues come from relocating an intricate project of this kind. Considering the sophistication and details involved in this AR, a generous amount of leeway is due for any small glitches. Otherworldly and thought-provoking, Symbiosis transport you to a wholly different time, place, and body.
In Conclusion
With limitless possibilities, XR is a frontier in experiential storytelling that deserves much more attention than it is receiving. SXSW is wise to shine a light on these new concepts in education and entertainment. The more support that pioneering production companies have, the more incredible these experiences will become. It’s magical to think that the Holodeck may become reality, and with XR technology, we are seeing that very concept come to life.
