Time Detectives AR by Picture This Productions is a site-specific AR trail experience meant to be seen at the Mary Rose Museum in Portsmouth, UK, but you can also experience it remotely as a table-top experience at home. Time Detectives AR asks users to investigate the sinking of the Mary Rose in July 1545 using a mobile phone as “a magical spyglass to reveal secrets from the past and complete your mission for King Henry VIII.”
The site-specific version has multi-sensory component using a backpack scent dispersal at 3-4 spots on the AR trail. I spoke to Charlotte Mikkelborg about her journey into immersive storytelling, and the process of developing this AR storytelling experience designed to provide families a more interactive experience with additional layers of story.
Rough transcript is down below.
Download the audio here.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
MIKKELBORG: My name is Charlotte Mikkelborg. I have been working in immersive storytelling since I think 2016, 15, 16 around there I started making 360 films and then I kind of played around with 360 films that would also work as multisensory experiences using more senses than just the audio visual. And then I moved into interactive VR with a big piece I did for British Airways to celebrate their centenary called Fly, which is very exciting.
It won best domestic experience in the UK based by the UK government in 2020 and then coming into COVID. I was just looking at the fact that there was going to be the kind of budgets we’d had to make something like fly available because obviously everyone was kind of tightening their belts and we just got to a point where people were getting to experience amazing immersive experiences.
The technology was there, and so I didn’t want audiences to lose that effectively, also us to lose our ability to tell their stories. So I started thinking about whether delving into AR because AR because you’re not building obviously those entire walls, but just elements of those worlds is naturally going to be less expensive to make. So yeah. So I started exploring with a prototype of this experience what we could do basically with augmented reality. And this was my first sort of experience of storytelling in augmented reality.
BYE: Okay, yeah, maybe you can give a bit more context as to your background and your journey into XR.
MIKKELBORG: Yeah. So, so I was originally a BBC foreign correspondent, so I was at BBC a one of the BBC China correspondents until 2009, and I left the BBC to start making longform because I was really just doing news and current affairs. So I left the BBC to start making longer form film, mostly documentary. I made my first sort of feature length documentary film in 2009 2000 173.
I also was entertaining me on a fiction film, although I was not directing. That was like producing it starring Matthew McConaughey, which we made around the same time. So I was nine on one grand jury at South by Southwest, but I was working largely in documentary film between leaving the BBC and moving into the world of kind of XR in 2015-16.
And the reason I moved into XR I guess, or the kind of the tipping point was I was having a meeting with a producer, Tragedian Connolly, in LA. He’s the producer for Lucy Walker. He’s an Oscar winning documentary maker. And I asked what he was working on, and he’d mentioned she was working on this 360 film. And I said, Oh, lots, 360 film in this at home.
So he explained it to me and showed it to me and I said, Oh, wow, this makes a lot of sense. So obviously, as we know, the tech wasn’t there, but just that idea that instead of experiencing film on kind of a rectangular screen, just because that sort of technology had gotten us to, we could now experience it in the round. The way we experience everything else in life just made sense to me. So I figured that this was going somewhere and this would be worth kind of exploring in more depth.
BYE: Yeah. And the first piece that I had a chance to see of yours was Fly that was remote during the pandemic. Everything that was going to be at these film festivals started to be in these virtual film festivals. So I saw a virtual representation of it, which is just the pure VR part. But I understand that there was a whole other installation component that gave it this whole multi-sensory experience of it.
So maybe you could give a bit more context as to that project of Fly and how you started to move from working in 2D storytelling into more of this specialized experiential and multi-sensory storytelling.
MIKKELBORG: So, so yeah, the idea of Fly was we wanted to take you on this journey through when humankind first really started to think critically about whether it was going to be possible to get humankind into the air. So we decided to start that with a young Leonardo da Vinci, who, of course, did not well arguably did not succeed in sustained human flight, but did make some serious inroads.
And so we start with him, and we know from his childhood diaries that he had this recurrent childhood dream, that he would turn into a kite or a kite would land on his chest. The bird that is not the one that you fly in the wind and that you would kind of take on the spirit and fly and take flight.
So this kind of plotting out that storyline and all the way back to Leonardo and eventually, you know, 100 years into the future of Fly at the end of the experience, what I knew was that I wanted people to actually feel that they were flying inside in key moments in this narrative. Otherwise it just wasn’t going to have the awe and wonder, I guess, that I wanted it to have.
So I was looking at different sort of flying rigs and I came across this thing that Sandra Bullock was put in for gravity. So I got talking to Neil Coupland, who won the Oscars for Gladiator back in the day, actually one of my favorite films and Gravity more recently. And we looked at that bag and of course, it wasn’t ever going to be practical actually, because it took ages to get into and then land kind of had to use it.
So then we started looking at motion platforms and whether we could actually make a motion platform, genuinely feel like you’re flying because quite often they use more of, as, you know, the kind of special effects in the movies where you’ve got a helicopter crashing in the platform, spinning or whatever. But whether it can actually give you that sense of elevation.
So we kind of started playing around with a platform that the Neil’s team had won of. Neil’s Right Hands is a man called Glenn Winchester, who I work with most closely and actually feeling that yes, we could when we, you know, did certain things that we could give you that sense of levitation. And we started designing a kind of lean boards that you would lean your body down in to when we wanted you to be prone and obviously flying like a bird.
And that opening sequence with a young Leonardo. But then later, as planes sort of evolved and you went into this more sedentary position that you could then sit back but you didn’t come out of a headset. We just plotted those things within the virtual space so that you could accurately find them and sit back down. And around that, I envisage this large egg and the egg was was playing several levels.
Obviously, it was from a health and safety perspective. It was kind of keeping you away from the moving platform. We didn’t want people getting hands on things, but going to storytelling perspective, it was kind of a metaphor for when human energy is directed in the right ways, we can be capable of incredible things. It’s not always directed in the right ways, but that’s without a story.
But, you know, the fact that we could go from obviously being a land based animals to fly was quite credible. So I wanted this to be kind of a metaphor of humankind making flight possible. So if you approached and put your hands on the front of the egg, it would warm to the touch like you were incubating it effectively with your hands.
And we had this kind of for lack of a more technical expression, kind of like a rainfall shower of sound that came down on you and you could hear that incubation and that heartbeat of life within. And then eventually the sort of cracking open of the egg and the color of the egg also change from this, you know, sort of frigid blue to to a warmer reds as you were warming and eventually to a kind of bright white as light flooded in.
But it was also the egg was also a little bit of a play on Leonardo when he went into his real fever pitch obsession with flight. He was also painting a painting called Later in The Swan, which has since been stolen. We don’t know where it is in the world, but the sketches still exist. And, you know, it was this human woman birthing like, you know, eggs.
The picture is an egg. The, you know, the eggs of her feet and the cracking open and revealing human children with this kind of, let’s say, a bit of a play on that. And there was a second part that you wanted me to answer, Kent, and I’m just trying to remember what that was.
BYE: Oh, well, you know, I guess we’re speaking here in the context of your journey into XR and immersive storytelling and the multi-sensory part of it, and you’re releasing a new experience of the Mary Rose ship that sunk nearly 500 years ago. And so the whole multi-sensory aspect of it and what you thought the multi-sensory dimension of it was adding to the process of telling stories as you’re moving from 2D and to the more spatial aspects, but also the multi-sensory aspects.
MIKKELBORG: Yeah. So as you know, the beauty of VR is that you do have some of these complete attention. They’re in the headset, you’ve got their entire peripheral vision cone, which happens very rarely these days in any aspect of life. Most people are on their phones while watching TV, while doing so, I guess. And the beauty of that from multi-sensory storytelling perspective is obviously that they then don’t see if you’ve placed cunning little devices to make it feel like the sun is shining on their faces or that the wind is blowing.
But one really important scene in Fly is the Wright Brothers scene, where you become Wilbur or you take Wilbur Wright’s place on the wing of the Wright Flyer and you get to pilot that first human powered flight. So I wanted to try, like I was saying before, because we didn’t expect the kind of budgets that have been available to fly to be available, coming in against COVID, to see if we could still play with the other senses while actually just playing an augmented reality kind of game on the phones.
So originally we would try to R&D, these notes of wearable scent badges effectively and actually the R&D on those wasn’t successful. I’m not to say that it couldn’t be. I’m sure that we’ll get there, but within the time frame it wasn’t possible. So we went to actually a device that you do out of school backpack that are hidden under the strap and the scent kind of emanates. Its dry scent.
It doesn’t kind of spray in your face, but it emanates of here on the chest. And scent was important to me in particular because I think probably of obviously all of our senses absolutely key. But scent is the only one that speaks directly to the limbic parts of our brain. And the limbic part of the brain is the part that forms memories.
So it was that opportunity to possibly, quite literally, create more memorable experiences. I mean, obviously the other senses do make it to the limbic part of the brain, but they just do it more slowly and more indirectly. And so we wanted to see if we could bring that sort of personalized sensory experience together with the audio visuals of the game to make for more immersive augmented reality experience, basically.
BYE: Yeah. And so as we start to talk about this new experience that you have, I’m wondering if you could give a bit more context as to the Mary Rose and the museum and how this is both a site specific experience, but also if people want to see it at home, they can see it at home. I personally saw it at home. I have not been able to see the site specific in Portsmouth, United Kingdom. So maybe you could give a bit more context as to the museum and how this project came about.
MIKKELBORG: Yeah, sure. So the Mary Rose was Henry the favorite warship, and she was perfectly seaworthy for the best part of 30 years and then kind of sank all of a sudden very quickly, a summer’s day, 19th of July 1545, when she was literally just starting to go into battle against the French and she sank, killing almost all of the 500 soldiers and sailors on board.
And why she sank has largely remained a historical mystery. So that was quite exciting for something that ultimately we’d sort of branded Time Detectives. We wanted to tell you it’s these kind of time traveling detectives, as they tell you in historical mysteries, is pretty much what we were looking for. So it was a great fit from that perspective. The boat itself was raised.
There’d been a few diving missions, including one immediately after the sinking in 1545, a few diving missions down to the wreck. But she was brought up out of the ocean in 1982 and 40 years ago this year. I’m, in fact, the lead historian on the boat is still there today with the lady who put up is still the lead historian in the museum.
Her life has been the Mary Rose and she knows everything there is to know. So she’s been an incredibly valuable resource, obviously, in developing the game. When we got the funding to make the first episode of Time Detectives, we actually didn’t get it specifically for the Mary Rose. And so we were talking to a few different venues, including like the usual suspects at Tower of London, Hampton Court, Shakespeare’s birthplace and some others.
And actually, it really became the Mary Rose because we were given quite a limited timeline to develop the game. We had six months if we wanted to make use of the funding we were offered. And so the Mary Rose was slightly smaller team, they were able to be a bit more nimble. And so, yes, if you visit the Mariners Museum, which is out in Portsmouth in Portsmouth’s historic dockyard, still on the seafront. And they also have The Victory, which was Lord Nelson’s ship there.
And a couple of the historical monuments. They built this sort of custom built museum around the wreck of the Mary Rose. And she is obviously humidity controlled and she’s kept sprayed down with chemicals because she is the walls, but she’s within this custom built, very modern museum space — a very dark museum space as well. So that was had its own interesting challenges with developing AR and AR markers and all of that.
But, you know, the wonder of her is that despite several damage to the site, because she was in such a busy shipping lane, in a way, the heart of how the sank under the sail, under the sands got incredibly well preserved. And she is pretty much half a ship these days because a lot of what was above there then got battered away by the combination of the waves and other boats going over. And also the people, unfortunately, who died on that side.
And obviously that was probably more than half because that’s the way she sank were also kind of down there. The skeletons were down there. They found 179 skulls of the 500 anyway. But also lots of objects, like a lot of the cannon on board, a lot of personal items. So she really is such an amazing time capsule for developing a game around.
Did you want me to talk about how the museum game is different to what you would have experienced at home beyond the obvious scent component?
BYE: One, to set a little bit more context to, because I do want to know a little bit more in terms of as you go into the museum, this feels like it might be an add on, like where you would download the app and maybe buy the in-app purchase to get access to the content. But also there’s a backpack component.
So as I go to different museums, there’s often objects and as those objects are there understand that the Mary Rose has a lot of from the Tudor period in England, lots of different objects are on the boat. And so there’s a lot of historical capture of moments from history that people are to look at. But there’s also the ship that’s there.
But I guess if you’re a kid or if you’re someone who is going in there, maybe you don’t have a lot of interest in this. But it seems like this app would maybe be a way for either young adults or youth to be able to have an interactive AR app that has a bit more of a narrative component that they could have an interactive experience of this museum rather than is going in and looking and reading all the placards.
And so maybe you can give a bit more context if that was the catalyst or if you came to them saying, hey, we want to use this new technology. But if you’re targeting a specific demographic of people who may not otherwise want to go to the museum and read all the stuff, because if you go to a museum, there’s plenty of stuff to look at.
But maybe this is a new way of having immersive technology to give new access to it. So, yeah, if you could maybe start there and explain the workflow and what’s different than what people, if they were to see it at home, what they would experience.
MIKKELBORG: Sure, lots of things. Yes. So when we were designing this, we were aiming to create a family based experience largely because we started that development during COVID. And when we were prototyping. We’re actually prototyping it around the outdoor trail based on a different side of a place called Otford’s Palace, which Henry VII’s lost Queen’s Palace. And it was an outdoor trail.
And you could go as a family or it doesn’t have the family group to be a group of friends. But we kind of, you know, modeled it around that family experience. And in fact, when we ran tests on that site, we we expected families with kids between 8-14. And we actually had a lot of requests from 16, 17, 18 year olds, which really surprised by pleasantly surprised.
And when I was kind of asking them, Well, why did you come? They were like, Well, we just never heard of like a multi satisfying, I think for them to understand that we love this idea of scent. So that was interesting. So it felt like it tapped into a demographic that is notoriously difficult. We were kind of aiming at that audience.
But also I think that the US has a similarly bad record, as the UK does, in attracting as diverse an audience to the museums and the cultural heritage sites that there are versus the diversity of society at large. Like here in the UK our record for that is appalling, like we see much less diverse audiences in museums. And interestingly, also in terms of if you look at someone like the National Gallery, their online following is a lot of young people.
But in terms of the people that come through the doors, that percentage is tiny, you know, the majority of over 55. So there’s a real disconnect, right, for both young people and I think to more diverse audiences. So we wanted for the app to help. Obviously is is one small part of, you know, moving towards solving these massive issues.
But if we could tell stories that felt more diverse, but also in a way that was exciting and immersive and that appeal to younger audiences, then we could, like you say, just bring new people through the doors. Because it is a really cool thing to explore. But like you say, it’s not necessarily the thing that you’re automatically going to think of on a Saturday to do.
So yeah, but at the Mary Rose is also being quite forward thinking I think as a museum, I mean they’ve anyway got the very modern space because it was only built 11, 12 years ago as you go in. And now the company, Figment Productions, has produced an immersive experience which kind of replicates the experience of the sinking to a degree. Obviously it’s not super distressing, but it’s more of a projection mapping.
It features a hologram of Henry the VIII. So they’ve invested something in, you know, immersive experiences already. But the game, like you say, you can download either before you come down and when you get there is kind of an add on. But a lot of people who visit at the moment are people who have like annual membership for the dockyard as a whole. So you might come one day do Mary Rose one day and do Lord Nelson’s Victory whatever. So now they could just come again and play the game because it does just show the whole museum in a completely different light. Like you’re really on a story journey. You’re not just on that looking around the museum journey because it is just picking out certain elements.
It obviously isn’t showcasing huge amounts of the collection. It really is one storyline that kind of leads through. But in terms of what the game does for the museum. I mean we looked at examples like obviously Pokémon Go and Harry Potter Wizards Unite and other trail based AR stuff that was out there. And certain aspects of those games are great.
But what was dissatisfying for me was I really wanted more photorealistic characters, for example. So that was one thing we wanted to do in the museum. If we could try and place these characters from history as accurately as we knew we had details like the captain, who’s one of the key characters. We now a fair bit about him that we could base our storyline and scripts on.
We had the skull on the skeleton of a 17 year old who may not have the eight but the other character in our game. His job was to keep the ship watertight and both of his parents were North African. And in fact, a third of the people on the boat were non English white people basically. Well, everyone on the boat was men, so there was no female male diversity.
But in terms that, you know, there was about a third who is in the Mediterranean of North Africa, which was different to, I think, what people perceive. So it was great to be able to bring out Henry’s story both because, you know, from a storyline point of view, he gave you a bit more of what was going on beneath deck when the captain wasn’t around, but also from a sort of diversity angle as well.
So you’ve got those two key main characters who are obviously represented in a 3D photorealistic way, but then the other characters who when you play the game at home and looking at these characters, they all just represented this kind of 2D videos. In actual fact, they are photo fits, again, recreated from the skulls of people who were on board and their pictures all around the museum.
And so when you hold the phone up to those pictures, on the pictures come to life and they tell you their backstory, tell you what’s going on. And yes, the other additional element I suppose you get well, there’s a couple more additional elements actually that you get from being in the museum. One is that a lot of the artifacts obviously look old because they are because you’re traveling back in time to examine when it happened.
For example, the Maltese Cross, which in the museum is a kind of very dull barnacles thing that you can just loosely identify as being a cross. Now you can see it in all its glory, how it would have looked, you know, like a very nice piece of jewelry, for example. And then obviously you’ve got the scent element, so you wear the same backpack, it releases certain key moments in the narrative and just kind of further immerses you in the gameplay.
So I think that the key differences between not in the home game, but we’re hoping people will still have a lot of fun with the home game because we really keen that it reaches audiences, digital audiences that way beyond. You can obviously come to Portsmouth in the UK and experience the on location game and I’m sure we’ll learn as we go from this episode to hopefully others, you know, other ways in which we can make the home game even more immersive, to make it as immersive as the on location game, which hopefully will be the case.
BYE: Okay. Yes, I had a chance to play the at home version and it’s basically a mystery genre where you’re trying to figure out why the ship sank. And so you get introduced to either the captain or someone who’s below the deck, and then you make a choice as to who you want to see. You see volumetric capture of these character actors who are giving you context as to who they are and a little bit information about the ship and what was happening.
And then you start to go through different objects that I presume are going to be in the museum from the Tudor period that are helping to both put you into that time and place of what was happening on the ship, but also to give you clues as to what may have been happening. So because I did it at home, that was obviously very linear experience.
But I’m wondering if as you go through it spatially, taking you through the museum in a similar way of how it’s going from object to object to help bring that object to life and give a little bit more context. And then maybe you can speak a little bit more about the scents that as you’re going on this journey, how you’re using smell to be able to amplify these objects and to amplify the story.
MIKKELBORG: Yes, sure. So yeah, so when you’re in the museum is a trail based experience and you activate the different clues with AR markers, and in fact, it’s a really good point that you make that the at-home version is much more linear. And I think, you know, one thing that we could easily do to change that potentially would be to reveal that ship in AR as you see it as your kind of anchor.
Sorry, so many ship puns, they’ll just keep going. But then have all the clues that represented spatially around the ship at one time. And you can kind of choose the order that you experienced them in as one simple thing. But yes, the way that we’ve designed it in the museum is there is a single trail and you are kind of supposed to follow that route, but there’s nothing forcing you to follow that route.
It’s just the museum because they are very busy at times. They kind of have a one way system that takes you along the ground floor down under into the hull of the ship and up onto the top deck. So we’ve kind of designed the trail and the storyline to follow that line, but actually know one part of the museum was super busy. You could skip and do it a slightly different way.
But in terms of the scent, when you enter the game, when you meet those characters for the first time, we release the kind of smell of ocean air of kind of Sea Spray to immediately put you there with them on the ocean. And then depending on which character you’re playing as, for example, if you’re playing on the captain kind of monster gun inside of the narrative, there’s a clear related to the gun because we know the guns have been sabotaged in the past and there was a suggestion that the guns and the sabotaged at the time of the sinking.
And so at the time when you get to interactively fire the gun, we release the smell of gunpowder and Gunsmoke. So that’s another. And then if you’re playing your Henry side of the narrative, when you’re down with him fixing a leak in the ship’s hold, they use this kind of molten tar almost. It was actually made from tree sap, but it was cool pitch, but it smelt much like tar, slightly more organic version of tar.
And so we release that smell. And then what else is we got? We’ve got beer because the Turors drank a lot of beer, so they drank beer on board instead of water because it was generally considered safer because the brewing process kills some of the bacteria. So they drank, I think, about 14 pints of beer a day to keep hydrated.
But also this could have contributed to the whole sinking issue. Because that is actually more than they were drinking on board the other ships. So we did do a bit of a comparison and they did see you drinking more on the Mary Rose. And so we released another beer. And when you’re in the hole where the beer was stored, but also when you were at the backgammon table, you know, one of the ways in which they took time off was backgammon was common, dice was common, playing instruments.
There’s a few musicians on board and quite a few instruments actually found on board as well. So and then also this amount of this Tudor pomander, which is something that if you could have afford to, you would have a pomander, which is like a necklace with a sort of ball on the end. You can actually make it, if you were not from the kind of noble classes, you could make a pretty well, homemade one with like an orange. If you could get an orange and, you know, sticking some cloves in it the way sometimes kids do nowadays at Christmas. But otherwise, if you could afford one, you had a fancy silver ball that stores rose water or whatever it is, you would waft it and it would help clear, unpleasant smells from under your nose and on board with 500 soldiers and sailors you probably hadn’t washed in some weeks.
So I’m guessing that was a bad number, a nasty smell. So we didn’t quite give you the full experience of being on board the Mary Rose because we figured you probably wouldn’t want to go around smelling beer or for the entire game. But. But, yeah, we gave a flavor.
BYE: Yeah. It’s interesting to hear your approach to this because as I’ve done other immersive VR experiences, one of the challenges with smell is that once you introduce a smell into a space, it’s hard to remove that smell. But because you’re walking through a museum and it’s basically the smell is a device attached to an individual that’s locomoting through a space you’re able to kind of do the editing of the smell by walking into different site specific locations that then release the smell at those locations, and then you move on to the next location so you don’t have to challenge that you usually get with VR.
MIKKELBORG: No. And also the fact that we could release the smell so close. Well, I’m going to make it sound unpleasant, as we’ve got it so close to your face. But like I say, it really is. It’s a dry scent. It’s not something that’s spraying. So it’s kind of emanating and getting that amounts of scent right was quite key. But because it is emanating quite close to your face, actually, even the person next to you shouldn’t really be able to smell it.
And because it’s so close, we don’t need to release huge amounts of it. So so the dissipation of it is not really an issue because the amount that’s being released is relatively small. So we have learned that kind of static scent compared to mobile. And let you say this just — Although, it’s harder to get through health and safety to have people walk around with scent devices on it was also much better for the scent experience.
BYE: I, I wish I was able to fully experience this, because I only saw the at home version which obviously I don’t have the scent pack to be able to experience this tech, but, I think, yeah, overall my experience of it is looking at these cultural artifact sites and when you travel to places, you are trying to tap into the lore and the stories of the past.
And I feel like museums like this, like the Mary Rose, are able to take this ship that’s been recovered and pulled out of the sea and all these objects and create a whole museum around it that becomes a tourist destination. But to add these other layers of story on top of it. So it’s interesting to see places where immersive technologies are starting to be used.
You mentioned that there was an already immersive experience because I saw a video of it where you see these big screens and the ship sank. And so it’s trying to give you a sense of the experience of the ship sinking and but then you’re going into that. So is that something that’s automatically included for everybody that goes in? Is it —
MIKKELBORG: Yeah, that’s something that everybody who comes to the Mary Rose Museum goes to that immersive experience, which lasts about 8 minutes, I think. And you kind of walk in and you actually have the actress Judy Dench telling you a little bit on the back story. And then the first room is a holographic Henry the Eighth, and he’s setting the scene, giving you a bit of the backgrounds, further background story.
And then you kind of walk past Henry and you’re on to the gun deck as they’re just kind of heading into battle like just before she sank. And the walls are slightly curved to give you something of a sense of being on board. And that’s, like I said, a project mapped to experience that we did not make, just to make it clear that was, I think, Figment Productions in the UK.
So we didn’t make that one. And so yes, after you experience that ship then goes down onto the water in the projection mapping experience and then you exit into the museum. So yeah, it’s nice. It kind of sets you up for a slightly more immersive visit to the museum, but our game doesn’t really interfere with that, although you can have downloaded it for you guys to do that. It really kickstarts in the next space that you enter after that.
BYE: So it’s in the app of the Time Detectives that you have is an iOS app that you’re able to do these in-app purchases. Do you plan on expanding that out to other museums and have one app that you have a variety of different experiences? Is that kind of the same idea of being able to have an interactive experience where you’re trying to solve a mystery but have a way of adding the story on top of the existing objects around these cultural heritage sites around the UK. I’m just curious if you have plans to expand it out beyond the Mary Rose?
MIKKELBORG: Yeah, definitely. Like I say, we originally prototyped it around a different site and when we got the funding to make this one, we were speaking to various sites. Some of whom are still interested, but they just couldn’t work with our six month timeline because it was quite a tight timeline. We wanted to really see this one launch.
See how it does. See how many new visitors it can attract all of those things and kind of get a sense of all of that and then go back to these sites and say, look, this is what we think the game can do for you, not just because we want to commercialize as a company, because actually as a storyteller, that is less my objective of being honest and other aspects.
You know, I just enjoy the storytelling. But also there was an element of this that wanted to help these sites because I’m a great lover of history and they’d obviously seen like their numbers decimated by COVID. And so it was an opportunity to also see what we could bring to help those sites, to not only get new audiences, but also kind of, you know, just expand their digital reach and all of that, which is why, you know, I’d love to hear please send me an email, Kent, if you feel like there’s any adjustments, improvements to my game, I’d love to hear about it because I think, you know, the better we can make these also for our home audiences, even though for now the scent element won’t be at home, you know, the better because it’s just great to take this history like further a field, way beyond the people who can obviously come to Portsmouth in this case.
But yeah, we definitely want to expand it to other UK sites and I’d love to take it to some Italian sites. And who knows, maybe also US sites. So although we haven’t really started thinking about which ones yet. So again, any input? Welcome.
BYE: Yeah, two quick pieces of feedback is that I initially set it on the ground. I feel like it probably works a little bit better if you set it on a table just because.
MIKKELBORG: Yeah.
BYE: A little bit low if you’re trying to see these objects. And also there’s a choice you make to go through two storylines. I chose the captain up front, and I wanted to go through the other storyline. It’d just be nice to do a reset button. They kind of do it again through the other perspective because it kind of got stuck because I’d already had done it and already had all these objects in there.
MIKKELBORG: So so I think it might be the way that the inventory displays possibly because in actual fact the objects are in the inventory but grayed out was where you collect them. So actually you go quickly into the inventory. It might look like you have all everything, but actually some will probably be grayed out and others not because certain objects are on the other characters side of the narrative, not on the captain.
So let me know on that. But it should be that Henry’s objects would still be grayed out, even if you could see them, that you wouldn’t have collected them yet. And also let me know which phone you own, because, you know, we’ve been having pretty smooth inventories on the phones that we’ve been using. But every phone represents a new challenge, and that’s one of the hardest things I discovered about AR. Luckily my developer had worked on AR before but is just you’re not developing for one or even two devices anymore. You’re developing for like 1600 androids and you know, and luckily only like eight iPhones or something. But it’s a marvelous debugging challenge.
BYE: Great and, and finally what do you think the ultimate potential of immersive storytelling might be and what it might be able to enable?
MIKKELBORG: Oh my goodness. That’s a big question, Kent. I think the possibilities for immersive storytelling are almost limitless? I mean, personally, I quite like the idea of working and I think it will likely catch on more with big audiences when you can make immersive experiences less tech heavy actually, because I feel we open out to, you know, people who might be less tech friendly.
And I’m developing an experience right now which is an LED, cave-based experience that, you know, you walk into that space and walls, ceiling, floor are LED — interactive LED, and you interact with the story through your physical movement and sounds you make and such like, you know, I love the idea of that because everybody knows how to move around the space and how to make sound if they’re asked to make sound.
Whereas there’s always technical challenges with everything you put in somebody else’s hands sites. But yeah, for sure. I think there’s so much potential. I also have a great idea in development — Well, I think it’s a great idea. We’ll obviously see what others think. But for something interactive that I would love to bring to someone like Netflix, something like a Bandersnatch, but kind of taking that a step further and also designing it in such a way that it doesn’t give you that kind of FOMO.
Like I need to go back and relive 50 different routes through this narrative. I just think there’s so many different formats, different mediums within Immersive right now, and they’re all kind of exciting and going in different places. And I don’t want to predict. I think that they’ll be one in a golden nugget of an idea that leads to the success of immersive at large.
But I think augmented reality does for now just mean that we can open out to some much bigger audiences than VR, and that’s exciting. As I think more interactive things on the likes of Netflix and other platforms is exciting because it just means immersive storytellers can also bring their work to a bigger audience. And I don’t think the likes of Oculus and such are quite there yet in terms of the ease with which you can get onto the platform. It’s become much easier, but it’s still. And the way in which you can start to experience is and you know, it’s just not quite as clever yet as Netflix. And so yeah I think we still way to go, but there’s a lot of exciting stuff happening and we’ll get there.
BYE: Is there anything else that’s left unsaid that you’d like to say to the broader immersive community?
MIKKELBORG: I don’t think so. Thank you very much for the opportunity though. Thanks so much. Really enjoyed it.
BYE: Awesome. Yeah. Thanks for joining me today on the podcast.
MIKKELBORG: Thanks, Kent.
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR podcast. Hello, my name is Kent Bye and welcome to the Voices of VR podcast. It's a podcast that looks at the future of immersive storytelling and spatial computing and supported by Patreon. So if you'd like to support the podcast, then go over to patreon.com slash Voices of VR. So today's episode is about the Time Detectives AR experience. It's a site-specific AR experience, but you can also see it at home. So, the site-specific version of the experience actually has some multi-century dimensions. It has smells that are being released as you go through this museum of the Mary Rose. The Mary Rose is a ship that sank on July 19th, 1545. So, they were able to recover this ship and to kind of restore it and put a whole museum up in Portsmouth, UK. And it's already got a museum around it, but they wanted to find new ways of having like an immersive experience like an augmented reality experience to be able to add an additional layer of story as they're going through this experience to make it a little bit more interactive. So the time detectives AR is like you're trying to solve the mystery as to why this ship sank. So that's what we're covering on today's episode of the Voices of VR podcast. So this interview with Charlotte happened on Monday, July 18th, 2022. So with that, let's go ahead and dive right in.
[00:01:21.547] Charlotte Mikkelborg: My name is Charlotte Mikkelborg. I have been working in immersive storytelling since I think 2016, 15, 16, around there. I started making 360 films and then I kind of played around with 360 films that would also work as multi-sensory experiences. So using more senses than just the audio visual. Then I moved into interactive VR with a big piece I did for British Airways to celebrate their centenary called Fly, which was very exciting. It won Best Massive Experience in the UK, voted by the UK government in 2020. And then coming into COVID, I was just looking at the fact that there wasn't going to be the kind of budgets we'd had to make something like Fly available because obviously everyone was kind of tightening their belts and We just got to a point where people were getting to experience amazing immersive experiences. The technology was there and so I didn't want audiences to lose that effectively or for us to lose our ability to tell those stories. So I started thinking about whether delving into AR because AR because you're not building obviously those entire worlds but just elements of those worlds is naturally going to be less expensive to make. So, yeah. So I started exploring with a prototype of this experience, what we could do basically with augmented reality. And this was my first sort of experience of storytelling in augmented reality.
[00:02:35.479] Kent Bye: Okay. Yeah. And maybe you could give a bit more context as to your background and your journey into XR.
[00:02:41.762] Charlotte Mikkelborg: Yeah, sure. So, um, I was originally a BBC foreign correspondent, so I was a BBC or one of the BBC China correspondents until 2009. And I left the BBC to start making longer form because I was really just doing news and current affairs. So I left the BBC to start making longer form film, mostly documentary. I made my first sort of feature length documentary film in 2009 called Building 173. I also worked entertainingly on a fiction film, although I was not directing that, I was line producing it, starring Matthew McConaughey, which we made around the same time, 2009, and won Grand Jury at South by Southwest. But I was working largely in documentary film between leaving the BBC and moving into the world of kind of XR in 2015-16. And the reason I moved into XR, I guess, or the kind of The tipping point was I was having a meeting with a producer called Julian Caudley in LA. He's the producer for Lucy Walker, who's an Oscar-winning documentary maker. And I asked what Lucy was working on, and he'd mentioned she was working on this 360 film. And I said, oh, what's 360 film when it's at home? So he sort of explained it to me and showed it to me. And I was like, oh, wow, this makes a lot of sense. Obviously, as we know, the tech wasn't there, but just that idea that instead of experiencing film on kind of a rectangular screen, just because that's where the technology had gotten us to, we could now experience it in the round the way we experience everything else in life just made sense to me. So I figured that this was going somewhere and this would be worth kind of exploring in more depth.
[00:04:12.617] Kent Bye: Yeah, and the first piece that I had a chance to see of yours was Fly that was remote during the pandemic. Everything that was going to be at these film festivals started to be in these virtual film festivals. So I saw a virtual representation of it, which is just the pure VR part. But I understand that there was a whole other installation component that gave it this whole multi-sensory experience of it. So maybe you could give a bit more context as to that project of Fly and how you started to move from working in 2D storytelling into more of this spatialized experiential and multisensory storytelling?
[00:04:46.712] Charlotte Mikkelborg: Sure. So yeah, the idea with Fly was we wanted to take you on this journey through when humankind first really started to think critically about whether it was going to be possible to get humankind into the air. So we decided to start that with a young Leonardo da Vinci, who, of course, did not well, arguably did not succeed in sustained human flight, but did make some serious inroads. And so we start with him. We know from his childhood diaries that he had this recurrent childhood dream that he would turn into a kite or a kite would land on his chest. The bird that is not the one that you fly in the wind and that you would kind of take on the spirit of flight and take flight. So So kind of plotting out that storyline and all the way back from Leonardo and eventually, you know, a hundred years into the future of flight at the end of the experience, what I knew was that I wanted people to actually feel that they were flying at certain key moments in this narrative. Otherwise it just wasn't going to have the awe and wonder, I guess, that I wanted it to have. So I was looking at different sort of flying rigs and I came across this thing that Sandra Bullock was put in for gravity. So I got talking to Neil Corbald, who won the Oscars for Gladiator back in the day, actually one of my favorite films, and Gravity more recently, and we looked at that rig and of course it wasn't ever going to be practical actually because it took ages to get into and then learned kind of how to use it. Then we started looking at motion platforms and whether we could actually make a motion platform genuinely feel like you're flying, because quite often they're used more, as you know, for kind of special effects in the movies where you've got a helicopter crashing and the platform spinning or whatever, but whether it could actually give you that sense of elevation. So we kind of started playing around with the platform that Neil's team had. One of Neil's right hands is a man called Glenn Winchester, who I work with most closely. And actually feeling that, yes, we could, when we, you know, did certain things, we could give you that sense of levitation. And we started designing a kind of lean board that you would lean your body down into when we wanted you to be prone and obviously flying like a bird in that opening sequence with the young Leonardo. But then later as planes sort of evolved and you went into this more sedentary position that you could then sit back. We didn't come out of headset. We just plotted those things within the virtual space so that you could accurately find them and sit back down. And around that, I envisaged this large egg and the egg was playing several LOLs. Obviously it was, from a health and safety perspective, it was kind of keeping you away from the moving platform. We didn't want people getting hands caught and things, but from a storytelling perspective, it was kind of a metaphor for When human energy is directed in the right ways, we can be capable of incredible things. It's not always directed in the right ways, but that's another story. But, you know, the fact that we could go from obviously being land-based animals to fly was quite incredible. So I wanted this egg to be kind of a metaphor for humankind making flight possible. So if you approached and put your hands on the front of the egg, it would warm to the touch like you were incubating it effectively with your hands. And we had this kind of, for lack of a more technical expression, kind of like a rainfall shower of sound that came down on you. And you could hear that incubation and that heartbeat of the life within and eventually the sort of cracking open of the egg. And the colour of the egg also changed from this, you know, sort of frigid blue through to a warmer red as you were warming it and eventually to a kind of bright white as light flooded in. But it was also, the egg was also a little bit of a play on, you know, Leonardo, when he went into his real fever pitch of obsession with flight. He was also painting a painting called Later in the Swamp, which has since been stolen. We don't know where it is in the world, but the sketches still exist. And, you know, it was this human woman birthing, like, you know, eggs. The picture isn't of her birthing the eggs, but, you know, the eggs are at her feet and they're cracking open and revealing human children within. So it's kind of also a bit of a play on that. And there was a second part that you wanted me to answer, Kent, and I'm just trying to remember what that was.
[00:08:34.094] Kent Bye: Oh, well, you know, I guess we're speaking here in the context of your journey into XR and immersive storytelling and the multisensory part of it, and you're releasing a new AR experience of the Mary Rose ship that sunk nearly 500 years ago. And so the whole multisensory aspect of it and what you thought the multisensory dimension of it was adding to the process of telling stories as you're moving from 2D into the more spatial aspects, but also the multisensory aspects.
[00:09:02.049] Charlotte Mikkelborg: Yeah. So as you know, the beauty of VR is that you do have somebody's complete attention. They're in the headset. You've got their entire peripheral vision code, which happens very rarely these days in any aspect of life. Most people are on their phones while watching TV, while doing something else. And the beauty of that from a multi-sensory storytelling perspective is obviously that they then don't see if you've placed cutting little devices to make it feel like the sun is shining on their faces or that the wind is blowing. One really important scene in Fly is the Wright Brothers scene where you become Wilbur or you take Wilbur Wright's place on the wing of the Wright Flyer and you get to pilot that first. human-powered flight. So I wanted to try, like I was saying before, because we didn't expect the kind of budgets that have been available for flight to be available coming into COVID, to see if we could still play with the other senses while actually just playing an augmented reality kind of game on the phone. So originally we were trying to R&D these little sort of wearable scent badges effectively, And actually the R&D on those wasn't successful. I'm not to say that it couldn't be. I'm sure that, you know, we'll get there, but within the timeframe, it wasn't possible. So we went for actually a device that you do wear in a small backpack that are hidden under the strap and the scent kind of emanates. It's dry scent. It doesn't kind of spray in your face, but it emanates sort of here on your chest. And scent was important to me in particular because I think probably of, obviously all of our senses are absolutely key, scent is the only one that speaks directly to the limbic part of our brain. And the limbic part of the brain is the part that forms memory. So it was that opportunity to possibly quite literally create more memorable experiences. I mean, obviously the other senses do make it to the limbic part of the brain, but they just do it more slowly and more indirectly. And so we wanted to see if we could bring that sort of personalized scent experience together with the audio visuals of the game. to make for a more immersive augmented reality experience, basically.
[00:10:58.480] Kent Bye: Yeah. And so as we start to talk about this new experience that you have, I'm wondering if you could give a bit more context as to the Mary Rose and the museum and how this is both a site-specific experience, but also if people want to see it at home, they could see it at home. I personally saw it at home. I have not been able to see the site-specific in Portsmouth, United Kingdom. So maybe you could give a bit more context as to the museum and how this project came about.
[00:11:24.046] Charlotte Mikkelborg: Yeah, sure. So the Mary Rose was Henry VIII's favorite warship, and she was perfectly seaworthy for the best part of 30 years, and then kind of sank all of a sudden, very quickly, on a summer's day, 19th of July, 1545. when she was literally just starting to go into battle against the French and she sank, killing almost all of the 500 soldiers and sailors on board. And why she sank has largely remained a historical mystery, so that was quite exciting for something that ultimately we'd sort of branded time detectives, we wanted to turn you into these kind of time traveling detectives and so genuine historical mysteries is pretty much what we were looking for, so it was a great fit from that perspective. The boat itself was raised, there'd been a few diving missions, including one immediately after the sinking in 1545, a few diving missions down to the wreck, but she was brought up out of the ocean in 1982, so 40 years ago this year. And in fact, the lead historian on the boat is still there today. So the lady who brought it up is still the lead historian in the museum. Her life has been the Mary Rose and she knows everything there is to know. So she's been an incredibly valuable resource, obviously, in developing the game. When we got the funding to make the first episode of Time Detectives, we actually didn't get it specifically for the Mary Rose. And so we were talking to a few different venues. including like the usual suspects, Tower of London, Hampton Court, Shakespeare's Birthplace and some others. And actually it really became the Mary Rose because we were given quite a limited timeline to develop the game. We had six months if we wanted to make use of the funding we were offered. And so the Mary Rose, because they're a slightly smaller team, they were able to be a bit more nimble. And so, yeah, so if you visit the Mary Rose Museum, which is on Portsmouth, in Portsmouth's historic dockyards or on the seafront, And they also have the Victory, which was Lord Nelson's ship there and a couple of other, you know, historical monuments. They've built this sort of custom built museum around the wreck of the Mary Rose. And she is obviously humidity controlled and she's kept sprayed down with chemicals because she is so old. But she's within this custom built, very modern museum space, very dark museum space as well. So that has had its own interesting challenges with developing AR and AR markers and all of that. But, you know, the wonder of her is that despite several dives to the site, because she was in such a busy shipping lane in a way, the half of her that sank under the silt, under the sand, got incredibly well preserved. And she is pretty much half a ship these days, because a lot of what was above there then got battered away by the combination of the waves and other boats going over. And also the people, unfortunately, who died on that side, and obviously that was probably more than half because that's the way she sank, were also kind of down there. The skeletons were down there. They found 179 skulls of the 500 anyway. but also lots of objects, like a lot of the cannon that were on board, a lot of personal items. So she really is such an amazing time capsule for developing a game around. Did you want me to talk about how the museum game is different to what you would have experienced at home beyond the obvious scent component?
[00:14:22.037] Kent Bye: I wanted to set a little bit more context too, because I do want to know a little bit more in terms of as you go into the museum, this feels like it might be an add-on, but like where you would download the app and maybe buy the in-app purchase to get access to the content, but also there's a backpack component. So as I go to different museums, there's often objects and as those objects are there, I understand that the Mary Rose has a lot of, from the Tudor period in England, lots of different objects that were on the boat. And so there's a lot of historical capture of moments from history that people ever look at, but there's also the ship that's there. But I guess if you're a kid or if you're someone who's going in there, maybe you don't have a lot of interest in this, but it seems like this app would maybe be a way for either young adults or youth to be able to have an interactive AR app that has a little bit more of a narrative component that they can have an interactive experience of this museum, rather than just going in and looking and reading all the placards. And so maybe you could give a bit more context if that was the catalyst, or if you came to them saying, Hey, we want to use this new AR technology. But if you're targeting a specific demographic of people who may not otherwise want to go to the museum and read all the stuff, because if you go to a museum, there's plenty of stuff to look at. But maybe this is a new way of having versus of technology to give new access to it. So yeah, if you could maybe start there and explain the workflow and what's different than what people, if they were to see it at home, what they would experience.
[00:15:44.386] Charlotte Mikkelborg: Sure. Lots of things. Yeah. So when we were designing this, we were aiming to create a family-based experience largely because we started that development during COVID. And when we were prototyping, we were actually prototyping it around an outdoor trail based on a different site, a place called Oaklands Palace, which is Henry VIII's lost queen's palace. And it was an outdoor trail. And you could go as a family, or it doesn't have to be a family group. It could be a group of friends. But we kind of modeled it around that family experience. And in fact, when we ran tests on that site, We expected families with kids between 8 and 14. We actually had a lot of requests from 16, 17, 18 year olds, which we were really surprised by, pleasantly surprised. And when I was kind of asking them, well, why did you come? They were like, well, we just never heard of like a multisensory AR thing for them. And the scent, we love this idea of scent. So that was interesting. So it felt like it tapped into a demographic that is sort of notoriously difficult. We were kind of aiming at that audience, but also I think that the US has a similarly bad record as the UK does in attracting as diverse an audience to the museums and the cultural heritage sites that there are, versus the diversity of society at large. Here in the UK, our record for that is appalling. We see much less diverse audiences in museums. And interestingly, also in terms of, if you look at somewhere like the National Gallery, their online following is a lot of young people. But in terms of the people that come through the doors, that percentage is tiny, you know, the majority are over 55. So there's a real disconnect, right, for both young people and I think for more diverse audiences. So we wanted for the app to help, obviously it's one small part of, you know, moving towards solving those big massive issues, but If we could tell stories that felt more diverse, but also in a way that was exciting and immersive and that appealed to younger audiences, then we could, like you say, just bring new people through the doors. Because it is a really cool thing to explore, but like you say, it's not necessarily the thing that you're automatically going to think of on a Saturday to do. So yeah. But the Mary Rose has also been quite forward thinking, I think, as a museum. I mean, they've anyway gotten the very modern space because it was only built 11, 12 years ago. As you go in, another company, Figment Productions, has produced an immersive experience which kind of replicates the experience of the sinking to a degree. Obviously, it's not super distressing, but it's more of a projection mapping. It features a hologram of Henry VIII. So they've invested something in, you know, immersive experiences already. But the game, like you say, you can download either before you come or you can download when you get there. It is kind of an add-on, but a lot of people who visit at the moment are people who have like annual membership for the Dockyard as a whole. So you might come one day, do Mary Rose one day and do Lord Nelson's Victory or whatever. So now they could just come again and play the game because it does just show the whole museum in a completely different light. Like you're really on a story journey. You're not just on a looking around the museum journey because it is just picking out certain elements. It obviously isn't showcasing huge amounts of the collection. It really is one storyline that kind of leads you through. But in terms of what the game does for the museum, I mean, we looked at examples like obviously Pokemon Go and Harry Potter Wizards Unite and other trail based AR stuff that was out there. And certain aspects of those games were great. But what was dissatisfying for me was I really wanted more photorealistic characters, for example. So that was one thing we wanted to do in the museum, see if we could try and place these characters from history as accurately as we knew. We have details like the captain, who's one of the key characters. We know a fair bit about him that we could base our storyline and our scripts on. we had the skull and the skeleton of 17-year-old Henry, not Henry VIII, but the other character in our game, whose job was to keep the ship watertight. And both of his parents were North African. And in fact, a third of the people on the boat were non-English white people, basically. Well, everyone on the boat was men, so there was no female-male diversity, but in terms of, you know, there was about a third who were either from the Mediterranean or from North Africa. which was different to, I think, what people perceived. So it was great to be able to bring out Henry's story, both because, you know, from a storyline point of view, he gave you a bit more of what was going on beneath deck when the captain wasn't around, but also from a sort of diversity angle as well. So you've got those two key main characters who are obviously represented in a 3D photorealistic way, but then the other characters who, when you play the game at home, unfortunately, with these characters, they are just represented as kind of 2D videos. In actual fact, they are photo fits, again, recreated from the skulls of people who were on board. And their pictures are around the museum. And so when you hold the phone up to those pictures, obviously the pictures come to life and they tell you their backstory, tell you what's going on. And yes, so the other additional element, I suppose you get, well, there's a couple more additional elements actually that you get from being in the museum. One is that a lot of the artifacts obviously look old because they are, but because you're traveling back in time to examine when it happened, For example, our Maltese cross, which in the museum is a kind of very dull, barnacled thing that you can just like loosely identify as being a cross. You know, now you can see it in all its glory, how it would have looked, you know, like a very nice piece of jewellery, for example. And then obviously you've got the scent element. So you wear the scent backpack, it releases scent at certain key moments in the narrative and just kind of further immerses you, we hope, in the gameplay. I think they're the key differences between that and the home game, but we're hoping people will still have a lot of fun with the home game because we're really keen that it reaches audiences, digital audiences, like way beyond who can obviously come to Portsmouth in the UK and experience the on-location game. And I'm sure we'll learn as we go from this episode to hopefully others, you know, are there ways in which we can make the home game even more immersive to sort of make it as immersive as the on-location game, which hopefully will be the case.
[00:21:17.951] Kent Bye: Okay. Yeah. So I had a chance to play the at-home version and it's basically a mystery genre where you're trying to figure out why the ship sank. And so you get introduced to either the captain or someone who was below the deck, and then you make a choice as to who you want to see. You see volumetric capture of these character actors who are giving you context as to who they are and a little bit information about the ship and what was happening. And then you start to go through different objects. that I presume are going to be in the museum from the Tudor period that are helping to both set you into that time and place of what was happening on the ship, but also to give you clues as to what may have been happening. So because I did it at home, it was obviously a very linear experience, but I'm wondering if as you go through it, spatially taking you through the museum in a similar way of how it's going from object to object to help bring that object to life and give a little bit more context. And then maybe you could speak a little bit more about the sense that as you're going on this journey, how you're using smell to be able to amplify these objects and to amplify the story.
[00:22:20.641] Charlotte Mikkelborg: Yeah, sure. So yeah, so when you're in the museum, it is a trail-based experience and you activate the different clues with AR markers. And in fact, it's really a good point that you make that the at-home version is much more linear. And I think, you know, one thing that we could easily do to change that potentially would be to reveal that ship in AR as you see it as your kind of anchor. Sorry, so many ship puns, they'll just keep coming. But then have all the clues like represented spatially around the ship at one time and you can kind of choose the order that you experience them in as one simple thing. But yes, the way that we designed it in the museum is there is a single trail and you are kind of supposed to follow that route, but there's nothing forcing you to follow that route. It's just The museum, because they are very busy at times, they kind of have a one way system that takes you along the ground floor, down under into the whole of the ship and up onto the top deck. So we've kind of designed the trail and the storyline to follow that line. But actually, you know, one particular part of the museum was super busy. You could skip and do it a slightly different way. But in terms of the scent, when you enter the game, when you meet those characters for the first time, we release the kind of smell of ocean air of kind of sea spray to immediately sort of put you there with them on the ocean. And then depending on which character you're playing as, for example, if you're playing on the captain kind of master gunner side of the narrative, there's a clue related to the gun because we know the guns have been sabotaged in the past. And there was a suggestion that the guns were sabotaged at the time of the sinking. And so at the time when you get to interactively fire the gun, we release the smell of gunpowder and gun smoke. So that's another. And then if you're playing on Henry's side of the narrative, when you're down with him fixing a leak in the ship's hold, they use this kind of molten tar almost. It was actually made from tree sap, but it was called pitch. But it smelled much like tar, slightly more organic version of tar. And so we released that smell. And then what else have we got? We've got beer because the Tudors drank a lot of beer. So they drank beer on board instead of water because it was generally considered safer because the brewing process kills some of the bacteria. So they drank, I think, about 14 pints of beer a day. to keep hydrated, but also this could have contributed to the whole sinking issue. Because that is actually more than they were drinking on board the other ships. So we did do a bit of a comparison and they did seem to be drinking more beer on the Mary Rose. So we released the smell of beer and when you're in the hold where the beer was stored, but also when you were at the background table, you know, one of the ways in which they took time off was Backgammon was common, dice was common, playing instruments. There's a few musicians on board and quite a few instruments actually found on board as well. And then also the smell of Tudor pomander, which is something that if you could afford to, you would have pomander, which is like a necklace with a sort of ball on the end. You could actually make, if you were not from the kind of noble classes, you could make a pretty well homemade one with like an orange, if you could get your hands on an orange and you know, sticking some clothes in it the way sometimes kids do nowadays at Christmas. But otherwise, if you could afford one, you had a fancy silver ball that stores rose water, whatever. And as you would waft it, it would help clear unpleasant smells from under your nose. And on board with 500 soldiers and sailors who probably hadn't washed in seven weeks, I'm guessing there was a fair number of nasty smells. So we didn't quite give you the full experience of being on board the Merry Rose because we figured you probably wouldn't want to go around smelling BO for the entire game. we gave a flavor.
[00:25:51.198] Kent Bye: Yeah, it's interesting to hear your approach to this, because as I've done other immersive VR experiences, one of the challenges with smell is that once you introduce a smell into a space, it's hard to remove that smell. But because you're walking through a museum and it's basically the smell is a device attached to an individual that's locomoting through a space, you're able to kind of do the editing of the smell by walking into different site-specific locations that then release the smell at those locations and then you move on to the next location. So you don't have the challenge that you usually get with VR.
[00:26:24.801] Charlotte Mikkelborg: No, and also the fact that we could release the smell so close Well, I'm going to make it sound unpleasant. I'm about to say so close to your face, but like I say, it really is, it's a dry scent. It's not something that's spraying. So it's kind of emanating and getting that amount of scent right was quite key. But because it is emanating quite close to your face, actually, even the person next to you shouldn't really be able to smell it. And because it's so close, we don't need to release huge amounts of it. So also the dissipation of it is not really an issue because the amount that's been released is relatively small. So we have looked at kind of static scent compared to mobile. And like you say, this just, although it's harder to get through health and safety to have people walking around with scent devices on, it was also much better for the scent experience.
[00:27:09.440] Kent Bye: I wish I was able to fully experience it because I only saw the at home version, which obviously I don't have the scent pack to be able to experience this tech. But I think, yeah, overall, my experience of it is looking at these cultural artifact sites. And when you travel to places, you are trying to tap into the lore and the stories of the past. And I feel like museums like this, like the Mary Rose, are able to take this ship that's been recovered and pulled out of the sea and all these objects and create a whole museum around it that becomes a tourist destination, but to add these other layers of story on top of it. So it's interesting to see places where immersive technologies are starting to be used. You mentioned that there was an already immersive experience? Because I saw a video of it where you see these big screens and, you know, the ship sank. And so it's trying to give you a sense of the experience of the ship sinking. But then you're going into that. So is that something that's automatically included for everybody that goes in?
[00:28:04.140] Charlotte Mikkelborg: Yeah, that's something that everybody who comes to Mary Rose Museum goes through that immersive experience, which lasts about eight minutes, I think. And you kind of walk in and you actually have the actress, Judi Dench, telling you a little bit on the backstory. And then the first room, There's a holographic Henry VIII and he's setting the scene, giving you a bit of the background, further background story. And then you kind of walk past Henry and you're onto the gun deck as they're kind of just heading into battle, like just before she sank. And the walls are slightly curved to give you something of a sense of being on board. And that's, like I say, a projection mapped experience that we did not make just to make it 100% clear. That was, I think, Figment Productions in the UK, so we didn't make that one. And so, yeah, so after you experience that ship then goes down under the water in the projection mapping experience and then you exit into the museum. So, yeah, it's nice. It kind of sets you up for a slightly more immersive visit, I think, to the museum. But our game doesn't really interfere with that, although you can download it before you go through that. It really kickstarts in the next space that you enter after that.
[00:29:04.474] Kent Bye: And the app of the time detectives that you have is an iOS app that you're able to do these in-app purchases. Do you plan on expanding that out to other museums and have one app that you have a variety of different experiences that are kind of the same idea of being able to have an interactive experience where you're trying to solve a mystery, but have a way of adding the story on top of the existing objects around these cultural heritage sites around the UK. I'm just curious if you have plans to expand it out beyond the Mary Rose.
[00:29:33.732] Charlotte Mikkelborg: Yeah, definitely. Like I say, we originally prototyped it around a different site. And when we got the funding to make this one, we were speaking to various sites, some of whom are still interested, but they just couldn't work with our six-month timeline because it was quite a tight timeline. We wanted to really see this one launch, see how it does, see how many new visitors it can attract, all of those things. and kind of get a sense of all of that and then go back to these sites and say, look, this is what we think the game can do for you, not just because we want to commercialize as a company, because actually, as a storyteller, that is less my objective, if I'm honest, than other aspects. You know, I just enjoy the storytelling. But also there was an animatist that wanted to help these sites, because I'm a great lover of history, and they've obviously seen like their numbers sort of decimated by COVID. And so it was an opportunity to also see what we could bring to help those sites to not only get new audiences but also kind of you know, just expand their digital reach and all of that, which is why, you know, I'd love to hear, please send me an email, Ken, if you feel like there's any adjustments, improvements to the remote game, I'd love to hear about it, because I think, you know, the better we can make these also for at home audiences, even though for now the scent element won't be at home, you know, the better, because it's just great to take this history like further afield, way beyond like the people who can obviously come to Portsmouth in this case. But yeah, we definitely want to expand it to other UK sites. And I'd love to take it to some Italian sites and maybe also US sites, although we haven't really started thinking about which ones yet. So again, any input, welcome.
[00:31:00.855] Kent Bye: Yeah, two quick pieces of feedback is that I initially set it on the ground, and I feel like it probably works a little bit better if you set it on a table, just because it's a little bit low if you're trying to see these objects. And also there's a choice you make to go through two storylines, and I chose the captain up front, and I wanted to go through the other storyline. It'd be just nice to do a reset button to kind of do it again through the other perspective, because it kind of got stuck because I'd already done it, and I already had all these objects in there, so.
[00:31:27.619] Charlotte Mikkelborg: So I think it might be the way that the inventory displays, possibly, because in actual fact, the objects are in the inventory, but grayed out before you collect them. So actually, if you go quickly into the inventory, it might look like you have all everything, but actually some will probably be grayed out and others not, because certain objects are on the other character's side of the narrative, not on the captain's. So let me know on that, but it should be that Henry's objects would still be grayed out, even if you could see them, but you won't have collected them yet. And also let me know which phone you're on, because You know, we've been having pretty smooth run throughs on the phones that we've been using, but every phone represents a new challenge. And that's one of the hardest things I discovered about AR. Luckily, my developer had worked on AR before, but it's just, you're not developing for one or even two devices anymore. You're developing like 1600 Androids and, you know, luckily only like eight iPhones or something, but it's a marvelous debugging challenge.
[00:32:20.179] Kent Bye: And finally, what do you think the ultimate potential of immersive storytelling might be and what it might be able to enable?
[00:32:29.153] Charlotte Mikkelborg: Yes, that's a big question, Kent. I think the possibilities for immersive storytelling are almost limitless. I mean, personally, I quite like the idea of working and I think it will likely catch on more with bigger audiences when you can make immersive experiences less tech heavy, actually, because I feel we open it out to people who might be less tech friendly. And I'm developing an experience right now, which is an LED cave based experience that You know, you walk into that space and wall, ceiling, floor of LED, interactive LED, and you interact with the story through your physical movement and sounds you make and such like. You know, I love the idea of that because everybody knows how to move around the space and how to make sound if they want to, and if they're asked to make sound, whereas there's always technical challenges with everything you put in somebody else's hands. So, but yeah, for sure. I think there's so much potential. I also have a great idea in development. Well, I think it's a great idea. We'll obviously see what others think. But for something interactive that I would love to bring to somewhere like Netflix, something like a Bandersnatch, but kind of taking that a step further and also designing it in such a way that it doesn't give you that kind of FOMO like I need to go back and relive 50 different routes through this narrative. I just think there's so many different formats, different mediums within immersive right now and they're all kind of exciting and going in different places and I don't want to predict I think. that there'll be one, you know, golden nugget of an idea that leads to the success of Immersive at large. But I think augmented reality does, for now, just mean that we can open it out to much bigger audiences than VR, and that's exciting, as I think more interactive things on the likes of Netflix and other platforms is exciting because it just means immersive storytellers can also bring their work to a bigger audience. And I don't think the likes of Oculus and such are quite there yet in terms of the ease with which you can get onto the platform. It's become much easier, but it's still, and the way in which you can search through experiences and, you know, it's just not quite as clever yet as Netflix. And so, yeah, I think we've still a way to go, but it's, there's a lot of exciting stuff happening and we'll get there.
[00:34:34.478] Kent Bye: And is there anything else that's left unsaid that you'd like to say to the broader immersive community?
[00:34:39.221] Charlotte Mikkelborg: I don't think so. Thank you very much for the opportunity though. Thanks so much. Really enjoyed it.
[00:34:43.964] Kent Bye: Awesome. Yeah. Thanks for joining me today on the podcast. Thanks Kent. So, that was Charlotte Meikeborg. She's been doing immersive storytelling since 2015 or 2016 as a part of Picture This Productions. So, I have a number of different takeaways about this interview is that, first of all, well, you can download this app and be able to experience it at home, and I recommend seeing it on a table. And there's two different paths you can go through, and I had to go through it twice, so you have to kind of choose which are the main protagonist characters that you're following, either the captain that's above the top of the ship or someone who's underneath the ship who's got a little bit more connection to what's happening underneath the ship. You have to go through it twice in order to see all the different objects, but with each of the objects that you're collecting, you can gather and collect these objects, and then at the top, these three objects that you can rotate around, and they have a little passage. They're also giving you little clues to what was happening leading up to why this ship sank. Then, at the end, you have to take a quiz to be able to see if you were able to solve the mystery as to why the Mary Rose sank. Taking something that would otherwise be a cultural heritage site with all these different objects from the Tudor period and something that may not be as interesting or interactive for a whole family experience. Trying to create something that's targeted for youth but also have the whole family be able to add an additional layer of story on top of going to a museum like this. So check it out to see some of the different stories yourself. They're basically using tabletop AR doing what looks to be something like a depth kit volumetric capture of two characters they've got the 3d object of the ship that is a fairly linear story for each of the different instances and Sounds like in the actual experience as well. You're going through a fairly linear, although you can have a a little bit more of a non-linear if you jump around. There's QR codes that are there on-site that launch into these different experiences. There are a number of different places within the site-specific version where there's scent that's being released with this backpack that you're wearing and somehow is only able to be within your own realm of smelling it. I have to check that out because I haven't seen technology that's able to really pull that off. I'd be really curious to experience that someday to see how that actually works. But because you're moving from room to room, it's more of a site-specific trail experience where you're moving through space, and so it makes it a little bit easier to use something like smell in that case, whereas in VR contained to a specific location, then it's a little bit more difficult to do that. It's an app that you can download to be able to see both the Mary Rose, and it's able to potentially expand out to other locations, as well. You're able to have a remote experience. You don't have to be on-site to be able to enjoy this as an experience and to get a bit of the story of somebody visiting and actually seeing all these objects. It's able to not have all the overhead of a virtual reality experience, but able to continue to do this type of immersive storytelling through the lens of augmented reality. So very interesting to see how the structures and forms of immersive storytelling are continuing to develop outside of the VR container, but also moving into augmented reality as well. So that's all I have for today, and I just wanted to thank you for listening to the Voices of ER podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and consider becoming a member of the Patreon. This is a list of supported podcasts, and so I do rely upon donations from people like yourself in order to continue to bring you this coverage. So you can become a member and donate today at patreon.com slash Voices of ER. Thanks for listening.
