Jane Crayton is an immersive educator at the ARTSLab University of New Mexico who teaches and creates immersive dome experiences. She’s collaborated with Charles Veasey from the The Digital Dome at Institute of American Indian Arts in creating the vDome open source software, which is multi-channel projection software that provides real-time warping and slicing of content designed for immersive domes.
Jane describes how you could take content developed in Unity and project it onto a 20ft dome with one computer, and a TripleHead2Go to drive three projectors. Producing content for domes used to require a lot of rendering time, but can now be done in real-time using vDome or Blendy Dome VJ.
The desire to do live VJ performances in an immersive dome is what catalyzed some of these technological breakthroughs including with two other groups working on this including the Société des arts technologiques (SAT) in Montreal, Recursive Function Immersive Dome (RFID) in the UK.
Jane talks about some of the educational uses of immersive domes including how she’s using it to recreate archaeological sites. Domes also allow for collective experiences that could be shared in groups, and that she expects to see Unity playing a bigger role in producing content for domes moving forward. She sees that fully immersive domes have the potential to change your perspective and alter your frame of reference, since you leave behind your point of view and it allows you understand material in a new way.
TOPICS
- 0:00 – Intro – Work in fully immersive dome. Teaching digital production for a full dome environment using technologies like spherical photography, photogrammetry, and building up 3D environments to be fully immersed in the dome environment and interact with it. At the University of Mexico arts lab, and got a grant to develop a curriculum to best product multi-projection, full dome format. Creating a 4000px x 4000px format. Blending photography with virtual objects with textures. Focusing on creating on new and interactive tool within the full dome. Technology has been innovating to change how multiple-projection digital planetariums are produced. vDome open source software written by Charles Veasey, which provides real-time warping and slicing for domemaster input. Developed it in order to do live VJ performances, and bringing in contemporary club culture into the immersive domes. Being able to build out virtual places that you can explore and interact with each other. vDome transformed how they use the dome since it doesn’t have to be pre-rendered so that they can see it immediately on the dome. It’ll change how dome content is produced. Still in the R&D phase. Other groups creating dome software include Société des arts technologiques (SAT) in Montreal, Recursive Function Immersive Dome (RFID) in the UK, Blendy Dome VJ in Brazil. All of the groups were motivated by wanting to do live VJ in immersive domes.
- 7:55 – Immersive dome vs immersive VR in a HMD. Some are 360-degrees and others are 180-degrees or 270-degrees. It allows you to look around and see out of your peripheral vision. You can engage audience with surround-sound audio. Use sound as an instigator for what to pay attention. Engaging emotionally and physically and do it with a live audience. You can sit in different perspectives within the dome. Consider how the audience will be seated and how they’ll be looking at the dome
- 10:35 – Educational component to domes. First experience within a dome was in a planetarium, and it got her interested in science, optics and computers. Slide projectors used within the dome. It’s not just about astronomy in the dome any more. Teaching photography and videography from a different perspective. Dome offers a lot to students and teachers to engage with each other. Your perspective changes when you’re immersed
- 13:11 – Content beyond astronomy. Cartoons. Film. Working with on a NSF grant to document archeological sites and building out a virtual archeological sites to be experienced in an immersive dome. Looking at applications beyond astronomy. Teaching photography, videography and 3D skills
- 15:40 – What one would need to set up a dome. Download vDome software. A 20ft dome would require 3 projectors. Need a computer. Would need a TripleHead2Go to drive three projectors.
- 16:53 – Digital planetariums used to use a $5k computer per projector x7. Today it’s a lot easier. A computer with two video cards could drive up to six projectors with two TripleHead2Go devices.
- 18:50 – How does Unity game engine fit in? Can pipe in Unity environments onto immersive dome environments. Movement can be difficult since moving too quickly will make the audience sick. Unity is up-and-coming platform for the dome
- 20:20 – What to avoid to minimize motion sickness. There’s a sweet spot on the dome where they audiences’ eye naturally rest. Take everything a bit slower and watch what you’re producing in the dome. Slow pans, animations and moves, and can be easy to get sick. Trojan commercial with pigs on a roller coaster that made people sick.
- 23:00 – Spherical video solutions to bring video into an immersive dome. High-learning curve on these technologies. 360Heros is probably the most affordable solution. Uses similar software pipeline.
- 25:48 – Full dome has the potential to change your perspective and alter your frame of reference, leave behind your point of view and understand material in a new way.
Theme music: “Fatality” by Tigoolio
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast.
[00:00:11.973] Jane Crayton: My name is Jane Creighton, and I am from the University of New Mexico. And I'm based out of Denver, Colorado. And I work in immersive technology, primarily in the full dome fields. And my focus is on teaching digital production for the immersive environment. So I'm working with teaching undergraduates how to produce digital productions for the full dome and how they might use some of the current technologies to do that. So for instance, one of the main things we teach is how to capture spherical photography. how to do photogrammetry, and also how to build these things up in 3D environments like Maya, Blender, and Unity, so that we can actually put ourselves into that environment and augment ourselves in the full dome. So we're fully immersed in the dome environment, but then we also take it a step further often by making ourselves become a part of the world and interact within it. I am a part of an immersive design team, our innovation team, and a teaching team from the University of New Mexico in the Arts Lab. We received a grant in collaboration with the Institute of American Indian Arts, funded by the Department of Defense. And as a part of that grant, we had a couple main focuses. The first focus was to develop a curriculum to help teach undergrad students how to get the skills to best produce on these complex planetarium productions. Multi-projection, full-dome environment. is very complex for imaging. You're working, instead of an HD format or SD formats, you're working in formats that are way beyond. We're talking 4,000 pixels by 4,000 pixels. So how do you take an HD camera and capture 4,000 pixels? It's impossible. And in fact, what you have to do is use multiple cameras or use multiple ways of tricking the viewer to think that you had multiple cameras. So that's where using 3D environments really comes into effect. So oftentimes when we're creating digital productions for the planetarium, we're incorporating actual photography, maybe of the outside and the sky, but then we include 3D objects like tables or furniture or buildings or you know, whatever that we will place into the production and texture it with the actual texture from photographs. And so that starts to build up an environment that you can actually model and walk through and animate. And that's really the gist of being able to teach dome production for undergrads. The other part of our grant was really to focus on creating a new and interactive tool that would really enable us to solve some of the design and interactive problems that we have in the full dome. One of the main problems with the full dome for the past decade since the development of multi-projection digital planetariums has really been that everything has had to be pre-rendered and pre-sliced on multiple computers at once. With the recent advancement of video processors, and computers and the way that they're able to talk to each other through network cards. A lot of that has changed, and primarily in the last few years, and ironically, really in the last two years. And in fact, there's been four or five really great teams done internationally that kind of cracked this code all at once. And we all approached it from very different aspects, which is kind of a unique story that, you know, you may not be aware of. And so, the New Mexico team that I'm a part of, one of my colleagues, our team developed this product called V-Dome. It's an open source, free application that you can install on your computer and it will help you slice in real time and distribute your images or videos onto the dome. The reason we developed it was for a couple different reasons. We wanted to develop the interactive live part so that we could do things like VJ live. That's one of my things that I really like to do. I wanted to bring back the Lazarium kind of feel to the Dome, but more with a kind of a contemporary context of club culture. and I come from club culture also and have been an active VJ, so when I was brought into the team, that was immediately one of the things that I wanted to do. Another thing is also just playing in virtual worlds with being able to move about. And this has become really important in a lot of the work that we're doing culturally, especially with the Institute of American Indian Arts, where they can start to train their students to really build things like the Smithsonian is doing here, building out virtual worlds and virtual exhibits, being able to explore these places, and not only virtually on a flat screen, but now you can do it in a dome, multi-projection dome. And it's not just one person who's experienced that. It's actually a whole group of people at that time. And so they can start to interact together and learn together and ultimately even experience their own views, but as a group. So it's a really powerful experience. And for our students, we've seen V-Dome really transform the way that they use the Dome. They used to have to take several hours and days and processing time and rendering time and all these things to render out their stuff on six different computers, six different slices, six different views, and that was a real pain. Now our students are able to take their spherical imagery, drop it right on the computer and see it instantly in the dome. And that has cut the workload down for them immensely. And I think it's going to change the way that planetariums are producing. A lot of planetarium systems, especially the big museums and the professional full-dome theaters, are still using these older technologies. The newer technologies we've developed definitely are not refined enough to really bring out to the marketplace and sell it as an item. We're really in the research and development phase of this. And so are all the other teams. Some of the other teams have come up with similar but different ways to do real live interactivity in the Full Dome are the Society of Arts and Technology out of Montreal, the RFID group, I believe they're in the UK, the Blendy Dome group, they're from Brazil. And then our V-Dome group from New Mexico really are the groups who pioneered somewhat similar technology. And ironically, all the groups really did focus on VJing as one of their interesting niches that all of them wanted to do. which is pretty interesting and maybe it's because we're all younger club kids or something is where we all came from this. You know we're all technology junkies. So I think that almost is the most interesting story that you know technology came to this point in the last two years where a bunch of us were able to figure it out in different ways and I think that's pretty interesting.
[00:07:54.729] Kent Bye: How would you describe the differences of the type of the immersive experiences that you could have within a full immersive dome versus something that is just in a virtual reality head-mounted display?
[00:08:06.156] Jane Crayton: Well they're great because first off with the full dome environment there's a lot of different domes that have been created so of course they're 360 degrees all the way around but some domes are like 180 from the top down so they come about to your halfway length of your eyes right it's almost like an equatorial plane Some domes go further, they go 270 degrees, and that gives you even more immersion. So the physicality of the dome, because the display unit isn't attached to your head, it actually allows you to look around and see, and you still have that peripheral vision to the side of you where you can see things outside of your field of view that you might not see if you had a head-mounted display. The other thing that's really important with full-dome production is engaging the audience with audio. So most of these theaters are equipped with surround sound systems, and so all of the undergrads that we're teaching, we're not only teaching them how to create great things visually in surround, we're also teaching them how to use sound as an instigator as a marker for something's happening behind you. You might want to turn your head and look at that. And so the dome has the potential to not only immerse you emotionally and physically, it also has that ability to take you there with another person. And I think that's really important. The head-mounted displays, you're alone and there's not that sense of camaraderie between the audience. or even just looking around, are you experiencing the same thing I am? And sitting in different places of the dome is also another important aspect. You see things completely different if you sit in the front row in the center, or if you sit in the middle to the side, or in the very back to the side. Every seat in the dome has a different perspective And it's a really important thing to consider when producing a dome production as to how your audience is going to be seated, how they're going to be looking at the dome, and the potentials of the images and the audio to change their perception by their movement or their interactivity within that space.
[00:10:35.287] Kent Bye: And we're here at the Immersive Education Initiative's immersion. There's obviously some sort of educational component to domes. I'm curious, what does that look like?
[00:10:44.856] Jane Crayton: Well, actually, domes for me, my first experience in a dome was at the Planetarium in Eugene, Oregon. And I was young. I was probably 9 or 10. And my mom signed me up for a summer camp. And I spent the week learning. how to Apple script and I learned about all the optics of the dome planetarium. It was like a very basic optic generator back then. It just shined lights through lenses basically and it would create stars and stuff on the dome. I remember taking this summer camp and it was wholly immersive. It completely captured my interest and was a game changer for me from then on. I was interested in science, I was interested in computers, I was interested in optics and light and all these things. And I think that the planetarium is really the driver of what helped us get to this and the limited ability of it. We started to see that slide projectors were being introduced into planetariums. I worked at Fisk Planetarium at UC Boulder where I got my undergrad. And, you know, when I worked there I was cleaning slides for their slide projectors to project images of cosmos and all this astronomy stuff. But it's changed. Now the Digital Dome is not all about astronomy and it's actually about a far greater reach of potential for teaching other contexts, other subjects beyond that. You know, for instance, even in teaching digital production, we're teaching photography, we're teaching videography, and we're teaching it from a different angle though. So these students are able to apply some of their prior knowledge in these fields, but they have to learn it a little different to be able to make it successful in the Dome. I think the Dome offers a lot for students and educators to really engage with each other and it's an opportunity to also really understand how your perspective and your awareness of something or of yourself changes when you are immersed. And I think that's the most important for the artist who's producing and for the viewer.
[00:13:10.095] Kent Bye: And so beyond astronomy, what type of educational or what type of content is actually being shown in the dome productions?
[00:13:17.949] Jane Crayton: Well there's a lot of different content being shown in the dome. They've expanded it to some cartoons and I believe the first full dome feature film was recently produced. So some of the other things that I'm working on for the full dome beyond planetarium and astronomy knowledge. Right now I'm working with my professor Arthur Joyce from the University of Colorado on a National Science Foundation grant. And part of that grant was to document some of his archaeology sites down in the lower Rio Verde Valley. And his colleagues and him had written several grants to bring me down to document these sites. And in part of my documentation, I suggested that we shoot some photogrammetry and spherical photography of the sites. I want to build out a virtual archaeological site of the Acropolis and some of the other outlying sites. And my professor at first was kind of baffled at why I wanted to do this, but then when I showed him how he could jump down into a pit that I had photogrammetried all of the sides of, and he could see the stratigraphy of that pit in great detail, he was like, Wait, you mean I could show this in my classroom and basically teach my archaeology students about stratigraphy right here, virtually? And I was like, yes, you could. So those are the kinds of transformations that we're looking at now in the dome on how the full dome could actually be used beyond astronomy and other types of learning environments. And so I'm exploring taking this further with Professor Joyce and Sarah Barber and their colleagues to look at transforming their archaeology project into a learning project for their undergrad and graduate students. And as well in my own work, you know, teaching photography and videography and 3D to students and being able to share with them that knowledge but in a full-dome context is a really unique opportunity beyond astronomy. I think the planetarium is growing beyond what it was initially made for.
[00:15:39.742] Kent Bye: And if someone wanted to set up one of these immersive domes in their backyard or at a music festival, what all type of equipment would they need to actually set it up?
[00:15:50.585] Jane Crayton: Well, the first thing they would need to do is download the V-Dome software, since it's free and open source. And you would need a small dome and probably more than one projector. You could do a small, say, 20-foot dome with three projectors easily, and you would set that up on a computer. You'd want to make sure it was a computer that had a pretty good video processing card and a lot of memory and things like that. And there's some, I believe that my colleague Charles has listed out the specifications required to run VDOME. So you should make sure that your whatever computer you choose meets those requirements. And basically you hook up a few projectors and there's a lot of different ways you can use it as far as hardware. A triple head to go is probably one of the easiest hardware to use for making a digital dome. And yeah, it's not that hard.
[00:16:53.017] Kent Bye: Is this all being driven by one computer then and being split out, or is it multiple computers driving each projector?
[00:16:59.672] Jane Crayton: Digital Planetarium started out using multiple computers to slice and so there was one dedicated computer per projector and then there was a main computer that kind of ran the whole thing. So you're looking at you know a minimum of seven or more computers and we're talking top-of-the-line computers decked out. So seven times five grand, I mean we're talking top-of-the-line five grand computers each. You're talking a lot of money. Today, it's not that difficult. Today, with the VDOME software, you can run one computer and run six projectors if you want. You have two video cards, two triple heads to go that run three projectors each. There you go. Bam! Install the VDOME software. Unfortunately, the other groups I mentioned who have created software, most of them haven't released it. I believe that Blendy did most recently release their full-dome version, but I believe it's probably several hundred dollars to purchase. I haven't heard much from the RFID crew about what their plans or program art and the SAT people from Montreal, their team is private and I believe it's all proprietary and paid for by a private company. So they're not releasing that software yet either or that technology. So currently if you want to run a digital dome with one computer, VDome is your solution. It's a little glitchy sometimes, but, you know, we're there to help. And especially Charles, who's great. Charles Veazey is the developer. He wrote the code and initially wrote it all in Max and recently rewrote it in C. So it's running much better now.
[00:18:49.684] Kent Bye: And so you mentioned Unity. I'm curious about how the game engines tie in, especially since a lot of people who listen to this podcast are game developers and are working with either Unreal Engine 4 or Unity. So I'm curious about if people had a project already built out, what would they need to do to output that and put it into a dome?
[00:19:07.102] Jane Crayton: Well, it's not that hard. You really, what you need to do is run the VDOME software and open Unity and you send Unity into it, I believe with a siphon connection. I'm not exactly sure Charles would be able to verify that, but it's not hard. We did do a Unity test and the whole idea behind that is that The full dome is this immersive environment but it naturally works with 3D worlds because three-dimensional worlds are completely 3D and surrounding you and they're immersive and that's exactly what the dome is. And so it's really easy to view that kind of environment in the dome. The one thing that is difficult is movement. Oftentimes, if you move too quickly, you'll make your audience sick or yourself sick. So that definitely is something that needs to be worked out a little better, maybe on the controls. Maybe we modify the controls for Fulldome so they're not so easy to manipulate and go crazy. But other than that, I think that unity is a really up and coming platform to be used in the Dome. I expect to see a lot more unity happening in the full Dome in the next three to five years, big time.
[00:20:25.470] Kent Bye: Yeah, and one lesson from virtual reality is that anytime you take control over the viewer's viewpoint and start moving it around and their body's not moving, then it does create a formula for motion sickness. I'm curious, when you mention that, what type of things should you avoid when you're creating dome experiences that you've found to be really nauseating or cause motion sickness in the audience?
[00:20:45.528] Jane Crayton: Well, one of the most important things we take into consideration when developing full dome productions is the sweet spot. And that is the area on the full dome that the audience's eye naturally rests. And it's actually a pretty small portion of the dome. And so most of our action will actually happen in that part of the dome. And then there'll be like peripheral stuff. It's kind of like your normal eye. The thing that you need to be aware of when producing full dome content is really to take everything just a tad bit slower and really check yourself and make sure you're going into the dome and watching what you're producing. Movements that might be normal on an HD screen or an SD screen will be way too fast in the dome. You have to be very careful about how you pan and move Because you're right, the audience is not in control of that movement. And so it's really easy to get sick. And I remember there was a hilarious Trojan condom commercial previewed at the Immersa Summit a couple of years ago, I don't remember. And I literally got sick and had to leave the dome, take care of my business in the bathroom. It wasn't pretty how sick I was. And I could not go back into the dome for the rest of the night. I mean, that's how sick I got from this. And a lot of people liked it, but I think half the people got up and left that dome because of the way the motion and the video and the animation were driving through it. It was a roller coaster scene of these pigs riding a roller coaster. I don't remember the context of how it was a Trojan commercial, but you know, pigs don't use condoms, I guess, or something. So anyway, yeah, their commercial stuck out. And to this day, I use that commercial actually when I'm teaching to show my students that if they move too quickly, if they don't slow down their pan, slow down their animations just a tad, that they can definitely make their audience sick. And just moving stuff too much around. If you have action in front and then all of a sudden behind and the audience has to move their head quickly, that can make them sick too.
[00:23:01.043] Kent Bye: And with the resurgence of virtual reality, there's starting to be all sorts of solutions for spherical video, like putting 12 GoPros in a printed sphere and other solutions that are taking multiple HD cameras and creating spherical video. So I'm curious, with producing Dome content, what type of solutions have you looked at in terms of bringing in video content into a fully spherical environment?
[00:23:25.982] Jane Crayton: Yeah, the video content is the next level of where we need to start teaching. Unfortunately, with the project that I'm currently working on at the Arts Lab in collaboration with the Institute of American Indian Arts, we did not have the funds to kind of delve into the immersive full-dome video. We would have loved to play with a 360 Hero, we would have loved for you to donated one to us but yeah it didn't happen. We do have a ladybug which has I think the ladybug has five cameras in it. It's got a high learning curve and some of these technologies have quite high learning curves They're new. And so, of course, as researchers and innovators and users and instructors of this technology, we're constantly checking out all that and trying to see what people are coming up with that we might be able to use to pass on to our students. And right now, I think as far as cost-effectiveness, the 360 HERO probably has everyone beat. And that is, you know, the HERO camera mounted on the 3D printed. It's a six-sided cube that takes six HEROs. And so, our current teaching model integrates HD video with spherical photography and animation to kind of trick the viewer to think that they're in a fully video immersive environment. It's much different than the full 360 video. That is the next step and it uses a lot of the same technology and hardware to actually process it, like the stitching of those 360 Hero content uses the same software that you would use to stitch your spherical photography. So the technology is there. I think it's a matter of the schools getting the funds to buy the equipment and that technology is about a year, year and a half old. So eventually it will trickle down to the university systems and we'll be incorporating that into the curriculum for sure.
[00:25:49.285] Kent Bye: And finally, what do you see as the ultimate potential for fully immersive virtual reality?
[00:25:56.355] Jane Crayton: I've always seen the full dome as the ultimate potential to change your perspective and alter your frame of reference. And I think that's probably one of the most important facets of immersive technology is that we all come into these experiences with our own frames of reference, our own personal histories, our own personal viewpoints, our own perspectives, and the immersive full-dome environment allows us to kind of leave that behind and experience it in a different environment, in a different point of view, and actually in a point of view that you can often change and manipulate. And ultimately, that gives you a new frame of reference to learn and understand the material. And I think that that is one of the most important things that planetariums in the past and digital planetariums today have to offer students and educators and their audiences is that it changes you.
[00:27:04.550] Kent Bye: Great. Well, thank you.
[00:27:07.612] Jane Crayton: Thank you. I appreciate it.
