The developers at Harmonix didn’t set out to create an open-ended music creativity tool with Rock Band VR, but once they realized how compelling it was to simulate the feeling of being a rock star on stage in VR, then they completely pivoted the production of their game. They discovered that it wasn’t compelling to force users to focus on any single gameplay mechanic, and so they focused on allowing users to look around to cultivate deep feelings of immersion and stage presence.
Rather than rewarding precision of playing the perfect studio session, they wanted to recreate what it felt like to give a live musical performance that allowed users a lot of agency in expressing their own musical creativity and giving an entertaining embodied performance. I had a chance to catch up with Jonathan Pardo, QA Audio Lead for Rock Band VR, to talk about their design process of how they were able to incrementally teach players how to play their game as well as some of the fundamental components of music theory for what chord combinations tend to work well within their set of 60 different songs.
LISTEN TO THE VOICES OF VR PODCAST
One of the most impressive things about Rock Band VR is that they’ve actually recorded live guitar sounds of every chord type and rhythm with the proper guitar sounds for all 60 of their songs. This means that you can push a few buttons while strumming and have it sound like you’re playing the proper chords within the context of playing within a band. There are seven different chords with the following finger combinations: Single notes (1), Muted Power Chord (1,2), Bar Chord (1,2,3), Power Chord (1,3), Muted Arpeggio (1,2,4), Arpeggio (1,3,4), & Octave (1,4). This instructional video from Harmonix gives a great overview of the basic game play mechanic:
Harmonic realized that the most immersive experience would be to not have any artificial gameplay visualization, but this mode would be virtually impossible for anyone to organically figure out how to play the game if they weren’t a game developer or professional musician. So the created a Performance Mode with more instructions and guidelines. The Virtuoso Mode can be unlocked after a tutorial, and provides more explicit chord following to help train your muscle memory and understand some of what chord combinations work well together. Then the goal is to eventually be able to play the Monster Mode, which is the most immersive since there isn’t any gameplay visualizations happening but you have to know all of the music theory and listen to the band and follow along — much like a real musician would have to do in a live performance.
Fans of the original Rock Band will be happy to hear that they’ve also included a Classic Mode that can be played in VR:
Harmonix has a lot of musicians who have played a lot of live shows, and so they’ve created 15 different environments in order to simulate what it feels like to play on stage in front of a wide range of audience sizes and contexts. They change how it sounds in each environment and have different pedals to modulate the sound that are activated by looking at the pedal and using your whammy bar.
Even if you don’t want to learn and play all of the different chord combinations, Pardo says that there are other things that you can do in order to earn points by doing an embodied performance that cultivates your sense of stage presence that include things like turning the guitar upright, getting on your knees, doing a head bang, or jumping up and down.
Adding more social features is a top priority for Harmonix, but there are various network latency challenges that will make it difficult to synchronize live performances. Pardo says that they often will hold back on features if they know that it’ll make a worse networked gameplay experience, and that there are some features like live streaming of the audio that previous console versions of Rock Band disable due to sync problems due to latency. They’re actively exploring network workaround solutions as well as how to track other instruments, but these challenges explain why Rock Band VR is only launching with guitar support.
While Harmonix didn’t originally set out to create a live performance simulator emulating what it feels like to be a rock star, the unique affordances of VR slowly led them down this path. Pardo says they kind of accidentally created a music creativity tool, and that it was more about designing a game to be fully embodied and present in the moment on stage rather than the type of precision that you’d want if you were in the recording studio. The game play is not easy, but neither is being able to play music. Musicians should have an advantage in learning to play Rock Band VR, and it will be exciting to how games like Rock Band VR will help train and inspire gamers into learning how to play actual instruments.
You can learn more about the release of Rock Band VR and the compatible guitar controllers in this Oculus blog post.
Donate to the Voices of VR Podcast Patreon
Music: Fatality & Summer Trip
Rough Transcript
[00:00:05.452] Kent Bye: The Voices of VR Podcast. My name is Kent Bye, and welcome to the Voices of VR Podcast. So before GDC this year, Oculus was showing 12 different VR games, and the one that really stood out for me was Rock Band VR. So Rock Band VR is a little bit different than classic Rock Band. In this interview with Jonathan Pardo, he's the QA audio lead for Rock Band, He talks about their design process where they started on one track, but then they discovered this open-ended performance mode where it's a little bit more of a music creativity tool, ended up being the most compelling. And so he talks about some of the design considerations that led him to this path and how they did a pivot in the middle of production into really focusing on creating this feeling that you were on stage and becoming a rock star. So, that's what we'll be covering on today's episode of the Voices of VR podcast. But first, a quick word from our sponsor. Today's episode is brought to you by VRLA. VRLA is the world's largest immersive technology expo with over 100 VR and AR experiences. They'll have tons of panels and workshops where you can learn from industry leaders about the future of entertainment and storytelling. I personally love seeing the latest motion platforms and experiences that I can't see anywhere else. Passes start at $30, but I actually recommend getting the Pro Pass so you can see a lot more demos. VRLA is taking place on April 14th to 15th, so go to VirtualRealityLA.com and get 15% off by using the promo code VRLA underscore VoicesOfVR. So this interview with Jonathan happened at the beginning of GDC on February 27, 2017 in San Francisco, California. So with that, let's go ahead and dive right in.
[00:01:57.875] Jonathan Pardo: I'm Jonathan Pardo, I'm the audio QA lead on Rock Band VR. We've been working on this project for a little over a year now. We're co-publishing it with Oculus, and it's just about to be released. The company is really focused on VR right now. We already came out with Harmonix Music VR for the Sony headset, and we are actually also showing, it's called Sync Space, it's a karaoke game, it's for the Gear VR, and it's a multiplayer game that you can play with up to five people.
[00:02:27.303] Kent Bye: So maybe you could tell me about you know evolving this game of a rock band VR to be well suited into VR and Trying to take the classic lessons of the original and what did and did not work.
[00:02:38.734] Jonathan Pardo: Yeah. Yeah, so that's absolutely something that was a process for us trying to get from classic rock band that everybody knows and loves the track coming down the gems that you smash with the frets and you're basically emulating one-to-one with the original guitar part. And as we were talking about, that's great for console games. It is perfect. But the only problem is it really guides your eye, and you really have to stare right at the track the entire song. Not great for VR. So you want to be able to look around. You have this whole world that we put in front of you now. with the drummer behind you and the crowd in front of you, these cool venues, like we have a huge theater that you want to be able to appreciate it. So looking at a track is just no good. We still have classic mode just because people want it. Like it's like legit, like it's a great game. We've made it for a long time, but we realized that that couldn't be the core experience. So what we came up with was performance mode. Performance mode is basically giving the agency of the guitar part, creating that guitar part for these songs over to the player. So you are creating your own chords, the different button combinations on the left hands for the fret buttons make different chord sounds. So like one will sound like a power chord, one will sound like a bar chord, and you can kind of mix and match those as you like. Every once in a while we'll guide you to certain ones that we think are good for that moment, but the majority of the song you kind of get to do whatever you want. while you're doing whatever you want, we also are asking you to do specific things that we think always sound good. So these are combos, and they're just patterns, musical patterns, like ABAB, ABCD, these kind of patterns that we guide you to. If you can do as many as you can in a row, As many of these combos, basically you're trying to get a perfect section. So if you play combos throughout the entire verse of the song, you get a bonus score for doing those musical choices perfectly. If you can do that throughout the entire song, then you get another bonus for just like playing perfectly the entire song. So that's the way to get gold stars. We also have another mode. So that's basically what we call rock star mode. We have another mode that's a little bit more difficult. It's called Virtuoso Mode. It has the exact same features that I just explained, but also we have this mode called Chord Following, this feature called Chord Following. What you're doing there is you're actually moving your hands up and down the frets in time with the chord changes of the actual song. So if you can do that while also making combos, which is very difficult, easier said than done, I can speak from experience because I've had to do a lot of the gold star passes and it is tough. Our game is hard. But it's really fun if you can do it right. And it takes a lot of, you know, replays and, oh shit, I missed that one chord and now I screwed up my combo and now I'm not going to get a perfect section and now I missed the chord follows and now I need to restart. And then there's top tier. Monster mode. That's all of those things that I just explained. No HUD. No UI. So you're basically on your own. You need to like understand all of those things in your head, counting out the numbers, counting out the bars, trying to do it perfectly. It's very musical if you're naturally musical. If you're a musician, it's going to come a little bit easier. If you're not, it's going to take some time.
[00:05:46.907] Kent Bye: Yeah, the really interesting thing that I see here is that, you know, it's actually very difficult to play an instrument and to understand a lot of these musical theory elements to be able to actually be on stage and play along. But yet you're turning this into a game. And so you're doing these abstractions such that, you know, you're trying to simplify it and make it simple for anybody to kind of casually get up and start playing. But yet, if you have that music background and you have experience with playing a guitar, you would perhaps presumably just jump in and maybe be able to play it better than someone who might not have that. And so I'm curious to hear that process of trying to translate something that is overall pretty complicated into something that's a fun and enjoyable game.
[00:06:26.748] Jonathan Pardo: Yeah, that's a great point. I mean, it's kind of the thing that we, it's the challenge that we love and it's the bane of our existence, right? Like being able to create a tutorial that teaches somebody how to play the game is one thing, but being able to start with, here's how music works, that's a big place to start. And there's a lot of ways to mess that up. Our tutorial has gone through multiple iterations and it's taken us months because we didn't want to shove all of these music jargon in people's faces. We didn't want to have to tell people like all the different musical terms that we're actually utilizing. We just wanted it to be natural. We just wanted people to like naturally understand it. So we still do have to teach like here's how you strum in time and here's the different Like, you know, you could strum an eighth notes. You can strum a sixteenth notes. Like, these are things that, like, we kind of, if we're going to give the agency of the guitar creation part over to the player, we need to, like, give them a little bit to start with. Whereas in classic rock band, it's really just hit note now. So that's, you know, you still, if you can start to get the rhythm pretty consistently, it's going to help you in the long run. But at the same time, it's not really a requirement to teach people what an eighth note is, what a sixteenth note is. So yeah, it is definitely a hurdle that we sometimes have to face, but at the same time, the amazing thing about this game is the fact that If you don't know anything about music, you can have a fucking amazing time on like on stage. There's actually we acknowledge if you're doing certain movements that are kind of like stage presence movements. You know, if you do an upright, like hold your guitar up, if you like get low on your knees, like we acknowledge that these are ways that you can actually deploy overdrive. So we classically have like put your guitar headstock up in the air and that's how you deploy overdrive. We have a few different ways you can do that now. and then you can extend it by doing a motion like headbang or jumping up and down. So it's kind of bringing more than just the musicality part, but also just the stage presence part. So if you don't happen to be very musical, you can kind of play the game that way. But yeah, there's a lot to teach the player in the beginning, and we tried to like parse it out as much as we could to not throw too much in your face. Me explaining all the different features that we have in our gameplay is like kind of obvious. Like there's a lot to digest. and we've somehow made an amazing first-time user experience that we're really proud of.
[00:08:55.766] Kent Bye: Yeah, you know, I had a great time playing the song, and I think that, you know, there's a certain amount of putting the experience on rails such that, you know, it's in the proper chords and, you know, whatever I'm playing, it sounds like it actually goes with the song, you know. But, you know, the question that I have is that as a new user, you know, I felt like I may have been able to get it and maybe perhaps plateau, you know, as to I've got to a certain level of playing through it for the first time. How do you break that down into being able to take the user on this journey of learning, on this learning curve, and allow them to see progress as they're getting better? Because you have, in the background, you have like these cues and these suggestions, and so it's maybe sort of a matter of trying to go step by step and to slowly learn the fundamental building blocks of playing along these songs. I'm just curious to hear that process of, you know, that learning curve that new users have. Because as game developers, you may already get up to a certain point. It's difficult for you to necessarily test it yourself if you already understand it. But to, you know, do all this user testing, to be able to actually do this game progression of this game.
[00:10:01.776] Jonathan Pardo: So yeah, that's a great point. I mean, it's a great question. We spent tons of hours playtesting this, specifically trying to figure out how much information we give the player and then how much we cue the player to do things. And that, I think, really guided us in terms of trying to figure out what to start with and then what the entire gameplay is. Because we have a lot of different features that we kind of jammed into one game, and we didn't want to force players to look at something. That was really the thing that was driving us. We wanted to be able to give the players the opportunity to look around while still playing the game. So cueing people to do something every moment of the entire song is not the way to do that. So we tried to bring that out. When you brought that out and you weren't cueing people to play anything, And now they were just creating something and trying to remember where they were in the song in their head. Now you had an opposite problem. Like now people don't even know if they're playing this song right, if they're doing anything right. So then we needed to start to give them a little bit more information. So you saw on the song map that as you were playing the song, you were building up those combos and you were laying down at the end of every measure the different chords that you played. So we realized we did have to give some feedback. It's not really required to look up there, but it helps the player know that they're almost there making a combo or something. So that was the first piece, understanding, OK, we need to give the player something, but we don't want to guide them all the time. And then also there was this idea of court following, like being able to have that top level play experience. It was something that we thought was going to be in just the main mode. And then we realized that it confused people when they didn't know what combos were. They thought it had to do with combos. And it annoyed people that they weren't good enough to play it. So we kind of were just like, you know what, we need to make a new mode. that the only difference is that it has cord following, and that's an unlockable mode. There's a virtuoso tutorial that teaches you how to cord follow. Once you do that tutorial, you unlock that, and now you can play that in the campaign if you want. A lot of it is just parsing out the different possibilities that you have. Monster mode was that mode that I'm saying, like, was where we were at the beginning. There was zero feedback, like you didn't really know if you were making combos until you finished making the combo. So that's actually one of the versions that we started with, was just this really hard mode that only if you were the developer of the game could you really understand what was happening. So we eventually brought that back, because we were like, you know what, people are going to want that. It's probably one of the most experiential versions of the game, because there's nothing to look at anymore. So you really are focusing in your brain on what you're playing, like a musician actually would be, but you also get to look around at your band much more. So if people can get to that level, then that's great. But parsing out all the different difficulty tiers is probably the best way that we could have gone about helping the player learn all these different features, get comfortable with them. There is kind of a difficulty ramp as you go through the story mode. We have the more difficult songs later in the game. So once you start getting more comfortable and feel like you understand it, then you're going to come across some more difficult songs.
[00:13:16.217] Kent Bye: Yeah, the thing that's really interesting about this project is that it's kind of training people to become musically minded in a certain way, teaching these different principles and how to, in an open form way, you know, having enough agency to kind of make the song your own. Now, you had a number of different finger combinations on this guitar. You have like five keys, essentially, and you have your four fingers, but you have the ability to do these different combinations, and I'm curious if you could kind of break out these different combinations that are in the game and how that translates into some sort of actual chord theory in the process of creating music.
[00:13:51.323] Jonathan Pardo: Yeah, so we have probably, I think it's eight different chord shapes that we accept and make audio from. The first one is just single notes. That'll just play a single note. One fret held down plays a single note. Two frets held down next to each other makes a muted power chord. Three buttons held down next to each other makes a bar chord. So that's like, as you kind of add more buttons, it gets a little bit chunkier. It adds a little bit more notes. Muted power chord is literally two notes. Bar chord has up to like six notes because it's the full breadth of all the strings of the guitar. You have power chord, which is two frets with one in between not held. And so that's the same thing as the muted power chord, except unmuted. So it's like full, a little bit louder, a little bit stronger. You have 1-3-4, so that's three frets held down with one in between. That's an arpeggio, so that's playing different notes of the chord one at a time. You have the same thing as that except muted arpeggios the same kind of like the muted power chord verse power chord and that's one two four and Then the final one is one four So two frets held down with two in between and that is an octave chord So basically just the same note one an octave higher than the other one
[00:15:07.606] Kent Bye: Yeah, so you're essentially abstracting out something that would normally be very complicated, you know, finger arrangements that you're doing, but you're doing it by just putting a button, but having your fingers kind of in different combinations of these buttons, but being able to get the musical quality of that and having a direct experience of that. And I guess what that may lead to is people actually being inspired to understand the theory enough to then actually do and play on actual music eventually, perhaps.
[00:15:35.628] Jonathan Pardo: Yeah, I think that's true of many of our games. We've heard so many stories about people that play rock band that then get inspired to actually pick up that instrument for real. I think the drums are an obvious actual parallel to drums in terms of you have sticks and you're hitting pads and if you are playing at expert level, you could probably play drums. That's the long and short of it. I mean, it's interesting, for our game, we actually have a guy that playtested our game, Matty Sudevan, he's a producer at Harmonix, and he playtested Rock Band VR, and when he got out, he was just like, man, I miss playing shows. He was like, I'm sad, it makes me sad, because this is what it's like. So, you know, musicians, this resonates with them. It feels real, the creativity that you give the player, if they know what to do with it, It's really exciting. My brother Steve Pardo, he's actually the audio lead on the project, and he went to NYU to do a talk about Rock Band VR, and he actually got on stage with two actual musicians, a guitarist and a drummer, that were actually playing the parts. and he had the Rock Band VR guitar and he was in the headset and they played the song and he actually soloed on the plastic guitar. And it was, I mean, you know, it was one of those things that the only way people could describe it was bizarre, but it was also like a real musical performance. Like he was actually making musical choices. I mean, he's the audio lead, so he knows the system more than anybody. But at the same time, anybody can get up there and play a solo on this thing and very intentionally create something that feels like a real Rippin' Solo.
[00:17:14.307] Kent Bye: Yeah, and you know, you have the two different octaves where you have, you know, at the end of the guitar, you have the low notes. And then if you go down near the actually where you're strumming, you get the high notes. And so that's where I think you get a lot more flair when it comes to the different solos, you know, the guitar solos that people really play. So can you talk a bit about the differences between the audio that you have between what's playing on those two different octaves of those buttons?
[00:17:39.251] Jonathan Pardo: Yeah, so actually they're very similar. It's literally taking what is happening in the lower frets, bringing it up an octave. So the guitar parts that are coming out of the guitar, there was a huge amount of work done by the audio folks that created this content. to actually emulate these guitar parts from these songs that we all know so well as closely as possible. So for every single song, they had to record every single chord playing through the entire song in every single strum speed, every single possible strum speed, make sure it sounded good, make sure it sounded right, it was all the right notes, and then use the correct amp sims to make sure it sounds like the guitar from that original part. And all of that work, that was like a huge ton of work, It's given us the ability to experiment in each song and make each song a unique performance. I mean, it's kind of breathtaking the amount of variety you can do with every version of a song. I mean, I've been working on this game for over a year now. And it's still inspiring that I can get in there, make a solo that sounds completely different than any other solo I've used. There's four pedals on the ground. I have the whammy bar. I could really get it kind of out and weird. Because we try to make it so you can always make it sound good. And if you know what to do, you can make it sound pretty out and bizarre. I wouldn't say bad, but you can make it sound pretty bizarre. So it's like accidentally a creativity tool in a lot of ways. It's not really just a game.
[00:19:07.889] Kent Bye: Yeah, I think that having it open enough like that, I think that was the thing that was really striking to me was how open-ended it is. And as you were talking there, are you kind of implying that you have to actually record on actual guitars and then take that and then synthesize it? Or I'm just wondering the limits of being able to synthesize things electronically and the difference in the sound quality that you're able to do between something that's completely constructed versus something that's, you know, recorded with some of the actual equipment.
[00:19:36.077] Jonathan Pardo: Yeah, I mean, we could have done it a little bit more synthesized, but it actually is full recordings on a guitar, every song, every part, every chord type, every strum speed. And we have 60 songs, so that's a lot of work. The mode itself, performance mode, was actually not the plan for the game in the beginning. We had other ideas that we were floating to each other and messing around with for a very long time. This mode was supposed to be an extra little feature. You press down a pedal and now you get to play whatever you want, rhythm guitar part. And it was kind of like a pet project for the audio folks. Once we got our hands on it, after months and months and months of it being worked on, kind of on the side in a silo, we brought it in, we tried it, and everybody loved it. And it was like, this should be the game. And we brought it to Oculus and they felt the exact same way. They said, We know you've been working on this other thing, but this is the game. And that was before we even had gameplay. It was just a creativity tool at the beginning. It was supposed to be like a power-up or something. And so then we had to, you know, we were already in production. So then we had to make the decision like, okay, what are we going to do with this mode? What is this game? And it was a lot of long hours and a lot of long meetings. And I think that that's why it is so unique because it wasn't the kind of thing that you could explain and say, yes, this is what the game should be. We made it. And then we realized that we had kind of struck gold. And so we couldn't turn away from it. And, you know, now here we are.
[00:21:07.727] Kent Bye: I think the feeling of actually being up on the stage, it's really playing into a lot of the unique affordances of VR, which is to transport you to another place. You have this crowd that's in front of you, and you're kind of playing in a small club. It's not like you're on the middle of a huge stadium. But I think the actual environments that you're playing in could actually change the way that you interact and play the game. And so I'm curious if there's just one environment, and maybe some of the process that you had in terms of creating this environment to give that feedback of actually performing in the small club.
[00:21:42.690] Jonathan Pardo: Yeah, so we actually have 15 different venues. We have one venue for every single campaign show that we have in the story mode. It ranges from small little clubs, to bowling alleys, to a crab shack, and then of course the big theater that everybody wants. Playing in a theater is insane. People definitely, playtesters have gotten stage fright. It is, you know, the binaural audio of being in a huge theater is exhilarating. Seeing, like, just, you know, a huge swarm of people is amazing. So working on that is, yeah, well it's like what you said, making a game that is, somebody is a rock star on stage, it seems kind of obvious, it's one of those things that's like, well no doubt it's gonna be fun. And so even for people that have no interest in gaming, like getting into this mode, or you know, getting into this version of Rock Band, people just immediately love it and they kind of understand that they can just go buck wild. Like, there's a lot of people that, they just ignore the game. They play the guitar, they do the motions and stances even if they didn't think that they were supposed to be doing motions and stances, they're getting credit for it. And, you know, you can actually play the game pretty well if you do that. Accidentally get like, you know, two and a half stars doing that. You're not going to get five stars at all. But yeah, that environment is a huge part of the game. Transporting, giving people the ability to be a rock star is very obvious and people love it.
[00:23:10.752] Kent Bye: Do you have any personal favorite stories or memories of playing in Rock Band VR?
[00:23:16.352] Jonathan Pardo: Ah, personal stories. Yeah, I mean, I can think of one time that I loved demoing it. There was this guy who was trying it and we had this thing at the time called missions. So they had little, like three missions per song that was on the ground. It was like next to your pedal board. And one of them was headbang plus any phrase. So that's if you did the motion headbang up and down with your head while playing a combo. But for some reason he thought he was supposed to sing that, so he got up to the mic, which you can actually talk into and it goes through the speakers, and he goes, headbang plus an E phrase, headbang plus an E phrase. And I mean, God bless him. He did it for like 30 seconds. And I don't know why, but you know, hey, he was having a hell of a time. So if it works for him.
[00:24:07.197] Kent Bye: So I didn't notice the foot pedals. Are there foot pedals that you can engage with? And how do you actually trigger them if you're not tracking your feet?
[00:24:14.199] Jonathan Pardo: Yeah, so there's four pedals at your feet. There's a whammy pedal. There is echo. There is vibe. And there is one more. But basically, they don't do anything other than just affect the audio. They're great during solos. Like, if you want to get a rip and solo, then you can do your own unique combination of the four. And yeah, I mean, it's a toy. There's a few things like that in our game that are just things to mess around with the audio or just with the environment. You can hit the headstock of your guitar on the crash cymbals if you're next to the drummer. You can teleport around with the whammy bar. And that's the way that you turn it on is use the whammy bar on the guitar while looking at either the teleport spots or the pedals.
[00:24:58.205] Kent Bye: Great. So what do you want to experience in VR?
[00:25:01.432] Jonathan Pardo: Oh, what do I want to experience in VR? I really like the idea of being somebody that I can't be. I actually am in a rock band, or in a folk rock band, so I've experienced that. And so it's nice to be able to contribute in terms of like, this is what it actually feels like to be on stage. A lot of people at Harmonix are former or current rock stars, so we have a lot of experience in that world. But everything that I've tried that has me being somebody that I'm not. I think that that is the future of VR. That, and being able to go places that you can't go. That seems like the obvious two things, right? Like, I did Google Maps, and I got emotional going to Florence. I went to Florence, and me and my wife went there for a really long vacation one time, and being able to, like, revisit those places, revisit my childhood home, like, it was an emotional experience, and, you know, they just did such a good job, and it's almost like a prototype, but it's still, like, just perfect. Yeah, I think like being able to experience like being, you know, I don't know, a monster or even like a member of the opposite sex or like a child, like being able to like experience being put in somebody else's shoes quite literally. There's something about VR that does that so much more than console games. Just like actually having that empathy, having that eye contact, it makes it so much more real.
[00:26:26.564] Kent Bye: Awesome. And finally, what do you think is kind of the ultimate potential of virtual reality and what it might be able to enable?
[00:26:34.945] Jonathan Pardo: I think socially, that's where it's really going to win. If we can start connecting people just in the same way that we did with FaceTime. I mean, it was just one piece of technology that gave you the ability to be next to a family member at any time. I remember when that came out, it felt life-changing. And I think that VR is going to be able to do that tenfold. Being able to be in the same room as somebody and playing a game or not, you know, just like being there and like goofing around. I think that there's a lot of apps that are capitalizing on that. I think that we're going to go there because it seems like obvious next step for our kind of games are a little bit more casual, and social is usually where we thrive. That's where we went with Rock Band 1. Like, why just have guitar when you could have all four instruments? Like, get the whole band together. and it became the ultimate party game. So being able to do something like that in VR, whether it's Rock Band or something else. that's where the bread and butter is. I mean, that's what Sync Space is doing right now. They have five people singing together, and it's surprisingly enjoyable, even when it's the simplest game, because you're talking to each other, and you're singing with each other, and that is one of the things that makes VR so unique, is it's bringing you together with people in a way that console games just don't really deliver.
[00:27:57.328] Kent Bye: I know that there's tracking issues with getting a full band within Rock Band VR, drumsticks, and you could have two lead guitars. Are there other big barriers in terms of audio latency when it comes to networked Rock Band in VR? Because I think that's the big limitation not a lot of people have. four VR-ready PCs in the same area, and so you have to deal with networking issues. And so, is that the biggest limitation, or is that something that you're also looking into, like how to get tracked drumsticks and how to get a full band within VR?
[00:28:30.379] Jonathan Pardo: Yes. I mean, absolutely. Whether it's VR or not, just being able to have a latency-free multiplayer music experience is very difficult. And you know, there's workarounds for a lot of it. If you have a track that's just playing, we can control where the audio is at that time. Like one of the struggles that we had with console rock band is just like vocals. You can't really stream those vocals and get it perfectly in time with the song. So we actually just don't stream the vocals when you're playing a multiplayer rock band over the network. It is something that we're talking about all the time, being able to overcome that. I think that we have some good ideas on how to accomplish that and how to get past it or at least work with it, work within those, whatever those milliseconds are that we have. And I think it also kind of guides the way we design these games. You know, sometimes we realize, okay, maybe we can't have that feature because it's just not going to work with online multiplayer. And, you know, having those decisions made early on so we don't get ourselves into a place where it's impossible to go back, that's important.
[00:29:38.706] Kent Bye: Awesome. Well, thank you so much.
[00:29:40.147] Jonathan Pardo: Yeah, it was great to talk to you.
[00:29:42.128] Kent Bye: So that was Jonathan Pardo. He works at Harmonix as the QA audio lead for Rock Band VR. So I have a number of different takeaways about this interview is that, first of all, Rock Band VR, I think, is probably one of the more compelling VR experiences that I've seen so far. And I think there's a couple of reasons for that. First of all, I think having the actual haptic feedback of playing a guitar within this VR experience actually makes a huge difference of just creating this sense of immersion. There's this little clip that's on the end where you put an Oculus Touch controller, and then when you're in the VR experience, you have this one-to-one tracking of the guitar. And you also just get this sense of embodied presence as you're on stage and in these different environments playing to a live audience. So to me the other really fascinating thing is just the design process that they went to to be able to come to you know having the performance mode be really the central part of the game. So what they realized is that if they did the more classic mode of Rock Band where you basically have the notes coming down and you're trying to do these timings that You end up just focusing on that and you're not able to really kind of look around and really appreciate what it feels like of being on stage. In the trailer, the video for Rock Band VR, they say that the classic mode for Rock Band VR is a little bit more about getting the perfect performance for recording in the studio. And that this Rock Band VR is a lot more about doing a live show where you're able to just innovate and be creative and understand the basic principles of music and then be able to take those constraints and then still work in the bounds of the song but be able to do this live performance. So there's something that they're really tapping into that I think is super powerful. One is that they're using these seven different types of chords where you have either a single note, a muted power chord, a bar code, a power chord, a muted arpeggio, an arpeggio, as well as an octave. So there's like these seven different finger combinations and you can go from one to the other and they teach you throughout the story mode of kind of what works musically and if you're able to do a number of different combos sequentially then you get bonus points. So the thing I find really fascinating about this is that they're kind of like teaching these principles of music theory. You know, one of the things that Jonathan said is that it's one thing to teach somebody how your game works and how to play the game, but it's a whole other entirely different thing to teach someone how music works. And I think that Rock Band VR is actually starting to do that with all these different modes, starting with the performance mode, and then they have the virtuoso mode where you do a little bit more chord following. And then eventually, I think the top tier is this monster mode where you don't have any cues. When you're in the performance mode you have these different visualizations of the song as it's unfolding and they're giving like little tips and you're able to kind of follow that and take their suggestions or not or you can kind of do your own performance. But I think the goal is to get good enough about understanding the principles of music so that you can eventually just go into the monster mode and just feel like you're playing on stage like any other musician would have to where they have to kind of keep all this information in their mind. So I think this is just a fascinating project and really interesting that they're able to really just discover this and to kind of organically find that this was actually the game, even though it was sort of a side project that was being worked off in its own silo and was going to be an add-on. But, you know, being able to take this, playtest it within their company, but also show it to Oculus and get the feedback of, you know what, this should actually be the entire game. And so it sounds like in the middle of production, they had to do a huge pivot and really focus on making the performance mode the game. So it's just really interesting to also just hear about their process of trying to create this game progression of being able to slowly teach people all these different mechanics to allow them to eventually get to this point where they could move into this monster mode. So at the end of it, they've kind of accidentally created this music creativity tool and focusing on the realism of, you know, playing all these different songs and all the different chord types and all the different strum speeds. So that when you're playing it, it's not synthesized. It's actually sounds like you're actually playing the guitar. So you're doing like these abstractions where you're just pushing, you know, two or three buttons, but you're able to play the full chords that are much more complicated chords that are being playing in the song. You know, the story that Jonathan says of his brother Steve playing at NYU where he's able to do this guitar solo and to be able to just use the game to be able to actually give a live musical performance. I think this is like super powerful and even being able to do like different modulations of, you know, the different pedals that are there as well as, you know, just having the sound sound different depending on where you're playing it. So I'm actually really looking forward to playing this game more and to just learn more both about music theory, but also just to, you know, play the game and get better at the game and hopefully have that, you know, by the end of it, just be inspired to learn more about music theory and to just understand more about the process of creating music. So it sounds like on the long horizon for harmonics, they're going to be looking into trying to integrate more social features. But the issues of network latency and trying to play synchronized with each other, I think, is one of the biggest open problems that they're trying to find different workarounds for. And that, you know, they're trying to really constrain the features that they're adding and to be able to do this in a multiplayer experience. So I think that's why they probably started just with the guitar at first and so that they can start there and perhaps figure out some workarounds for how to include a whole multiplayer band social experience. So that's all I have for today. I just wanted to thank you for listening to the Voices of VR podcast. And if you enjoy the podcast, then please do spread the word, tell your friends, and become a donor. Just a few dollars a month makes a huge difference. So donate today at patreon.com slash Voices of VR. Thanks for listening.
