Social VR experiences and telepresence applications are going to be one of the most compelling use cases for virtual reality, and watching sports events within a VR space with your friends has a lot of potential. Virtually Live is taking a really innovative approach to showing live sports events within VR in that they’re not using 360-degree cameras. Instead they’re using optically-tracked statistical data from Stats.com and then transposing it into a virtual arena. They’re able to re-create the positions and movements of all of the players and the ball in a 3D space, and therefore recreate the experience of going to a sporting event with your friends.
Tom Impallomeni is the CEO of Virtually Live and he was at SVVRCon showing off a demo of their technology in action. They’re starting with soccer, but planning on expanding to other sports presumably including baseball, basketball, and tennis as is shown in their trailer for their company.
Virtually Live – Intl. from Metaverse Holdings on Vimeo.
Part of the advantage of using optically-tracked data is so that they can create a virtual camera and put it anywhere to have the best perspective, even start to give you the perspective of what a specific player might have been able to see.
Tom says that we’re still a long ways away to being able to completely recreate what it feels like to be at the stadium with tens of thousands of people, but that they’re getting closer and closer. They’re able to create a shared social space with your friends that goes beyond watching it a game on a 2D screen.
It’s also still a bit of an open question of how realistic their VR recreation will feel, especially when there’s a lot of emotion, body language, and body movements that may be too nuanced for cameras to be able to track at such a long distance. But Tom says that the technology is continuing to improve, and that they’re even looking at tracking facial expressions and limb tracking as a part of their R&D initiatives.
They’re also able to visualize the raw data from stats.com in a way that is improving it. There’s a lot of noisy data from the raw feed, and they’re able to apply some algorithms to help to improve upon the data that’s being collected. Using VR to visualize the data is helping to provide feedback and improve upon these existing optical tracking techniques.
Some of the use cases that Tom sees are going to be compelling is to be able to first be able to share a virtual environment with your friends. He also imagines that it might be really interesting for people to watch a game from their favorite seat in the stadium if they’re using to seeing the games from that perspective. They’ll also be able to create views that were impossible, and so I’d imagine that a tabletop view may be a perspective that’s really new and likely to be super compelling.
Tom says that they also plan on fusing live footage into the experience as well and so there will likely be a way to watch the action in a more traditional way and then have the VR environment be able to recreate the feeling of being with at the stadium, in a room with your friends, and have the sports commentary all piped into whatever combination best suits your preferences.
At the moment there aren’t clear broadcasting rights to data-driven reconstructions within VR, but I imagine that there will be new licensing frameworks if this type of approach starts to catch on.
Virtually Live has some really interesting and innovative technology, and so I’ll be curious to see where these types of reconstructed VR experiences will go in the future. I think there’s a lot of interesting use cases beyond just sports, and it could be applied to many other live performances and concerts, especially when there and better and better ways of capturing action with arrays of 3D depth sensor cameras and other emerging technologies.
Become a Patron! Support The Voices of VR Podcast Patreon
Theme music: “Fatality” by Tigoolio
Subscribe to the Voices of VR podcast.
Rough Transcript
[00:00:05.412] Kent Bye: The Voices of VR Podcast.
[00:00:11.992] Tom Impallomeni: I'm Tom Impalemany, I'm CEO of Virtually Live and what we do is we broadcast live sports games and events but the way that we do this is through 3D optically tracked long distance data which we gather through a six camera rig within the stadium through our partnership with Stats.com, the big Chicago based data tracking company and then we reconstruct that in a virtual venue. So you have a quasi-avatar but in-stadium experience. And the benefit that that gives is complete freedom of viewing position. So you can watch from any seat in the house. You can even stand in the middle of the field or the arena and watch the game live. And also it's a social experience, so you can attend with your friends, which we feel for live sporting events and other events is the key thing really, that actually you don't go to a stadium just to watch a game from that position, you go there to experience it, to watch it with your friends. So that's really what we do, and we have offices in London, in Switzerland, in Spain, and we've just opened up our office here in San Francisco a couple of months ago. So it's my first SVVR, and it's a fantastic event.
[00:01:25.015] Kent Bye: Great, and so I took a quick look at the demo yesterday, and what it looked like was a soccer game with like avatars moving around with the ball and everything, and so What I'm extrapolating is that you're using a service that is already collecting statistics on what is happening in the game, like where the ball's at, who's possessing it, all the other things that sports has broken into, all these different numbers, and you're taking those numbers and translating them into a VR experience that is somewhat delayed from actually being really live, but maybe rendered in a slight delay, but still kind of real-time, pretty much live.
[00:01:59.364] Tom Impallomeni: That's pretty much what we're doing. We actually work very closely with stats and we're essentially helping refine that track data. So if you think about it, if you're tracking from long distance, you're tracking down to a certain square on the pitch and that can mean, you know, for example, that if you take the raw data, the ball will wobble around. We've developed algorithms to kind of smooth that data out in real time. We're also producing a pretty complex game engine to go with it to kind of take that data and render it as quickly as possible. So the latency at the moment is about six seconds from live. It's reducing pretty quickly. But there are two big opportunities here. The first one is data tracking is nowhere near where it can be. There's a huge opportunity. The data is getting deeper and deeper. You know, soon we're going to be able to track, for example, facial movements and hand movements and so on. And the second is really the front-end production side, where we're seeing an improvement in graphics all the time. And so, if you follow this as logical conclusion, what you see now in terms of what we've built and where this can go, I mean, where the potential is pretty mind-blowing.
[00:03:04.866] Kent Bye: Well, yeah, I guess, because at the moment, you're not really necessarily tracking any of the limb or body movements of the players, are you?
[00:03:12.832] Tom Impallomeni: There's some basic tracking, but at the moment not to that level, but that's what we are working on. And it's not far off, let's put it that way. And the other thing is it's, you know, the social element of the experience is really important. So the way that you communicate with your friends, you know, attending live sports events is really a social experience. and we've adopted an approach of layering sound, so you have the background noise of the stadium, and then the sound of the interactions with your friends, and then obviously the commentary layer as well. If you're watching a live sports game at home, it's helpful to be told who's on the ball and so on, or who's in possession. So that kind of audio element as well is super important to the event.
[00:03:53.873] Kent Bye: And so there's the issue of broadcast rights that comes to mind in terms of if you're there shooting a video and broadcasting it live, you have to get a lot of permission. But if you're taking 3D depth data and all this numbers and translating it and recreating the experience, is there any licensing issues that you have to get permission to create these virtual experiences of the same sporting event?
[00:04:15.133] Tom Impallomeni: I think the conversation about sporting rights is a very long conversation but certainly one thing that we like to kind of emphasize is this is a complementary new form of watching sport that really augments the whole experience and enables teams and leagues to In February this year, the cricket World Cup game between India and Pakistan had a global TV audience of a billion people. Almost, what, 8x the Super Bowl. You've got Premier League games in the UK that get audiences of 650 million people plus, so you've got this huge audience that are just watching passively on TV and it's an unprecedented opportunity to offer those audiences even more ways to engage with the game. And that comes with extra revenue streams that broadcasters and sports teams don't currently access. So when you have those discussions, then people start to see where this can go.
[00:05:21.594] Kent Bye: Yeah, I guess, you know, when I think of this, it's pretty amazing that you're even able to do this. And first of all, just take all these numbers and create these compelling VR experiences out of it. But there's something about it that seems to be lost from actually watching the players, their expressions, their emotions, their movements that aren't being picked up at this point, and may perhaps never really be fully picked up from actually seeing it on a video. So I just wonder, what is lost, but what is gained?
[00:05:46.183] Tom Impallomeni: So, what is lost, I guess, you're never going to be able to completely replicate an in-stadium experience because, you know, the feeling of actually being there and hearing that atmosphere and so on, it's never going to be perfectly replicated, but you can get close. And as tracking improves, you know, we're starting to see much better capture of, you know, facial expressions and so on. You're going to start really getting, capturing that emotion of the game. The other thing to kind of emphasize is that we also embed actual game footage alongside the recreated footage so you have the best of both worlds and this is something that we find actually helps because it both places what we've recreated in a real context and gives that kind of emotional element as well. What is gain? There's huge gains in terms of being able to attend matches with your friends from around the world in ways that you couldn't otherwise do. You can get closer to the action than you ever can before and we like to think that those are very good reasons for our existence and what we're trying to produce here.
[00:06:46.155] Kent Bye: Yeah, and one of the other reactions I had from the demo is that I'm in the audience, everybody's really tiny, pixelated, can barely see them, and my thought was like, I don't think this is necessarily the best view to be able to see this experience. And so I'd imagine that perhaps maybe a kind of tabletop, you're looking down at everything as if it was a toy, that would be something that I would find really compelling. I'm curious of what types of different views you've personally experimented with, like where to put the camera, if it's from the stadium, if you're actually on the field. Where do you find is the best experience to view this recreated live sports events in your system?
[00:07:22.403] Tom Impallomeni: So it depends. I guess one of the things you've hit on there is, I guess, one of the limitations of a fixed seat position in the stadium. In that, say you're behind at one end of the field. When players stand at the other end of the field, you're somewhat limited. You can't see the players as closely as you would otherwise be able to. So there are a bunch of different ways that we've experimented with, which is really with our platform you can sit in any seat in the house. So you can move around, you can try different viewing perspectives. We've also had overhead viewing perspectives, so, you know, elevated, and that gives you a different view. But it's interesting because it depends on the actual user. So, for example, the person like myself, I have a season ticket to a Premier League soccer team back in the UK, which now I'm in San Francisco, I can't really use. But I know that my seat in that stadium, I know the view and it's familiar. And for me, recreating that is really important. But that wouldn't apply for everyone. So it really depends on the user as to their preference, but we like to give them the flexibility to choose their viewing position, as well as giving them some curated options.
[00:08:33.983] Kent Bye: Yeah, I've definitely had feedback from people who've either attended the Super Bowl or NCAA Finals, and they have seats where they end up just watching the whole broadcast on the screen because they can't really see anything. And so I think there's certainly limitations to having a physical location and watching the sport events that sometimes it's actually better just to watch it on TV. And so I'm just trying to think of, well, what about in VR? How would you make that so that you could be anywhere, but it's actually the best optimal position rather than the limitations of the physical location?
[00:09:02.615] Tom Impallomeni: So one way of doing that is the approach that I mentioned before, which is sort of embedding the TV footage alongside the VR experience. And that gives you an option to choose a flick between the two, which I guess is the VR equivalent of having a screen in the stadium. But yeah, I think there are ways that we are looking at kind of optimizing this that really, I think the opportunity here is to give people better ways of watching sport. And, you know, it's still a nascent market, but there are huge opportunities in terms of flexibility of viewpoint.
[00:09:36.795] Kent Bye: So what are the different sports that you cover? You mentioned Stats.com. So what do they track?
[00:09:40.918] Tom Impallomeni: So Stats track a whole bunch of things. They track basketball, so they have partnerships with the NBA. They work with a whole series of sports. We've shown a demo here of a soccer game. And being British, people think that's a little bit of a cliche. But actually, our platform is usable across a multitude of different sports. And we're having conversations with lots of different players across different markets. Being in the U.S., we are hopeful of being able to announce U.S. sports soon. So I think certainly the opportunity with our platform is huge. It's not limited just to soccer. It's applicable across a multitude of sports.
[00:10:20.428] Kent Bye: Great. And finally, what do you see as the ultimate potential of virtual reality and what it might be able to enable?
[00:10:27.334] Tom Impallomeni: I think, you know, the ultimate potential, you only have to see the demos here of the latest headsets that are coming out and to really sort of see the variety of approaches that people are taking and being able to transport people to places that they might not otherwise be able to experience. But also, I guess, two important things. The first one is really the social aspect. So, enabling people to really interact with close friends and relatives and bringing that kind of emotional connection to VR. And the second is to really give them experiences that they might never otherwise be able to experience. You know, we're doing that to some degree with our in-field viewpoint. So the fact that you could become Cristiano Ronaldo and watch what he sees on the pitch at a specific point in time, that's something that's never really been done. So it's kind of new perspectives for people that, you know, you're not creating new behaviors, but you're creating new experiences and improvements to what they're really experiencing. I think that's super exciting and having been here a couple of days, there's massive potential in this market. We haven't even got started.
[00:11:31.082] Kent Bye: Yeah, there's something about those memories and those physical things of being able to recreate them and revisit them. I think it's going to be pretty compelling. Yeah, anything else that's left unsaid that you'd like to say?
[00:11:41.586] Tom Impallomeni: Thanks to the organizers here for putting on a great event. And, you know, the energy here is something else. So looking forward to the next one.
[00:11:50.271] Kent Bye: Awesome. Thank you. Thanks a lot. And thank you for listening. If you'd like to support the Voices of VR podcast, then please consider becoming a patron at patreon.com slash voices of VR.
