The decentralized metaverse will require the transferring of A LOT of data. Distributed, peer-to-peer file systems like IPFS could provide the backbone of the decentralized metaverse.
I talk with David Dias who is one of the developers of IPFS about how they’ve been able to transform your web browser into distributed file system node, how Filecoin could create an incentive mechanism to support a robust decentralized file system and mesh networks, why content deliver networks aren’t of much use if a region doesn’t have a viable ISP network, and the challenges of making a peer-to-peer file system more robust and reliable.
LISTEN TO THIS EPISODE OF THE VOICES OF VR PODCAST
This is a listener-supported podcast through the Voices of VR Patreon.
[00:02:52.206] Kent Bye: So IPFS, the Interplanetary File System, the dream of being able to have files that are decentralized, how do you make that work?
[00:03:00.870] David Dias: Yeah, so the first thing that you really need to do is really understand the reason why the web breaks when connectivity is not so good is because you rely on knowing the location of the content in order to fetch it. You didn't have to rely on the location if you relied on something primitive that is stronger, like cryptographic hash, which is kind of like a fingerprint of data, like something that if you look at the fingerprint, you know that you are looking at the right content that you're looking for. then you don't longer need to go to that location to fetch the content. You can just ask the network, ask someone that might be next to you. And once you receive it, you'll get the fingerprint and you realize that, yes, it is the right data. So you don't need to rely anymore on a central point of authority. A very fun example to think about this, I really like to explain this one, is imagine I want to recommend you a book, and instead of giving you the title and the author, I tell you, oh, go to the library in Lisbon and go to Corridor 3, Shelf 4, the third book from the right. And then you go and travel, you take a plane, you go there, you spend a day to get there. And then when you go to that library, you realize that the book is no longer there because someone requested it. So now you have a 404. Or someone changed the book to another location. Now you're going to read a book that is actually not the one that I recommended to you. Or you look at the book and you realize, oh, I had this book in my backpack all along. I read it 10 times. I didn't have to go to Lisbon to get the book. And this is how the web works today. A location-based web means you don't know before you get things. and so you lose a lot of optimizations.
[00:04:36.805] Kent Bye: Great. So I guess right now I have a podcast, The Voices of VR. I upload it to Amazon Web Services, in part because I know that it's reliable, it's going to be fast, it's going to be distributed across the world. And with something like IPFS, is there a persistence there? Is there going to be something that is, I guess, I think of it as a metaphor of BitTorrent, where if lots of people are torrenting something, the popular things stay around. But if people aren't actively seeding that content, then it's going to essentially disappear. So how do you prevent things from disappearing on something like IPFS?
[00:05:08.979] David Dias: Totally. And I'll answer that question in two parts. The first one is really just to break that assumption that by uploading it to Amazon, you get instant free access and that is available everywhere in the world. But because the reality is Amazon is really good at providing services to developed countries. Like if you go even to a mountain here, you will have flaky connectivity and suddenly the thing doesn't work. And there's a lot of countries that need access to this kind of knowledge and they cannot because the network infrastructure is not there yet. And so, yeah, we need to break that assumption like Amazon or Google Cloud or Microsoft Azure just doesn't work to every single place in the planet. The second part of your question is, okay, how can I make sure it persists? So you can do it in multiple ways. You can also, like, rent some Amazon machines and, like, rent some IPFS nodes there. And in the same way that you paid Amazon in the past to, like, store your content, you are still paying. The difference is now it is also being served by IPFS. And so if other people replicate it, you still get access from those, like, you can access the files from those people. You don't have to go always to the Amazon servers. But the next thing, or the thing that it builds on top of IPFS, that is a new protocol of its own, is really Filecoin. Filecoin is a cryptocurrency incentivized storage network. What this means is, essentially, you create a free market. A market where anyone that has some storage space, can be your phone, can be your laptop, can be your tablet, whatever, can offer to the network its location, its bandwidth, and its storage capacity. And so if I'm here and I have a video on my phone, or if I have some 10 gigs available on my phone, and people here want to watch a video from some provider, the provider is now interested in paying me, rewarding me to use my storage so that the video gets very close to the users. And this way, you kind of like bootstrap an entire new network, or you get access to a lot of latent storage that is out there. Today, there's like data centers all over the world, or even just like hard drives all over the world that are not being used, because there is no protocol to actually ask, can I use your hard drive? And with Filecoin, you make this conversation, like this deal, happen very smoothly. You just say to the network, I want to store these files, and then the network can tell you, oh, I have this storage available with these conditions in this location.
[00:07:26.351] Kent Bye: Yeah, as people start to move away from having PCs and laptops and move more to mobile, mobile phone has a limited amount of battery, limited amount of potentially, depending on the internet service provider, you may be bandwidth limited. So you can only have like five gigabytes of up or down. And so is the idea with the IPFS that eventually will move away from some of these centralized ISPs and something that's more like a a decentralized mesh net that is somehow building in through this cryptocurrency file coin an incentive such that people could actually create an economy that is a decentralized way of distributing files.
[00:08:01.026] David Dias: Yeah, totally. So you read my mind. I was going to literally talk about that. Once you get a fully decentralized network where you can find content from any place and leverage that, make the web feel faster. Because another problem that is happening right now is the amount of data that we produce is increasing very rapidly. And the pipes of the internet are not increasing that rapidly. And so it's going to start to feel that the web is getting slower. Because we are getting 4K video, 8K video, 12K video, and so on. And because the pipes are not getting faster, it's going to be hard to move that data. We'll go back to the scenario where it's going to be faster to actually do package switching, where you just ship hard drives, than actually shipping things to the internet. The same way that you do that for storage, you can also do that for bandwidth. You can have a mesh network where people are incentivized to deploy these nodes to create local connectivity, local mesh networks, where instead of going to the ISP and then back, or going through some server and then back, you just find your neighbors that have the content that you're looking for, and you establish direct connections, and you ask them directly, bypassing completely the regular internet infrastructure. Yeah.
[00:09:10.229] Kent Bye: So it sounds like that with Filecoin, it's less of a speculative cryptocurrency in the sense that the ones that are more of an asset that is not doing any actual work, that you're, it may be more susceptible to the market dynamics that could be either overtaken or controlled by normal market forces. But it seems like you're creating a little bit more of a economy that is given a take that is more equitable in some ways. Do you have a sense of that this is creating a fundamentally new or different economic model in that way?
[00:09:39.191] David Dias: Yeah, totally. What the Filecoin network is really about is creating a free market that multiple players can now offer their services. They don't have to ask for permission. You can think of it as like Airbnb or Uber, where multiple companies from multiple countries start recruiting businesses to just provide their service through the network. And the network would facilitate the conversation to happen between clients and service providers. And Filecoin creates two markets. One for storage, one to make sure your files are there. And the other one that is really interesting is the retrieval market. Think of it as one being Amazon S3 and the other one being like CDN, like a cloud fair or something. So you can pay the miners, you can pay the people in the network for their location. If you want to explore a new market, a new geographic location, today you have to wait for some cloud provider to build it at the center so that you have access to machines that are close to those users. With Filecoin, you can recruit local entrepreneurs to create local businesses. distribute the wealth, distribute the opportunity, and create the service that you need to reach new economies, to reach new markets. And so this is like a very powerful model. It basically aligns everyone that is participating on the network with the network itself, because they know the more they invest, the more time they spend, the more people that they bring to the network, the currency that the network generates for them, the token, will appreciate in value. So then it becomes up to them, they can cash out and go on with their lives, or they can reinvest. And the more they invest in the network, the network grows. It's kind of like an instrument that has some currency properties, some bearer token properties, some stock properties. It's not like any of them, but it has all of them mixed up, like really empower the users, the contributors on the network, and really powering the early adopters. Because the people that will join the network first will get a special reward, like the block reward will help them grow their deployment much, much, much faster. And one other key thing on the PowerCoin network is really that we don't have a wasteful proof of work. The way that you store files is actually the way that you give security to the network. So when you are storing a file, you are sealing a thing that we call a container, and that's a cryptographic proof. And that's what you give to the network to make sure that the network is secure and the consensus is reached. There is no need to buy expensive GPUs or CPUs or ASICs to run crypto puzzles just for the sake of running a network.
[00:11:58.170] Kent Bye: Yeah, and there seems to be trade-offs whenever you talk about centralization versus decentralization. And I wanted to focus on the metaphor that you gave, which is you flying to Lisbon and finding the book. We can kind of think of that as an HTTP request where you're sending it out and it's being sent back to you. And that has a certain speed, but also computational resources to be able to do that. Now, when you go to the decentralized system, If there is a cryptographic hash that's out there, and there's many different people serving it, you may actually download it faster. But that may be at the cost of other computational resources that are having to do both the searching and the discovery of that, and then the seeding up and the distribution. So when you think about this in terms of the time that it takes to serve a file versus the computational resources, how do you think about those trade-offs between the two?
[00:12:44.297] David Dias: Yeah, totally. It's a very important question. But the reality is Moore's Law still applies. And those things are getting cheaper and faster. And doing hashing today is very inexpensive compared to bandwidth, for example. Right now, bandwidth is one of the most expensive resources in computing. And if you have a way to fetch the file from someone that has a file locally, or even in a 10-hop range, rather than traversing the whole world multiple times to get the bits of the file, then you are going to save a ton. And what we expect to see is, as computing advances and as these systems develop, we'll see hardware implementations that then will basically make sure that all of these expensive computations that we see today are just a fragment of the costs. And what we end up having is saving a ton in the bandwidth. And again, saving us from feeling that the internet is getting slower, because then we can actually fetch the content in a way that it's much faster and stream it from multiple pipes instead of just one.
[00:15:07.882] Kent Bye: RETO MEIER And so in the session today, the lightning talks of IPFS, you mentioned libpdp. What is libpdp, and how is that related to IPFS?
[00:15:17.570] David Dias: All right, so that's a great question as well. Leap Peer-to-Peer is a modular networking stack. And it's basically the networking stack of IPFS refactored out and packaged with its own ecosystem, with its own team, with its own API, so that other projects on the peer-to-peer world, on the D-Web world, can also use it and leverage all the solutions that we had to come up with for the peer-to-peer web. The long story short is there have been multiple peer-to-peer applications in the past, but they all make a lot of assumptions about what the network looks like. And so we were missing a network stack that is modular, that is upgradable. We peer-to-peer basically tells you, OK, there is this set of interfaces that are really meshed together very well. But if you are building an application for the browser, you can use properties that are available on the browser. If you are developing an application for a desktop, you can use transports that exist only for the desktop. Or if you are going to develop an application for IoT, you can use things like radio, like ZigBee, or Bluetooth. and still have the same API, the same primitives, the same process addressing primitive, and connect to the rest of the network without having to redesign everything from scratch. So it's really like a platform for innovation in the network's land, which is something that we have been wanting to do for a while. It took us a while to get all interfaces right, but now it's very quick to go into new platforms, new runtimes, and just like build the custom modules for the runtime features, and just drop entire applications on top. We have seen the surge of many applications where it runs on a browser, and then they can also run on Electron. And when they run on Electron, they get more features, because Electron just runs on your machine. Or they can run on a hardware device, and it's always the same application, always the same code. You don't have to touch anything. You get that process addressing primitive, and that's it.
[00:17:07.402] Kent Bye: And so here at the Decentralized Web Summit, there was a series of lightning talks about IPFS. And a number of different people either announcing new things or talking about their projects. Maybe you could talk about some of the highlights that you take from this series of lightning talks here at the Decentralized Web Summit.
[00:17:21.765] David Dias: Yeah, totally. We are doing multiple lightning talks on IPFS and leap year to peer. And we also have workshops on how to build a distributed application on top of all of these technologies. And I guess the key takeaway is, yes, the web exists. It's here to stay. And it's possible to have fully distributed applications. on the browser without requiring any server. And you still can get things like identity. You can still get things like real time. You can still get things like collaboration without having any coordination server. If you are new to all this world, I always recommend checking out an application called PeerPad. And it's essentially Google Docs without Google. You don't need the central server to coordinate. And it uses this very cool data structure that came from academia that was kind of lost in a drawer. But we picked it up, and we developed it, and we implemented it. And so you can have multiple machines going on and off. And without any interaction or live interaction, it can write something on a document, go offline, and another machine can come later and, like, fetch the changes, and they can agree on what is the state of the doc. So you can have true collaboration, again, the Google Docs without Google, true collaboration without any central point.
[00:18:47.595] David Dias: Yeah, so both cases happen. Either they develop the web version, like the thing that loads on your Firefox, your Chrome. But sometimes, because they want an application, like for example, a photo viewer, and because it's a photo viewer, and they want to have a copy of all their photos locally, it means they require a lot of storage. And so one thing that you don't get from the browser is access to your whole hard drive. You just get a quota, a very small quota, and so some apps provide the experience on a browser, but then when users want to reuse it heavily, they just repackage it on Electron. And fortunately, repackaging an app from a browser to Electron is very easy. And you just use IPFS again. You don't have to change any of the way that you use IPFS. It just adapts very quickly. And then you have access to new features that the OS can give you. For example, file storage, large amounts of file storage.
[00:19:38.797] Kent Bye: So from your perspective, do you see that the decentralized web and these applications are well on their way, and you have everything that you need? I know that file storage seems to be a key component of that, but are there other things that are out there that are part of that ecosystem that we're kind of waiting on to have everything kind of fit together in the way that it needs to?
[00:19:58.217] David Dias: Yeah, so I think a lot of the important ideas to make all this work are already there, and we see a lot of them implemented. Right now, what I think we are is, comparing to the 90s, when the web was just getting started, and we were seeing some web pages and some applications on the web, we are kind of like late 90s. We know it's possible. We know we can do it. There are some examples. There are some companies running on top of these technologies. But we still haven't really reached the consensus on what is the convention of how to build this app? What is the good API? What is the experience that a web developer needs to see in order to build really powerful experiences on the web? So think about before Gmail appearing with AJAX, and doing AJAX requests, and having a full app running on a browser. I think we are in that era, before having those primitives really figured out. So right now, what we need is a lot of experimentation. The more, the merrier. And we are seeing a ton of here. In this summit, there's apps appearing everywhere. There's multiple talks, multiple prototypes, multiple demos. People are experimenting really hard on this. And we need even more people to try and tell us what doesn't work. What are they missing in order to build the app of their dreams?
[00:21:06.928] Kent Bye: Yeah, I saw a couple of people mention the awesome.ipfs.io, which is, I guess, some curation of some of the paradigmatic examples of use cases for using IPFS in these decentralized applications. Do you have any personal favorites of applications that are out there?
[00:21:21.857] David Dias: Oh, OK. So I have many. I get super excited just scrolling down the list. And again, it's an open resource. Everyone can just submit a pull request, and I have another app, another article. But for example, one of my favorites is D2. It's the decentralized YouTube. It's pretty much like a mirror, like a clone of YouTube. but again, over IPFS, and it has a mechanism to reward content creators. It has a token that they can give to the content creators, and users, by commenting, by reviewing, by helping each other, actually get rewarded as well. So it's a very live, vibrant community, and you see tons of videos, terabytes getting added every week. to YouTube and people really picking it up and getting really excited about the ability to see all these videos, get rewarded, and also see it work in locations where YouTube doesn't reach right now. Because, again, there are no servers there to serve those videos.
[00:22:11.157] Kent Bye: That's great. And so for you, what are some of the either biggest problems that you're trying to solve or open questions that you're trying to answer?
[00:22:20.382] David Dias: So there's still a lot of questions in identity, like what is the right UX to help users not to have to manage keys? Because we know there have been many problems with crypto wallets and even government identities. In Europe, you are used to get a citizen card and it has your private key and you can log into your government services with that. But the UX is just not there. And also, when we develop all these things, one thing that we need to care about is, OK, if we are going to rebuild the web from bottom up, better create better primitives. And so let's enable people to have personas. Let's make sure that people can have privacy. Let's make sure that people can navigate the web and be free. Don't be afraid of what they are checking. And so there's a lot of open questions there, and a lot of teams experimenting. And now it's all about building the apps, and getting the users, and seeing what are the patterns, what are the behaviors that people expose, so that we can polish and improve. DUSTIN KIRKLANDER Yeah, I'm curious.
[00:23:14.859] Kent Bye: Because I cover virtual reality, if you've heard of other companies, whether it's High Fidelity, or Janus VR, or some of the web VR things that are happening, where you see the fusion between IPFS and the immersive web in the metaverse?
[00:23:29.384] David Dias: Yeah, totally. And so VR, by default, is a thing that requires a ton of data. And the data has to move very, very fast. VR worlds are huge. And so we have seen a lot of examples, one with GeniusVR, where it would be just worlds that would be added to IPFS nodes. And the experience was when you were close to an IPFS node that had worlds. you could see a wormhole, and you could just traverse through. So it's kind of like discovering these virtual worlds, and really having the experience of, as you move in the real world, you also have access to different virtual worlds. The other one is augmented reality. There is a project from Holocene where they just use IPFS to transfer artifacts. from digital archives, and they are able to build rooms with a museum in their own house by pulling all these artifacts from these digital archives. Or going to Rome and going to the Colosseum and just painting the Colosseum as it was millennia ago, 2,000 years ago, and not giving the user the perception that they are living in the past, but through augmented reality. And again, loading these assets that are really big through IPFS.
[00:24:39.458] Kent Bye: Great. And finally, what do you think is kind of the ultimate potential of all these decentralized systems and what they might be able to enable?
[00:24:49.023] David Dias: Yeah, it's freedom of access to world's knowledge. Make sure that the platforms that we create to distribute opportunity and really balance the access to information and power is evenly distributed all over the world. One of the things that always worries me is that as technologists, as entrepreneurs, as people that want to make the world better, we keep creating these platforms to distribute knowledge like Coursera, Khan Academy, YouTube, Wikipedia. But when you go to the places of the world where people need this the most, it's where it fails. And so it just creates power walls all over again. It's just the developers in countries that have access to these platforms, and everyone else that actually could really use this information is just missing out. So we need better protocols. We need to get information everywhere. Everyone should have equal access to the whole world's information.
[00:25:36.660] Kent Bye: Is that why it's called the Interplanetary File System?
[00:25:38.581] David Dias: Exactly. The reason why, first it's an homage to J.C.R. Licklider, because in one of his notes from a very long time ago, he had this idea for the intergalactic network, and that kind of like inspired the creation of the internet. And so the Interplanetary File System comes as an homage and also as a realization. By making a system that is capable to work in multiple planet scale, you actually solve a lot of the problems that we have in our own planet. A lot of these Offline scenarios, a lot of like lack of connectivity, like reliability, like the networks that don't really work well are sometimes worse than actually like beaming data to some satellite. And so by solving one, you solve all the other use cases as well. Awesome.
[00:26:19.771] Kent Bye: Is there anything else that's left unsaid that you'd like to say to the decentralized community?
[00:26:23.632] David Dias: Oh, well, like keep hacking, like show up your work. show demos, get everyone excited. We have been seeing a surge of examples and more and more developers jumping in. But overall, I guess you can see from my face, I'm just super excited and super happy for this to be happening and to be part of it. So thank you so much. Awesome. Yeah, thank you. Yeah, thank you. Thank you. This was great.