5 ms·

We built an alternative to IPFS called Skynet that attempts to solve a couple of the major issues with IPFS. The biggest one being data durability and uptime.

by Taek 6y ago

We built an alternative to IPFS called Skynet that attempts to solve a couple of the major issues with IPFS. The biggest one being data durability and uptime. On Skynet, pinning doesn't mean hosting the file from your machine, it means paying a bunch of service providers to host the file for you. When you pin content to Skynet, you can turn off your computer 5 minutes later and the data will still be available globally. Just like IPFS, data is content-addressed and anyone can choose to re-pin the content. Service providers are held to 95% uptime each, which means doing something like 10-of-30 erasure coding can get you 99.99% uptime on the file as a whole. The low uptime requirement for individual providers dramatically cuts costs and allows amateurs to be providers on the network. The erasure coding algorithms ensure data availability despite a relatively unreliable physical layer. The other problem Skynet set out to solve is performance. In our experience with IPFS, if you aren't talking directly to a node that's pinning the content, IPFS is very slow. We've heard stories of lookups taking greater than 10 minutes for data that isn't pinned on a major gateway. Skynet uses a point-to-point protocol rather than a DHT, which not only makes it faster, it's also more robust to abuse. DHTs are pretty famously fragile to things like DDoS and active subversion, and Skynet has been designed to be robust and high performance even when malicious actors are trying to interrupt the network. Other than that, we've tried to make sure Skynet has feature parity to IPFS. Content-addressed data, support for building DAGs, support for running applications and static web pages, and then we've added a couple of elements of our own, such as APIs that allow applications and web pages to upload directly to Skynet from within the application. https://siasky.net https://siasky.net if you want to give it a try for yourself.

vin047 6y ago
Interesting project, though to be clear, you're using the Sia network and not IPFS right? Also do you have Swift bindings for use in an iOS app?
- Taek 6y ago
  Yes Skynet leverages the Sia network for storage and retrieval. We don't have swift bindings but other developers have been able to make iOS apps without much trouble - the API is pretty clean. Example: https://github.com/tetek/skynet-ios https://github.com/tetek/skynet-ios
georgyo 6y ago
Say what you will about IPFS and ProtocolLabs, but I really admire that they made IPFS completely blockchain agnostic. There is no requirement to buy or receive coins to use IPFS. And multiple different implementations of persistent stores can use the same IPFS network to ensure durability. This means that FileCoin, other coins, static hosting providers, self-hosting can all coexist and strengthen the same network. Every other distributed content addressable file store seems to only be interested in supporting their own coin only. For that reason alone, I find IPFS the most interesting.
- Ericson2314 6y ago
  Yeah the layering in Protocol Lab's work is really nice. The tendency of funded / for-profit software is to eschew layering, so it's easy to try out but ultimately ill-fitting and inflexible. This is a core problem of capitalism with IT. IPFS, libp2p, IPLD, Filecoin etc. all resist that temptation, and I think it will help them greatly in the end.
  - whyrusleeping 6y ago
    I really appreciate hearing things like this :) We put a ton of work into making the layering of our projects just right so that they can be reused over and over even if other things we're working on fail. It often does make it slower for us to make progress (could take soooo many shortcuts if we tried to collapse the stack) but I really do believe it's worth the effort.
brabel 6y ago
This looks interesting. Why is the payment method based on a crypto coin? How does using a blockchain help make Skynet and Sia work? I am wary of crypto coins as they tend to have wild swings in value over time as they are used primarily for speculation... can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?
- hinkley 6y ago
  A common theme in distributed filesystem conversations is the idea of "socializing" the costs of intermittent loads. If I 'pay in' a little more than my mean traffic capacity, then my surplus during low traffic cancels out some of my peak traffic. Peak shaving and trough filling. If you are doing business in India, you get payed in rupees. If the workers are in India, you just pay them in rupees. If you have to exchange currencies you end up with several types of friction that just create headaches and potential losses. If you periodically cash out or inject cash it's easier to deal with than on every transaction. Denominating file replication services in a "coin of the realm" just seems like the same sort of rationale. One of the problems with capacity planning is that you get punished for being wrong in either direction. You bought too much hardware or not enough, too soon or too late. With an IPFS or a Skynet, putting your hardware online two months before you need the capacity at least affords you some opportunity to make use of the hardware while your Development or PR team figures out how to cross the finish line.
- kordlessagain 6y ago
  > Why is the payment method based on a crypto coin? A blockchain gives a way to make a payment and provide identity/ecrypt functions to keep the resource private while it is active. Yes, one could create a system that attaches other authentication (user/pass or oauth), but then one has to create/connect a payment system which then uses that login information in conjunction with credit card information. To sell a product online taking credit cards requires an excess of 10 pieces of information that must be provided by the user. In the case of compute resources, I may want to deal with 100s of providers to host my resources (blog, images, video, code) and I'd need to use an intermediary if I wanted to be efficient about it. With something like Lightning payments, a system like this can provide resources without the need for a "signup" process or intermediary. > can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less? What does the future value of a fiat currency have to do with the current rate of storage on something like AWS? Would those prices not go up if the currency was undergoing deflation? Would you not have paid less integer values before the deflation? Would you not pay more integer values after? Where does something like AWS provide cost protection, other than spot instances? If you create a crypto contract and put the funds in escrow, then there is ZERO ways for the cost to go up over the life of the contract. Other than a bad actor scenario, which is why having multiple providers is the way to go.

by Taek 6y ago

vin047 6y ago

Interesting project, though to be clear, you're using the Sia network and not IPFS right? Also do you have Swift bindings for use in an iOS app?

Taek 6y ago
Yes Skynet leverages the Sia network for storage and retrieval. We don't have swift bindings but other developers have been able to make iOS apps without much trouble - the API is pretty clean. Example: https://github.com/tetek/skynet-ios https://github.com/tetek/skynet-ios

georgyo 6y ago

Say what you will about IPFS and ProtocolLabs, but I really admire that they made IPFS completely blockchain agnostic. There is no requirement to buy or receive coins to use IPFS. And multiple different implementations of persistent stores can use the same IPFS network to ensure durability. This means that FileCoin, other coins, static hosting providers, self-hosting can all coexist and strengthen the same network. Every other distributed content addressable file store seems to only be interested in supporting their own coin only. For that reason alone, I find IPFS the most interesting.

Ericson2314 6y ago
Yeah the layering in Protocol Lab's work is really nice. The tendency of funded / for-profit software is to eschew layering, so it's easy to try out but ultimately ill-fitting and inflexible. This is a core problem of capitalism with IT. IPFS, libp2p, IPLD, Filecoin etc. all resist that temptation, and I think it will help them greatly in the end.
- whyrusleeping 6y ago
  I really appreciate hearing things like this :) We put a ton of work into making the layering of our projects just right so that they can be reused over and over even if other things we're working on fail. It often does make it slower for us to make progress (could take soooo many shortcuts if we tried to collapse the stack) but I really do believe it's worth the effort.

brabel 6y ago

This looks interesting. Why is the payment method based on a crypto coin? How does using a blockchain help make Skynet and Sia work? I am wary of crypto coins as they tend to have wild swings in value over time as they are used primarily for speculation... can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?

hinkley 6y ago
A common theme in distributed filesystem conversations is the idea of "socializing" the costs of intermittent loads. If I 'pay in' a little more than my mean traffic capacity, then my surplus during low traffic cancels out some of my peak traffic. Peak shaving and trough filling. If you are doing business in India, you get payed in rupees. If the workers are in India, you just pay them in rupees. If you have to exchange currencies you end up with several types of friction that just create headaches and potential losses. If you periodically cash out or inject cash it's easier to deal with than on every transaction. Denominating file replication services in a "coin of the realm" just seems like the same sort of rationale. One of the problems with capacity planning is that you get punished for being wrong in either direction. You bought too much hardware or not enough, too soon or too late. With an IPFS or a Skynet, putting your hardware online two months before you need the capacity at least affords you some opportunity to make use of the hardware while your Development or PR team figures out how to cross the finish line.
kordlessagain 6y ago
> Why is the payment method based on a crypto coin? A blockchain gives a way to make a payment and provide identity/ecrypt functions to keep the resource private while it is active. Yes, one could create a system that attaches other authentication (user/pass or oauth), but then one has to create/connect a payment system which then uses that login information in conjunction with credit card information. To sell a product online taking credit cards requires an excess of 10 pieces of information that must be provided by the user. In the case of compute resources, I may want to deal with 100s of providers to host my resources (blog, images, video, code) and I'd need to use an intermediary if I wanted to be efficient about it. With something like Lightning payments, a system like this can provide resources without the need for a "signup" process or intermediary. > can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less? What does the future value of a fiat currency have to do with the current rate of storage on something like AWS? Would those prices not go up if the currency was undergoing deflation? Would you not have paid less integer values before the deflation? Would you not pay more integer values after? Where does something like AWS provide cost protection, other than spot instances? If you create a crypto contract and put the funds in escrow, then there is ZERO ways for the cost to go up over the life of the contract. Other than a bad actor scenario, which is why having multiple providers is the way to go.