7 ms·

"distributed, scalable, redundant" describes AWS pretty well, and if I can pay much less for a faster more reliable service but sacrifice "trustless and anonymo

by hudon 9y ago

"distributed, scalable, redundant" describes AWS pretty well, and if I can pay much less for a faster more reliable service but sacrifice "trustless and anonymous", I'll stick to it.

Ajedi32 9y ago
What makes you think AWS would be necessarily be faster or more reliable? If done correctly, think a decentralized system could be significantly faster and more reliable than even the best centralized storage solution. For example, with a decentralized system files could be broken up into shards and distributed across multiple nodes, so unless a significant percentage of computers on the planet all melt down at once your data would remain accessible (i.e. zero downtime). That would also allow downloads to be parallelized across multiple connections like with BitTorrent, meaning bandwidth wouldn't be an issue either. And that's not even considering the effect that a commoditized pricing system would have on the costs of a distributed storage solution. I imagine it'd be significantly cheaper than AWS too, as storage would be sold at a price very close to costs.
- viraptor 9y ago
  If the decentralised network is popular and stable and clients happen to be next to you and they happen to have parts of your files, then yes. It could be fast and reliable. On the other hand, you could spend some money and guarantee it instead: https://aws.amazon.com/directconnect/ https://aws.amazon.com/directconnect/
  - hbk1966 9y ago
    Also if people ever stop using the service your files could be gone forever.
- Zahlmeister 9y ago
  Bittorrent at least has has much higher latency than a protocol like HTTP with simple auth. As for price, AWS has some of the cheapest operating costs due to scale, something which smaller players will have trouble competing with. Moving data within AWS is fast and sometimes free. Smaller players also pay the highest cost for bandwidth. If some plans don't have bandwidth caps yet (many already do), they will get caps as soon as serving bandwidth as an individual becomes profitable. Pretty much all distributed systems have the following in common: You pay for resilience with overhead. If it wasn't that way, everything would become as distributed as possible over time.
  - dx034 9y ago
    > Smaller players also pay the highest cost for bandwidth. If some plans don't have bandwidth caps yet (many already do), they will get caps as soon as serving bandwidth as an individual becomes profitable. Isn't it the other way around? Small providers offer low traffic costs while the big cloud providers (Google, AWS, Azure) charge significantly for bandwidth.
    - Zahlmeister 9y ago
      Small providers pay the most for traffic to their ISPs, because they get smaller volume discounts. What Google/AWS/Azure charge is a different topic entirely, they themselves pay much less.
- guy_c 9y ago
  I had been recently looking at Siacoin, but still a lot of work is needed before it can displace something like AWS. Took nearly a day with lots of stalls just to sync the blockchain. Also I am not sure about how price competitive it will ultimately be. Amazon has huge buying power which I am sure means they pay less per TB then consumers. Also other economies of scale. I bet the cost of labour to bring online each 1TB on S3 is very low.
  - guy_c 9y ago
    Siacoins calculator at http://sia.tech/ http://sia.tech/ is claiming a price of $2/TB/month vs $23/TB/month for Amazon S3. However it neglects to include Backblaze's B2 storage which is only $5/TB/month - https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html That is maybe a not a big enough price saving to convince conservative corporations to switch. Imagine going to your accounts department to request they purchase a cryptocurrency so you can use it to pay for data storage.
    - jhall1468 9y ago
      Does backblaze have a single enterprise client that anyone knows of? That's Sia's target market. They are targeting companies that use S3, not a company that's pretty clearly targeting personal and SMB's.
    - Zahlmeister 9y ago
      I don't know how using Sia works in practice, but comparing it with regular S3, which is connected to the entire AWS ecosystem (including EC2), is absurd. Amazon S3 also offers "glacier" for longterm storage with few accesses, which is $4 per TB.
      - dx034 9y ago
        But that's the problem. S3 is cheap if you use it with other AWS services and it's designed to be only cheap that way. Bandwidth is expensive to avoid that people pick services from a combination of Azure/AWS/Google. If you're just looking for storage, S3 is certainly not the cheapest provider. Glacier is a bit different but clearly just for archives.
- deleted 9y ago
  [deleted]
- hudon 9y ago
  > files could be broken up into shards and distributed across multiple nodes You can setup any old distributed database to do this... If there was a benefit to breaking down a file rather than replicating it, AWS would be using this efficiency to provide a better service. > the effect that a commoditized pricing system would have on the costs of a distributed storage solution It'll always be cheaper for someone to sell their hard drive than to sell access to their hard drive, just like it is cheaper to buy bitcoin instead of mining it. Here is why: let's say Alice is profit-driven and greedy and is making $1 profit for 1gb of space. She can use those profits to buy more hardware, make more profit, buy more hardware, and so on and so forth until she has 100,000gb of space being offered. At that point, because of economies of scale, it will cost Alice less to maintain 100,000gb then the total cost of 100,000 people hosting 1gb each (eg. reduce total electricity cost because of bulk purchase). Now, Alice can grow her operation so big (say, 100 petabytes) that it actually becomes no longer profitable for Bob to host 1gb. Alice has so much lower costs per gb that she was able to drive the price per gb down to a point where the price is lower than Bob's costs. This is basic economy of scale so far, right? And it's the same reason why data centers filled with ASICs make it useless for you to mine Bitcoin on your laptop. OK now that Alice is hosting say 100petabytes and has priced out all the individuals on the network, she will be subject to data policy rules and will have to obey law enforcement to kick out users that are using her hard drives for illegal activity. Alice is now AWS.
  - baddox 9y ago
    > If there was a benefit to breaking down a file rather than replicating it, AWS would be using this efficiency to provide a better service. Are you suggesting that AWS is doing every possible thing that would increase efficiency, or in other words, that AWS' services are optimally efficient? That seems very unlikely.
    - hudon 9y ago
      Yes. This is how capitalism works. If there is an inefficiency in the market, there is an opportunity for a competitor to beat you on price or quality. If AWS is not incorporating these efficiencies into their tech, then Microsoft Azure will and beat them by providing either a cheaper or higher quality service. If FileCoin actually worked as a better technology, Google or some big name would say they will start using it to beat AWS in the data storage game. I'm not saying FileCoin is useless, it is useful to store 3D printed gun designs and illegal media, because it is outside the reach of the law. But to think it is a more efficient technical solution to data storage is a bit naive...
xtracto 9y ago
Ooohh we thought that as well, until the latest S3 outage.
- hudon 9y ago
  Ask the CTO of any Fortune 100 company what they're doing to mitigate such an outage. What do you think they will answer? Option A: cross-region replication, regular backups, and secondary service provider failover plan Option B: decentralize everything on people's laptops
  - jhall1468 9y ago
    There's nothing I love more than a chicken or egg argument. Nobody is doing B because there isn't a company that has a scalable B option yet. But hey, your argument is totally valid, which is why everyone is still programming in Cobalt!
    - deleted 9y ago
      [deleted]
    - hudon 9y ago
      Sarcasm aside, option A works and option B does not, as you admit. What's your counter-argument? If it is that option B will one day be better than option A, you will need to provide evidence because that's a claim that is not addressed in the posted filecoin.pdf.
      - jhall1468 9y ago
        > If it is that option B will one day be better than option A, you will need to provide evidence because that's a claim that is not addressed in the posted filecoin.pdf. Argument to the Future. You can't provide evidence of a future event and asking for it is a joke. Maybe Sia/File/Storj are the future maybe it's something else, but asking for "evidence" of a future event is a joke.
        deleted 9y ago

by hudon 9y ago

"distributed, scalable, redundant" describes AWS pretty well, and if I can pay much less for a faster more reliable service but sacrifice "trustless and anonymous", I'll stick to it.

Ajedi32 9y ago

What makes you think AWS would be necessarily be faster or more reliable? If done correctly, think a decentralized system could be significantly faster and more reliable than even the best centralized storage solution. For example, with a decentralized system files could be broken up into shards and distributed across multiple nodes, so unless a significant percentage of computers on the planet all melt down at once your data would remain accessible (i.e. zero downtime). That would also allow downloads to be parallelized across multiple connections like with BitTorrent, meaning bandwidth wouldn't be an issue either. And that's not even considering the effect that a commoditized pricing system would have on the costs of a distributed storage solution. I imagine it'd be significantly cheaper than AWS too, as storage would be sold at a price very close to costs.

viraptor 9y ago
If the decentralised network is popular and stable and clients happen to be next to you and they happen to have parts of your files, then yes. It could be fast and reliable. On the other hand, you could spend some money and guarantee it instead: https://aws.amazon.com/directconnect/ https://aws.amazon.com/directconnect/
- hbk1966 9y ago
  Also if people ever stop using the service your files could be gone forever.
Zahlmeister 9y ago
Bittorrent at least has has much higher latency than a protocol like HTTP with simple auth. As for price, AWS has some of the cheapest operating costs due to scale, something which smaller players will have trouble competing with. Moving data within AWS is fast and sometimes free. Smaller players also pay the highest cost for bandwidth. If some plans don't have bandwidth caps yet (many already do), they will get caps as soon as serving bandwidth as an individual becomes profitable. Pretty much all distributed systems have the following in common: You pay for resilience with overhead. If it wasn't that way, everything would become as distributed as possible over time.
- dx034 9y ago
  > Smaller players also pay the highest cost for bandwidth. If some plans don't have bandwidth caps yet (many already do), they will get caps as soon as serving bandwidth as an individual becomes profitable. Isn't it the other way around? Small providers offer low traffic costs while the big cloud providers (Google, AWS, Azure) charge significantly for bandwidth.
  - Zahlmeister 9y ago
    Small providers pay the most for traffic to their ISPs, because they get smaller volume discounts. What Google/AWS/Azure charge is a different topic entirely, they themselves pay much less.
guy_c 9y ago
I had been recently looking at Siacoin, but still a lot of work is needed before it can displace something like AWS. Took nearly a day with lots of stalls just to sync the blockchain. Also I am not sure about how price competitive it will ultimately be. Amazon has huge buying power which I am sure means they pay less per TB then consumers. Also other economies of scale. I bet the cost of labour to bring online each 1TB on S3 is very low.
- guy_c 9y ago
  Siacoins calculator at http://sia.tech/ http://sia.tech/ is claiming a price of $2/TB/month vs $23/TB/month for Amazon S3. However it neglects to include Backblaze's B2 storage which is only $5/TB/month - https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html That is maybe a not a big enough price saving to convince conservative corporations to switch. Imagine going to your accounts department to request they purchase a cryptocurrency so you can use it to pay for data storage.
  - jhall1468 9y ago
    Does backblaze have a single enterprise client that anyone knows of? That's Sia's target market. They are targeting companies that use S3, not a company that's pretty clearly targeting personal and SMB's.
  - Zahlmeister 9y ago
    I don't know how using Sia works in practice, but comparing it with regular S3, which is connected to the entire AWS ecosystem (including EC2), is absurd. Amazon S3 also offers "glacier" for longterm storage with few accesses, which is $4 per TB.
    - dx034 9y ago
      But that's the problem. S3 is cheap if you use it with other AWS services and it's designed to be only cheap that way. Bandwidth is expensive to avoid that people pick services from a combination of Azure/AWS/Google. If you're just looking for storage, S3 is certainly not the cheapest provider. Glacier is a bit different but clearly just for archives.
deleted 9y ago
[deleted]
hudon 9y ago
> files could be broken up into shards and distributed across multiple nodes You can setup any old distributed database to do this... If there was a benefit to breaking down a file rather than replicating it, AWS would be using this efficiency to provide a better service. > the effect that a commoditized pricing system would have on the costs of a distributed storage solution It'll always be cheaper for someone to sell their hard drive than to sell access to their hard drive, just like it is cheaper to buy bitcoin instead of mining it. Here is why: let's say Alice is profit-driven and greedy and is making $1 profit for 1gb of space. She can use those profits to buy more hardware, make more profit, buy more hardware, and so on and so forth until she has 100,000gb of space being offered. At that point, because of economies of scale, it will cost Alice less to maintain 100,000gb then the total cost of 100,000 people hosting 1gb each (eg. reduce total electricity cost because of bulk purchase). Now, Alice can grow her operation so big (say, 100 petabytes) that it actually becomes no longer profitable for Bob to host 1gb. Alice has so much lower costs per gb that she was able to drive the price per gb down to a point where the price is lower than Bob's costs. This is basic economy of scale so far, right? And it's the same reason why data centers filled with ASICs make it useless for you to mine Bitcoin on your laptop. OK now that Alice is hosting say 100petabytes and has priced out all the individuals on the network, she will be subject to data policy rules and will have to obey law enforcement to kick out users that are using her hard drives for illegal activity. Alice is now AWS.
- baddox 9y ago
  > If there was a benefit to breaking down a file rather than replicating it, AWS would be using this efficiency to provide a better service. Are you suggesting that AWS is doing every possible thing that would increase efficiency, or in other words, that AWS' services are optimally efficient? That seems very unlikely.
  - hudon 9y ago
    Yes. This is how capitalism works. If there is an inefficiency in the market, there is an opportunity for a competitor to beat you on price or quality. If AWS is not incorporating these efficiencies into their tech, then Microsoft Azure will and beat them by providing either a cheaper or higher quality service. If FileCoin actually worked as a better technology, Google or some big name would say they will start using it to beat AWS in the data storage game. I'm not saying FileCoin is useless, it is useful to store 3D printed gun designs and illegal media, because it is outside the reach of the law. But to think it is a more efficient technical solution to data storage is a bit naive...

xtracto 9y ago

Ooohh we thought that as well, until the latest S3 outage.

hudon 9y ago
Ask the CTO of any Fortune 100 company what they're doing to mitigate such an outage. What do you think they will answer? Option A: cross-region replication, regular backups, and secondary service provider failover plan Option B: decentralize everything on people's laptops
- jhall1468 9y ago
  There's nothing I love more than a chicken or egg argument. Nobody is doing B because there isn't a company that has a scalable B option yet. But hey, your argument is totally valid, which is why everyone is still programming in Cobalt!
  - deleted 9y ago
    [deleted]
  - hudon 9y ago
    Sarcasm aside, option A works and option B does not, as you admit. What's your counter-argument? If it is that option B will one day be better than option A, you will need to provide evidence because that's a claim that is not addressed in the posted filecoin.pdf.
    - jhall1468 9y ago
      > If it is that option B will one day be better than option A, you will need to provide evidence because that's a claim that is not addressed in the posted filecoin.pdf. Argument to the Future. You can't provide evidence of a future event and asking for it is a joke. Maybe Sia/File/Storj are the future maybe it's something else, but asking for "evidence" of a future event is a joke.
      - deleted 9y ago