12 ms·
Author here. A lot of these numbers are drawn from experience in the mining world, where people realized that when cost is the ultimate bottom line, a lot of co
by Taek 7y ago
Author here. A lot of these numbers are drawn from experience in the mining world, where people realized that when cost is the ultimate bottom line, a lot of corners can be cut.
Sia systems don't need a ton of networking. I ran the networking buildout costs by some networking people, and again it comes down to cutting corners. If you only need 10 gbps per rack, if you don't mind having extra milliseconds added, etc, you can get away with very scrappy setups. The whole point is that it's not a highly reliable facility.
- anonymousiam 7y agoThe third sentence of your Medium article says; "Despite this, the Sia network is able to achieve 99.9999% uptime for files." How do you achieve this in a "not a highly reliable facility"?
- prepend 7y agoRedundancy, I assume. Lots of unreliable facilities and nodes can be very durable in aggregate.
- deleted 7y ago[deleted]
- jakewalker 7y agoSounds kinda like the people who thought that bundling a bunch of bad mortgage debt together in slices could get it rated AAA.
- __jal 7y agoReminds me of a saying from the first dot.com crash. (Or at least I heard it then first.) Tying two bricks together doesn't make them float.
- kybernetikos 7y agoIt's a great saying, but aren't all large ships these days made out of components that don't float individually?
- mcdevilkiller 7y agoNo, they are mostly made of air, which floats (Partial sarcasm, as that is what makes them float)
- feross 7y agoYour mind is going to be blown when you learn how TCP/IP works.
- allendoerfer 7y agoOr computers for that matter. Deep down it is not really about 1s and 0s, more like thresholds in between 1 and 0. To me a big part of computer science is abstracting away unreliable details to make them seem reliable.
- deleted 7y ago[deleted]
- statictype 7y agoExcept the people who put those sub prime mortgages in probably didn’t apply rigorous statistical analysis to their situation.
- rpdillon 7y agoIt does sound like that! The mistake there was to assume the failure of one mortgage was statistically independent of the failure of another, which is obviously incorrect for many failure scenarios. In this case, it it would be similar if all nodes were in, say, the same datacenter. That doesn't appear to be the case, but there may be other dimensions on which the network lacks the required diversity to support the reliability claims (disk vendor and age...? There must be others.)
- justinmeiners 7y agoNot really. Just probability. If you have fully redundant services then ALL of them have to go down to have an outage. Suppose you 5 copies with 75% uptime each. The probability that all of them are down is 0.25^5 ~ 0.0009 Now of course that assumes they are uncorrelated, but since Sia nodes are distributed across the internet, that's likely as opposed to multiple servers at a few data centers like AWS. Turns out mortgage bonds tend to be correlated.
- iampims 7y agoThey all run the same software stack though, so despite being deployed on diverse hardware, so they can’t claim to only have independent failure modes.
- justinmeiners 7y agoTrue. It would be interesting to analyze what others aspects are correlated.
- zchrykng 7y agoI mean, the concept actually works, but you have to understand what is actually going into the bundled product. There is no reason you couldn't bundle 100 million in mortgages that had a 10% default risk and sell 10 million of that bundle as AAA. It is when you just start bundling everything and then saying the entire thing is AAA that the trouble starts.
- Aunche 7y agoRedundancy is kind of meaningless unless if you have sufficient diversity. If all your replicas are located on the same rack, you're screwed if there is a minor disaster like a sprinkler going off.
- im3w1l 7y agoYou will be able to boast of 99.9999% uptime up until that sprinkler goes of. And with luck, it could be decades until it does.
- teknopurge 7y agoThis is why the post is a red-herring. You cannot use storage if you cannot get to it. Talking about the price of disk space/mth is absolutely useless without talking about the bandwidth required to use it, which means, if you want to go big-boys-math, 95% bandwidth and peering agreement pricing. That $1.50 with all the network chatter between hosts for replication/repair/etc., not to mention actually _downloading_ of the data from the Sia network will take that $1.50/TB/mth to $70/TB/mth. Talking about the disk-space pricing is, IMO, disingenuous. Talk about the math for an all-in use-case of the service. I appreciate what the OP is going for, however, hand-waving around uncomfortable critical items like bandwidth cost for the Sia network is not a good look.
- Dylan16807 7y ago> not to mention actually _downloading_ of the data from the Sia network will take that $1.50/TB/mth to $70/TB/mth You really need to back that math up. Here's my math: 10mbps is over 3TB per month at max utilization, so let's say it's good enough for either 2TB throughput with semi-even use, or 1TB throughput if the use is really focused on part of the day. When buying transit in bulk in the US, Google tells me that 10mbps was very roughly $8.50 a month in 2017, $4.50 a month in 2018, and $2.50 a month in 2019. Are you expecting the data to be rebuilt every single day or something?
- deleted 7y ago[deleted]
- arcticbull 7y agoHere's the issue. We know that due to economy of scale and domain experience, AWS will always have the lowest cost (to Amazon) for storage -- whether that's totally-reliable storage, or sorta-reliable. If there was a demand for sorta-reliable, they'd build a sorta-reliable S3 and undercut you. Then, blockchain adds inefficiency. Therefore, it's basically impossible for any blockchain solution to have a lower total cost to provide storage.
- throwaway9878 7y agoEverybody with more money than you can always undercut you in anything you ever do so why bother ever trying to do anything
- nemonemo 7y agoI think the point is that the cost should not be the main motivator. I would agree that there needs to be other differentiators in addition to cost, which could provide sufficient moat against other competitors, big or small.
- jessriedel 7y agoIt's possible that there could be non-obvious innovations in how to save money with a low reliability threshold, which Amazon might not be able to effortlessly copy.
- res0nat0r 7y agoIt should also be noted that you can get S3 storage for $1/TB/month already if you use the Glacier Deep Archive storage class.
- paulryanrogers 7y agoDid that include network IO to retrieve your data?
- kmod 7y agoSure, let's dig into networking. Who pays for rereplication traffic? If you do 64-of-96 RS encoding, that means for every failure you need to transfer 64x the lost storage capacity. If you're targeting a "low individual uptime but high aggregate uptime" model this means you need to be storing data in multiple sites -- and dedicated cross-geo bandwidth is expensive. I agree that in the happy case you can use low-bandwidth cheap equipment, but to get good reliability you need to provision for larger clustered failures such as rack- and row-level outages.
- DuskStar 7y agoI think his point is that he's targeting low aggregate uptime, too.
- Taek 7y agoDefinitely not, aggregate uptime is extremely high. We've never seen downtime do to network outages, only software bugs. And even then, only some users were impacted by the bugs, we've never in 5 years had a broad outage.
- DuskStar 7y agoGotcha, and your reply to the GP clarified a lot for me!
- Taek 7y agoSure! First up, we don't do repairs every time one host goes down. Standard practice on the network is to wait to do a repair until a full 25% of the redundancy is missing (in 64-of-96, that would be 8 hosts offline). Then you repair all 8 at once, significantly reducing the total amount of repair traffic. But secondly, offline doesn't usually mean dead and gone, with unstable datacenters like this they are usually back online before the user has lost a full 25% of their redundancy. Row level and rack level outages are handled by data randomization. The entire Sia system heavily depends on probabilistic techniques, both on the renting and hosting side. Row level failures will take out some of your data, but nobody should be disproportionately impacted by a cluster failure. On Sia, each piece is at a different site. So 64-of-96 implies that each chunk of data (96 pieces to a chunk) is located in 96 different places. This doesn't help with the geo-bandwidth, but as discussed above there are other techniques to handle that. Surprisingly, bandwidth pricing on the Sia network is even cheaper than storage pricing relative to centralized competition. That's a lot harder to model at scale though, so we aren't as confident the Sia bandwidth pricing will hold up at $1 / TB in the long term. And technically, most of this stuff is customizable per-customer. If your particular use case has a different optimal parameterization, it's fairly easy to tune your client to suit your particular needs.
- kllrnohj 7y agoThe buildout in the article doesn't work. You can't plug in that 4-lane SAS SFF-8087 splitter cable into that motherboard. You're only getting 8 hard drives per motherboard with that setup, not 32. That puts the cost of 192 TB at more like $6240, not $4945. Could be less if you find a good deal on mini-SAS PCIE cards, but still going to be substantially higher than $4945.
- notyourday 7y agoYou are going to get a 4U server that takes 60 disks and populate it with SAS cards driving them using SAS-2-SATA cables.
- ADefenestrator 7y agoEven if it had slots for the splitter cable, Intel and AMD onboard SATA explicitly doesn't support port multipliers as far as I know. You can buy PCIE SAS cards that do for relatively cheap, but then you have to find board with enough PCIe slots. Easy enough on the "gamer" boards but if you want ECC (and you probably do, for storage) and IPMI (you probably do, if you have more than a few dozen servers) your options get much more limited. Other than 1 or 2 Asrock Rack boards, you pretty much have to move into Epyc 7000-series or Xeon Silver or above. Often dual-socket on the Xeons to get a board with lots of PCIe. In theory something like an Epyc 3000-series with lots of PCIe or onboard SATA that supports port multipliers would work great, but I don't think anyone actually makes that.
- samstave 7y agoWhy the heck would one want to store data in a non highly reliable anything run by a team who is building around cutting corners??? Wtf am i missing here? —- So, I assume you’re quite young. ~31 or so... I say this because youre not speaking with a level of professionalism one would ostensibly have when talking about storing (and providing access to) another’s data... And if your actual business model is focused on cutting corners to save costs in exchange for reliability - then the obvious is that you seek to cut corners and costs in all other aspects of your business and are telling people “STAY THE FUCK AWAY” from your business... unless your customers are of similar ilk, and if not - then their data isn't even important enough to persist, your systems or theirs...
- justinmeiners 7y agoYou're taking "cutting corners" out of context. He is describing how individual nodes don't need to run at the standards of regular data servers and are hence cheaper, but in aggregate provide a reliable service. > So, I assume you’re quite young. So you do you have a technical criticism?
- toolslive 7y agoOTOH, your object storage can be even cheaper if you don't do Reed-Solomon erasure coding, but use rateless erasure codes. For example, online codes[1] have been used by Amplidata to have more reliable storage with lower overhead. There are downsides however (no partial reads, no mutability, ....) [1] https://en.wikipedia.org/wiki/Online_codes https://en.wikipedia.org/wiki/Online_codes