3 ms·

> Large storage systems already do this. RAID is a type of erasure coding. Various enterprise storage systems use hashing for data integrity. > Distributing th

> Large storage systems already do this. RAID is a type of erasure coding. Various enterprise storage systems use hashing for data integrity. > Distributing the data over a larger number of systems can actually be more efficient because the percentage of erasure blocks you need goes down as the number of independent devices increases. For example, if you only have two devices and want redundancy, you need a mirror and lose 50% of your capacity. If you spread the data across 200 devices you might achieve an even higher level of resilience by sacrificing only 25% of capacity so you can lose any 50 devices without data loss. You may get this down even further by periodically checking if a device is still available and replacing it, so the number of devices you can lose can be as low as the maximum number of devices you expect to lose simultaneously. You're mixing things up so bad it's hard to correct… For starter regarding filecoin the number of nodes you must expect to lose is “almost all of them” because they can stop operating if the economics isn't even good enough to cover their OpEx (They seem to be fine not covering the CapEx for now, but who knows for how long). It's almost like putting all your data in a datacenter owned by a nearly broke provider: if they go bankrupt you're screwed so you need a plan B. > You can now defect, but you have a 10% chance of being detected each time, so you can expect to only get to do it 10 times. So you get a 90% payment ten times without doing work, then have to go back to getting a 50% payment 100 times. 10 times 90% is way less than 100 times 40%, so if you do this you lose money. Again, you're mixing things up. The problem isn't that a node could defect and not do the work (this is assuming non-byzantine fault tolerance), the problem is that a node could voluntarily fuck up the calculations when/if it advantages them. And it could be far less than 10% of the time while still being a nuisance. I don't need to fuck up 10% of the back-propagation calculations in a neural-network training to make it completely unusable/to make the person training it spend way more resource than they should in the training process (which gets me more usage as a node operator). Adversarial threat modeling is hard, I work with people who do this on a daily basis and I can clearly tell you're oversimplifying things a lot. > Which is a self-solving problem. The unprofitable providers go out of business until the price makes it profitable. But that seems like it should happen quicker than this if the profitability isn't there, because storage is fungible. Even if you bought a bunch of drives to do this when the price was higher, you could sell them and go put the money in a traditional investment. Or if you're speculating on the value of Filecoin going up, sell your storage and use the money to buy Filecoin. So the people still doing it are presumably turning a profit even at current prices, whether through economies of scale or because they had "free" storage to use. And yet, it hasn't self-solved itself since 2021… The problem is that node operators get paid roughly what they spend in OpEx, and their capital is essentially illiquid[1] so there's no good reason to stop operating. Of course now the only reason for them to keep invest is the hope that the token price increases again, but because this is crypto it is fueled by the “fantasy of the bull run”, not by an expected uptick in usage (which interestingly enough isn't happening even though the storage price is very cheap). > It causes very inexpensive storage to be available, which is useful. Yet barely anyone uses it, which empirically question its “usefulness”. [1] and I don't get were you got the idea that storage was “fungible”, the failure rate going up exponentially over time makes storage a poor fit for the second-hand market, especially if people know that you've been running stressful proof of space-time on it, and if you're trying to fire-sale a Petabyte of storage, chances are high that people will figure that out

AnthonyMouse 3y ago
> For starter regarding filecoin the number of nodes you must expect to lose is “almost all of them” because they can stop operating if the economics isn't even good enough to cover their OpEx You don't expect to lose "almost all of them" at the same time. Even if the price crashes, you would expect capacity to go down over a period of days or weeks, not minutes. And then if a fraction of the data is lost but is less than the number of erasure blocks, you promptly reconstruct it and put it on a different node. Meanwhile you would expect an equilibrium here. The price going down forces some providers out of the market, but providers leaving the market brings the price back up. As long as the customer is offering as much as some providers need to stay in the market, somebody is hosting the data. That only stops if the customer won't bid what the providers need to get, at which point the customer transfers their data out of the system so they can stop paying more than they're willing to. Now, you can screw this up if you make your system sufficiently convoluted so the price signal doesn't make it from the customer to the provider or vice versa, and I'm not familiar enough with the specific implementation in Filecoin to comment, but screwing that up isn't inherently necessary for this category of system. > the problem is that a node could voluntarily fuck up the calculations when/if it advantages them. Which is why you duplicate some of them at random, and don't tell them when you're going to do it. The calculations are deterministic. If you distribute one to two random nodes and they don't get the same result, but they've each signed their own result, now you know one of them defected and can prove which one by doing that calculation yourself or doing some other potentially expensive operation that only happens when there is an inconsistency. At which point the defector is found out, you can prove it, and their reputation is in ashes. > Yet barely anyone uses it, which empirically question its “usefulness”. It seems to be storing more than a exabyte of data for someone. > their capital is essentially illiquid[1] > [1] and I don't get were you got the idea that storage was “fungible”, the failure rate going up exponentially over time makes storage a poor fit for the second-hand market, especially if people know that you've been running stressful proof of space-time on it, and if you're trying to fire-sale a Petabyte of storage, chances are high that people will figure that out It's fungible because a used hard drive is a commodity product with a wide customer base. That new ones cost more than the used ones doesn't make it not a commodity; you could have bought the used ones to begin if you're content to continue running them at their current age. Let me know if you're aware of some place you can buy working >=16TB drives, used or otherwise, for less than ~$100 each in 2023.

AnthonyMouse 3y ago

> For starter regarding filecoin the number of nodes you must expect to lose is “almost all of them” because they can stop operating if the economics isn't even good enough to cover their OpEx You don't expect to lose "almost all of them" at the same time. Even if the price crashes, you would expect capacity to go down over a period of days or weeks, not minutes. And then if a fraction of the data is lost but is less than the number of erasure blocks, you promptly reconstruct it and put it on a different node. Meanwhile you would expect an equilibrium here. The price going down forces some providers out of the market, but providers leaving the market brings the price back up. As long as the customer is offering as much as some providers need to stay in the market, somebody is hosting the data. That only stops if the customer won't bid what the providers need to get, at which point the customer transfers their data out of the system so they can stop paying more than they're willing to. Now, you can screw this up if you make your system sufficiently convoluted so the price signal doesn't make it from the customer to the provider or vice versa, and I'm not familiar enough with the specific implementation in Filecoin to comment, but screwing that up isn't inherently necessary for this category of system. > the problem is that a node could voluntarily fuck up the calculations when/if it advantages them. Which is why you duplicate some of them at random, and don't tell them when you're going to do it. The calculations are deterministic. If you distribute one to two random nodes and they don't get the same result, but they've each signed their own result, now you know one of them defected and can prove which one by doing that calculation yourself or doing some other potentially expensive operation that only happens when there is an inconsistency. At which point the defector is found out, you can prove it, and their reputation is in ashes. > Yet barely anyone uses it, which empirically question its “usefulness”. It seems to be storing more than a exabyte of data for someone. > their capital is essentially illiquid[1] > [1] and I don't get were you got the idea that storage was “fungible”, the failure rate going up exponentially over time makes storage a poor fit for the second-hand market, especially if people know that you've been running stressful proof of space-time on it, and if you're trying to fire-sale a Petabyte of storage, chances are high that people will figure that out It's fungible because a used hard drive is a commodity product with a wide customer base. That new ones cost more than the used ones doesn't make it not a commodity; you could have bought the used ones to begin if you're content to continue running them at their current age. Let me know if you're aware of some place you can buy working >=16TB drives, used or otherwise, for less than ~$100 each in 2023.