4 ms·

Hard disks have been getting larger capacity but i/o capability is fairly flat. The result is that price/GB is dropping but price/iops is not. So _cold_ storage

Hard disks have been getting larger capacity but i/o capability is fairly flat. The result is that price/GB is dropping but price/iops is not. So _cold_ storage is where you would expect to see pricing fall. I don't follows AWS pricing closely but I've seen a lot of news around Glacier over the year so that might reflect this fact.

aledalgrande 4y ago
There's now a new Glacier product, called Glacier Instant Retrieval, which supposedly makes it cheaper and faster to retrieve from Glacier, if you do it a limited number of times.
- pwarner 4y ago
  Yeah this is magic price wise
didgetmaster 4y ago
The capacity and speed curves of drives is quite different. When the first TB drives were released about 15 years ago, the fastest ones were about 100MB/s. Now you can get 20TB drives (20x capacity) and the fastest ones are about 300MB/s (3x speed). It is just far easier and cheaper to make a drive twice as big than it is to make it twice as fast. SSDs are a different animal, but have some similar characteristics. Within the same generation (e.g. m.2 pcie gen 4), you can get drives that have a lot more capacity but have roughly the same access speeds (i.e. the 2TB version is very similar to the 1TB version). The speed increases between generations is much better than with HDD. Pcie gen 3 drives seemed to max out about 3500MB/s while the gen 4 drives are about double that. I have seen reports that gen 5 drives might double it again With HAMR technology we might get HDD drives with capacities in the 50TB-100TB range. You can bet that the speed won't be 5x current technology even if they get dual-actuators in them. There will need to be some kind of breakthrough technology to improve it significantly. This is why we need better data management systems. If the meta-data (e.g. file table) is only 1% of the data that is still a lot of data to read in and store in RAM. We need better systems where the file records are much smaller. https://didgets.substack.com/p/where-did-i-put-that-file https://didgets.substack.com/p/where-did-i-put-that-file

aledalgrande 4y ago

There's now a new Glacier product, called Glacier Instant Retrieval, which supposedly makes it cheaper and faster to retrieve from Glacier, if you do it a limited number of times.

pwarner 4y ago
Yeah this is magic price wise

didgetmaster 4y ago

The capacity and speed curves of drives is quite different. When the first TB drives were released about 15 years ago, the fastest ones were about 100MB/s. Now you can get 20TB drives (20x capacity) and the fastest ones are about 300MB/s (3x speed). It is just far easier and cheaper to make a drive twice as big than it is to make it twice as fast. SSDs are a different animal, but have some similar characteristics. Within the same generation (e.g. m.2 pcie gen 4), you can get drives that have a lot more capacity but have roughly the same access speeds (i.e. the 2TB version is very similar to the 1TB version). The speed increases between generations is much better than with HDD. Pcie gen 3 drives seemed to max out about 3500MB/s while the gen 4 drives are about double that. I have seen reports that gen 5 drives might double it again With HAMR technology we might get HDD drives with capacities in the 50TB-100TB range. You can bet that the speed won't be 5x current technology even if they get dual-actuators in them. There will need to be some kind of breakthrough technology to improve it significantly. This is why we need better data management systems. If the meta-data (e.g. file table) is only 1% of the data that is still a lot of data to read in and store in RAM. We need better systems where the file records are much smaller. https://didgets.substack.com/p/where-did-i-put-that-file https://didgets.substack.com/p/where-did-i-put-that-file