4 ms·

What is the point of that speed if the rest of your system is a bottle neck and the drives will fail in a year if you use that speed continously? I think we ne

by bullen 5y ago

What is the point of that speed if the rest of your system is a bottle neck and the drives will fail in a year if you use that speed continously? I think we need to go back and look at cycles per bit (DWPD is confusing because you need to know the number of years and sometimes it's not mentioned on the product page). Complexity can increase some performance but there is always a cost to complexity. Optane seemed promising because they increased the speed (that I personally will never need) and preserved the resiliance. Time will tell, but I bought 30x new 50nm SLC 64GB X25-E drives from 2011 on ebay for $100 each that I will mount on 12x SATA motherboards (only got cable/power for 8+1 per board) instead of buying one 480GB Optane drive for $1500! Cheaper, slower and maybe more robust? Atleast it is simpler! Atleast SATA I can use in my laptops/desktops/consoles etc. Also nobody is looking at CPU usage of NVMe vs. SATA!

consp 5y ago
I did something similar on a smaller scale: "Caching" ssd drives which are not the best but have a huge amount write bytes left are widely available on ebay and I bought 8 200GB SAS3 drives for dirt cheap (under 200$ total, drives from 2017). Not the fastest drives but good enough for my purpose (overkill even).
- bullen 5y ago
  What drives did you get?
  - consp 5y ago
    Sun/oracle branded HGST SS300 drives (HUSMH4020ASS210), they have the specs in the ballpark of the SS800 drives in endurance (according to the scsi data pages from the disks) but read/write performance of the SS300.
    - bullen 5y ago
      Trying to google it it seems unclear to me what the product is... SSD or hybrid? Also what tech (SLC, MLC, TLC, QLC) and how many nm? This is the real reason why I went all in on the X25-E; there is no clear winner since 10 years... either it's super expensive and might be good or cheap and guaranteed crap! Also the interfaces are all over the place, PCIe, SAS, SATA, NVMe (which is PCIe?), M.2 with double PCIe?! with all kinds of connectors... We have a serious problem with storage of data when magnetic tape is still the real solution!
      - consp 5y ago
        MLC SSD, Sorry for being unclear, googling it is difficult as they were never released to non DC providers (these are custom for oracle) so data sheets are hard to get (I only have partial data, rest is from the sas inquiry and mode pages). You can deduce some of the specs from the name as they are the same for all hgst drives: HUS - hgst uktrastar standard MH - MLC high endurance (4PB is claimed for the 200GB drives, verified this by looking at the lifetime counter and the bytes written) 4020 - max spec 400gb, device is 200gb A - generation S - 2.5" formfactor S2 - 12GB/s (sas3), dual port 1 - probably means custom spec all my hgst former netapp/sun/oracle drives have this set to 1 both SSD and hdd 0 - crypto sanetize Haven't seen them a lot on eBay last month but caching drives with low percentage writes do show up from time to time in batches.
        bullen 5y ago
        $25 per 200GB MLC used... I'd rather pay $100 per 64GB SLC unused.
user5994461 5y ago
For reference, you can buy a 2 TB SSD for $200 today. Same capacity as the 30 old drives, for much less space and cables.
Empf 5y ago
I'm lost on your comment. What specific problem die you see? It's much cheaper and easier to pay and use NVM than ram. You get best latency for databases across your data. The prices are already super cheap. I also not find it easier to buy a lot of small devices instead of one. Your NVM is doing the same thing you do but directly on the controller Check out the latency difference between sata and NVM. And yes CPU is idling often enough. Good thing if they have a little bit more io overhead but when you look at PS5 and Xbox, Sony solved this issue and Microsoft is bringing their new storage API to PC.
bob1029 5y ago
>I think we need to go back and look at cycles per bit I've started playing around with new storage architectures for databases and with flash storage in mind. One thing I encountered that seems incredibly powerful, when combined with a log-based store, is micro batching of operations. When you aggregate a few hundred/thousand transactions together and then append them as modified subtrees to the end of a log, you start to be able to say crazy shit like "Transactions per I/O operation or block" When working with objects in the 512b range, I can easily hit 2 million inserts per second on a single Samsung 960pro using this technique.
- jorangreef 5y ago
  We use a similar technique for TigerBeetle [1], a financial accounting database, to do a million financial transactions a second with full distributed persistence and reasonably low stable latency (< 20ms). We have around 10,000 financial transactions in a batch and amortize away the distributed consensus (Viewstamped Replication), the many assertions, syscalls and O_DSYNC so that they are almost free. Direct I/O helps to avoid flushing the CPU cache through memcpy's to the kernel's page cache for a roughly 7% improvement, and io_uring helps to amortize the syscalls themselves into a single io_uring_enter() for a doubling of throughput for small 4 KiB read/write units, but having a first-class batching interface from end to end is the biggest win by far. [1] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle
Severian 5y ago
This rack could be used for storage, not compute. You could add a Mellanox 400Gb NIC to the machine (assuming you have the required 32x PCI lanes for it), and then route that to your compute resource. This is a typical scenario.
luma 5y ago
Does this setup do 80M IOPS? I'm not sure what the point of your post is, everyone can make slow storage cheaply. The article is about fast storage which is also cheap (compared to similar solutions from flash storage array vendors).
tw04 5y ago
>What is the point of that speed if the rest of your system is a bottle neck and the drives will fail in a year if you use that speed continously? >I think we need to go back and look at cycles per bit (DWPD is confusing because you need to know the number of years and sometimes it's not mentioned on the product page). This work is for people developing dedicated storage arrays, not your laptop. Datacenter optane is good for 60 DWPD. Given that it is most frequently used as a caching layer, it's extremely unlikely you'd exceed that in a well designed system. https://www.intel.com/content/www/us/en/architecture-and-technology/optane-technology/delivering-new-levels-of-endurance-article-brief.html https://www.intel.com/content/www/us/en/architecture-and-tec... >Atleast SATA I can use in my laptops/desktops/consoles etc. Also nobody is looking at CPU usage of NVMe vs. SATA! Literally every storage vendor in the market is looking at CPU usage of NVMe vs. SATA. There's a reason they benchmark in "a standard 2U system" and not a Dell Laptop or Desktop.
- bullen 5y ago
  60 DWPD was the original optane development result, but the final product that we can buy is 10 DWPD. My issue with DWPD is that you can have the firmware complexity pretend the drive is good while losing space... I prefer to have really solid "100.000 writes per bit" 50nm SLC flash memory cells with less magic. If everyone is measuring NVMe vs. SATA how come I can't find anything interesting when googling? It's only popular articles about the high level difference, "the plug is different"... I would like benchmarks with apples to apples CPU usage while having alot of concurrent data in flight simoultaneously!
CyberDildonics 5y ago
You realize you could have bought at least 30 terabytes of SSD space for the money you spent right? If you wanted simplicity you could have bought a single 2 TB drive.

by bullen 5y ago

consp 5y ago

I did something similar on a smaller scale: "Caching" ssd drives which are not the best but have a huge amount write bytes left are widely available on ebay and I bought 8 200GB SAS3 drives for dirt cheap (under 200$ total, drives from 2017). Not the fastest drives but good enough for my purpose (overkill even).

bullen 5y ago
What drives did you get?
- consp 5y ago
  Sun/oracle branded HGST SS300 drives (HUSMH4020ASS210), they have the specs in the ballpark of the SS800 drives in endurance (according to the scsi data pages from the disks) but read/write performance of the SS300.
  - bullen 5y ago
    Trying to google it it seems unclear to me what the product is... SSD or hybrid? Also what tech (SLC, MLC, TLC, QLC) and how many nm? This is the real reason why I went all in on the X25-E; there is no clear winner since 10 years... either it's super expensive and might be good or cheap and guaranteed crap! Also the interfaces are all over the place, PCIe, SAS, SATA, NVMe (which is PCIe?), M.2 with double PCIe?! with all kinds of connectors... We have a serious problem with storage of data when magnetic tape is still the real solution!
    - consp 5y ago
      MLC SSD, Sorry for being unclear, googling it is difficult as they were never released to non DC providers (these are custom for oracle) so data sheets are hard to get (I only have partial data, rest is from the sas inquiry and mode pages). You can deduce some of the specs from the name as they are the same for all hgst drives: HUS - hgst uktrastar standard MH - MLC high endurance (4PB is claimed for the 200GB drives, verified this by looking at the lifetime counter and the bytes written) 4020 - max spec 400gb, device is 200gb A - generation S - 2.5" formfactor S2 - 12GB/s (sas3), dual port 1 - probably means custom spec all my hgst former netapp/sun/oracle drives have this set to 1 both SSD and hdd 0 - crypto sanetize Haven't seen them a lot on eBay last month but caching drives with low percentage writes do show up from time to time in batches.
      - bullen 5y ago
        $25 per 200GB MLC used... I'd rather pay $100 per 64GB SLC unused.

user5994461 5y ago

For reference, you can buy a 2 TB SSD for $200 today. Same capacity as the 30 old drives, for much less space and cables.

Empf 5y ago

I'm lost on your comment. What specific problem die you see? It's much cheaper and easier to pay and use NVM than ram. You get best latency for databases across your data. The prices are already super cheap. I also not find it easier to buy a lot of small devices instead of one. Your NVM is doing the same thing you do but directly on the controller Check out the latency difference between sata and NVM. And yes CPU is idling often enough. Good thing if they have a little bit more io overhead but when you look at PS5 and Xbox, Sony solved this issue and Microsoft is bringing their new storage API to PC.

bob1029 5y ago

>I think we need to go back and look at cycles per bit I've started playing around with new storage architectures for databases and with flash storage in mind. One thing I encountered that seems incredibly powerful, when combined with a log-based store, is micro batching of operations. When you aggregate a few hundred/thousand transactions together and then append them as modified subtrees to the end of a log, you start to be able to say crazy shit like "Transactions per I/O operation or block" When working with objects in the 512b range, I can easily hit 2 million inserts per second on a single Samsung 960pro using this technique.

jorangreef 5y ago
We use a similar technique for TigerBeetle [1], a financial accounting database, to do a million financial transactions a second with full distributed persistence and reasonably low stable latency (< 20ms). We have around 10,000 financial transactions in a batch and amortize away the distributed consensus (Viewstamped Replication), the many assertions, syscalls and O_DSYNC so that they are almost free. Direct I/O helps to avoid flushing the CPU cache through memcpy's to the kernel's page cache for a roughly 7% improvement, and io_uring helps to amortize the syscalls themselves into a single io_uring_enter() for a doubling of throughput for small 4 KiB read/write units, but having a first-class batching interface from end to end is the biggest win by far. [1] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle

Severian 5y ago

This rack could be used for storage, not compute. You could add a Mellanox 400Gb NIC to the machine (assuming you have the required 32x PCI lanes for it), and then route that to your compute resource. This is a typical scenario.

luma 5y ago

Does this setup do 80M IOPS? I'm not sure what the point of your post is, everyone can make slow storage cheaply. The article is about fast storage which is also cheap (compared to similar solutions from flash storage array vendors).

tw04 5y ago

>What is the point of that speed if the rest of your system is a bottle neck and the drives will fail in a year if you use that speed continously? >I think we need to go back and look at cycles per bit (DWPD is confusing because you need to know the number of years and sometimes it's not mentioned on the product page). This work is for people developing dedicated storage arrays, not your laptop. Datacenter optane is good for 60 DWPD. Given that it is most frequently used as a caching layer, it's extremely unlikely you'd exceed that in a well designed system. https://www.intel.com/content/www/us/en/architecture-and-technology/optane-technology/delivering-new-levels-of-endurance-article-brief.html https://www.intel.com/content/www/us/en/architecture-and-tec... >Atleast SATA I can use in my laptops/desktops/consoles etc. Also nobody is looking at CPU usage of NVMe vs. SATA! Literally every storage vendor in the market is looking at CPU usage of NVMe vs. SATA. There's a reason they benchmark in "a standard 2U system" and not a Dell Laptop or Desktop.

bullen 5y ago
60 DWPD was the original optane development result, but the final product that we can buy is 10 DWPD. My issue with DWPD is that you can have the firmware complexity pretend the drive is good while losing space... I prefer to have really solid "100.000 writes per bit" 50nm SLC flash memory cells with less magic. If everyone is measuring NVMe vs. SATA how come I can't find anything interesting when googling? It's only popular articles about the high level difference, "the plug is different"... I would like benchmarks with apples to apples CPU usage while having alot of concurrent data in flight simoultaneously!

CyberDildonics 5y ago

You realize you could have bought at least 30 terabytes of SSD space for the money you spent right? If you wanted simplicity you could have bought a single 2 TB drive.