7 ms·

SSD speeds are nothing short of miraculous in my mind. I come from the old days of striping 16 HDDs together (at a minimum number) to get 1GB/s throughput. Depe

SSD speeds are nothing short of miraculous in my mind. I come from the old days of striping 16 HDDs together (at a minimum number) to get 1GB/s throughput. Depending on the chassis, that was 2 8-drive enclosures in the "desktop" version or the large 4RU enclosures with redundant PSUs and fans loud enough to overpower arena rock concerts. Now, we can get 5+GB/s throughput from a tiny stick that can be used externally via a single cable for data&power that is absolutely silent. I edit 4K+ video as well, and now can edit directly from the same device the camera recorded to during production. I'm skipping over the parts of still making backups, but there's no more multi-hour copy from source media to edit media during a DIT step. I've spent many a shoot as a DIT wishing the 1s&0s would travel across devices much faster while everyone else on the production has already left, so this is much appreciated by me. Oh, and those 16 device units only came close to 4TB around the time of me finally dropping spinning rust. The first enclosure I ever dealt with was a 7-bay RAID-0 that could just barely handle AVR75 encoding from Avid. Just barely to the point that only video was saved to the array. The audio throughput would put it over the top, so audio was saved to a separate external drive. Using SSD feels like a well deserved power up from those days.

Spooky23 1y ago
Totally. I spent a lot of time 15-20 years ago building out large email systems. I recently bought a $17 SSD for my son’s middle school project that was speced to deliver like 3x what I needed in those days. From a storage perspective, I was probably spending $50 GB/mo all-in to deploy a multi million dollar storage solution. TBH… you’d probably smoke that system with used laptops today.
- 392 1y ago
  If only folks wrote software with this in mind.
S04dKHzrKT 1y ago
In a similar vein, I had Western Digital Raptors striped in my gaming PC in the mid 2000s. I remember just how amazed I was after moving to SSDs.
gchamonlive 1y ago
> I come from the old days of striping 16 HDDs together (at a minimum number) to get 1GB/s throughput Woah, how long would that last before you'd start having to replace the drives?
- dylan604 1y ago
  Depending on the HDD vendor/model. We had hot spares and cold spares. On one build, we had a bad batch of drives. We built the array on a Friday, and left it for burn-in running over the weekend. On Monday, we came in to a bunch of alarms and >50% failure rate. At least they died during the burn-in so no data loss, but it was an extreme example. That was across multiple 16-bay rack mount chassis. It was an infamous case though, we were not alone. More typically, you'd have a drive die much less frequently, but it was something you absolutely had to be prepared for. With RAID-6 and a hot spare, you could be okay with a single drive failure. Theoretically, you could lose two, but it would be a very nervy day getting the array to rebuild without issue.
  - gchamonlive 1y ago
    I asked because I did a makeshift NAS for myself with three 4tb ironwolf, but they died before the third year. I didn't investigate much, but it was most likely because of power outages and a lack of a nobreak PSU at that time. It's still quite a bit of work to maintain physical hard drives and the probability of failure as I understand tend to increase the more units the array has because of inverse probability (not the likelihood of one of them failing but the likelihood of none of them failing after a period of time, which is cumulative)
    - dylan604 1y ago
      Any electronic gear that you care about must be connected to a UPS. HDDs are very susceptible to power issues. Good UPS are also line conditioners so you get a clean sine wave rather than whatever comes straight from the mains. If you've never seen it, connect a meter to an outlet in your home and what how much fluctuations you get throughout the day. Most people think about spikes/surges, while forgetting that dips and under-volting is damaging as well. Most equipment have a range of acceptable voltage, but you'd be amazed at the number of times mains will dip below that range. Obviously location will have an affect on quality of service, but I hear my UPSes kick in multiple times a week to cover a dip if only for a couple of seconds. The fun thing about storage pools is that they can lull you into thinking they are set it and forget it. You have to monitor SMART messages. Most drives will give you a heads up if you know where to look. Having the fortitude to have a hot spare instead of just adding it to the storage pool goes a long way from losing data.
lvl155 1y ago
You should try now-discontinued Intel Optane especially p5800x. I got my OS running on them and they are incredible.
- the8472 1y ago
  Just a few more years until we get MRAM as viable storage technology. And affordable fusion, and hovercars.
- OptionOfT 1y ago
  The endurance on those drivers is amazing. I have (stupidly) used a too small Samsung EVO drive as a caching drive, and that is probably the first computer part that I've worn out (bar a mouse & keyboard).
- MrDrMcCoy 1y ago
  I'm running 12 of them for ZFS cache/log/special, and they are fast/tough enough to make a large array on a slow link feel fast. I shake my fist at Intel and Micron for taking away one of the best memory technologies to ever exist.
geerlingguy 1y ago
This hits home even more since I started restoring some vintage Macs. For the ones new enough to get an SSD upgrade, it's night and day the difference (even a Power Mac G4 can feel fresh and fast just swapping out the drive). For older Macs like PowerBooks and classic Macs, there are so many SD/CF card to IDE/SCSI/etc. adapters now, they also get a significant boost. But part of the nostalgia of sitting there listening to the rumble of the little hard drive is gone.
- brailsafe 1y ago
  > For older Macs like PowerBooks and classic Macs, there are so many SD/CF card to IDE/SCSI/etc. Would those be bandwidth limited by the adapter/card or CPU? Can you get throughput higher than say, a cheap 2.5" SSD over Sata 3/4?
  - eptcyka 1y ago
    You are limited at first by the IDE/SCSI interface, so below SATA speeds.
    - brailsafe 1y ago
      Oh I must have misread the comment initially as PCIE/SCSI
- dylan604 1y ago
  I had a 2011 MBP that I kept running by replacing the HDD with an SSD, and then removed the DVD-ROM drive with a second SSD. The second SSD had throughput limits because it was designed for shiny round disc, so it had a lower ability chip. I had that until the 3rd GPU replacement died, and eventually switched to second gen butterfly keyboard. The only reason it was tolerable was because of the SSDs, oh and the RAM upgrades
  - geerlingguy 1y ago
    Did you ever have the GPU issue? My sister had a 2011, I had to desolder a resistor (or maybe two?) on it to bypass the dGPU since it was causing it to boot loop. But now it's still running and pretty happily for some basic needs!
jorvi 1y ago
It's not really the SSDs themselves that are incredibly fast (they still are somewhat), it's mostly the RAM cache and clever tricks to make TLC feel like SLC. Most (cheap) SSDs their performance goes off a cliff once you hit the boundary of these tricks.
- dylan604 1y ago
  [flagged]
  - redman25 1y ago
    It can be good to know that SSDs are fast until you exhaust the cache by copying gigs of files around. It doesn’t hurt to be aware of the limitations even if for the common case things are much better.
    - dylan604 1y ago
      We're talking about devices capable of >2GB/s throughput, and acquiring footage <.5GB/s. No caching issues, but I'm not buying el cheapo SSDs either. These are all rated and approved by camera makers. It wasn't brought up because it's not an issue. For people that are wondering why camera makers approve or not particular recording media, this is part of the answer. But nobody was asking that particular question and instead the reply tried to rain on my parade.
  - jorvi 1y ago
    Not sure what warranted such an aggressive response. I grew up in the 90s, on 56kb modems and PCs that rumbled and whined when you booted them up. I was at the tail end of using floppies. I never said I didn't love the speed of SSDs, and when they just started to become mainstream it was an upgrade I did for everyone around me. I made my comment in part because you mentioned dumping 4K into the SSD and/or editing it. It can be a nasty surprise if you're doing something live, and suddenly your throughout plummets, everything starts to stutter and you have no idea why.
    - 1y ago
bob1029 1y ago
The latency of modern NVMe is what really blows my mind (as low as 20~30 uS). NVMe is about an order of magnitude quicker than SAS and SATA. This is why I always recommend developers try using SQLite on top of NVMe storage. The performance is incredible. I don't think you would see query times anywhere near 20uS with a hosted SQL solution, even if it's on the same machine using named pipes or other IPC mechanism.
- MortyWaves 1y ago
  Meanwhile a job recently told me they are on IBM AS400 boxes “because Postgress and other sql databases can’t keep up with the number of transactions we have”… for a company that has a few thousand inserts per day… Obviously not true that they’d overwhelm modern databases but feels like that place has had the same opinions since the 1960s.
- Weryj 1y ago
  I’m still buying old Optane drives for that latency when it matters. RockDB loves it
  - mhuffman 1y ago
    >Optane drives This is the secret hack a lot of people don't know about!
- Numerlor 1y ago
  Then there's optane that got ~10us with. The newest controllers and nand is inching closer with randoms but optane is still the most miraculous ssd tech that's normally obtainable
  - robaato 1y ago
    But they've retired it??
  - teaearlgraycold 1y ago
    Eventually we'll have machines with unified memory+storage. You'll certainly have to take a bit of a performance hit in certain scenarios but also think about the load time improvements. If you store video game files in the same format they'd be needed at runtime you could be at the main menu in under a second.
wslh 1y ago
So, now someone can strip several SSDs to gain more performance as well for other purposes than video editing, right?
- AStonesThrow 1y ago
  [dead]

Spooky23 1y ago

Totally. I spent a lot of time 15-20 years ago building out large email systems. I recently bought a $17 SSD for my son’s middle school project that was speced to deliver like 3x what I needed in those days. From a storage perspective, I was probably spending $50 GB/mo all-in to deploy a multi million dollar storage solution. TBH… you’d probably smoke that system with used laptops today.

392 1y ago
If only folks wrote software with this in mind.

S04dKHzrKT 1y ago

In a similar vein, I had Western Digital Raptors striped in my gaming PC in the mid 2000s. I remember just how amazed I was after moving to SSDs.

gchamonlive 1y ago

> I come from the old days of striping 16 HDDs together (at a minimum number) to get 1GB/s throughput Woah, how long would that last before you'd start having to replace the drives?

dylan604 1y ago
Depending on the HDD vendor/model. We had hot spares and cold spares. On one build, we had a bad batch of drives. We built the array on a Friday, and left it for burn-in running over the weekend. On Monday, we came in to a bunch of alarms and >50% failure rate. At least they died during the burn-in so no data loss, but it was an extreme example. That was across multiple 16-bay rack mount chassis. It was an infamous case though, we were not alone. More typically, you'd have a drive die much less frequently, but it was something you absolutely had to be prepared for. With RAID-6 and a hot spare, you could be okay with a single drive failure. Theoretically, you could lose two, but it would be a very nervy day getting the array to rebuild without issue.
- gchamonlive 1y ago
  I asked because I did a makeshift NAS for myself with three 4tb ironwolf, but they died before the third year. I didn't investigate much, but it was most likely because of power outages and a lack of a nobreak PSU at that time. It's still quite a bit of work to maintain physical hard drives and the probability of failure as I understand tend to increase the more units the array has because of inverse probability (not the likelihood of one of them failing but the likelihood of none of them failing after a period of time, which is cumulative)
  - dylan604 1y ago
    Any electronic gear that you care about must be connected to a UPS. HDDs are very susceptible to power issues. Good UPS are also line conditioners so you get a clean sine wave rather than whatever comes straight from the mains. If you've never seen it, connect a meter to an outlet in your home and what how much fluctuations you get throughout the day. Most people think about spikes/surges, while forgetting that dips and under-volting is damaging as well. Most equipment have a range of acceptable voltage, but you'd be amazed at the number of times mains will dip below that range. Obviously location will have an affect on quality of service, but I hear my UPSes kick in multiple times a week to cover a dip if only for a couple of seconds. The fun thing about storage pools is that they can lull you into thinking they are set it and forget it. You have to monitor SMART messages. Most drives will give you a heads up if you know where to look. Having the fortitude to have a hot spare instead of just adding it to the storage pool goes a long way from losing data.

lvl155 1y ago

You should try now-discontinued Intel Optane especially p5800x. I got my OS running on them and they are incredible.

the8472 1y ago
Just a few more years until we get MRAM as viable storage technology. And affordable fusion, and hovercars.
OptionOfT 1y ago
The endurance on those drivers is amazing. I have (stupidly) used a too small Samsung EVO drive as a caching drive, and that is probably the first computer part that I've worn out (bar a mouse & keyboard).
MrDrMcCoy 1y ago
I'm running 12 of them for ZFS cache/log/special, and they are fast/tough enough to make a large array on a slow link feel fast. I shake my fist at Intel and Micron for taking away one of the best memory technologies to ever exist.

geerlingguy 1y ago

This hits home even more since I started restoring some vintage Macs. For the ones new enough to get an SSD upgrade, it's night and day the difference (even a Power Mac G4 can feel fresh and fast just swapping out the drive). For older Macs like PowerBooks and classic Macs, there are so many SD/CF card to IDE/SCSI/etc. adapters now, they also get a significant boost. But part of the nostalgia of sitting there listening to the rumble of the little hard drive is gone.

brailsafe 1y ago
> For older Macs like PowerBooks and classic Macs, there are so many SD/CF card to IDE/SCSI/etc. Would those be bandwidth limited by the adapter/card or CPU? Can you get throughput higher than say, a cheap 2.5" SSD over Sata 3/4?
- eptcyka 1y ago
  You are limited at first by the IDE/SCSI interface, so below SATA speeds.
  - brailsafe 1y ago
    Oh I must have misread the comment initially as PCIE/SCSI
dylan604 1y ago
I had a 2011 MBP that I kept running by replacing the HDD with an SSD, and then removed the DVD-ROM drive with a second SSD. The second SSD had throughput limits because it was designed for shiny round disc, so it had a lower ability chip. I had that until the 3rd GPU replacement died, and eventually switched to second gen butterfly keyboard. The only reason it was tolerable was because of the SSDs, oh and the RAM upgrades
- geerlingguy 1y ago
  Did you ever have the GPU issue? My sister had a 2011, I had to desolder a resistor (or maybe two?) on it to bypass the dGPU since it was causing it to boot loop. But now it's still running and pretty happily for some basic needs!

jorvi 1y ago

It's not really the SSDs themselves that are incredibly fast (they still are somewhat), it's mostly the RAM cache and clever tricks to make TLC feel like SLC. Most (cheap) SSDs their performance goes off a cliff once you hit the boundary of these tricks.

dylan604 1y ago
[flagged]
- redman25 1y ago
  It can be good to know that SSDs are fast until you exhaust the cache by copying gigs of files around. It doesn’t hurt to be aware of the limitations even if for the common case things are much better.
  - dylan604 1y ago
    We're talking about devices capable of >2GB/s throughput, and acquiring footage <.5GB/s. No caching issues, but I'm not buying el cheapo SSDs either. These are all rated and approved by camera makers. It wasn't brought up because it's not an issue. For people that are wondering why camera makers approve or not particular recording media, this is part of the answer. But nobody was asking that particular question and instead the reply tried to rain on my parade.
- jorvi 1y ago
  Not sure what warranted such an aggressive response. I grew up in the 90s, on 56kb modems and PCs that rumbled and whined when you booted them up. I was at the tail end of using floppies. I never said I didn't love the speed of SSDs, and when they just started to become mainstream it was an upgrade I did for everyone around me. I made my comment in part because you mentioned dumping 4K into the SSD and/or editing it. It can be a nasty surprise if you're doing something live, and suddenly your throughout plummets, everything starts to stutter and you have no idea why.
  - 1y ago

bob1029 1y ago

The latency of modern NVMe is what really blows my mind (as low as 20~30 uS). NVMe is about an order of magnitude quicker than SAS and SATA. This is why I always recommend developers try using SQLite on top of NVMe storage. The performance is incredible. I don't think you would see query times anywhere near 20uS with a hosted SQL solution, even if it's on the same machine using named pipes or other IPC mechanism.

MortyWaves 1y ago
Meanwhile a job recently told me they are on IBM AS400 boxes “because Postgress and other sql databases can’t keep up with the number of transactions we have”… for a company that has a few thousand inserts per day… Obviously not true that they’d overwhelm modern databases but feels like that place has had the same opinions since the 1960s.
Weryj 1y ago
I’m still buying old Optane drives for that latency when it matters. RockDB loves it
- mhuffman 1y ago
  >Optane drives This is the secret hack a lot of people don't know about!
Numerlor 1y ago
Then there's optane that got ~10us with. The newest controllers and nand is inching closer with randoms but optane is still the most miraculous ssd tech that's normally obtainable
- robaato 1y ago
  But they've retired it??
- teaearlgraycold 1y ago
  Eventually we'll have machines with unified memory+storage. You'll certainly have to take a bit of a performance hit in certain scenarios but also think about the load time improvements. If you store video game files in the same format they'd be needed at runtime you could be at the main menu in under a second.

wslh 1y ago

So, now someone can strip several SSDs to gain more performance as well for other purposes than video editing, right?

AStonesThrow 1y ago
[dead]