4 ms·

> I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental The problem with spinning disks is that speed

by darkr 11y ago

> I wonder if there will be 100TB spinning read/write drives by 2020, an exponential leap instead of incremental The problem with spinning disks is that speed and reliability has not increased in line with capacity. There comes a point at which it doesn't make sense to make them any bigger (capacity-wise). Just creating a filesystem on an 8TB disk takes hours. If you bring in disk (block level) encryption and the requirement to fill the disk with random data and then encrypt before creating the filesystem, you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. On the reliability front - a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss, as well as ~ 8x more data to recover/re-distribute for every failure

SixSigma 11y ago
> you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. That's what adding value looks like.
__david__ 11y ago
> Just creating a filesystem on an 8TB disk takes hours. Are you sure you mean TB? $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop8 /dev/loop9 $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9} mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md8 started. $ sudo time mke2fs /dev/md8 [lots of output snipped] real 42m0.590s user 0m2.470s sys 1m0.440s $ du -sh . # just for fun 65G . That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.
- ck2 11y ago
  Is that defaulting to EXT4 or XFS ?
  - winter_blue 11y ago
    He used `mke2fs`, which defaults to Ext4.
bsdetector 11y ago
> Just creating a filesystem on an 8TB disk takes hours. Am I being dense here, but why would you create a filesystem on the hard drive? The hard drive should only store file contents. There's all kinds of seeking and syncing and random access needed for the filesystem. Seeks from file contents can't be avoided, but ones due to the filesystem can be. If there's no metadata-on-ssd + data-on-disk filesystem for Linux, there should be.
- winter_blue 11y ago
  So what you're advocating is that instead of creating an ext4/zfs/xfs file system on each drive, have a sort of distributed filesystem that has raw access to each drive. And the i-node of the distributed filesystem could be always-available on RAM with SSD backing. My guess is, Google probably already does this.
- woodman 11y ago
  ZFS allows for that level of control: assign a cache device to the pool, then set your secondarycache flag to metadata. Yes the metadata will hit the primary cache first (memory), but it will spill onto the secondary as it is pressured out of ARC. I've got a SD card playing that role in my build machine, it works really well for keeping track of all the tiny files that make up the FreeBSD base and ports trees.
- pjc50 11y ago
  Interesting, I don't think much work has been done in this area. It's not a common way of dividing the workload, even though it sounds obvious.
- darkr 11y ago
  Lots of distributed filesystems (I guess I'm talking about distributed object stores here rather than an actual clustered network filesystem) rely on a regular POSIX filesystem for the storage of blobs. A huge amount of work has gone into making ext4/zfs/xfs etc fast and reliable, plus you get other benefits like journaling, filesystem caching, fsync(), metadata caching (a la ZFS), so there are credible arguments for not going down the path of NIH.
jo909 11y ago
> a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss I don't understand that argument, not if you think at scale. You need to store 8000TB. You can run 8000 x 1TB, or 1000 x 8TB. You loose 10% of the disks per year (to keep the math simple). So after a year, you have either had to redistribute 800 x 1TB or 100 x 1TB, which is the same to a software defined distributed storage. There are operational upsides to fewer disks, power usage, manpower, rack space etc. But of course more disks are faster. Google wants to have more options than just bigger vs faster available. Since they know exactly, and can influence, how their software behaves, they can gain much from using specialized disks that balance things differently.
jabl 11y ago
> Just creating a filesystem on an 8TB disk takes hours. Xfs creates such a filesystem in seconds. As another reply showed ext4 is quite a bit slower since it preallocates the inodes; more modern designs allocate them on the fly as needed.

by darkr 11y ago

SixSigma 11y ago

> you're looking at a multi-day task. Expanded to 100TB, you could be looking at a month just to bring a disk online. That's what adding value looks like.

__david__ 11y ago

> Just creating a filesystem on an 8TB disk takes hours. Are you sure you mean TB? $ for i in $(seq 8); do truncate -s 1T 8tb.$i.img; losetup --find --show 8tb.$i.img; done /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop8 /dev/loop9 $ sudo mdadm --create --level linear -n 8 /dev/md8 /dev/loop{2..9} mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md8 started. $ sudo time mke2fs /dev/md8 [lots of output snipped] real 42m0.590s user 0m2.470s sys 1m0.440s $ du -sh . # just for fun 65G . That's certainly not fast, but it's not "hours" either. It's also a magnetic disk that was actively being used by other processes. Given how much data it wrote, it averaged 25MB/s, which is not really that fast. An 8TB SSD would be much faster.

ck2 11y ago
Is that defaulting to EXT4 or XFS ?
- winter_blue 11y ago
  He used `mke2fs`, which defaults to Ext4.

bsdetector 11y ago

> Just creating a filesystem on an 8TB disk takes hours. Am I being dense here, but why would you create a filesystem on the hard drive? The hard drive should only store file contents. There's all kinds of seeking and syncing and random access needed for the filesystem. Seeks from file contents can't be avoided, but ones due to the filesystem can be. If there's no metadata-on-ssd + data-on-disk filesystem for Linux, there should be.

winter_blue 11y ago
So what you're advocating is that instead of creating an ext4/zfs/xfs file system on each drive, have a sort of distributed filesystem that has raw access to each drive. And the i-node of the distributed filesystem could be always-available on RAM with SSD backing. My guess is, Google probably already does this.
woodman 11y ago
ZFS allows for that level of control: assign a cache device to the pool, then set your secondarycache flag to metadata. Yes the metadata will hit the primary cache first (memory), but it will spill onto the secondary as it is pressured out of ARC. I've got a SD card playing that role in my build machine, it works really well for keeping track of all the tiny files that make up the FreeBSD base and ports trees.
pjc50 11y ago
Interesting, I don't think much work has been done in this area. It's not a common way of dividing the workload, even though it sounds obvious.
darkr 11y ago
Lots of distributed filesystems (I guess I'm talking about distributed object stores here rather than an actual clustered network filesystem) rely on a regular POSIX filesystem for the storage of blobs. A huge amount of work has gone into making ext4/zfs/xfs etc fast and reliable, plus you get other benefits like journaling, filesystem caching, fsync(), metadata caching (a la ZFS), so there are credible arguments for not going down the path of NIH.

jo909 11y ago

> a spinning 8TB disk is probably about as reliable as a 1TB disk, so that means you have a 8x increase in probability of data loss I don't understand that argument, not if you think at scale. You need to store 8000TB. You can run 8000 x 1TB, or 1000 x 8TB. You loose 10% of the disks per year (to keep the math simple). So after a year, you have either had to redistribute 800 x 1TB or 100 x 1TB, which is the same to a software defined distributed storage. There are operational upsides to fewer disks, power usage, manpower, rack space etc. But of course more disks are faster. Google wants to have more options than just bigger vs faster available. Since they know exactly, and can influence, how their software behaves, they can gain much from using specialized disks that balance things differently.

jabl 11y ago

> Just creating a filesystem on an 8TB disk takes hours. Xfs creates such a filesystem in seconds. As another reply showed ext4 is quite a bit slower since it preallocates the inodes; more modern designs allocate them on the fly as needed.