4 ms·

Realistically, I think most workloads would end up moving to pure copy-on-write filesystems. There won't be much rewriting.

by nknight 15y ago

mistercow 15y ago
A copy-on-write file system still does plenty of rewriting; if it didn't, you'd run out of space before your harddrive was full.
- nknight 15y ago
  You seem to be in an entirely different universe from the point. Most workloads would take weeks, months, even years to write out 60TB of data. It will be unnecessary for the filesystems to garbage collect old data with any frequency, and you may very well not want them to -- when you have so much space, why not have an available record of every single I/O transaction back to the beginning of time?
  - mistercow 15y ago
    > Most workloads would take weeks, months, even years to write out 60TB of data Today, yes, but this isn't going to give us 60TB drives today, or even tomorrow. It's going to give us 6TB drives. It might give us 60TB drives at some point down the line, and it would be unwise to make any strong predictions about workloads at indeterminate points in the future.
    - nknight 15y ago
      60TB is 2+ years of HD video at typical bitrates. It's two months at a patently absurd 100mbps. You could write every bit of traffic passing through a 10gigabit link for 14 hours. Assuming a 100-year lifespan, if every second of your life were recorded at 5mbps, you would need just 30 such drives. Yeah, I'm pretty confident about my prediction as to most workloads.
      - tjoff 15y ago
        I get 9 months at 20 MBit/s [1] which is kind of low for a blu-ray[2] or a high-end consumer camera, today. I find it kind of difficult discussing 60 TB since it will be soo faar in the future. By then we will at least have 4k and probably a bitrate in ~100 MBit/s (since the cap is at 50 MBit/s today that is aiming low but there will probably be a limit where you say that it is enough (then again, 3D imagery could potentially demand much more)), a small drive (same density but ~2") would hold about 2 weeks of footage which IMO sounds reasonable for a camera in an age where you don't have to compromise that much. It will put some constraints on usage patterns, but I don't feel that 60 TB will be that much more data than 4 TB is today. Sure, for text and audio you basically will have an infinite amount of space (as a consumer) but that is the case even today. 60 TB is only 15 times more than we have to day in a single drive, and it will take many many years to get there. And I can barely fit my grandfathers VHS camera tapes on a 4 TB drive today, and since I want some redundancy 4 TB isn't enough for even that. 1. http://www.wolframalpha.com/input/?i=60+TB+at+20+Mb%2Fs http://www.wolframalpha.com/input/?i=60+TB+at+20+Mb%2Fs 2. http://forum.blu-ray.com/showthread.php?t=3338 http://forum.blu-ray.com/showthread.php?t=3338
        nknight 15y ago
        > I get 9 months at 20 MBit/s [1] which is kind of low for a blu-ray Blu-Ray bitrates are bloated for various reasons, like older encoders, film transfer, and a lack of anything better to do with the space. You don't need 20mbps with modern H.264 High encoders, it's just a waste. Anything past 10 is at best questionable > By then we will at least have 4k and probably a bitrate in ~100 MBit/s Even if 50mbps were something that we would ever need at 1080p (it's not), 4k would not justify a doubling in compressed size, there is no reason to have 100mbps compressed video. 4k is also way beyond reasonable for home display. Just 3840x2160 on a 60" display would be something like 23 arc seconds at 3 meters, you physically cannot see that. > 60 TB is only 15 times more than we have to day in a single drive 4TB is already orders of magnitude more than most workloads will ever need. I was using video to demonstrate the absurdity of thinking typical workloads would actively use 60TB, because it's not a typical workload, it's a worst-case scenario.
  - wmf 15y ago
    People won't buy drives much bigger than their workload can use. It's possible that 60TB drives would only be used in massive storage systems, not PCs.
    - vidarh 15y ago
      My DVD collection includes about 3TB of data with no HD content... 60TB isn't that much.
      - nknight 15y ago
        Your DVD collection uses the largely obsolete MPEG-2. DVD-quality video in h.264 is a fraction of the size.
        vidarh 15y ago
        But any new content I'm likely to acquire will not be DVD quality video. A backup of a Bluray takes up vastly more than a DVD. Even my phone records video at a higher bitrate than most DVD's use, and my camera even more.
    - Groxx 15y ago
      People will, if it comes bundled with their computer. Nearly every grandparent I've run across has a >100GB hard drive (if not >500GB). They're all using roughly 20GB of that drive, after a couple years. With only very rare exceptions, people don't choose what goes into their computer. They buy what looks good for a price that they're willing to pay, for the simple reason that buying a computer is full of more buzz-words, acronyms, and useless numbers than any other purchase I've ever made. It's insane, and it gets people to buy more than they need, so the sellers won't stop doing it.
      - roc 15y ago
        > "People will, if it comes bundled with their computer." But what will a computer look like in 10 years? What I've been noting, from the whole mobile explosion, is that people really don't care if their computer only has 64GB of storage. Of all the complaints that people have about tablets and phones replacing 'real' computing, disk space rarely enters the discussion. So even if tablets and smartphones don't notably replace traditional PCs, their effect on consumer perception of the necessity of large disks can't be ignored. Sure, it's easy to market 'more gigabytes than your old machine' to a regular person. But so is "faster boot" and "less waiting". That's why laptop makers can't seem to ditch spinning disks fast enough. And what motivation does a PC builder have, to continue bundling increasingly-capacious hard drives with little markup? Why not bundle increasingly-faster and increasingly-cheaper SSDs and save the spinning drives for higher-margin positions as add-ons and external storage? I'd be incredibly surprised if the market for spinning disks doesn't shrink over the next 10 years. They'll certainly remain, even for consumers. But there's no good argument that they'll still be standard.
  - Freaky 15y ago
    Quick data point: DragonFlyBSD's HAMMER filesystem uses a periodic cronjob to garbage collect and otherwise maintains full history of writes.

Realistically, I think most workloads would end up moving to pure copy-on-write filesystems. There won't be much rewriting.

by nknight 15y ago

Realistically, I think most workloads would end up moving to pure copy-on-write filesystems. There won't be much rewriting.

mistercow 15y ago

A copy-on-write file system still does plenty of rewriting; if it didn't, you'd run out of space before your harddrive was full.

nknight 15y ago
You seem to be in an entirely different universe from the point. Most workloads would take weeks, months, even years to write out 60TB of data. It will be unnecessary for the filesystems to garbage collect old data with any frequency, and you may very well not want them to -- when you have so much space, why not have an available record of every single I/O transaction back to the beginning of time?
- mistercow 15y ago
  > Most workloads would take weeks, months, even years to write out 60TB of data Today, yes, but this isn't going to give us 60TB drives today, or even tomorrow. It's going to give us 6TB drives. It might give us 60TB drives at some point down the line, and it would be unwise to make any strong predictions about workloads at indeterminate points in the future.
  - nknight 15y ago
    60TB is 2+ years of HD video at typical bitrates. It's two months at a patently absurd 100mbps. You could write every bit of traffic passing through a 10gigabit link for 14 hours. Assuming a 100-year lifespan, if every second of your life were recorded at 5mbps, you would need just 30 such drives. Yeah, I'm pretty confident about my prediction as to most workloads.
    - tjoff 15y ago
      I get 9 months at 20 MBit/s [1] which is kind of low for a blu-ray[2] or a high-end consumer camera, today. I find it kind of difficult discussing 60 TB since it will be soo faar in the future. By then we will at least have 4k and probably a bitrate in ~100 MBit/s (since the cap is at 50 MBit/s today that is aiming low but there will probably be a limit where you say that it is enough (then again, 3D imagery could potentially demand much more)), a small drive (same density but ~2") would hold about 2 weeks of footage which IMO sounds reasonable for a camera in an age where you don't have to compromise that much. It will put some constraints on usage patterns, but I don't feel that 60 TB will be that much more data than 4 TB is today. Sure, for text and audio you basically will have an infinite amount of space (as a consumer) but that is the case even today. 60 TB is only 15 times more than we have to day in a single drive, and it will take many many years to get there. And I can barely fit my grandfathers VHS camera tapes on a 4 TB drive today, and since I want some redundancy 4 TB isn't enough for even that. 1. http://www.wolframalpha.com/input/?i=60+TB+at+20+Mb%2Fs http://www.wolframalpha.com/input/?i=60+TB+at+20+Mb%2Fs 2. http://forum.blu-ray.com/showthread.php?t=3338 http://forum.blu-ray.com/showthread.php?t=3338
      - nknight 15y ago
        > I get 9 months at 20 MBit/s [1] which is kind of low for a blu-ray Blu-Ray bitrates are bloated for various reasons, like older encoders, film transfer, and a lack of anything better to do with the space. You don't need 20mbps with modern H.264 High encoders, it's just a waste. Anything past 10 is at best questionable > By then we will at least have 4k and probably a bitrate in ~100 MBit/s Even if 50mbps were something that we would ever need at 1080p (it's not), 4k would not justify a doubling in compressed size, there is no reason to have 100mbps compressed video. 4k is also way beyond reasonable for home display. Just 3840x2160 on a 60" display would be something like 23 arc seconds at 3 meters, you physically cannot see that. > 60 TB is only 15 times more than we have to day in a single drive 4TB is already orders of magnitude more than most workloads will ever need. I was using video to demonstrate the absurdity of thinking typical workloads would actively use 60TB, because it's not a typical workload, it's a worst-case scenario.
- wmf 15y ago
  People won't buy drives much bigger than their workload can use. It's possible that 60TB drives would only be used in massive storage systems, not PCs.
  - vidarh 15y ago
    My DVD collection includes about 3TB of data with no HD content... 60TB isn't that much.
    - nknight 15y ago
      Your DVD collection uses the largely obsolete MPEG-2. DVD-quality video in h.264 is a fraction of the size.
      - vidarh 15y ago
        But any new content I'm likely to acquire will not be DVD quality video. A backup of a Bluray takes up vastly more than a DVD. Even my phone records video at a higher bitrate than most DVD's use, and my camera even more.
  - Groxx 15y ago
    People will, if it comes bundled with their computer. Nearly every grandparent I've run across has a >100GB hard drive (if not >500GB). They're all using roughly 20GB of that drive, after a couple years. With only very rare exceptions, people don't choose what goes into their computer. They buy what looks good for a price that they're willing to pay, for the simple reason that buying a computer is full of more buzz-words, acronyms, and useless numbers than any other purchase I've ever made. It's insane, and it gets people to buy more than they need, so the sellers won't stop doing it.
    - roc 15y ago
      > "People will, if it comes bundled with their computer." But what will a computer look like in 10 years? What I've been noting, from the whole mobile explosion, is that people really don't care if their computer only has 64GB of storage. Of all the complaints that people have about tablets and phones replacing 'real' computing, disk space rarely enters the discussion. So even if tablets and smartphones don't notably replace traditional PCs, their effect on consumer perception of the necessity of large disks can't be ignored. Sure, it's easy to market 'more gigabytes than your old machine' to a regular person. But so is "faster boot" and "less waiting". That's why laptop makers can't seem to ditch spinning disks fast enough. And what motivation does a PC builder have, to continue bundling increasingly-capacious hard drives with little markup? Why not bundle increasingly-faster and increasingly-cheaper SSDs and save the spinning drives for higher-margin positions as add-ons and external storage? I'd be incredibly surprised if the market for spinning disks doesn't shrink over the next 10 years. They'll certainly remain, even for consumers. But there's no good argument that they'll still be standard.
- Freaky 15y ago
  Quick data point: DragonFlyBSD's HAMMER filesystem uses a periodic cronjob to garbage collect and otherwise maintains full history of writes.