5 ms·

TL;DR: "simple lack of foresight combined with the age-old problem of the temporary solution becoming de-facto permanent"

simonblack 6y ago
Exactly. Just another case of "640K should be enough for anybody". But we have to admit that storage has leapt hugely in size and very rapidly. In 1990 or thereabouts, a common hard disk size was about 30 megabytes. Today I have files that are many times that size, and we think that that is quite normal. A mere five or so years later around 1996, that size had jumped to 4000 megabytes! These days my laptop has 6 million megabytes. We can't blame filesystem planners for lack of foresight. They were blind-sided by progress just like the rest of us were.
- mnw21cam 6y ago
  I have a single file that is 6 million megabytes in size.
  - Thristle 6y ago
    who/what reads that file?
    - mnw21cam 6y ago
      tar
      - AnIdiotOnTheNet 6y ago
        That stretches the definition of "a single file" when it is basically a filesystem itself.
        bilekas 6y ago
        Not necessarily, it's all relative really. But the point is, the limitations or expected limitations are only relevant until they're not.. Also why not tar a drive for archive if I want :}
    - jcranmer 6y ago
      I haven't managed a 6TB file before, but I did have a >100GB log file I ended up parsing with awk just to be sure that the memory consumption on the consuming process wouldn't go crazy.
    - petschge 6y ago
      Not my file, but I have produced HDF5 files in that size range using simulation codes on supercomputers and paraview can open them and render 3d view as video from it (eventually).
- ben509 6y ago
  He had already seen exponential growth in storage capacity and could easily extrapolate from that. Moore's law was coined in the '60s, after all. This wasn't a lack of foresight, this was being prudent. If I can't come close to testing an upper limit, it's correct to acknowledge it may degrade in a way I haven't anticipated. Especially since he was writing a GUI which should generally avoid handing users footguns, and which can be updated over time.
jandrese 6y ago
More directly: He only expected his code to be used in Windows NT and made the fatal flaw of thinking that just because he slapped it together in an hour it would be easy for someone to rewrite in the next version of Windows to up the limit. Instead the code sat untouched for decades as the limit slowly became more and more unreasonable. IMHO the real issue is the artificial limit. Instead of straight up disallowing you from creating large filesystems, it should have had a dialog box warning about small files being inefficient but letting you make it anyway.
- jackric 6y ago
  >Instead the code sat untouched for decades as the limit slowly became more and more unreasonable. My projects likely won't span decades but when I know I'm putting in some opinionated value I leave behind a comment like: // XXX picking 32gb because it'll probably be 5 years until we have >32gb hard drives. Change this when the world changes!

simonblack 6y ago

Exactly. Just another case of "640K should be enough for anybody". But we have to admit that storage has leapt hugely in size and very rapidly. In 1990 or thereabouts, a common hard disk size was about 30 megabytes. Today I have files that are many times that size, and we think that that is quite normal. A mere five or so years later around 1996, that size had jumped to 4000 megabytes! These days my laptop has 6 million megabytes. We can't blame filesystem planners for lack of foresight. They were blind-sided by progress just like the rest of us were.

mnw21cam 6y ago
I have a single file that is 6 million megabytes in size.
- Thristle 6y ago
  who/what reads that file?
  - mnw21cam 6y ago
    tar
    - AnIdiotOnTheNet 6y ago
      That stretches the definition of "a single file" when it is basically a filesystem itself.
      - bilekas 6y ago
        Not necessarily, it's all relative really. But the point is, the limitations or expected limitations are only relevant until they're not.. Also why not tar a drive for archive if I want :}
  - jcranmer 6y ago
    I haven't managed a 6TB file before, but I did have a >100GB log file I ended up parsing with awk just to be sure that the memory consumption on the consuming process wouldn't go crazy.
  - petschge 6y ago
    Not my file, but I have produced HDF5 files in that size range using simulation codes on supercomputers and paraview can open them and render 3d view as video from it (eventually).
ben509 6y ago
He had already seen exponential growth in storage capacity and could easily extrapolate from that. Moore's law was coined in the '60s, after all. This wasn't a lack of foresight, this was being prudent. If I can't come close to testing an upper limit, it's correct to acknowledge it may degrade in a way I haven't anticipated. Especially since he was writing a GUI which should generally avoid handing users footguns, and which can be updated over time.

jandrese 6y ago

More directly: He only expected his code to be used in Windows NT and made the fatal flaw of thinking that just because he slapped it together in an hour it would be easy for someone to rewrite in the next version of Windows to up the limit. Instead the code sat untouched for decades as the limit slowly became more and more unreasonable. IMHO the real issue is the artificial limit. Instead of straight up disallowing you from creating large filesystems, it should have had a dialog box warning about small files being inefficient but letting you make it anyway.

jackric 6y ago
>Instead the code sat untouched for decades as the limit slowly became more and more unreasonable. My projects likely won't span decades but when I know I'm putting in some opinionated value I leave behind a comment like: // XXX picking 32gb because it'll probably be 5 years until we have >32gb hard drives. Change this when the world changes!