5 ms·

Developers can do a lot to fix this by simply choosing SQLite to store all the local things. Performing backups of our production apps used to take hours (esp

Developers can do a lot to fix this by simply choosing SQLite to store all the local things. Performing backups of our production apps used to take hours (especially in cheap clouds) because of all the loose files. Today, it takes about 3-5 minutes since there are just a handful of consolidated files to worry about.

bfgoodrich 4y ago
One of the trends on Hacker News right now is the assumption that SQLite fixes everything. It's the Flex Tape of the software world. There is zero reason a process would have more than tiny slowdowns with even millions of files in a folder. Finder has problems if you're trying to look at that folder, for obvious reasons, but it's a bit of a self-own for a backup co to claim that 200,000 files causes their solution to break. That speaks to serious algorithmic issues. DISCLAIMER: This comment will be auto-dead because of moderation choices by dang (e.g. his pernicious need to pander to the anti-science, far-right crowd). This is a badge of honor. Never vouch for my comments.
- meindnoch 4y ago
  Wow. Indeed, practically all of your comments are dead. None of them seem off-topic though. How is this possible? Is there a gang of users who auto-flag all of your comments?
  - yorwba 4y ago
    Account was banned in 2020: https://news.ycombinator.com/item?id=25399750 https://news.ycombinator.com/item?id=25399750
    - dx034 4y ago
      But why would that affect new comments if they’re able to create comments?
      - dang 4y ago
        Banned accounts can post comments; the comments are killed by default.
    - meindnoch 4y ago
      Then how come I see their comments?
      - dang 4y ago
        You turned showdead on in your profile.
- bob1029 4y ago
  > It's the Flex Tape of the software world. I assume you are using Flex Tape as a derogatory comparison here. That said, I do view SQLite as a kind of "fix all" in the software world. It's cheap (free), fast, everywhere and applicable to virtually every type of problem domain. AAA game assets to B2B line of business app storage. It's the most tested and used software on earth. It doesn't fix everything, but it certainly gives you a fighting chance to make it to the next step.
  - username135 4y ago
    A solution to a problem is still a solution regardless of what it is. Might not be the most glorious or flashy solution, but like you mention, getting to the next step is all that really matters, IMO.
  - antisthenes 4y ago
    Once people realized that 90% of things aren't actually #BIGDATA, and SSDs became commonplace, it turned out SQLite is just good enough to solve those 90% things. Things like contact lists or game assets or even web history on individual computers will never grow to multi-TB sizes, hence no reason to over engineer them.
- pjc50 4y ago
  This is a weird fight to pick. It's usually a problem for any software that enumerates a directory for any reason, because that ends up being O(n) and often at least N system calls to e.g. get file information.
- majkinetor 4y ago
  > There is zero reason a process would have more than tiny slowdowns with even millions of files in a folder. What kind of BS is this? Have you ever worked with one such folder? If you did, you would know that almost every app not doing some magic slows down to the point of being unsable (or even not working as described in original post). This is true for both Linux and Windows file systems.
  - didgetmaster 4y ago
    Exactly. Applications that are searching for files that have a certain extension (e.g. *.jpg) must read every directory entry and compare the file extension for that pattern. There is no magic call to the FS that says "Give me all the photos" or "Give me all the .jpg files). The application (including any command processors) have to do all that work manually.
- toast0 4y ago
  If you've got 1M files, your alternatives include: a) 1M files in a single directory b) 1k directories with 1k files in each c) 1k aggregate files with 1k files in each d) 1 aggregate file with 1M files in it I think there's some filesystems that will work with option a, but any filesystem should work ok with the other options. Options c and d will make rsync much faster as you eliminate millions of syscalls.
- dang 4y ago
  Your comments are auto-dead because we banned your account. We told you that when we banned you: https://news.ycombinator.com/item?id=25399750 https://news.ycombinator.com/item?id=25399750. We banned your account because you were egregiously breaking the site guidelines.
gmuslera 4y ago
Or at least using a multilayered approach i.e. all the files starting with A in the A subfolder. You still have to deal with that amount of files, but you don't lose so much time dealing with filesystem/directory operations. And that might be better for accessing individual files outside the app instead of a big binary blob that you may have no clue on its format (or corruption/deletion of that single file). Thats why there are no universal solutions, there are many different use cases.
- bob1029 4y ago
  The multilayered approach is something we also do in one area, but at a certain point you are still playing dangerous games with the filesystem. The most recent innovation for us is to use a lot of smaller SQLite databases, each scoped to a specific customer, session or unit of work. This is still far superior to loose records on disk, but you also get clear separation and reasonable firewalls between system entities.
- aargh_aargh 4y ago
  I think the question should be why don't we have filesystems handle this implicitly using a tree structure behind the scenes. Are we so many layers above our hardware that it simply doesn't occur to us as the right solution?
  - londons_explore 4y ago
    The unix filesystem API doesn't let you "give me a list of all files in this directory matching this pattern". Nor does it let you start iterating through a directory with getdents(), then pause, then come back to iterate the rest of the files later. You have to receive every file name in the order the OS wants to give them to you. That prevents applications really being able to make use of clever storage mechanisms - you can never find the most recent 3 files in a big directory with anything cheaper than a linear scan.
  - dahauns 4y ago
    Well, NTFS works with B-Trees...so nope, it's not a silver bullet either, the bottleneck will just be somewhere else in the storage stack :) (But we have impressively speedy tools like Wiztree or Everything thanks to it.)
NovemberWhiskey 4y ago
Historically, one of the reasons not to create one massive blob with all your stuff in it is containment of the blast radius of any kind of failure. The filesystem at least has (probably) reasonable consistency checking and repair tools, and you're likely to see integrity problems discovered at a file level, rather than experience catastrophic losses.

bfgoodrich 4y ago

One of the trends on Hacker News right now is the assumption that SQLite fixes everything. It's the Flex Tape of the software world. There is zero reason a process would have more than tiny slowdowns with even millions of files in a folder. Finder has problems if you're trying to look at that folder, for obvious reasons, but it's a bit of a self-own for a backup co to claim that 200,000 files causes their solution to break. That speaks to serious algorithmic issues. DISCLAIMER: This comment will be auto-dead because of moderation choices by dang (e.g. his pernicious need to pander to the anti-science, far-right crowd). This is a badge of honor. Never vouch for my comments.

meindnoch 4y ago
Wow. Indeed, practically all of your comments are dead. None of them seem off-topic though. How is this possible? Is there a gang of users who auto-flag all of your comments?
- yorwba 4y ago
  Account was banned in 2020: https://news.ycombinator.com/item?id=25399750 https://news.ycombinator.com/item?id=25399750
  - dx034 4y ago
    But why would that affect new comments if they’re able to create comments?
    - dang 4y ago
      Banned accounts can post comments; the comments are killed by default.
  - meindnoch 4y ago
    Then how come I see their comments?
    - dang 4y ago
      You turned showdead on in your profile.
bob1029 4y ago
> It's the Flex Tape of the software world. I assume you are using Flex Tape as a derogatory comparison here. That said, I do view SQLite as a kind of "fix all" in the software world. It's cheap (free), fast, everywhere and applicable to virtually every type of problem domain. AAA game assets to B2B line of business app storage. It's the most tested and used software on earth. It doesn't fix everything, but it certainly gives you a fighting chance to make it to the next step.
- username135 4y ago
  A solution to a problem is still a solution regardless of what it is. Might not be the most glorious or flashy solution, but like you mention, getting to the next step is all that really matters, IMO.
- antisthenes 4y ago
  Once people realized that 90% of things aren't actually #BIGDATA, and SSDs became commonplace, it turned out SQLite is just good enough to solve those 90% things. Things like contact lists or game assets or even web history on individual computers will never grow to multi-TB sizes, hence no reason to over engineer them.
pjc50 4y ago
This is a weird fight to pick. It's usually a problem for any software that enumerates a directory for any reason, because that ends up being O(n) and often at least N system calls to e.g. get file information.
majkinetor 4y ago
> There is zero reason a process would have more than tiny slowdowns with even millions of files in a folder. What kind of BS is this? Have you ever worked with one such folder? If you did, you would know that almost every app not doing some magic slows down to the point of being unsable (or even not working as described in original post). This is true for both Linux and Windows file systems.
- didgetmaster 4y ago
  Exactly. Applications that are searching for files that have a certain extension (e.g. *.jpg) must read every directory entry and compare the file extension for that pattern. There is no magic call to the FS that says "Give me all the photos" or "Give me all the .jpg files). The application (including any command processors) have to do all that work manually.
toast0 4y ago
If you've got 1M files, your alternatives include: a) 1M files in a single directory b) 1k directories with 1k files in each c) 1k aggregate files with 1k files in each d) 1 aggregate file with 1M files in it I think there's some filesystems that will work with option a, but any filesystem should work ok with the other options. Options c and d will make rsync much faster as you eliminate millions of syscalls.
dang 4y ago
Your comments are auto-dead because we banned your account. We told you that when we banned you: https://news.ycombinator.com/item?id=25399750 https://news.ycombinator.com/item?id=25399750. We banned your account because you were egregiously breaking the site guidelines.

gmuslera 4y ago

Or at least using a multilayered approach i.e. all the files starting with A in the A subfolder. You still have to deal with that amount of files, but you don't lose so much time dealing with filesystem/directory operations. And that might be better for accessing individual files outside the app instead of a big binary blob that you may have no clue on its format (or corruption/deletion of that single file). Thats why there are no universal solutions, there are many different use cases.

bob1029 4y ago
The multilayered approach is something we also do in one area, but at a certain point you are still playing dangerous games with the filesystem. The most recent innovation for us is to use a lot of smaller SQLite databases, each scoped to a specific customer, session or unit of work. This is still far superior to loose records on disk, but you also get clear separation and reasonable firewalls between system entities.
aargh_aargh 4y ago
I think the question should be why don't we have filesystems handle this implicitly using a tree structure behind the scenes. Are we so many layers above our hardware that it simply doesn't occur to us as the right solution?
- londons_explore 4y ago
  The unix filesystem API doesn't let you "give me a list of all files in this directory matching this pattern". Nor does it let you start iterating through a directory with getdents(), then pause, then come back to iterate the rest of the files later. You have to receive every file name in the order the OS wants to give them to you. That prevents applications really being able to make use of clever storage mechanisms - you can never find the most recent 3 files in a big directory with anything cheaper than a linear scan.
- dahauns 4y ago
  Well, NTFS works with B-Trees...so nope, it's not a silver bullet either, the bottleneck will just be somewhere else in the storage stack :) (But we have impressively speedy tools like Wiztree or Everything thanks to it.)

NovemberWhiskey 4y ago

Historically, one of the reasons not to create one massive blob with all your stuff in it is containment of the blast radius of any kind of failure. The filesystem at least has (probably) reasonable consistency checking and repair tools, and you're likely to see integrity problems discovered at a file level, rather than experience catastrophic losses.