4 ms·

Are there any heuristics for when you should switch from easy data storage, like parsing flat files, to a simple database, then to a complex database? Like, "if

by mpbm 10y ago

Are there any heuristics for when you should switch from easy data storage, like parsing flat files, to a simple database, then to a complex database? Like, "if it's breaking" seems obvious. And "it depends" doesn't clarify what it depends on and how much. Comparing databases, setting them up, and learning what's what has a cost. When is the benefit worth it?

otoolep 10y ago
Sheer scale is a key reason. The simple, and easy to understand, format of a text file won't work well when you've got a very large number of records to read, write, and locate, quickly.
- mpbm 10y ago
  Yeah, that seems to fall under the "if it breaks" category. But even then it seems like you could just split the monolithic file into different files with smaller scope. Like all the keys that start with "x" go into the "x's" file. You could probably get a lot of mileage out of just splitting the files by index whenever they reach a certain number of records. It doesn't really matter how many files you have if their naming convention is simple, like if the key starts with "xyz" it's in the "xyz's" file. Intuitively it seems like that solution should scale pretty much as far as you need. As long as some other feature, like simultaneous access, didn't become important you'd be fine. I guess that's basically just using the existing functions in the OS. Hmmm...okay, so that implies a heuristic for investing in a database could be if the product needs high performance in a function the OS doesn't already have. In that case you'd have to implement it yourself anyway, and a database would probably do it better. That would also help constrain the search for an appropriate database since it would make sense to focus on that function. OS's already provide the function of allowing multiple users to access the same information; that's what the file system is. But something like simultaneous access is different. I think they generally just lock the file. So I'd have to write an middle-man function to work with the file in memory when multiple users need to read/write simultaneously. That could easily get complicated fast and clearly justify a database.
  - otoolep 10y ago
    In some ways, that is what a database is. Complex files, each with their own structure.

by mpbm 10y ago

otoolep 10y ago

Sheer scale is a key reason. The simple, and easy to understand, format of a text file won't work well when you've got a very large number of records to read, write, and locate, quickly.

mpbm 10y ago
Yeah, that seems to fall under the "if it breaks" category. But even then it seems like you could just split the monolithic file into different files with smaller scope. Like all the keys that start with "x" go into the "x's" file. You could probably get a lot of mileage out of just splitting the files by index whenever they reach a certain number of records. It doesn't really matter how many files you have if their naming convention is simple, like if the key starts with "xyz" it's in the "xyz's" file. Intuitively it seems like that solution should scale pretty much as far as you need. As long as some other feature, like simultaneous access, didn't become important you'd be fine. I guess that's basically just using the existing functions in the OS. Hmmm...okay, so that implies a heuristic for investing in a database could be if the product needs high performance in a function the OS doesn't already have. In that case you'd have to implement it yourself anyway, and a database would probably do it better. That would also help constrain the search for an appropriate database since it would make sense to focus on that function. OS's already provide the function of allowing multiple users to access the same information; that's what the file system is. But something like simultaneous access is different. I think they generally just lock the file. So I'd have to write an middle-man function to work with the file in memory when multiple users need to read/write simultaneously. That could easily get complicated fast and clearly justify a database.
- otoolep 10y ago
  In some ways, that is what a database is. Complex files, each with their own structure.