3 ms·

This article seems to be from the perspective of using databases, whereas I expected from the title it would be from someone who created database software. Sin

by btrask 10y ago

This article seems to be from the perspective of using databases, whereas I expected from the title it would be from someone who created database software. Since learning how databases work from the ground up, my understanding of all software has changed a lot. When people argue about a database server being good or bad, or which ORM is best, I can only roll my eyes. Almost all of the popular advice around them is the cargo cult of an echo chamber. It only took me around 6 months of studying, using, and building databases to learn the fundamentals. Database technology may seem boring at first, but it's actually a microcosm of computer science. If you want to learn more, dive in!

otoolep 10y ago
OP here. I did create database software. I spent almost 2 years as part of the core team at InfluxDB, as outlined in the article.
- mpbm 10y ago
  Are there any heuristics for when you should switch from easy data storage, like parsing flat files, to a simple database, then to a complex database? Like, "if it's breaking" seems obvious. And "it depends" doesn't clarify what it depends on and how much. Comparing databases, setting them up, and learning what's what has a cost. When is the benefit worth it?
  - otoolep 10y ago
    Sheer scale is a key reason. The simple, and easy to understand, format of a text file won't work well when you've got a very large number of records to read, write, and locate, quickly.
    - mpbm 10y ago
      Yeah, that seems to fall under the "if it breaks" category. But even then it seems like you could just split the monolithic file into different files with smaller scope. Like all the keys that start with "x" go into the "x's" file. You could probably get a lot of mileage out of just splitting the files by index whenever they reach a certain number of records. It doesn't really matter how many files you have if their naming convention is simple, like if the key starts with "xyz" it's in the "xyz's" file. Intuitively it seems like that solution should scale pretty much as far as you need. As long as some other feature, like simultaneous access, didn't become important you'd be fine. I guess that's basically just using the existing functions in the OS. Hmmm...okay, so that implies a heuristic for investing in a database could be if the product needs high performance in a function the OS doesn't already have. In that case you'd have to implement it yourself anyway, and a database would probably do it better. That would also help constrain the search for an appropriate database since it would make sense to focus on that function. OS's already provide the function of allowing multiple users to access the same information; that's what the file system is. But something like simultaneous access is different. I think they generally just lock the file. So I'd have to write an middle-man function to work with the file in memory when multiple users need to read/write simultaneously. That could easily get complicated fast and clearly justify a database.
      - otoolep 10y ago
        In some ways, that is what a database is. Complex files, each with their own structure.

by btrask 10y ago

otoolep 10y ago

OP here. I did create database software. I spent almost 2 years as part of the core team at InfluxDB, as outlined in the article.

mpbm 10y ago
Are there any heuristics for when you should switch from easy data storage, like parsing flat files, to a simple database, then to a complex database? Like, "if it's breaking" seems obvious. And "it depends" doesn't clarify what it depends on and how much. Comparing databases, setting them up, and learning what's what has a cost. When is the benefit worth it?
- otoolep 10y ago
  Sheer scale is a key reason. The simple, and easy to understand, format of a text file won't work well when you've got a very large number of records to read, write, and locate, quickly.
  - mpbm 10y ago
    Yeah, that seems to fall under the "if it breaks" category. But even then it seems like you could just split the monolithic file into different files with smaller scope. Like all the keys that start with "x" go into the "x's" file. You could probably get a lot of mileage out of just splitting the files by index whenever they reach a certain number of records. It doesn't really matter how many files you have if their naming convention is simple, like if the key starts with "xyz" it's in the "xyz's" file. Intuitively it seems like that solution should scale pretty much as far as you need. As long as some other feature, like simultaneous access, didn't become important you'd be fine. I guess that's basically just using the existing functions in the OS. Hmmm...okay, so that implies a heuristic for investing in a database could be if the product needs high performance in a function the OS doesn't already have. In that case you'd have to implement it yourself anyway, and a database would probably do it better. That would also help constrain the search for an appropriate database since it would make sense to focus on that function. OS's already provide the function of allowing multiple users to access the same information; that's what the file system is. But something like simultaneous access is different. I think they generally just lock the file. So I'd have to write an middle-man function to work with the file in memory when multiple users need to read/write simultaneously. That could easily get complicated fast and clearly justify a database.
    - otoolep 10y ago
      In some ways, that is what a database is. Complex files, each with their own structure.