8 ms·

Hmm. You are not thinking in business terms. You run a software house: do you want your developers reinventing data storage on each application? Or using a fai

Hmm. You are not thinking in business terms. You run a software house: do you want your developers reinventing data storage on each application? Or using a fairly decent data storage that is RDBMS. Most of the time RDBMS is a very good choice. Think about the tooling, support, hiring knowledgeable people etc. Lets face it RDBMS are good at the very small (single table, replacing a text file) up to the very large. In the RDBMS/SQL world you have everything from sqllite to save as a single file, to Oracle for your Enterprise app.

sklogic 12y ago
Most of the time a plain text file is good enough. Databases are overrated.
- ams6110 12y ago
  This is simply not true, otherwise we would all be using text files. Text files can work in some situations. I disagree that this is "most of the time."
  - sklogic 12y ago
    We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.
    - duaneb 12y ago
      The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?
      - sklogic 12y ago
        Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.
        superuser2 12y ago
        It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?
        sklogic 12y ago
        > It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly? > Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own? Parsing?!? In ORMs and DBMSes?!? C'mon, try to parse me some C++ with your Oracle.
        SamReidHughes 12y ago
        > Parsing?!? In ORMs and DBMSes?!? Yes. Stuff stored in files has to be parsed when you read it out of files.
        sklogic 12y ago
        And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?
        SamReidHughes 12y ago
        The grandparent was talking about the serialization and parsing of the data that you'd be storing in flat files, not parsing of C++ files. Work that you avoid by using a DBMS.
        12y ago
    - bbcbasic 12y ago
      Enterprise applications typically need a data store that offers multiple connections, transactions and atomicity. Even for very simple apps though I like using a DB to store data, and then text files for import/export to other programs if necessary. Once you have a DB you get so much useful functionality without having to code.
      - sklogic 12y ago
        The fact that enterprisey coders got some weird habits does not mean they're doing it all the right way. Outside of the enterprise broken mindset there is very little use for the databases.
    - XorNot 12y ago
      Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite. A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already very good at.
      - sklogic 12y ago
        > Try doing any remotely large data handling. I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer. No databases whatsoever. Only streams, no random access.
        SamReidHughes 12y ago
        What's an example of work you've done that used database systems that could have been better done using flat files?
        sklogic 12y ago
        That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use for them in most of the data analysis tasks - stream storage works much better here. No RDBMSes in the system level (OSes, compilers, linkers, etc.). No RDBMSes in the scientific computing.
        SamReidHughes 12y ago
        Why does iOS ship with SQLite then? I used it for an iPhone app that I wrote. It was more convenient than making my own flat file format. > No RDBMSes in the system level (OSes, compilers, linkers, etc.). OS X has them. Any OS with file search needs some kind of indexing technology. As do mail clients, web browsers, and a bunch of other stuff that works with data. It's a lot easier to add fast indexing after-the-fact when you didn't design up-front to use flat files.
    - SamReidHughes 12y ago
      How do you randomly access variable length fields in a text file and modify them and add new ones?
      - sklogic 12y ago
        Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. Filesystem driver may want to do something like this, some large scale caching proxy, probably. In most cases it's just an optimisation (like a berkeley db records backing text files such as /etc/passwd).
        SamReidHughes 12y ago
        > Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. If by "niche" you mean every interactive website does it.
      - sklogic 12y ago
        Yes, web is a niche thing. Get over it.
    - virmundi 12y ago
      How many programs in a typical Unix installation are complex? Not many. Most of designed around the concept of pipes and filters. They don't need a database; they are transformative maps. Complex systems often require complex data storage. If you have more than one process storing information into a file or set of files, you have to control that type of information. When faced with the option of writing complex, ACID like code or dropping in a PG or MySQL or SQLite option that takes from 1 MB to maybe 128 MB, most will go with using existing tools at the expense of RAM. Is Oracle the solution for every project? Sure, if you're Oracle. However, one shouldn't throw out databases for alternatives simply because they are a bit complex. In fact, I think that raw file IO is more complex because it is not the norm.
      - sklogic 12y ago
        And how many programs need to be complex? Most systems are better off when built from a number of simple components, communicating via text protocols. If multiple entities want to store data in the same place, most likely you're doing something wrong. I very rarely met things that really needed a database. And in most such cases it was not really a RDBMS. For example, a CAD needs a hierarchical database, relational does not fit there at all.
- edraferi 12y ago
  JWZ wrote a great article about choosing between flat files and a database for Netscape Mail: http://www.jwz.org/doc/mailsum.html http://www.jwz.org/doc/mailsum.html In Netscape 4.0, the new team went both C++ and Database happy, threw away the tightly-tuned mail summary files I had designed, and generally screwed the pooch raw. My code had summary files that were on the order of 2% of the size of the folder they were summarizing, and was blazingly fast in all respects. The 4.0 code had an overhead closer to 30% (last time I checked) and was insanely slow, not to mention extremely fragile: their summary files got corrupted all the time.
threeseed 12y ago
I really question your experience with databases. Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different. I do agree that SQLite is an excellent choice for most small applications.
- bbcbasic 12y ago
  "I really question your experience with databases. " I have used SQL Server, Oracle, MySQL, SQLite, MS Access(!), Sybase and ... CouchDB! I have optimized queries in SQL Server and Oracle. Most of them commercially (not CouchDB). "Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different." Spot on. Not sure how it relates to my post.
jbergens 12y ago
But if the storage is a storage library the developers wouldn't be reinventing it every time. And if a new technolgy came along it might be easier to start using it than to say that all applications in an organization must use the same (often relational) db. The latter seems to be very common. And the dba's don't seem to push the use of newer technology much, or look into how much time it would save the developers. Regarding tooling, you are right, it is a bit lacking with the newer database solutions but they seem to be working on it and I think it would help if everyone involved tried to help describe what kind of tooling is needed/wanted. Now a lot of discussions stops with saying the tooling isn't good enough and noone should ever use the new database solutions.
arielby 12y ago
First-Order Logic (and therefore SQL, which is based on it) sucks at describing graphs and highly heterogeneous structures. If you're working with these, you should have a better representation.

sklogic 12y ago

Most of the time a plain text file is good enough. Databases are overrated.

ams6110 12y ago
This is simply not true, otherwise we would all be using text files. Text files can work in some situations. I disagree that this is "most of the time."
- sklogic 12y ago
  We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.
  - duaneb 12y ago
    The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?
    - sklogic 12y ago
      Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.
      - superuser2 12y ago
        It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?
        sklogic 12y ago
        > It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly? > Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own? Parsing?!? In ORMs and DBMSes?!? C'mon, try to parse me some C++ with your Oracle.
        SamReidHughes 12y ago
        > Parsing?!? In ORMs and DBMSes?!? Yes. Stuff stored in files has to be parsed when you read it out of files.
        sklogic 12y ago
        And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?
        SamReidHughes 12y ago
        The grandparent was talking about the serialization and parsing of the data that you'd be storing in flat files, not parsing of C++ files. Work that you avoid by using a DBMS.
        12y ago
  - bbcbasic 12y ago
    Enterprise applications typically need a data store that offers multiple connections, transactions and atomicity. Even for very simple apps though I like using a DB to store data, and then text files for import/export to other programs if necessary. Once you have a DB you get so much useful functionality without having to code.
    - sklogic 12y ago
      The fact that enterprisey coders got some weird habits does not mean they're doing it all the right way. Outside of the enterprise broken mindset there is very little use for the databases.
  - XorNot 12y ago
    Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite. A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already very good at.
    - sklogic 12y ago
      > Try doing any remotely large data handling. I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer. No databases whatsoever. Only streams, no random access.
      - SamReidHughes 12y ago
        What's an example of work you've done that used database systems that could have been better done using flat files?
        sklogic 12y ago
        That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use for them in most of the data analysis tasks - stream storage works much better here. No RDBMSes in the system level (OSes, compilers, linkers, etc.). No RDBMSes in the scientific computing.
        SamReidHughes 12y ago
        Why does iOS ship with SQLite then? I used it for an iPhone app that I wrote. It was more convenient than making my own flat file format. > No RDBMSes in the system level (OSes, compilers, linkers, etc.). OS X has them. Any OS with file search needs some kind of indexing technology. As do mail clients, web browsers, and a bunch of other stuff that works with data. It's a lot easier to add fast indexing after-the-fact when you didn't design up-front to use flat files.
  - SamReidHughes 12y ago
    How do you randomly access variable length fields in a text file and modify them and add new ones?
    - sklogic 12y ago
      Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. Filesystem driver may want to do something like this, some large scale caching proxy, probably. In most cases it's just an optimisation (like a berkeley db records backing text files such as /etc/passwd).
      - SamReidHughes 12y ago
        > Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. If by "niche" you mean every interactive website does it.
    - sklogic 12y ago
      Yes, web is a niche thing. Get over it.
  - virmundi 12y ago
    How many programs in a typical Unix installation are complex? Not many. Most of designed around the concept of pipes and filters. They don't need a database; they are transformative maps. Complex systems often require complex data storage. If you have more than one process storing information into a file or set of files, you have to control that type of information. When faced with the option of writing complex, ACID like code or dropping in a PG or MySQL or SQLite option that takes from 1 MB to maybe 128 MB, most will go with using existing tools at the expense of RAM. Is Oracle the solution for every project? Sure, if you're Oracle. However, one shouldn't throw out databases for alternatives simply because they are a bit complex. In fact, I think that raw file IO is more complex because it is not the norm.
    - sklogic 12y ago
      And how many programs need to be complex? Most systems are better off when built from a number of simple components, communicating via text protocols. If multiple entities want to store data in the same place, most likely you're doing something wrong. I very rarely met things that really needed a database. And in most such cases it was not really a RDBMS. For example, a CAD needs a hierarchical database, relational does not fit there at all.
edraferi 12y ago
JWZ wrote a great article about choosing between flat files and a database for Netscape Mail: http://www.jwz.org/doc/mailsum.html http://www.jwz.org/doc/mailsum.html In Netscape 4.0, the new team went both C++ and Database happy, threw away the tightly-tuned mail summary files I had designed, and generally screwed the pooch raw. My code had summary files that were on the order of 2% of the size of the folder they were summarizing, and was blazingly fast in all respects. The 4.0 code had an overhead closer to 30% (last time I checked) and was insanely slow, not to mention extremely fragile: their summary files got corrupted all the time.

threeseed 12y ago

I really question your experience with databases. Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different. I do agree that SQLite is an excellent choice for most small applications.

bbcbasic 12y ago
"I really question your experience with databases. " I have used SQL Server, Oracle, MySQL, SQLite, MS Access(!), Sybase and ... CouchDB! I have optimized queries in SQL Server and Oracle. Most of them commercially (not CouchDB). "Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different." Spot on. Not sure how it relates to my post.

jbergens 12y ago

But if the storage is a storage library the developers wouldn't be reinventing it every time. And if a new technolgy came along it might be easier to start using it than to say that all applications in an organization must use the same (often relational) db. The latter seems to be very common. And the dba's don't seem to push the use of newer technology much, or look into how much time it would save the developers. Regarding tooling, you are right, it is a bit lacking with the newer database solutions but they seem to be working on it and I think it would help if everyone involved tried to help describe what kind of tooling is needed/wanted. Now a lot of discussions stops with saying the tooling isn't good enough and noone should ever use the new database solutions.

arielby 12y ago

First-Order Logic (and therefore SQL, which is based on it) sucks at describing graphs and highly heterogeneous structures. If you're working with these, you should have a better representation.