6 ms·

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text f

by sklogic 12y ago

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

duaneb 12y ago
The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?
- sklogic 12y ago
  Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.
  - superuser2 12y ago
    It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?
    - sklogic 12y ago
      > It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly? > Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own? Parsing?!? In ORMs and DBMSes?!? C'mon, try to parse me some C++ with your Oracle.
      - SamReidHughes 12y ago
        > Parsing?!? In ORMs and DBMSes?!? Yes. Stuff stored in files has to be parsed when you read it out of files.
        sklogic 12y ago
        And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?
        SamReidHughes 12y ago
        The grandparent was talking about the serialization and parsing of the data that you'd be storing in flat files, not parsing of C++ files. Work that you avoid by using a DBMS.
        sklogic 12y ago
        Parsing text files is not a problem at all. Never been a problem. And there are far too many benefits of having flat text streams vs. some inaccessible stupid binary ad hoc formats hidden inside some enterprisey RDBMS guts.
      - duaneb 12y ago
        > Unix is a pretty general purpose thing. Yea, until you want to query a normalized data set. Then unix is pretty miserable, because text is very ill-suited for indices and rapid and/or random access.
        sklogic 12y ago
        Turns out, you need to do such things pretty rarely. Not "most of the time", as RDMBS proponents claim.
      - superuser2 12y ago
        >Parsing?!? In ORMs and DBMSes?!? What exactly do you think a database does? Provides a way to durably store tabular data on disk and read it back. Without a database, I have to make that myself by writing serialization routines to put my data structures on disk and parsing routines to go from "stream of characters" to my data structures. DMBS places some limits on your data structures but does all that for you.
  - eropple 12y ago
    Maybe you can replace a document database with the file system (though reinventing indexes for reasonable performance--which in so many use cases becomes remarkably important past a few hundreds of thousands of records--will be fun for you), but modeling relational concepts on the file system is both hard and time-consuming. And I'm comfortable saying that most nontrivial business applications are fundamentally relational.
    - sklogic 12y ago
      > And I'm comfortable saying that most nontrivial business applications are fundamentally relational. Now try to assess, how many applications are supposed to be "nontrivial business applications". It's quite a niche thing, and that's where all this DBMS stuff belongs to. Please do not impose this crap on the rest of the world, we don't need your CRUD.
      - eropple 12y ago
        Why are you so randomly hostile about this? Your tone and attitude are super shitty throughout this thread. Chill. As for CRUD--I'm a platform and devops engineer. I'm not a web developer (except incidentally). But I do often write relational things--like, say, "I have this node, running this application, using this artifact from that data store"--that are best modeled by a relational data store. They're otherwise trivial applications, but reinventing the relational data store would take orders of magnitude longer than writing the application against SQLite or Postgres. How do you propose to model relations across the file system in a way that doesn't make it the tail wagging the dog? While I'm thinking about it, I'm pretty comfortable going further than "nontrivial business applications" to "a majority of networked applications". The world is relational. Modeling it relationally makes a lot of sense for a lot of use cases.
        sklogic 12y ago
        No, it's RDBMS proponents who are hostile. Their attitude is "you will be assimilated, resistance is futile, we already won, so shut up and obey". This cannot and will not be tolerated. I was pressured many times into giving up doing things the right way and adopting RDBMSes, and that was not pleasant at all. In some cases someone else who picked up that projects tried to move to RDBMSes (from hierarchical or document-oriented storages used before) and failed miserably. So, again, I never came across anything that would need a relational storage. Not even when I was doing heavily distributed HPC (which is as "networked" as it gets). Yes, of course I did a lot of things that are relational naturally - but all of them were in Datalog, with no storage required. Btw., wtf is "devops"? I've heard this a number of times recently, but nobody could explain the meaning.
bbcbasic 12y ago
Enterprise applications typically need a data store that offers multiple connections, transactions and atomicity. Even for very simple apps though I like using a DB to store data, and then text files for import/export to other programs if necessary. Once you have a DB you get so much useful functionality without having to code.
- sklogic 12y ago
  The fact that enterprisey coders got some weird habits does not mean they're doing it all the right way. Outside of the enterprise broken mindset there is very little use for the databases.
XorNot 12y ago
Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite. A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already very good at.
- sklogic 12y ago
  > Try doing any remotely large data handling. I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer. No databases whatsoever. Only streams, no random access.
  - SamReidHughes 12y ago
    What's an example of work you've done that used database systems that could have been better done using flat files?
    - sklogic 12y ago
      That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use for them in most of the data analysis tasks - stream storage works much better here. No RDBMSes in the system level (OSes, compilers, linkers, etc.). No RDBMSes in the scientific computing.
      - SamReidHughes 12y ago
        Why does iOS ship with SQLite then? I used it for an iPhone app that I wrote. It was more convenient than making my own flat file format. > No RDBMSes in the system level (OSes, compilers, linkers, etc.). OS X has them. Any OS with file search needs some kind of indexing technology. As do mail clients, web browsers, and a bunch of other stuff that works with data. It's a lot easier to add fast indexing after-the-fact when you didn't design up-front to use flat files.
SamReidHughes 12y ago
How do you randomly access variable length fields in a text file and modify them and add new ones?
- sklogic 12y ago
  Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. Filesystem driver may want to do something like this, some large scale caching proxy, probably. In most cases it's just an optimisation (like a berkeley db records backing text files such as /etc/passwd).
  - SamReidHughes 12y ago
    > Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. If by "niche" you mean every interactive website does it.
- sklogic 12y ago
  Yes, web is a niche thing. Get over it.
virmundi 12y ago
How many programs in a typical Unix installation are complex? Not many. Most of designed around the concept of pipes and filters. They don't need a database; they are transformative maps. Complex systems often require complex data storage. If you have more than one process storing information into a file or set of files, you have to control that type of information. When faced with the option of writing complex, ACID like code or dropping in a PG or MySQL or SQLite option that takes from 1 MB to maybe 128 MB, most will go with using existing tools at the expense of RAM. Is Oracle the solution for every project? Sure, if you're Oracle. However, one shouldn't throw out databases for alternatives simply because they are a bit complex. In fact, I think that raw file IO is more complex because it is not the norm.
- sklogic 12y ago
  And how many programs need to be complex? Most systems are better off when built from a number of simple components, communicating via text protocols. If multiple entities want to store data in the same place, most likely you're doing something wrong. I very rarely met things that really needed a database. And in most such cases it was not really a RDBMS. For example, a CAD needs a hierarchical database, relational does not fit there at all.

by sklogic 12y ago

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

duaneb 12y ago

The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?

sklogic 12y ago
Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.
- superuser2 12y ago
  It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?
  - sklogic 12y ago
    > It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly? > Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own? Parsing?!? In ORMs and DBMSes?!? C'mon, try to parse me some C++ with your Oracle.
    - SamReidHughes 12y ago
      > Parsing?!? In ORMs and DBMSes?!? Yes. Stuff stored in files has to be parsed when you read it out of files.
      - sklogic 12y ago
        And how exactly RDBMS crap could help with parsing? Mind parsing a C++ source file with your RDBMS?
        SamReidHughes 12y ago
        The grandparent was talking about the serialization and parsing of the data that you'd be storing in flat files, not parsing of C++ files. Work that you avoid by using a DBMS.
        sklogic 12y ago
        Parsing text files is not a problem at all. Never been a problem. And there are far too many benefits of having flat text streams vs. some inaccessible stupid binary ad hoc formats hidden inside some enterprisey RDBMS guts.
    - duaneb 12y ago
      > Unix is a pretty general purpose thing. Yea, until you want to query a normalized data set. Then unix is pretty miserable, because text is very ill-suited for indices and rapid and/or random access.
      - sklogic 12y ago
        Turns out, you need to do such things pretty rarely. Not "most of the time", as RDMBS proponents claim.
    - superuser2 12y ago
      >Parsing?!? In ORMs and DBMSes?!? What exactly do you think a database does? Provides a way to durably store tabular data on disk and read it back. Without a database, I have to make that myself by writing serialization routines to put my data structures on disk and parsing routines to go from "stream of characters" to my data structures. DMBS places some limits on your data structures but does all that for you.
- eropple 12y ago
  Maybe you can replace a document database with the file system (though reinventing indexes for reasonable performance--which in so many use cases becomes remarkably important past a few hundreds of thousands of records--will be fun for you), but modeling relational concepts on the file system is both hard and time-consuming. And I'm comfortable saying that most nontrivial business applications are fundamentally relational.
  - sklogic 12y ago
    > And I'm comfortable saying that most nontrivial business applications are fundamentally relational. Now try to assess, how many applications are supposed to be "nontrivial business applications". It's quite a niche thing, and that's where all this DBMS stuff belongs to. Please do not impose this crap on the rest of the world, we don't need your CRUD.
    - eropple 12y ago
      Why are you so randomly hostile about this? Your tone and attitude are super shitty throughout this thread. Chill. As for CRUD--I'm a platform and devops engineer. I'm not a web developer (except incidentally). But I do often write relational things--like, say, "I have this node, running this application, using this artifact from that data store"--that are best modeled by a relational data store. They're otherwise trivial applications, but reinventing the relational data store would take orders of magnitude longer than writing the application against SQLite or Postgres. How do you propose to model relations across the file system in a way that doesn't make it the tail wagging the dog? While I'm thinking about it, I'm pretty comfortable going further than "nontrivial business applications" to "a majority of networked applications". The world is relational. Modeling it relationally makes a lot of sense for a lot of use cases.
      - sklogic 12y ago
        No, it's RDBMS proponents who are hostile. Their attitude is "you will be assimilated, resistance is futile, we already won, so shut up and obey". This cannot and will not be tolerated. I was pressured many times into giving up doing things the right way and adopting RDBMSes, and that was not pleasant at all. In some cases someone else who picked up that projects tried to move to RDBMSes (from hierarchical or document-oriented storages used before) and failed miserably. So, again, I never came across anything that would need a relational storage. Not even when I was doing heavily distributed HPC (which is as "networked" as it gets). Yes, of course I did a lot of things that are relational naturally - but all of them were in Datalog, with no storage required. Btw., wtf is "devops"? I've heard this a number of times recently, but nobody could explain the meaning.

bbcbasic 12y ago

Enterprise applications typically need a data store that offers multiple connections, transactions and atomicity. Even for very simple apps though I like using a DB to store data, and then text files for import/export to other programs if necessary. Once you have a DB you get so much useful functionality without having to code.

sklogic 12y ago
The fact that enterprisey coders got some weird habits does not mean they're doing it all the right way. Outside of the enterprise broken mindset there is very little use for the databases.

XorNot 12y ago

Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite. A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already very good at.

sklogic 12y ago
> Try doing any remotely large data handling. I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer. No databases whatsoever. Only streams, no random access.
- SamReidHughes 12y ago
  What's an example of work you've done that used database systems that could have been better done using flat files?
  - sklogic 12y ago
    That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use for them in most of the data analysis tasks - stream storage works much better here. No RDBMSes in the system level (OSes, compilers, linkers, etc.). No RDBMSes in the scientific computing.
    - SamReidHughes 12y ago
      Why does iOS ship with SQLite then? I used it for an iPhone app that I wrote. It was more convenient than making my own flat file format. > No RDBMSes in the system level (OSes, compilers, linkers, etc.). OS X has them. Any OS with file search needs some kind of indexing technology. As do mail clients, web browsers, and a bunch of other stuff that works with data. It's a lot easier to add fast indexing after-the-fact when you didn't design up-front to use flat files.

SamReidHughes 12y ago

How do you randomly access variable length fields in a text file and modify them and add new ones?

sklogic 12y ago
Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. Filesystem driver may want to do something like this, some large scale caching proxy, probably. In most cases it's just an optimisation (like a berkeley db records backing text files such as /etc/passwd).
- SamReidHughes 12y ago
  > Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. If by "niche" you mean every interactive website does it.
sklogic 12y ago
Yes, web is a niche thing. Get over it.

virmundi 12y ago

How many programs in a typical Unix installation are complex? Not many. Most of designed around the concept of pipes and filters. They don't need a database; they are transformative maps. Complex systems often require complex data storage. If you have more than one process storing information into a file or set of files, you have to control that type of information. When faced with the option of writing complex, ACID like code or dropping in a PG or MySQL or SQLite option that takes from 1 MB to maybe 128 MB, most will go with using existing tools at the expense of RAM. Is Oracle the solution for every project? Sure, if you're Oracle. However, one shouldn't throw out databases for alternatives simply because they are a bit complex. In fact, I think that raw file IO is more complex because it is not the norm.

sklogic 12y ago
And how many programs need to be complex? Most systems are better off when built from a number of simple components, communicating via text protocols. If multiple entities want to store data in the same place, most likely you're doing something wrong. I very rarely met things that really needed a database. And in most such cases it was not really a RDBMS. For example, a CAD needs a hierarchical database, relational does not fit there at all.