4 ms·

Please don't use this for anything significant. Shared global state and synchronous file I/O pretty much guarantee that it will blow up in any non-trivial use.

pizza 12y ago
What demarcates trivial and non-trivial applications? # of machines, guaranteeing xyz?
- couchand 12y ago
  When I say trivial I mean one machine, one user, no errors. The shared global state (see here: https://github.com/typicode/lowdb/blob/481cf43d6b0a52c1cb996644f53763e630ad7da8/src/index.coffee#L59 https://github.com/typicode/lowdb/blob/481cf43d6b0a52c1cb996...) means that attempting to use more than one collection will cause issues but it will silently succeed. The synchronous file I/O will grind your process to a halt. If you are serving a mock API for the purpose of local development you might be fine. If you're trying to build any web application at all, this isn't the right choice.
  - TheZenPsycho 12y ago
    Do you think in theory that it is possible to create a flat file based system that is multi-thread/multiuser safe? My somewhat ignorant inclination is to suggest that maildir fits this description.
    - couchand 12y ago
      Certainly. SQLite is a great example of one. It handles the tradeoff elegantly: reads can be done in parallel but writes have to be sequential. I don't believe maildir would be described as a flat file.
      - TheZenPsycho 12y ago
        Well I suppose then maybe I have the wrong idea of what "flat file database" means. To you anyway. I would not actually describe sqlite as a flat file database system... Since the data cannot be read safely directly from the file, using a text editor, or -- since a flat file could be binary data, any arbitrary program that just knows the format. And "flat" implies there is no special data structure involved. I would think that would not be the case for sqlite given how well it performs, it must use some kind of binary tree or clever indexing system of some sort. You must go via the actual sqlite process. On the other hand maildir, composed of multiple ascii compatible text files, can be read by any email client- provided it follows the protocol for reading and writing of files in the maildir spec.
        couchand 12y ago
        The term "flat file database" conventionally refers to a database that is serialized to a single file, whether it is plain text or a binary. The emphasis here is on the single file aspect: it's really easy to transfer the entire database between machines.
        mappu 12y ago
        MyISAM serializes each table to a single file on disk (well, three files, one each for the definition, data and indexes). Would you consider MySQL to be a flat file database?
        deleted 12y ago
        [deleted]
        couchand 12y ago
        I wouldn't, but I'm not really an expert here. If every table is three files, that implies a non-trivial database would be dozens of files. I suppose the other use of "flat file" I've heard is a flat directory with no sub-directories, but I've never heard that applied to a "flat file database".
        12y ago
    - xxs 12y ago
      There is zero issue to have a flat file database that's all multi. MySQL InnoDB, the transactional one engine, can be a single file database. [Small exception - part of the schema is stored on two separates files for easier access, there won't be a big deal to store them in the big file. Transaction log is also not in the same file for obvious reasons] If you want a full serialization of the memory to the 'flat' file - then no. It just makes no sense. Yet, a single file databases that's multi-user, mulch-threaded and mult-transaction is all viable.
- efsavage 12y ago
  How much someone is paying you to write it and how much it costs to replace are good criteria to start with.
hliyan 12y ago
Perhaps this could benefit from ArangoDB's (1) architecture -- using memory mapped files in append mode and writing to file asynchronously with a new revision number for each write. Or conversely, ArangoDB could benefit from a simple API like this one. (1) https://www.arangodb.org https://www.arangodb.org

pizza 12y ago

What demarcates trivial and non-trivial applications? # of machines, guaranteeing xyz?

couchand 12y ago
When I say trivial I mean one machine, one user, no errors. The shared global state (see here: https://github.com/typicode/lowdb/blob/481cf43d6b0a52c1cb996644f53763e630ad7da8/src/index.coffee#L59 https://github.com/typicode/lowdb/blob/481cf43d6b0a52c1cb996...) means that attempting to use more than one collection will cause issues but it will silently succeed. The synchronous file I/O will grind your process to a halt. If you are serving a mock API for the purpose of local development you might be fine. If you're trying to build any web application at all, this isn't the right choice.
- TheZenPsycho 12y ago
  Do you think in theory that it is possible to create a flat file based system that is multi-thread/multiuser safe? My somewhat ignorant inclination is to suggest that maildir fits this description.
  - couchand 12y ago
    Certainly. SQLite is a great example of one. It handles the tradeoff elegantly: reads can be done in parallel but writes have to be sequential. I don't believe maildir would be described as a flat file.
    - TheZenPsycho 12y ago
      Well I suppose then maybe I have the wrong idea of what "flat file database" means. To you anyway. I would not actually describe sqlite as a flat file database system... Since the data cannot be read safely directly from the file, using a text editor, or -- since a flat file could be binary data, any arbitrary program that just knows the format. And "flat" implies there is no special data structure involved. I would think that would not be the case for sqlite given how well it performs, it must use some kind of binary tree or clever indexing system of some sort. You must go via the actual sqlite process. On the other hand maildir, composed of multiple ascii compatible text files, can be read by any email client- provided it follows the protocol for reading and writing of files in the maildir spec.
      - couchand 12y ago
        The term "flat file database" conventionally refers to a database that is serialized to a single file, whether it is plain text or a binary. The emphasis here is on the single file aspect: it's really easy to transfer the entire database between machines.
        mappu 12y ago
        MyISAM serializes each table to a single file on disk (well, three files, one each for the definition, data and indexes). Would you consider MySQL to be a flat file database?
        deleted 12y ago
        [deleted]
        couchand 12y ago
        I wouldn't, but I'm not really an expert here. If every table is three files, that implies a non-trivial database would be dozens of files. I suppose the other use of "flat file" I've heard is a flat directory with no sub-directories, but I've never heard that applied to a "flat file database".
        12y ago
  - xxs 12y ago
    There is zero issue to have a flat file database that's all multi. MySQL InnoDB, the transactional one engine, can be a single file database. [Small exception - part of the schema is stored on two separates files for easier access, there won't be a big deal to store them in the big file. Transaction log is also not in the same file for obvious reasons] If you want a full serialization of the memory to the 'flat' file - then no. It just makes no sense. Yet, a single file databases that's multi-user, mulch-threaded and mult-transaction is all viable.
efsavage 12y ago
How much someone is paying you to write it and how much it costs to replace are good criteria to start with.

hliyan 12y ago

Perhaps this could benefit from ArangoDB's (1) architecture -- using memory mapped files in append mode and writing to file asynchronously with a new revision number for each write. Or conversely, ArangoDB could benefit from a simple API like this one. (1) https://www.arangodb.org https://www.arangodb.org