4 ms·
FleetDB
- va_coder 17y agoIt looks like the first schema-free database written in Clojure has arrived
- mmcgrana 17y agoClojure is very compelling for implementing a main-memory database; if you use the persistent data structures and concurrency primitives in the right way, you get atomicity, consistency, and isolation essentially for free.
- pradocchia 17y agoDo the index maps contain pointers back to the base records? The examples I saw on the implementation page showed a nested copy of the data; I assume this is just a projection?
- mmcgrana 17y agoThat's a good question; indeed the index maps just contain pointers back to the records.
- netghost 17y agoLooks neat, certainly something to watch.
- mhendric 17y agocongrats mark!
- patrickgzill 17y agoWhat happens if/when the database runs out of RAM? Does the OS then start swapping to disk?
- mmcgrana 17y agoYes; the current version of FleetDB is designed for datasets that fit in RAM.
- AnneTheAgile 17y agoIs it not true that a major purpose of a DB is to handle datasets the are too big for RAM? If the items fit in RAM, isn't speed ~1/1000th (ie ratio of RAM access to HD access time) the issue it was?
- DrJokepu 17y agoThat really depends on the query and the data structures used in the database. You can query large amounts of data on the hard drive very quickly if you store your data and indexes in data structures manually tuned for the problem. Modern operating systems / databases will try to use up your available RAM anyway to cache hard drive so keeping the whole database in RAM (if it fits into the RAM) won't help that much anyway.
- mahmud 17y agoIf everything fits into RAM, you can use my proprietary ArrayDB storage system which has bindings for every non-toy programming language known to man. In fact, if you're a programmer, you already know the API for ArrayDB. No downloads, no installation, minimal investment and training required!
- z8000 17y agoThis looks rather neat! I like to think that I know enough Clojure to decipher that the database and metadata is locked during compaction. Is this true? See http://github.com/mmcgrana/fleetdb/blob/master/src/clj/fleetdb/embedded.clj#L99 http://github.com/mmcgrana/fleetdb/blob/master/src/clj/fleet... I wrote something called LogStore based on the general notion of log-structured data (then learned about the work of Ousterhout et al. in the 90s). For what it's worth, I avoided the need to lock the metadata and the database during compaction, allowing the log to grow during compaction. Instead of working through the locked metadata (offset per entry) and rewriting the entries to a new file, the compactor works from the end of the log file to the start of the log file. Each record descriptor is actually written after the record data to help the compactor easily skip over older versions of records it has processed already during the compaction iteration. Once the compactor reaches the start of the file, it checks to see if the file grew (more appends, new data) since the beginning of the compaction, and starts over from the new EOF but stopping at the previous EOF. This repeats until the compactor fully catches up. Then the database is temporarily locked, the new metadata is swapped in, and the existing log file is overwritten with the compacted log file.
- mmcgrana 17y agoThe database and metadata are not locked during compaction, at least to the extent that you seem to think they are. When a compaction is requested, the compact query enters the write queue. When the query reaches the head of queue, an asynchronous compaction is started on a snapshot of the database at that time, to a file in /tmp. Also, a buffer is added to the database metadata in which subsequent queries will be stored. After this essentially instant operation, the database can proceed to process write queries as normal. While the compaction is ongoing, write queries are appended to the buffer mentioned above. When the compaction thread that was spawned earlier finishes writing its snapshot, it inserts into the write queue a request to finalize compaction. When this request reaches the head of the queue, it appends all the buffered queries to the compacted database file. Finally, it swaps the compacted file in /tmp to the regular database path. So writes are blocked once for an instant and once for however long it takes to write those buffered queries, which shouldn't be long either. Note that reads are never blocked by compaction; indeed they are never blocked in general.
- admn_is_traitor 17y agoStopped reading at "optimized for agile development."..
- tedunangst 17y agoPerhaps a little editorializing of the headline is appropriate when it's one word that doesn't tell me a single thing about the link?
- deleted 17y ago[deleted]
- DrJokepu 17y agoSo what does this offer over a key-value store (such as BerkeleyDB) other than the JSON query/object abstraction layer?