3 ms·
i think that topic is larger than the presentation thus far. hopefully memory mapped flash will obviate the need for alot of this but: o you need to make an
by convolvatron 9y ago
i think that topic is larger than the presentation thus far. hopefully memory mapped flash will obviate the need for alot of this but:
o you need to make an on-disk structure (i.e. btree) to let you search your tables (indices)
o that has to be fronted by an in memory cache
o your on-disk structure should maintain consistency even if the power fails and some writes get lost
- often but not always this is facilitated by keeping a separate write-optimized structure called a write-ahead-log
- if you use a WAL, you'll also need to implement the replay mechanisms to get the primary indices back up to date on a failure
o because the disk operations are expensive and high latency you'll need to start managing concurrency explicitly
back ends are alot of work. unfortunately this is a place where the lack of decent concurrency mechanisms in your language and OS interface can really cause alot of headache.
pedagogically, i guess you would start with a completely synchronous non fault tolerant btree? or a maybe just the log, and introduce trees as a read accelerator?
- noam87 9y agoWhat are some good readings / places to start on this topic?
- convolvatron 9y agothats a really good question, unfortunately everything I learned was by working with people who knew more than I did. its pretty sad that alot of systems work is that way. this book looks to be pretty comprehensive, and explicitly discusses on-disk structures, write ahead logging, and recovery. http://infolab.stanford.edu/~ullman/dscb.html relational languages and transactions get alot more playin the academy..i think its because you can make sense of them in some abstract way without getting sucked into involved discussions about caching heuristics and OS write guarentees, scheduling and fussy performance characteristics
- jwhitlark 9y agoI like http://dataintensive.net/ http://dataintensive.net/, though it's a slightly different focus.