4 ms·
Hmm, I thought this pattern was really common? That is, appending everything to a file and reading back from the file when there's a reboot. I constantly use it
by michaelmcmillan 8y ago
Hmm, I thought this pattern was really common? That is, appending everything to a file and reading back from the file when there's a reboot. I constantly use it when a database (or redis for that matter) is simply overkill for my use case.
Here's a 34 line implementation I use on a node production system. It writes struct-like JavaScript objects that represents events to disk. When reading it back I do a fold (or .reduce) to build the state.
And yes –– it could be way smarter (writing to memory and disk), but YAGNI has been working out pretty well so far.
class EventStore {
constructor(file) {
this.file = file;
this.cache = null;
}
async appendEvent(event) {
// Purge the cache for entries since we mutated the store.
this.cache = null;
return new Promise((resolve, reject) => {
createWriteStream(this.file, { flags: 'a' })
.on('error', reject)
.on('close', resolve)
.end(`${JSON.stringify(event)}\n`);
});
}
async readEvents() {
if (this.cache !== null) {
return this.cache;
}
try {
const data = await readFile(this.file, 'utf-8');
const lines = data.split('\n').filter(line => line);
const events = lines.map(line => JSON.parse(line));
this.cache = events;
return events;
} catch (error) {
return [];
}
}
}
- jrockway 8y agoIt's pretty common. All the wonderful proprietary file formats from the 90s (and probably before, and probably after) basically boil down to writing raw C structs to disk. You can try it yourself... mmap a file, memcpy some structs there, do the reverse ... and enjoy! (Obviously depending on the memory layout of one C compiler on one architecture does not make for portable files. But that was never a design goal of this system.)
- beat 8y agoI spent part of the mid-1990s hacking a proprietary data streaming protocol written in lexx and yacc. It was what we had back then.
- lioeters 8y agoI've been using (and developing a fork of) NeDB [0] that does exactly what you describe: an in-memory database with append-only logs of changes for file-system persistence. On startup, and optionally at regular intervals, it "compacts" the database by reducing all events to a single JSON string. The README links to an article by @antirez, Redis Persistence Demystified [1]. It's been educational studying how it works. [0] https://github.com/louischatriot/nedb#persistence https://github.com/louischatriot/nedb#persistence [1] http://oldblog.antirez.com/post/redis-persistence-demystified.html http://oldblog.antirez.com/post/redis-persistence-demystifie...
- elif 8y ago> Hmm, I thought this pattern was really common? while true, redis predates node, so your example does not support your use of past tense.
- michaelmcmillan 8y agoA pattern is not specific to a language. Read again.