11 ms·

Paul Graham inspired the creation of Redis

411 points by oneowl 8y ago

techie128 8y ago
When the application would be restarted, reading back the log would recreate the in memory data structures. I thought that it was cool, and that databases themselves could be that way instead of using the programming language data structures without a networked API. That is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.
- kijin 8y ago
  Except that most databases don't store their data in anything resembling "programming language data structures". You get tables, rows, and columns (or maybe a bit of JSON if you're lucky) instead of native integers, strings, lists, sets, and dictionaries.
  - threeseed 8y ago
    Object and document databases have been around for decades. Likewise ORMs which allow for higher order types etc. have been around since WebObjects i.e. also decades.
    - kijin 8y ago
      The primary purpose of an ORM is to overcome the "impedance mismatch" between relational databases and programming language data structures. There's no need for an ORM if you can store your data structures directly in the database.
      - TeMPOraL 8y ago
        If this is a primary purpose of an ORM, then I wish I knew of one that isn't utterly failing at that. One thing I've learned about this "impedance mismatch" is that it isn't a syntax thing, it's a fundamental difference in the way of viewing the world. The way you store data about the world is different from the way you model that world dynamically, with objects. I find it safer to always split out the "business model" from the storage layer, so that those different views don't interfere - and once you do that, you may as well implement the storage layer in a relational way.
        icebraining 8y ago
        .. and the code you implement to connect that business layer to that storage layer is an ORM. The idea that the ORM forces the storage layer to a particular representation of the business layer is only true if they implement the ActiveRecord pattern, which isn't universal.
      - neokantian 8y ago
        > ... if you can store your data structures directly in the database. ABSTRACT. Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). It provides a means of describing data with its natural structure only—that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other. E.F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM 13, 6 (June 1970), 377-387. You propose to reintroduce a problem that they absolutely wanted to get rid off 40 years ago. Just imagine that you first have to figure out how to painstakingly parse serialized Python dictionaries before you can access the data in another program written in e.g. Rust. It clearly amounts to UNSOLVING a problem that is now SOLVED ALREADY.
        kijin 8y ago
        Well, Redis allows me to store a JavaScript array, add some more with a Ruby client, remove some items with a PHP client, and finally read it back as a Python list just fine. What's the problem that has been unsolved? :)
        neokantian 8y ago
        https://msgpack.org https://msgpack.org is absolutely fine. I wasn't criticizing msgpack. You can also serialize the bytes of an array of C structs, if you see what I mean.
      - deleted 8y ago
        [deleted]
      - mamcx 8y ago
        That is an angle. Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. Exist 2 main reasons for the "impedance mismatch": - Paradigms. 2 different paradigms will be at odds. Example: Functional and OO. This is ok. - Limitations: The relational model is absolutely superior and more expressive at manipulate data than OO/Functional. You need A LOT of machinery to recover that power. This is not ok. However, this not change the fact that OO is ok.Similar how a KV store is fine, but certainly, a RDBMS store is much more capable.
        heavenlyblue 8y ago
        How is relational more powerful than OO/Functional if you can’t define an infinite dataset with it?
        mamcx 8y ago
        You can't? Most people only see the relational model as is inside a RDBMS. That is how judge the OO for what you can do on Mongo. RDBMS have some weird and well considered restrictions for their case. But read a little about the relational model, and none of it depend on SQL or say anything about how is the storage. P.D: Is important to note for where is MORE powerful. Remember that the creator of Pascal say: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures... Algorithms + Data Structures = Programs You can say OO/Functional lean more to the "Algorithms" side but RM to the "Data Structures". OO/Functional not say much how operate on data, most is a exercise for the reader. Instead, RM give a clear answer and defined operations for that. You need to "spice up" things to make the one be more useful to the other part of the equation. RM, too pure, certainly is incredible limited, not even can print "hello world!", but that is just too give a solution about how transform data...
        notduncansmith 8y ago
        > Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. This is why I love Clojure (and a particular style of Javascript). Destructuring and a good library of object/array manipulation functions make an environment well-suited to transforming data structures (which is precisely what I want to do in most programs I write), and I find I do not need an ORM where this type of data-focused programming is supported.
  - jinjin2 8y ago
    It does if you use an object database like Realm.
- elvinyung 8y ago
  I think you're slightly missing the point -- for me, a certain unique "awesomeness" lies in specifically being able to literally pipe stdout back to stdin and get back the exact same data structures.
  - snr 8y ago
    Can be easily done by printing out the stdin to stdout in the first place?
    - elvinyung 8y ago
      ok, without doing that :p
- Illniyar 8y ago
  Relational databases (except maybe MemQL) treat the file system as the source of truth. And usually the file system is bigger then the memory so they need to constantly update their cache with more relevant data. Redis loads everything to memory. And doesn't keep the structure in the file system, only the log, recreating it from log+snapshot.
threeseed 8y ago
Strange title. Does merely recalling a language pattern really count as inspiring ?
- tomhoward 8y ago
  In the tweet, the author of Redis says "[Paul Graham] ... inspired the creation of Redis". So it probably counts!
  - dang 8y ago
    Ok, we put that in the title above for disambiguity.
viach 8y ago
Probably the next logical idea would be to apply to YC with Redis?
stretchwithme 8y ago
Doesn't AWS's Aurora database sort of work that way too?
- bpicolo 8y ago
  The pattern in general is used by more or less every database in the form of write-ahead logging. https://en.wikipedia.org/wiki/Write-ahead_logging https://en.wikipedia.org/wiki/Write-ahead_logging
konamicode 8y ago
I'd be keen to see a simple example of this pattern in Lisp (or another language). Does anybody have a good link?
- pavlov 8y ago
  I use it by default for new Node.js projects (most of which are experiments but some end up in production, and have been running without problems for years). It's a simple pattern to implement. To make it a bit easier to use repeatedly, I've got a small helper class called JournaledCollection. You pass it serialize+deserialize callbacks for your item type, and it takes care of persistence in event logs. For a while I was thinking about releasing my helpers as a project called LAUF, short for "Lame-Ass Un-Framework". Then one could say: "Most of my projects are LAUFable, I don't need anything more serious." (Awful dad jokes are a solid reason to publish open source, right?) Never got around to it though, but if you're interested, I could put together an example.
  - chrisweekly 8y ago
    I like the name -- and it sounds worth sharing!
- michaelmcmillan 8y ago
  Here's one I'm currently using in node (wrote it as another comment): https://news.ycombinator.com/item?id=19499799 https://news.ycombinator.com/item?id=19499799
- stevelosh 8y ago
  sjl at alephnull in ~/Desktop ><((°> sbcl [SBCL] CL-USER> (defparameter *name* "World") *NAME* [SBCL] CL-USER> (defun foo () (format t "Hello, ~A~%" *name*)) FOO [SBCL] CL-USER> (sb-ext:save-lisp-and-die "session.core") sjl at alephnull in ~/Desktop ><((°> sbcl --core session.core [SBCL] CL-USER> (foo) Hello, World NIL
jokoon 8y ago
Is this really a brilliant idea though? Everybody knows RAM became cheaper cheaper, while mechanical disk can't be made faster, and SSD have reliability limits. It seems entirely logical to expect databases to work on RAM first and then commit on disk for a large performance improvement.
- antirez 8y ago
  When I started Redis people were like FTW data in memory?!
  - jokoon 8y ago
    I was talking to a database teacher, and I tried arguing about ACID on an in memory database, arguing that there was a minimal window for database corruption in a transaction system, since the worst case scenario would seem to be a loss of very few transaction. He was not really listening to what I was saying or my arguments, because a system like redis seems like a more than acceptable compromise. It still seems a few people are reluctant to an in-memory database.
    - benmmurphy 8y ago
      VoltDB is a probably a better example of an in-memory database with ACID semantics. Redis usually won't be deployed in a way where it fsyncs after each write operation because it runs too slow. You need to do some tricks like batching fsyncs to get decent performance and I don't think Redis has support for this. However, SSD fsync performance seems very high. I remember benchmarking an EC2 i3 instance that was giving 20000 fsync/s but if redis is giving you like 80k writes/s then even fsync this fast is going to be a bottleneck. [https://redis.io/topics/benchmarks https://redis.io/topics/benchmarks] I think people who are deploying redis are willing to tolerate lower durability guarantees for the extra performance. A lot of the time redis is some kind of cache and there is a way of reconstructing the real data from more durable storage in the case of failure. Or people are willing to lose a second of data or whatever fsync interval people are using.
  - finnh 8y ago
kukabynd 8y ago
Paul Graham inspired many people. His Hackers & Painters is something I enjoy going through now and then.
- dredmorbius 8y ago
  Counterpoint: https://idlewords.com/2005/04/dabblers_and_blowhards.htm https://idlewords.com/2005/04/dabblers_and_blowhards.htm
  - jimbokun 8y ago
    Those footnotes are pretty funny.
userSumo 8y ago
The comment should probably be still around right? Does anyone have the link?
- tim333 8y ago
  Maybe https://news.ycombinator.com/item?id=14605 https://news.ycombinator.com/item?id=14605 ?
  - adlpz 8y ago
    Here's the one, I believe. Linked to parent for context: https://news.ycombinator.com/item?id=14754 https://news.ycombinator.com/item?id=14754 PS: They're basically the same, really, just adding another candidate to yours.
    - tim333 8y ago
      Wonder if HN still has all 20 million comments in a hash table?
andy_ppp 8y ago
Of course Erlang has had ETS (and of course you can use gen_servers of various kinds for this) built in forever. Redis is fantastic but I think there have been many examples of prior art before this tweet!
- pushpop 8y ago
  Lots of prior art but the tweet didn’t say PG inspired the paradigm; rather just Redis specifically. Anecdotally, I’ve used plenty of in memory DBs (and written some too) but Redis has been by far my personal favourite.
- zimpenfish 8y ago
  Perl's `Storable` springs to mind here - I know many places who have mini-"databases" that are effectively straight dumps of Perl variables to disk that get loaded in, worked on, then saved back out. I guess Smalltalk's images are the ur-example here?
michaelmcmillan 8y ago
Hmm, I thought this pattern was really common? That is, appending everything to a file and reading back from the file when there's a reboot. I constantly use it when a database (or redis for that matter) is simply overkill for my use case. Here's a 34 line implementation I use on a node production system. It writes struct-like JavaScript objects that represents events to disk. When reading it back I do a fold (or .reduce) to build the state. And yes –– it could be way smarter (writing to memory and disk), but YAGNI has been working out pretty well so far. class EventStore { constructor(file) { this.file = file; this.cache = null; } async appendEvent(event) { // Purge the cache for entries since we mutated the store. this.cache = null; return new Promise((resolve, reject) => { createWriteStream(this.file, { flags: 'a' }) .on('error', reject) .on('close', resolve) .end(`${JSON.stringify(event)}\n`); }); } async readEvents() { if (this.cache !== null) { return this.cache; } try { const data = await readFile(this.file, 'utf-8'); const lines = data.split('\n').filter(line => line); const events = lines.map(line => JSON.parse(line)); this.cache = events; return events; } catch (error) { return []; } } }
- jrockway 8y ago
  It's pretty common. All the wonderful proprietary file formats from the 90s (and probably before, and probably after) basically boil down to writing raw C structs to disk. You can try it yourself... mmap a file, memcpy some structs there, do the reverse ... and enjoy! (Obviously depending on the memory layout of one C compiler on one architecture does not make for portable files. But that was never a design goal of this system.)
  - beat 8y ago
    I spent part of the mid-1990s hacking a proprietary data streaming protocol written in lexx and yacc. It was what we had back then.
- lioeters 8y ago
  I've been using (and developing a fork of) NeDB [0] that does exactly what you describe: an in-memory database with append-only logs of changes for file-system persistence. On startup, and optionally at regular intervals, it "compacts" the database by reducing all events to a single JSON string. The README links to an article by @antirez, Redis Persistence Demystified [1]. It's been educational studying how it works. [0] https://github.com/louischatriot/nedb#persistence https://github.com/louischatriot/nedb#persistence [1] http://oldblog.antirez.com/post/redis-persistence-demystified.html http://oldblog.antirez.com/post/redis-persistence-demystifie...
DonHopkins 8y ago
That's exactly how HyperCard works, and we all know what inspired HyperCard! https://www.mondo2000.com/2018/06/18/the-inspiration-for-hypercard/ https://www.mondo2000.com/2018/06/18/the-inspiration-for-hyp... https://twit.tv/shows/triangulation/episodes/247?autostart=false https://twit.tv/shows/triangulation/episodes/247?autostart=f...
antirez 8y ago
I want to clarify a couple of things. I'm not saying that Paul Graham invented this pattern. Actually after he mentioned it, I remembered a friend of my father to implement exactly that in QUICKBASIC in the late 80s :-) The point is that maybe the Redis design was already inside me, but I needed a trigger: I often think of good things after being triggered. And smart people are more likely to tell about good ideas, old and new. That was the point. Similarly I believe there are a lot of simple fundamental ideas that can be re-applied to today's technology, as the landscape changes many things become relevant again.
- michaelmcmillan 8y ago
  I'll just take the opportunity to say how grateful I am that the idea of inventing Redis struck you -– regardless of how it originated. I use it all the time, both professionally and in my free time. Awesome piece of software. An idea is worthless by itself, execution is everything.
  - markbnj 8y ago
    Very much this. Redis is pretty much the swiss army knife of persistence around our shop.
    - technics256 8y ago
      I've been meaning to learn redis for quite some time. Do you have favorite resources for this? Thank you.
      - markbnj 8y ago
        Just the documentation to be honest. The basic functions of redis are quite simple to learn and use either using the redis-cli client or a language binding. Basically you PUT keyname value, and GET keyname to retrieve value. There's a ton of additional features and types of structures but the basic use of it is as a key/value store.
  - antirez 8y ago
lucideer 8y ago
To offer a slightly alternative perpective on this, I actually think this type of "article"/("listicle"/"tweetacle"?) can have negative effects. In my mind, it lends credence to the very toxic notion of "value of ideas" over "value of execution". The former is something that has all sorts of knock-on effects: backward IP laws "protecting" ideas, perverse incentives within large corporations with outspoken "idea men" being promoted ahead of doers, non-technical founders with "high-potential ideas" sucking up investment and expecting to execute with technical hires on untested theories. I'm not saying any of the above applies in this case of course, but the fact is that it is you who built Redis, not pg, nor many others who've had similar ideas, and I think writing the above tweet thread lends undue weight to many of above negative trends in our industry (and also in general in recorded history of IP/invention-credit battles).
- antirez 8y ago
  It's a matter of interpretation. IMHO the tweet shows how valuable Hacker News itself is, not Paul Graham ideas (but then HN was created by PG, so, yep, also gives credits). If you take a number of people that are good at doing things and put them together, this will result in more things created because of a natural process of ideas exchanges / triggering. I'm a example of a very isolated programmer, so this applies especially to folks in my condition, but at this point I guess there are quite a bit of "us".
- dragonwriter 8y ago
  > In my mind, it lends credence to the very toxic notion of "value of ideas" over "value of execution". Ideas are required for execution; they don't have independent value (idea without execution delivers nothing), but neither does execution (you have tohave something to execute.)
  - lucideer 8y ago
    Of course. I didn't say otherwise: what I'm talking about is value. Ideas are often (usually?) assigned greater value than execution itself, which is absurd. In actual fact, even beyond "inception", most execution requires ongoing iteration and innovation. No final product is solely the result of its inspiration.
quadcore 8y ago
Actually, I wonder why we dont have yet a programming language and runtime which, after a shutdown, reload exactly like it was.
- jon-wood 8y ago
  I believe you're thinking of Smalltalk there - https://en.wikipedia.org/wiki/Smalltalk#Image-based_persistence https://en.wikipedia.org/wiki/Smalltalk#Image-based_persiste...
- ovi256 8y ago
  eLisp (and others!) do this. It's very slow because memory and disk sizes got larger, but disk transfer bandwidth didn't keep up.
- tonyarkles 8y ago
  If I recall correctly, Emacs currently does this. During the build it (slowly) loads all of the elisp into memory and then dumps out the memory image of it after it’s all been compiled. I’m a bit hazy on the details, but I think it involves calling unexec(). https://lwn.net/Articles/673724/ https://lwn.net/Articles/673724/ Edit: just noticed a sibling comment mentioned this too...
  - Nullabillity 8y ago
    Only for built-in elisp, as part of the build process. User configuration (including packages) is reloaded from scratch on each start.
- golergka 8y ago
  What strategy do you use to migrate old data to new codebase? Different business logic would probably want different answers in this regard, so I doubt that there's a one size fits all solution.
- mc_mike 8y ago
  Some lisps do this. like sbcl's sb-ext:save-lisp-and-die function is a "quit" that also persists the memory state. When you load this image later, you get it as it was when you "quit" last time.
himynameisdom 8y ago
Man, some of the comments on here are really disheartening. What's the deal with trying to humble people with long-winded, tangential counterpoints, gotchas, and condescending questions? Constantly proving one's intellect seems to be a prevailing theme on HN and I'm not seeing how it adds to the quality of the content. I'm sure someone will unearth the irony in this and let me know soon enough.
- coleifer 8y ago
  Best not to read the comments. There's literally nothing you can do about the phenomenon you're describing.
chrchr 8y ago
“System prevalence[1] is a simple software architectural pattern that combines system images (snapshots) and transaction journaling to provide speed, performance scalability, transparent persistence and transparent live mirroring of computer system state.“ — https://en.m.wikipedia.org/wiki/System_prevalence https://en.m.wikipedia.org/wiki/System_prevalence
huahe 8y ago
PG inspires as usual!
tosh 8y ago
Redis is a great example for unbundling a pattern to a lib/service/business.
vshan 8y ago
PG was also involved in the inception of Reddit: It was PG who gave Alexis and Steve the idea to make something like reddit, and also gave them the tagline "the front page of the internet".[0] PG had vetoed their initial idea to create a food-delivery app and then called them back and asked them to come up with something new. [0]: https://www.youtube.com/watch?v=5rZ8f3Bx6Po https://www.youtube.com/watch?v=5rZ8f3Bx6Po

411 points by oneowl 8y ago

When the application would be restarted, reading back the log would recreate the in memory data structures. I thought that it was cool, and that databases themselves could be that way instead of using the programming language data structures without a networked API. That is literally how databases work. In Memory + WAL + Data Files on disk. You could, in theory, live without the Data Files and just a big WAL.

kijin 8y ago
Except that most databases don't store their data in anything resembling "programming language data structures". You get tables, rows, and columns (or maybe a bit of JSON if you're lucky) instead of native integers, strings, lists, sets, and dictionaries.
- threeseed 8y ago
  Object and document databases have been around for decades. Likewise ORMs which allow for higher order types etc. have been around since WebObjects i.e. also decades.
  - kijin 8y ago
    The primary purpose of an ORM is to overcome the "impedance mismatch" between relational databases and programming language data structures. There's no need for an ORM if you can store your data structures directly in the database.
    - TeMPOraL 8y ago
      If this is a primary purpose of an ORM, then I wish I knew of one that isn't utterly failing at that. One thing I've learned about this "impedance mismatch" is that it isn't a syntax thing, it's a fundamental difference in the way of viewing the world. The way you store data about the world is different from the way you model that world dynamically, with objects. I find it safer to always split out the "business model" from the storage layer, so that those different views don't interfere - and once you do that, you may as well implement the storage layer in a relational way.
      - icebraining 8y ago
        .. and the code you implement to connect that business layer to that storage layer is an ORM. The idea that the ORM forces the storage layer to a particular representation of the business layer is only true if they implement the ActiveRecord pattern, which isn't universal.
    - neokantian 8y ago
      > ... if you can store your data structures directly in the database. ABSTRACT. Future users of large data banks must be protected from having to know how the data is organized in the machine (the internal representation). It provides a means of describing data with its natural structure only—that is, without superimposing any additional structure for machine representation purposes. Accordingly, it provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other. E.F. Codd. 1970. A relational model of data for large shared data banks. Commun. ACM 13, 6 (June 1970), 377-387. You propose to reintroduce a problem that they absolutely wanted to get rid off 40 years ago. Just imagine that you first have to figure out how to painstakingly parse serialized Python dictionaries before you can access the data in another program written in e.g. Rust. It clearly amounts to UNSOLVING a problem that is now SOLVED ALREADY.
      - kijin 8y ago
        Well, Redis allows me to store a JavaScript array, add some more with a Ruby client, remove some items with a PHP client, and finally read it back as a Python list just fine. What's the problem that has been unsolved? :)
        neokantian 8y ago
        https://msgpack.org https://msgpack.org is absolutely fine. I wasn't criticizing msgpack. You can also serialize the bytes of an array of C structs, if you see what I mean.
    - deleted 8y ago
      [deleted]
    - mamcx 8y ago
      That is an angle. Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. Exist 2 main reasons for the "impedance mismatch": - Paradigms. 2 different paradigms will be at odds. Example: Functional and OO. This is ok. - Limitations: The relational model is absolutely superior and more expressive at manipulate data than OO/Functional. You need A LOT of machinery to recover that power. This is not ok. However, this not change the fact that OO is ok.Similar how a KV store is fine, but certainly, a RDBMS store is much more capable.
      - heavenlyblue 8y ago
        How is relational more powerful than OO/Functional if you can’t define an infinite dataset with it?
        mamcx 8y ago
        You can't? Most people only see the relational model as is inside a RDBMS. That is how judge the OO for what you can do on Mongo. RDBMS have some weird and well considered restrictions for their case. But read a little about the relational model, and none of it depend on SQL or say anything about how is the storage. P.D: Is important to note for where is MORE powerful. Remember that the creator of Pascal say: https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures... Algorithms + Data Structures = Programs You can say OO/Functional lean more to the "Algorithms" side but RM to the "Data Structures". OO/Functional not say much how operate on data, most is a exercise for the reader. Instead, RM give a clear answer and defined operations for that. You need to "spice up" things to make the one be more useful to the other part of the equation. RM, too pure, certainly is incredible limited, not even can print "hello world!", but that is just too give a solution about how transform data...
      - notduncansmith 8y ago
        > Other way to say it is that ORM is a workaround to the fact most languages are VERY poor at manipulate data. This is why I love Clojure (and a particular style of Javascript). Destructuring and a good library of object/array manipulation functions make an environment well-suited to transforming data structures (which is precisely what I want to do in most programs I write), and I find I do not need an ORM where this type of data-focused programming is supported.
- jinjin2 8y ago
  It does if you use an object database like Realm.
elvinyung 8y ago
I think you're slightly missing the point -- for me, a certain unique "awesomeness" lies in specifically being able to literally pipe stdout back to stdin and get back the exact same data structures.
- snr 8y ago
  Can be easily done by printing out the stdin to stdout in the first place?
  - elvinyung 8y ago
    ok, without doing that :p
Illniyar 8y ago
Relational databases (except maybe MemQL) treat the file system as the source of truth. And usually the file system is bigger then the memory so they need to constantly update their cache with more relevant data. Redis loads everything to memory. And doesn't keep the structure in the file system, only the log, recreating it from log+snapshot.

threeseed 8y ago

Strange title. Does merely recalling a language pattern really count as inspiring ?

tomhoward 8y ago
In the tweet, the author of Redis says "[Paul Graham] ... inspired the creation of Redis". So it probably counts!
- dang 8y ago
  Ok, we put that in the title above for disambiguity.

viach 8y ago

Probably the next logical idea would be to apply to YC with Redis?

stretchwithme 8y ago

Doesn't AWS's Aurora database sort of work that way too?

bpicolo 8y ago
The pattern in general is used by more or less every database in the form of write-ahead logging. https://en.wikipedia.org/wiki/Write-ahead_logging https://en.wikipedia.org/wiki/Write-ahead_logging

konamicode 8y ago

I'd be keen to see a simple example of this pattern in Lisp (or another language). Does anybody have a good link?

pavlov 8y ago
I use it by default for new Node.js projects (most of which are experiments but some end up in production, and have been running without problems for years). It's a simple pattern to implement. To make it a bit easier to use repeatedly, I've got a small helper class called JournaledCollection. You pass it serialize+deserialize callbacks for your item type, and it takes care of persistence in event logs. For a while I was thinking about releasing my helpers as a project called LAUF, short for "Lame-Ass Un-Framework". Then one could say: "Most of my projects are LAUFable, I don't need anything more serious." (Awful dad jokes are a solid reason to publish open source, right?) Never got around to it though, but if you're interested, I could put together an example.
- chrisweekly 8y ago
  I like the name -- and it sounds worth sharing!
michaelmcmillan 8y ago
Here's one I'm currently using in node (wrote it as another comment): https://news.ycombinator.com/item?id=19499799 https://news.ycombinator.com/item?id=19499799
stevelosh 8y ago
sjl at alephnull in ~/Desktop ><((°> sbcl [SBCL] CL-USER> (defparameter *name* "World") *NAME* [SBCL] CL-USER> (defun foo () (format t "Hello, ~A~%" *name*)) FOO [SBCL] CL-USER> (sb-ext:save-lisp-and-die "session.core") sjl at alephnull in ~/Desktop ><((°> sbcl --core session.core [SBCL] CL-USER> (foo) Hello, World NIL

jokoon 8y ago

Is this really a brilliant idea though? Everybody knows RAM became cheaper cheaper, while mechanical disk can't be made faster, and SSD have reliability limits. It seems entirely logical to expect databases to work on RAM first and then commit on disk for a large performance improvement.

antirez 8y ago
When I started Redis people were like FTW data in memory?!
- jokoon 8y ago
  I was talking to a database teacher, and I tried arguing about ACID on an in memory database, arguing that there was a minimal window for database corruption in a transaction system, since the worst case scenario would seem to be a loss of very few transaction. He was not really listening to what I was saying or my arguments, because a system like redis seems like a more than acceptable compromise. It still seems a few people are reluctant to an in-memory database.
  - benmmurphy 8y ago
    VoltDB is a probably a better example of an in-memory database with ACID semantics. Redis usually won't be deployed in a way where it fsyncs after each write operation because it runs too slow. You need to do some tricks like batching fsyncs to get decent performance and I don't think Redis has support for this. However, SSD fsync performance seems very high. I remember benchmarking an EC2 i3 instance that was giving 20000 fsync/s but if redis is giving you like 80k writes/s then even fsync this fast is going to be a bottleneck. [https://redis.io/topics/benchmarks https://redis.io/topics/benchmarks] I think people who are deploying redis are willing to tolerate lower durability guarantees for the extra performance. A lot of the time redis is some kind of cache and there is a way of reconstructing the real data from more durable storage in the case of failure. Or people are willing to lose a second of data or whatever fsync interval people are using.
- finnh 8y ago

kukabynd 8y ago

Paul Graham inspired many people. His Hackers & Painters is something I enjoy going through now and then.

dredmorbius 8y ago
Counterpoint: https://idlewords.com/2005/04/dabblers_and_blowhards.htm https://idlewords.com/2005/04/dabblers_and_blowhards.htm
- jimbokun 8y ago
  Those footnotes are pretty funny.

userSumo 8y ago

The comment should probably be still around right? Does anyone have the link?

tim333 8y ago
Maybe https://news.ycombinator.com/item?id=14605 https://news.ycombinator.com/item?id=14605 ?
- adlpz 8y ago
  Here's the one, I believe. Linked to parent for context: https://news.ycombinator.com/item?id=14754 https://news.ycombinator.com/item?id=14754 PS: They're basically the same, really, just adding another candidate to yours.
  - tim333 8y ago
    Wonder if HN still has all 20 million comments in a hash table?

andy_ppp 8y ago

Of course Erlang has had ETS (and of course you can use gen_servers of various kinds for this) built in forever. Redis is fantastic but I think there have been many examples of prior art before this tweet!

pushpop 8y ago
Lots of prior art but the tweet didn’t say PG inspired the paradigm; rather just Redis specifically. Anecdotally, I’ve used plenty of in memory DBs (and written some too) but Redis has been by far my personal favourite.
zimpenfish 8y ago
Perl's `Storable` springs to mind here - I know many places who have mini-"databases" that are effectively straight dumps of Perl variables to disk that get loaded in, worked on, then saved back out. I guess Smalltalk's images are the ur-example here?

michaelmcmillan 8y ago

Hmm, I thought this pattern was really common? That is, appending everything to a file and reading back from the file when there's a reboot. I constantly use it when a database (or redis for that matter) is simply overkill for my use case. Here's a 34 line implementation I use on a node production system. It writes struct-like JavaScript objects that represents events to disk. When reading it back I do a fold (or .reduce) to build the state. And yes –– it could be way smarter (writing to memory and disk), but YAGNI has been working out pretty well so far. class EventStore { constructor(file) { this.file = file; this.cache = null; } async appendEvent(event) { // Purge the cache for entries since we mutated the store. this.cache = null; return new Promise((resolve, reject) => { createWriteStream(this.file, { flags: 'a' }) .on('error', reject) .on('close', resolve) .end(`${JSON.stringify(event)}\n`); }); } async readEvents() { if (this.cache !== null) { return this.cache; } try { const data = await readFile(this.file, 'utf-8'); const lines = data.split('\n').filter(line => line); const events = lines.map(line => JSON.parse(line)); this.cache = events; return events; } catch (error) { return []; } } }

jrockway 8y ago
It's pretty common. All the wonderful proprietary file formats from the 90s (and probably before, and probably after) basically boil down to writing raw C structs to disk. You can try it yourself... mmap a file, memcpy some structs there, do the reverse ... and enjoy! (Obviously depending on the memory layout of one C compiler on one architecture does not make for portable files. But that was never a design goal of this system.)
- beat 8y ago
  I spent part of the mid-1990s hacking a proprietary data streaming protocol written in lexx and yacc. It was what we had back then.
lioeters 8y ago
I've been using (and developing a fork of) NeDB [0] that does exactly what you describe: an in-memory database with append-only logs of changes for file-system persistence. On startup, and optionally at regular intervals, it "compacts" the database by reducing all events to a single JSON string. The README links to an article by @antirez, Redis Persistence Demystified [1]. It's been educational studying how it works. [0] https://github.com/louischatriot/nedb#persistence https://github.com/louischatriot/nedb#persistence [1] http://oldblog.antirez.com/post/redis-persistence-demystified.html http://oldblog.antirez.com/post/redis-persistence-demystifie...

DonHopkins 8y ago

That's exactly how HyperCard works, and we all know what inspired HyperCard! https://www.mondo2000.com/2018/06/18/the-inspiration-for-hypercard/ https://www.mondo2000.com/2018/06/18/the-inspiration-for-hyp... https://twit.tv/shows/triangulation/episodes/247?autostart=false https://twit.tv/shows/triangulation/episodes/247?autostart=f...

antirez 8y ago

I want to clarify a couple of things. I'm not saying that Paul Graham invented this pattern. Actually after he mentioned it, I remembered a friend of my father to implement exactly that in QUICKBASIC in the late 80s :-) The point is that maybe the Redis design was already inside me, but I needed a trigger: I often think of good things after being triggered. And smart people are more likely to tell about good ideas, old and new. That was the point. Similarly I believe there are a lot of simple fundamental ideas that can be re-applied to today's technology, as the landscape changes many things become relevant again.

michaelmcmillan 8y ago
I'll just take the opportunity to say how grateful I am that the idea of inventing Redis struck you -– regardless of how it originated. I use it all the time, both professionally and in my free time. Awesome piece of software. An idea is worthless by itself, execution is everything.
- markbnj 8y ago
  Very much this. Redis is pretty much the swiss army knife of persistence around our shop.
  - technics256 8y ago
    I've been meaning to learn redis for quite some time. Do you have favorite resources for this? Thank you.
    - markbnj 8y ago
      Just the documentation to be honest. The basic functions of redis are quite simple to learn and use either using the redis-cli client or a language binding. Basically you PUT keyname value, and GET keyname to retrieve value. There's a ton of additional features and types of structures but the basic use of it is as a key/value store.
- antirez 8y ago

lucideer 8y ago

To offer a slightly alternative perpective on this, I actually think this type of "article"/("listicle"/"tweetacle"?) can have negative effects. In my mind, it lends credence to the very toxic notion of "value of ideas" over "value of execution". The former is something that has all sorts of knock-on effects: backward IP laws "protecting" ideas, perverse incentives within large corporations with outspoken "idea men" being promoted ahead of doers, non-technical founders with "high-potential ideas" sucking up investment and expecting to execute with technical hires on untested theories. I'm not saying any of the above applies in this case of course, but the fact is that it is you who built Redis, not pg, nor many others who've had similar ideas, and I think writing the above tweet thread lends undue weight to many of above negative trends in our industry (and also in general in recorded history of IP/invention-credit battles).

antirez 8y ago
It's a matter of interpretation. IMHO the tweet shows how valuable Hacker News itself is, not Paul Graham ideas (but then HN was created by PG, so, yep, also gives credits). If you take a number of people that are good at doing things and put them together, this will result in more things created because of a natural process of ideas exchanges / triggering. I'm a example of a very isolated programmer, so this applies especially to folks in my condition, but at this point I guess there are quite a bit of "us".
dragonwriter 8y ago
> In my mind, it lends credence to the very toxic notion of "value of ideas" over "value of execution". Ideas are required for execution; they don't have independent value (idea without execution delivers nothing), but neither does execution (you have tohave something to execute.)
- lucideer 8y ago
  Of course. I didn't say otherwise: what I'm talking about is value. Ideas are often (usually?) assigned greater value than execution itself, which is absurd. In actual fact, even beyond "inception", most execution requires ongoing iteration and innovation. No final product is solely the result of its inspiration.

quadcore 8y ago

Actually, I wonder why we dont have yet a programming language and runtime which, after a shutdown, reload exactly like it was.

jon-wood 8y ago
I believe you're thinking of Smalltalk there - https://en.wikipedia.org/wiki/Smalltalk#Image-based_persistence https://en.wikipedia.org/wiki/Smalltalk#Image-based_persiste...
ovi256 8y ago
eLisp (and others!) do this. It's very slow because memory and disk sizes got larger, but disk transfer bandwidth didn't keep up.
tonyarkles 8y ago
If I recall correctly, Emacs currently does this. During the build it (slowly) loads all of the elisp into memory and then dumps out the memory image of it after it’s all been compiled. I’m a bit hazy on the details, but I think it involves calling unexec(). https://lwn.net/Articles/673724/ https://lwn.net/Articles/673724/ Edit: just noticed a sibling comment mentioned this too...
- Nullabillity 8y ago
  Only for built-in elisp, as part of the build process. User configuration (including packages) is reloaded from scratch on each start.
golergka 8y ago
What strategy do you use to migrate old data to new codebase? Different business logic would probably want different answers in this regard, so I doubt that there's a one size fits all solution.
mc_mike 8y ago
Some lisps do this. like sbcl's sb-ext:save-lisp-and-die function is a "quit" that also persists the memory state. When you load this image later, you get it as it was when you "quit" last time.

himynameisdom 8y ago

Man, some of the comments on here are really disheartening. What's the deal with trying to humble people with long-winded, tangential counterpoints, gotchas, and condescending questions? Constantly proving one's intellect seems to be a prevailing theme on HN and I'm not seeing how it adds to the quality of the content. I'm sure someone will unearth the irony in this and let me know soon enough.

coleifer 8y ago
Best not to read the comments. There's literally nothing you can do about the phenomenon you're describing.

chrchr 8y ago

“System prevalence[1] is a simple software architectural pattern that combines system images (snapshots) and transaction journaling to provide speed, performance scalability, transparent persistence and transparent live mirroring of computer system state.“ — https://en.m.wikipedia.org/wiki/System_prevalence https://en.m.wikipedia.org/wiki/System_prevalence

huahe 8y ago

PG inspires as usual!

tosh 8y ago

Redis is a great example for unbundling a pattern to a lib/service/business.

vshan 8y ago

PG was also involved in the inception of Reddit: It was PG who gave Alexis and Steve the idea to make something like reddit, and also gave them the tagline "the front page of the internet".[0] PG had vetoed their initial idea to create a food-delivery app and then called them back and asked them to come up with something new. [0]: https://www.youtube.com/watch?v=5rZ8f3Bx6Po https://www.youtube.com/watch?v=5rZ8f3Bx6Po