5 ms·

LMDB – First version of Redis written in Tcl

149 points by weeber 9y ago

dunkelheit 9y ago
This is really inspiring. Reminds me of the quote "A complex system that works is invariably found to have evolved from a simple system that worked." The interesting question of course is how to evolve a simple system into a successful complex system. One thing that seems important is nailing the right architecture early on - the simple system must contain all the right ideas in it and it is important to weed out bad ideas before they are entrenched in the code base. Still, planning the long-term evolution of a software system remains an arcane art to me. "Listen to your users and do what is necessary at the moment" is too simplistic.
- ben_jones 9y ago
  On the contrary I don't think you can plan architectural perfection from the beginning. IMO you have to throw it in a prod-like environment early (preferably as an internal app with a minimal SLO) so that you can iterate early and often.
  - dunkelheit 9y ago
    That's what I talked about in the last sentence of my post. It is important to do this, but it is not the whole story. How do you decide in which direction to iterate?
    - mk89 9y ago
      I think it takes experience. That's why I still believe that sw architects are really essential.
      - ben_jones 9y ago
        Similar to my sibling comment I think the value in sw architects isn't the ability to predict the future from experience, rather they should build systems that can accept reasonable change. Some practical examples would be to implement build systems into their stack that accept database migrations and rollbacks easily. Or realizing they are in a fast-paced team that could benefit from specific frameworks and languages (experience would play a big role here).
    - ben_jones 9y ago
      Make it feature-driven. When Facebook came out they didn't know photo sharing would be a definitive feature. When they put it out there they took a slug to the shoulder for not realizing it sooner and then implemented relevant features (like tagging). Similarly when Redis came out I doubt the creator realized everybody and their grandmother would use lists as a defacto message queue but now look at all the commands list has that facilitate just that. Key point is to accept and welcome that slug to the shoulder that pushes you in the right direction, to abuse the metaphor.
      - wand3r 9y ago
        Iirc Zuckerberg allowed profile pics but realized people kept changing them all the time so they implemented photos
- avaer 9y ago
  The ability to plan falls out of experience. All you have to do is listen to your users and do what's necessary if you have scars to remind you that sticking IO here will bring the system to its knees on a real user load, or that this customer is going to ask for LDAP integration next week so you better build your authentication accordingly.
  - dunkelheit 9y ago
    I agree that experience helps, but experience alone is insufficient. Experience should be properly reflected upon, or you can stick to suboptimal practices justifying them by saying "that's how I always do it, worked for me in the past". Experience can even be detrimental - the well-known second system syndrome comes to mind!
    - mk89 9y ago
      Yet lack of experience won't help either... so, experience and keeping up with what customers need
- isodude 9y ago
  If you are able to rewrite the code base in a short amount of time, the infrastructure problems are easier to handle. That way you can move forward faster and fix problems early on.
  - dunkelheit 9y ago
    Yes, when the code base is small, evolving it is much simpler. One of the challenges is deciding on the exact right moment to introduce big changes - too early and the changes are in the wrong direction or even useless, too late and the amount of work is prohibiting and you are forced to live with suboptimal architecture.
    - isodude 9y ago
      Yeah, I agree. I think the real challange is to keep the code base in such a good shape that a rewrite of the whole thing or parts of is always doable. Someone talked about never writing modules that can't be rewritten in two weeks.
    - anarazel 9y ago
      > Yes, when the code base is small, evolving it is much simpler. There's obviously some truth to that. But I also think it leaves or an important angle: Often larger, more complicated, features are easier to add if you have a lot of infrastructure pieces. Say, in a database, if you have no durability (journaling), resource management (errors, locks), error checking (deadlocks, days checksumming), it'll be a lot more work to add support for some new query language [feature]. Now databases are my thing, but the same it's true for a lot of areas. Disregarding that is IMO one of the reasons ground up rewrites fail so often: You'll get bogged down with infrastructure for a long time, without proper non toy users of the facility.
- dunkelheit 9y ago
  As a partial answer, here is the document that outlines the redis philosophy: http://oldblog.antirez.com/post/redis-manifesto.html http://oldblog.antirez.com/post/redis-manifesto.html I bet that coming up with this points and having them in mind (even subconsciously) has helped antirez evolve redis in a coherent direction.
js2 9y ago
antirez mentions that this forks to save, but I'm not seeing that. The backgroundsave procedure makes a copy of db to dbcopy, but then never saves dbcopy. I wonder how much further the TCL implementation was extended before antirez switched to C. BTW, I hadn't heard of LLOOGG (the L in LMDB) before. It was this: "LLOGG was web service I (Salvatore Sanfilippo) and my co-founder Fabio Pitrola ran for seven years for free." https://github.com/antirez/lloogg/blob/master/README.md https://github.com/antirez/lloogg/blob/master/README.md
- tyingq 9y ago
  Looks like the intention was there, they pull in TclX with a comment about it. https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e88a8#file-lmdb-tcl-L17 https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e... Guessing he just found some interim version that wasn't the one he actually wanted to post.
  - antirez 9y ago
    You are right, the intention was that but now I'm reminding that, in the middle of doing it, I thought, ok I reached the limits of the Tcl prototype, let's rewrite in C.
user5994461 9y ago
This naming is so confusing. For those not aware, LMDB is the memory mapped database that's powering OpenLDAP. https://symas.com/lightning-memory-mapped-database/ https://symas.com/lightning-memory-mapped-database/
- nurettin 9y ago
  LMDB is one of the backends that you can choose. BDB is more common in my experience. http://www.openldap.org/doc/admin24/backends.html http://www.openldap.org/doc/admin24/backends.html
  - deleted 9y ago
    [deleted]
- teacpde 9y ago
  To be fair, LLOOGG Memory DB is apparently earlier than Lightning Memory-Mapped Database
unwttng 9y ago
As someone who worked for a company for a few years on a vast Tcl codebase, a company in which almost every engineer lamented the fact that they were working in an unused and unknown language, it warms my heart to see that it was used for this.
- stevekemp 9y ago
  Antirez has long-been a TCL fan, as you can see by his piece here: http://antirez.com/articoli/tclmisunderstood.html http://antirez.com/articoli/tclmisunderstood.html I dabbled with it myself, writing a simple TCL interpreter in C, but I always find myself comparing it to FORTH a language I still find it easy to love.
KasianFranks 9y ago
Nice to see Tcl used here.
SonOfLilit 9y ago
If you enjoyed reading this, you might enjoy reading git's first self-hosted commit: https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23ca2e25604af290 https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23...

149 points by weeber 9y ago

This is really inspiring. Reminds me of the quote "A complex system that works is invariably found to have evolved from a simple system that worked." The interesting question of course is how to evolve a simple system into a successful complex system. One thing that seems important is nailing the right architecture early on - the simple system must contain all the right ideas in it and it is important to weed out bad ideas before they are entrenched in the code base. Still, planning the long-term evolution of a software system remains an arcane art to me. "Listen to your users and do what is necessary at the moment" is too simplistic.

ben_jones 9y ago
On the contrary I don't think you can plan architectural perfection from the beginning. IMO you have to throw it in a prod-like environment early (preferably as an internal app with a minimal SLO) so that you can iterate early and often.
- dunkelheit 9y ago
  That's what I talked about in the last sentence of my post. It is important to do this, but it is not the whole story. How do you decide in which direction to iterate?
  - mk89 9y ago
    I think it takes experience. That's why I still believe that sw architects are really essential.
    - ben_jones 9y ago
      Similar to my sibling comment I think the value in sw architects isn't the ability to predict the future from experience, rather they should build systems that can accept reasonable change. Some practical examples would be to implement build systems into their stack that accept database migrations and rollbacks easily. Or realizing they are in a fast-paced team that could benefit from specific frameworks and languages (experience would play a big role here).
  - ben_jones 9y ago
    Make it feature-driven. When Facebook came out they didn't know photo sharing would be a definitive feature. When they put it out there they took a slug to the shoulder for not realizing it sooner and then implemented relevant features (like tagging). Similarly when Redis came out I doubt the creator realized everybody and their grandmother would use lists as a defacto message queue but now look at all the commands list has that facilitate just that. Key point is to accept and welcome that slug to the shoulder that pushes you in the right direction, to abuse the metaphor.
    - wand3r 9y ago
      Iirc Zuckerberg allowed profile pics but realized people kept changing them all the time so they implemented photos
avaer 9y ago
The ability to plan falls out of experience. All you have to do is listen to your users and do what's necessary if you have scars to remind you that sticking IO here will bring the system to its knees on a real user load, or that this customer is going to ask for LDAP integration next week so you better build your authentication accordingly.
- dunkelheit 9y ago
  I agree that experience helps, but experience alone is insufficient. Experience should be properly reflected upon, or you can stick to suboptimal practices justifying them by saying "that's how I always do it, worked for me in the past". Experience can even be detrimental - the well-known second system syndrome comes to mind!
  - mk89 9y ago
    Yet lack of experience won't help either... so, experience and keeping up with what customers need
isodude 9y ago
If you are able to rewrite the code base in a short amount of time, the infrastructure problems are easier to handle. That way you can move forward faster and fix problems early on.
- dunkelheit 9y ago
  Yes, when the code base is small, evolving it is much simpler. One of the challenges is deciding on the exact right moment to introduce big changes - too early and the changes are in the wrong direction or even useless, too late and the amount of work is prohibiting and you are forced to live with suboptimal architecture.
  - isodude 9y ago
    Yeah, I agree. I think the real challange is to keep the code base in such a good shape that a rewrite of the whole thing or parts of is always doable. Someone talked about never writing modules that can't be rewritten in two weeks.
  - anarazel 9y ago
    > Yes, when the code base is small, evolving it is much simpler. There's obviously some truth to that. But I also think it leaves or an important angle: Often larger, more complicated, features are easier to add if you have a lot of infrastructure pieces. Say, in a database, if you have no durability (journaling), resource management (errors, locks), error checking (deadlocks, days checksumming), it'll be a lot more work to add support for some new query language [feature]. Now databases are my thing, but the same it's true for a lot of areas. Disregarding that is IMO one of the reasons ground up rewrites fail so often: You'll get bogged down with infrastructure for a long time, without proper non toy users of the facility.
dunkelheit 9y ago
As a partial answer, here is the document that outlines the redis philosophy: http://oldblog.antirez.com/post/redis-manifesto.html http://oldblog.antirez.com/post/redis-manifesto.html I bet that coming up with this points and having them in mind (even subconsciously) has helped antirez evolve redis in a coherent direction.

js2 9y ago

antirez mentions that this forks to save, but I'm not seeing that. The backgroundsave procedure makes a copy of db to dbcopy, but then never saves dbcopy. I wonder how much further the TCL implementation was extended before antirez switched to C. BTW, I hadn't heard of LLOOGG (the L in LMDB) before. It was this: "LLOGG was web service I (Salvatore Sanfilippo) and my co-founder Fabio Pitrola ran for seven years for free." https://github.com/antirez/lloogg/blob/master/README.md https://github.com/antirez/lloogg/blob/master/README.md

tyingq 9y ago
Looks like the intention was there, they pull in TclX with a comment about it. https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e88a8#file-lmdb-tcl-L17 https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e... Guessing he just found some interim version that wasn't the one he actually wanted to post.
- antirez 9y ago
  You are right, the intention was that but now I'm reminding that, in the middle of doing it, I thought, ok I reached the limits of the Tcl prototype, let's rewrite in C.

user5994461 9y ago

This naming is so confusing. For those not aware, LMDB is the memory mapped database that's powering OpenLDAP. https://symas.com/lightning-memory-mapped-database/ https://symas.com/lightning-memory-mapped-database/

nurettin 9y ago
LMDB is one of the backends that you can choose. BDB is more common in my experience. http://www.openldap.org/doc/admin24/backends.html http://www.openldap.org/doc/admin24/backends.html
- deleted 9y ago
  [deleted]
teacpde 9y ago
To be fair, LLOOGG Memory DB is apparently earlier than Lightning Memory-Mapped Database

unwttng 9y ago

As someone who worked for a company for a few years on a vast Tcl codebase, a company in which almost every engineer lamented the fact that they were working in an unused and unknown language, it warms my heart to see that it was used for this.

stevekemp 9y ago
Antirez has long-been a TCL fan, as you can see by his piece here: http://antirez.com/articoli/tclmisunderstood.html http://antirez.com/articoli/tclmisunderstood.html I dabbled with it myself, writing a simple TCL interpreter in C, but I always find myself comparing it to FORTH a language I still find it easy to love.

KasianFranks 9y ago

Nice to see Tcl used here.

SonOfLilit 9y ago

If you enjoyed reading this, you might enjoy reading git's first self-hosted commit: https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23ca2e25604af290 https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23...