13 ms·
The “Build Your Own Redis” Book Is Completed
- eBombzor 4y ago> Here are some examples that meet my criteria of the “building block”: NGINX, SQLite, PostgreSQL, Kafka, Linux kernel, etc. Kafka?
- nkozyra 4y agoGiven there's a focus on basic data structures it seems like a good fit.
- ibbtown 4y agoProbably MQ as example for a message bus was not OSS and hip enough;)
- Jolter 4y agoWhy not? I’m curious as to why it wouldn’t fit among those others. Not used broadly enough?
- x86hacker1010 4y agocheck out distributed services in Go and you'll see why Kafka would be in this list.
- Zvez 4y agokafka is a good fit when you design modern system. As your system grows larger, necessity of message queue becomes obvious. And kafka is probably the most widely used general-purpose high performance distributed q
- tikkabhuna 4y agoSorry for being a little off topic, but the Mini Redis tutorial [1] was really fun when I did it a few years ago. It has you implement a server and client in Rust using the Tokio library. I think Redis is a great server to “build yourself” as you don’t need to start with much to get it going. [1] https://tokio.rs/tokio/tutorial/setup https://tokio.rs/tokio/tutorial/setup
- cormacrelf 4y agoIf anyone is looking at this stuff and thinking it’s intimidating then think again: Redis is mostly a lot of fuss around a hash map. There are cool features sure but the idea is mainly “what if a hash map were accessible from two or more servers”. If you only have one or two servers, you only want a cache, and you don’t anticipate scaling up soon (be honest) you can just use a hash map with expiry checking and not have to learn how to deploy another piece of software.
- HideousKojima 4y agoAnd some languages will even have this built in to the standard library, like C#'s MemoryCache class.
- cormacrelf 4y agoIIRC there’s a nice abstraction in .NET Core for caching, which has a swappable backend that means you can upgrade from MemoryCache to Redis if you ever need it. If only all software was this good. One of the least-leaky abstractions of all time.
- tayo42 4y agoRedis offeres alot more then just a key value store. The fuss is the data structures it provides. And just using a hash map in memory isn't sufficient. You'll have unbounded growth. You need a max size and need to evict. Then you want to do that efficiently so your not wasting space on useless keys...
- suncherta 4y agoI am very happy to see more content like this (attempting to teach how to create a real system) becoming available. Few month ago there was a launch announcement of YC backed startup that sell you walk-through labs guiding you to build Redis, Docker, Git and others. (Can't find link to it now.) What is still missing, in my opinion, and is badly needed, is content or even an idea on how to teach taking such projects from toy prototype version to the production quality one.
- DylanSp 4y agoYou're probably thinking of https://codecrafters.io/ https://codecrafters.io/. I've been meaning to check them out with my education stipend from work, just haven't had the time to devote to it.
- lprd 4y agoCurious if anyone else has checked out their content yet? If so, hows the quality?
- pizza234 4y agoThe idea is very good - the tests approach is effective and stimulating; they also provide a range of products to study/experiment with. However, I think that the approach to the learning material is "love it or hate it". By design, the service doesn't provide any documentation; it provides references to existing technical documentation (of any kind, including blog posts). Those who expect a focused introduction to each topic will find it very tedious or hard to proceed (for example, the SQLite exercise has important details buried in a very large and confusing webpage), and likely hate it; those who like the challenge of understanding loads of raw documentation will love it.
- giancarlostoro 4y agoSounds like I might as well not pay them a penny though if its research that I can do myself instead.
- viksit 4y agoi love this! is there a book like this to learn rust with? it would be amazing to have a working system at the end. (not looking for a beginner, but say, intermediate level book).
- thebamby 4y agoOne of the top level comments recommends a tokio.rs tutorial. See here: https://news.ycombinator.com/item?id=34573028 https://news.ycombinator.com/item?id=34573028
- dairem 4y agoYou could try https://www.zero2prod.com/ https://www.zero2prod.com/ , which walks through building an newsletter service in rust.
- x86hacker1010 4y agodistributed services in go is intermediate/advanced programming book where you build a commit log like Kafka. It's pretty intense. VERY fun
- dandrust 4y agoYou should check out Rust in Action. It teaches rust through a bunch of system programming projects.
- osynavets 4y agoI've created a repository for all tutorials that I found in rust [1] I guess that's what you're looking for [1] -https://github.com/osynavets/build-your-own-x-in-rust https://github.com/osynavets/build-your-own-x-in-rust
- linhns 4y agoThanks for all your work finding these. Great job!
- messe 4y agoLooks neat and relatively self contained. I’m adding this to my list of projects to use to learn a new language. I might give it a shot in Kotlin and Zig (I know it and use it, but am not as familiar with its event loop implementation as I’d like to be) over the next few weeks.
- yazzku 4y agoA good read for sure, thank you. '4. Protocol Parsing', hand-writing protocol parsers in C looks particularly thrilling.
- 77pt77 4y agoWhat could go wrong? It's not like there are decades of paying the price of such decisions...
- yazzku 4y agoAt a cursory glance, they assert((size_t)rv <= n), but if that's the assert.h assert(), it's just void when NDEBUG is defined, and cmake seems to define that when building a Release target. Also, gcc does not seem to define it by default. So that's a bit of an overflow there.
- CyberDildonics 4y agoI think you're mixing up parsing a binary protocol with parsing strings.
- swatcoder 4y agoYou don't have to learn raw systems coding yourself, but if nobody bothers, the code of 2050 is going to be built on a lot of stale, brittle magic that nobody can reproduce or understand. This is exactly the sort of playground effort where those who do want to learn can get their hands dirty. If this is their first time doing this kind of project, they shouldn't use any of this code in production. There will be all sorts of cheated corners and vulnerabilities, and not just in obvious high-risk places like this. That's how hands-on learning goes.
- theideaofcoffee 4y agoThe redis wire protocol is simple enough where it's not very difficult. It's instructive to write it in C because there's a lot of translatable knowledge to other languages: read in a buffer, cast that buffer to the protocol-specific struct, read the various fields from memory, etc. Basic operations for a systems-level project like this.
- rickreynoldssf 4y agoThat's something I wish I had time for.
- Jolter 4y agoWhile I applaud the initiative and effort, I want to urge the author to consult a good proofreader. I’m only in chapter 2 yet I’ve already encountered enough friction in the form of poor English to put me off. I think the substance looks great, so it’s a bit wasteful to not walk the last mile and make it genuinely enjoyable to read by eliminating blatant errors like this one: “The Redis is an example of the server/client system.”
- m00dy 4y agoI would ask chatgpt to refactor his text rather than waiting him to do so... I think poor english as an excuse will be a thing of past.
- travisgriggs 4y agoInstead of saying “the singer sounds so autotuned” we’ll soon be whining “the text reads too autogpt” (I propose a pronunciation of “auto-jipped”).
- chimeracoder 4y ago> (I propose a pronunciation of “auto-jipped”). Probably want to pick another term, since that's a homonym of a racial slur.
- travisgriggs 4y agoSigh. I had to actually go look this up. Sometimes it's just easier not to talk, because all of the skeletons of old normalized words come to light. I truly was not trying to slur anyone, and apologize if my (unlucky) attempt at humor resonated sour for anyone.
- ly3xqhl8g9 4y agoI propose adding some Unicode sign like the double turnstile [1], ⊨ (ALT+8875), which can signify "is a semantic consequence of" or "is stronger than", after a polysemic word to imply that no racial/socially derogatory connotation is to be derived from the word in that phrase, for example, "the text reads too auto-jipped⊨" or "the master⊨ branch is fubar" in a Git context. Or perhaps an even better sign would be the negated double turnstile, ⊭, "to denote the statement 'does not entail'" [1], making it more explicit. Hence an example would look like "the text reads too auto-jipped(⊭racism)", which can be read "the text reads too auto-jipped and this word, auto-jipped, does not entail racism in this context". Ok, done, racism solved, your move David Guetta [2]. [1] https://en.wikipedia.org/wiki/Double_turnstile https://en.wikipedia.org/wiki/Double_turnstile [2] https://www.youtube.com/watch?v=dEI7oX0XxJw https://www.youtube.com/watch?v=dEI7oX0XxJw
- varispeed 4y agoOne of my pet peeves when I was learning programming is that I couldn't find any books that would teach how to write tests in real world. There was that chicken and egg problem then, when potential employers would skip my CV because I didn't have any TDD experience and I couldn't find anywhere how to learn this. Before I even learned that something like testing exist, I was so confident like "oh I could write something like that over the weekend, how come they needed months to do that?". Then my software would crash the first time someone other than me used it. Anyway - what I want to say that while this book sounds like a great idea, without showing TDD and how to write code so that it can be proven it works the way intended and that it can handle unhappy paths and edge cases, it won't teach someone trying to learn programming much and it doesn't actually stand over about million other books about programming that really just scratch the surface and don't show how to write production ready code. That's what is still missing on the market. It's almost like a well kept secret that only developers working at large corporations know. That skill was very very difficult to acquire.
- bilekas 4y agoWhile i have so many things i could say about your idea of what development means and the "only path" you seem to subscribe to. I think you're missing the point of the book. > it won't teach someone trying to learn programming much and it doesn't actually stand over about million other books about programming that really just scratch the surface and don't show how to write production ready code. It's not a book to teach people how to program, infact the author goes out of their way to mention only C and minor c++ has been used and that it may be beneficial for learning how to build out such a POC of redis, to DIY your own. This is not a book to hand over to an outsourcing company and expect production ready work. Nor was it described as such.
- x86hacker1010 4y agoI commented this above, but Distributed Services in Go does testing and low-level kafka-like implementation. It taught me so much, I'm probably going to reread at some point and implement in Rust.
- Zvez 4y agoI remember how tdd (together with scrum) was basically a religion in 'large corporations'. How a lot of people talked about 'the right way to do things'. Your comment made me smile nostalgically... But back to the topic. I'm pretty sure there are enough books about how to write tests. And more or less all engineers understand the value of testability and coverage (not necessary tdd!). At leas I wouldn't need a book for this. But books about building something closer to complex real world systems - that's a good stuff engineers would enjoy.
- rcarmo 4y agoI actually build minimal Redis clones in every new language or runtime, or when I want to explore threading models. It all started with https://github.com/rcarmo/miniredis https://github.com/rcarmo/miniredis (which I forked to add and experiment with pub/sub), and I just found myself doing it again and again because Redis is the quintessential network service: By implementing it, you learn about socket handling, event loop for a specific runtime, threading models, data representation, concurrency (if you want to do a multi-threaded version), etc. None of my "ports" are fully functional, but they all helped me sort out some of the above plus build tools, packaging, dependencies, etc. It's "hello world" for core cloud native microservices, if you will (and without having to do REST or JSON stuff).
- sillysaurusx 4y agoRESP3 is really nice too. It should probably be required reading. There’s a lot of interesting / subtle design choices worth studying. https://github.com/antirez/RESP3/blob/master/spec.md https://github.com/antirez/RESP3/blob/master/spec.md The best parts still haven’t been implemented.
- rcarmo 4y agoDepends on how much time I have. The Python tests work, so I’ve re-used them a couple of times. Sometimes I also explore the native testing.
- mLuby 4y agoI'm curious if you re-use the same tests across different languages? And is it actually on a network or are you pretending?
- helge5 4y agoI once built one in Swift 4, in part to see how multithreading via SwiftNIO and using copy-on-write datastructures would compare. It held up well against the C implementation, would be worth trying again against Swift 5.x. https://github.com/NozeIO/redi-s https://github.com/NozeIO/redi-s
- 4y ago
- zX41ZdbW 4y agoIt is nice to see more on "build your own database" topics. Although people constantly bug me on attempts to build their own ClickHouse. Someone is trying to do it with Apache Arrow and DataFusion. Folks from DuckDB are trying to build their own crippled version of ClickHouse. Friends from China doing it with their Apache Doris. InfluxDB is being rewritten to be closer to ClickHouse in an attempt to make it better and so on...
- joinbrita 4y agoIncase you're unaware, InfluxDB IOx (the rewrite) is built on Apache Arrow and Datafusion, and is the primary driver behind the project. I also think calling DuckDB a crippled version of Clickhouse disingenuous. That's like calling SQLite a cripped version of postgres. They have very different goals. That's also a bit like calling Clickhouse a build-your-own vectorwise/MonetDB because they did it first.
- tidwall 4y agoMaybe I'm missing something but the server application resulting from the source code in this book does not actually work with redis-cli or any Redis clients. It appears to be using a custom binary protocol, not valid RESP. I feel that the RESP protocol itself is the most challenging task of building ones own Redis.
- x86hacker1010 4y agoIt says on the site: > The end result is a mini Redis alike with only about 1200 lines of code. 1200 LoC seems low, but it illustrates many important aspects the book attempts to cover. The techniques and approaches used in the book are not exactly the same as the real Redis. Some are intentionally simplified, and some are chosen to illustrate a general topic. Readers can learn even more by comparing different approaches. I wouldn't emphasize the importance is understanding Redis per-se but the ideas around a system like Redis.
- kajaktum 4y agoWhat? I genuinely think RESP is the easy part. I used to use your redcon library to get bootstrapped but then rewrote it from scratch and it was really easy. Unfortunately Golang doesnt permit good abstraction sp i have to write the same thing over and over again.
- bkuehl 4y agoRedis is great, but sometimes it isn't always the best solution. We almost configured it, but instead implemented our own Cache web service and used the built in memory/cache management of that. Yes, it's only accessible via http but it's given us a lot of flexibility. We are primarily using it for caching of large datasets (hundreds of MBs). When service has to be restarted, it makes a call to get all the items it needs.
- Princesingh 4y ago[dead]
- victor106 4y ago> Redis could be considered one of the building blocks of modern computing Is Redis really that critical to modern computing?
- jjice 4y agoI'd say that for "modern computing", no. Modern web applications? I can see it. Redis rules and fixes a lot of problems, especially when you probably already have a server/cluster for something else. A building block of modern computing is a bit overzealous to me.
- KptMarchewa 4y agoDepends on how you define "critical" - humanity would survive without it - but a lot of large web systems depend on it one way or another.
- feichuan 4y agoI did something similar by building Redis in Rust a few years ago. It’s a really good learning and exercise to a lot of topics like network programming. https://github.com/huangjiahua/reredis https://github.com/huangjiahua/reredis
- Timpy 4y agoI saw this "Build Your Own Text Editor"[0] on HN a month or two ago, everyone was raving about it so I went through it and it really was fantastic. The learning experience was unparalleled. I'm a believer in the idea of "Build Your Own..." guides now, I hope this Redis guide is just as good as the kilo text editor. I'm definitely bookmarking this for a deep dive when the time is right. Any other top notch "Build Your Own" recommendations would be highly appreciated. [0] https://viewsourcecode.org/snaptoken/kilo/ https://viewsourcecode.org/snaptoken/kilo/
- mbwgh 4y agoThe site you linked (which is great by the way, thanks) seems to already be a great starting point: https://viewsourcecode.org/snaptoken/similarTutorials.html https://viewsourcecode.org/snaptoken/similarTutorials.html
- sarupbanskota 4y agoHere's a whole repository. https://github.com/codecrafters-io/build-your-own-x https://github.com/codecrafters-io/build-your-own-x
- pull_my_finger 4y agoPurchased this via Gumroad, but the epub version seems to not be formatted correctly. After about page 3 it decides the book is finished on my Kobo. Hopefully the author will get a fix out, I'm pretty excited to read it and pdfs are pretty hard to read on my device.
- weird_user 4y agoSorry to hear that. I don't have a Kobo so I can not troubleshoot this. Would a small-sized PDF be readable on your device?
- pull_my_finger 4y agok2pdfopt actually has an option for my model, cropping some of the excess margins helps a lot, but I will say epubcheck found some invalid markup in there, so it could be related to that.
- weird_user 4y agoI have updated the EPUB on Gumroad. Would you mind checking it again? I might add a PDF version with bigger a font size if it's not working.
- pull_my_finger 4y agoI tried and it still doesn't work. File shows 1/1 pages, which is blank.
- weird_user 4y agoAdded a cropped PDF with a larger font size. Hope that will work.
- akie 4y agoThis take on "building blocks of modern computing" is fairly opinionated though. They mention "NGINX, SQLite, PostgreSQL, Kafka, Linux kernel, etc." - of which I'd only consider NGINX and the Linux kernel (and Redis) as "building blocks". The others might be part of their own preferred stack, but if you mention Postgres, why not MySQL? If Kafka, why not RabbitMQ? But yeah, NGINX, Redis, and the Linux kernel are basically outside of discussion.
- chillfox 4y agoThey are all arbitrary (including nginx) in the sense that there are many stacks out there that don't use any of them. But I don't think it's a controversial selection as they are all extremely popular.
- cassepipe 4y agoThe book seems really interesting but I am quite poor so thanks for making a free web version available. I'll think of buying the epub when I earn a wage <3
- sandgiant 4y agoIn the first code snippet in "Introduction to Sockets", shouldn't it be `close(conn_fd)` in the while loop? That confused me a bit. Otherwise this looks interesting, thanks for sharing.
- danjc 4y agoKudos to the author for a very succinct, practical book. It would be nice to see a few chapters on high availability though - seems to me that failover and consistency guarantees are one of the really hard bits.
- avinassh 4y agoI am a big fan of Build Your Own X educational projects. Recently, I released a Go version of my build your own KV Store project. I have set up this project in TDD fashion with the tests. So, you start with simple functions, pass the tests, and the difficulty level goes up. There are hints if you get stuck (e.g. link). When all the tests pass, you will have written a persistent key-value store. go - https://github.com/avinassh/go-caskdb https://github.com/avinassh/go-caskdb python - https://github.com/avinassh/py-caskdb https://github.com/avinassh/py-caskdb
- Dowwie 4y agoIs anyone aware of an in-depth explanation about how key-value store persistence, such as Redis RDB, is built? A guide on compaction would be really interesting and useful.
- tr33house 4y agoI believe this book is such an explanation. At least for the key-value store persistence bit
- Dowwie 4y agoCould you point me to the chapter on persistence?
- fxnn 4y agoHighly recommending Martin Kleppmanns „Designing Data-Intensive Applications“ book (O‘Reilly). Covers virtually any topic around database implementation. Well written, big source of knowledge. Introduces concepts gradually and with many examples. Lots of references to actual software out there, and how they are done. KV persistence is covered in Chapter 3, right from the start. Redis is also mentioned as an example for an in-memory store with „weak durability by writing to disk asynchronously“.
- BossingAround 4y agoIs there a public repo we could send PRs/MRs/patches to? The prose has room for improvement and since the book is free, I'd love to send a couple of PRs your way.
- blacklion 4y agoAm I paranoid, or `container_of` macro is UB in modern C/C++, as it cast null pointer to some struct type and operates on members of this type?