6 ms·

Rich Hickey has some great talks on YouTube. If you are a software engineer: Build a simple database, a simple interpreter, and a simple web application (back

Rich Hickey has some great talks on YouTube. If you are a software engineer: Build a simple database, a simple interpreter, and a simple web application (backend and frontend). Read about “distributed systems”, “devops”, “Conway’s Game Of Life”, and “fractals”. Read about these technologies/techniques (in whatever order is comfortable), how they work, their pros and cons, why they were created, how they compare to alternatives: “TCP”, “Open Sound Control”, “Plan9”, “REST”, “Lisp”, “Erlang”, “Smalltalk”, “Forth”, “Datomic”, “event sourcing”, “reactive programming”, “communicating sequential processes”, “APL”/“J”/“K”, “Ansible”, “jq”, “graph databases”, “Apache Kafka” The idea is not to become an expert in any of these, but to digest the wisdom that lies in their design and surrounding literature. Each was created after lots of careful thought by brilliant people about how to build sustainable systems.

ioddly 9y ago
Where does one get started building a simple database?
- bordercases 9y ago
  Can you develop a file format plus API that is ACID compliant in C?
  - elvinyung 9y ago
    Careful with that word ("ACID"), it's ridiculously easy to get screwed by the filesystem arbitrarily lying about things like the order of system calls: https://queue.acm.org/detail.cfm?id=2801719 https://queue.acm.org/detail.cfm?id=2801719
    - jmalicki 9y ago
      The file system does not lie, it just does not provide guarantees of ordering.
- weavie 9y ago
  I found Martin Kleppmans book - Designing Data-Intensive Applications (http://martin.kleppmann.com/ http://martin.kleppmann.com/) to be a great primer.
  - goostavos 9y ago
    An absolutely _fantastic_ book. It was also one of the scariest books I've read recently. Made me realize just how much I don't know (and don't know that I don't know!) about the failure modes of distributed systems.
  - gothy 9y ago
    This is a must read book even if you're not building a database :)
- deleted 9y ago
  [deleted]
- tayo42 9y ago
  Something like a kv store with a btree or sstable. https://github.com/antirez/otree https://github.com/antirez/otree simple example I think
bauerd 9y ago
Huh? I currently work with some of the technologies you mention, but fail to see the connection to systems theory. Can you elaborate why e.g. jq or Erlang are relevant here?
- bordercases 9y ago
  Most of those technologies are known for either how they handle complexity, or the coordination between asynchronous actors with shared resources
  - gaius 9y ago
    Uhhh what? jq is basically an updated sed. One input stream, one output stream, no awareness of anything else
    - joncampbelldev 9y ago
      Perhaps they meant it as a good example of the type of component to build systems out of, simple input->output. Unsure though.
    - JshWright 9y ago
      You don't see how a simple input/output streaming model would be helpful in understanding systems theory and design? There are a _lot_ of systems out there that tie together various components with tools that behave a lot like jq (or literally are jq...)
      - gaius 9y ago
        Yes this is how every Unix command line utility works. There is nothing unique about jq in this regard, and there is nothing particularly “systems thinking” about any of them. You can and people have used them for decades without doing any special “systems thinking”.
        deleted 9y ago
        [deleted]
blueprint 9y ago
> how to build sustainable systems Well, except for reactive programming
tazjin 9y ago
Missing the thing that has had the single biggest impact on how I personally think about programming: Learning a properly typed language (e.g. Haskell). I'd also substitute Ansible with Nix/NixOS because the conceptual takeaways are much greater there.
thinkersilver 9y ago
This is how it worked out for me. I tried to build the system myself and note the abstractions, edge case. It didn't make me an expert in all things but gave me the right questions to ask. As you do enough of these your expertise will crystallise in a particular domain. Started by building my own build system, monitoring solutions, taking at stab orchestration problems, graph databases etc. The problem statement would be super simple like how do I install my application on 100 servers? And I will start off with bash, tar, ssh and work my way up. The goal isn't too finish. You want to discover the problems first hand, decide upfront how you would evaluate an acceptable solution, attempt a naive solution, and then look at an open source system, note the family of algorithms + data structures then loop back and move onto the next set of problems in the domain. Ideally the system you're building is distributed. Mostly I work from the outside going in and counterintuitively I start with the two non-functional requirements, as questions: how fast do you want it to go, secondly how badly can the system fail. I find that I'm peculiar in this approach. This won't work for everyone. It has for me because it's allowed me to build a handy library of interrogative and evaluative heuristics and develop strong understanding system architectures and design

ioddly 9y ago

Where does one get started building a simple database?

bordercases 9y ago
Can you develop a file format plus API that is ACID compliant in C?
- elvinyung 9y ago
  Careful with that word ("ACID"), it's ridiculously easy to get screwed by the filesystem arbitrarily lying about things like the order of system calls: https://queue.acm.org/detail.cfm?id=2801719 https://queue.acm.org/detail.cfm?id=2801719
  - jmalicki 9y ago
    The file system does not lie, it just does not provide guarantees of ordering.
weavie 9y ago
I found Martin Kleppmans book - Designing Data-Intensive Applications (http://martin.kleppmann.com/ http://martin.kleppmann.com/) to be a great primer.
- goostavos 9y ago
  An absolutely _fantastic_ book. It was also one of the scariest books I've read recently. Made me realize just how much I don't know (and don't know that I don't know!) about the failure modes of distributed systems.
- gothy 9y ago
  This is a must read book even if you're not building a database :)
deleted 9y ago
[deleted]
tayo42 9y ago
Something like a kv store with a btree or sstable. https://github.com/antirez/otree https://github.com/antirez/otree simple example I think

bauerd 9y ago

Huh? I currently work with some of the technologies you mention, but fail to see the connection to systems theory. Can you elaborate why e.g. jq or Erlang are relevant here?

bordercases 9y ago
Most of those technologies are known for either how they handle complexity, or the coordination between asynchronous actors with shared resources
- gaius 9y ago
  Uhhh what? jq is basically an updated sed. One input stream, one output stream, no awareness of anything else
  - joncampbelldev 9y ago
    Perhaps they meant it as a good example of the type of component to build systems out of, simple input->output. Unsure though.
  - JshWright 9y ago
    You don't see how a simple input/output streaming model would be helpful in understanding systems theory and design? There are a _lot_ of systems out there that tie together various components with tools that behave a lot like jq (or literally are jq...)
    - gaius 9y ago
      Yes this is how every Unix command line utility works. There is nothing unique about jq in this regard, and there is nothing particularly “systems thinking” about any of them. You can and people have used them for decades without doing any special “systems thinking”.
      - deleted 9y ago
        [deleted]

blueprint 9y ago

> how to build sustainable systems Well, except for reactive programming

tazjin 9y ago

Missing the thing that has had the single biggest impact on how I personally think about programming: Learning a properly typed language (e.g. Haskell). I'd also substitute Ansible with Nix/NixOS because the conceptual takeaways are much greater there.

thinkersilver 9y ago

This is how it worked out for me. I tried to build the system myself and note the abstractions, edge case. It didn't make me an expert in all things but gave me the right questions to ask. As you do enough of these your expertise will crystallise in a particular domain. Started by building my own build system, monitoring solutions, taking at stab orchestration problems, graph databases etc. The problem statement would be super simple like how do I install my application on 100 servers? And I will start off with bash, tar, ssh and work my way up. The goal isn't too finish. You want to discover the problems first hand, decide upfront how you would evaluate an acceptable solution, attempt a naive solution, and then look at an open source system, note the family of algorithms + data structures then loop back and move onto the next set of problems in the domain. Ideally the system you're building is distributed. Mostly I work from the outside going in and counterintuitively I start with the two non-functional requirements, as questions: how fast do you want it to go, secondly how badly can the system fail. I find that I'm peculiar in this approach. This won't work for everyone. It has for me because it's allowed me to build a handy library of interrogative and evaluative heuristics and develop strong understanding system architectures and design