10 ms·

Things I learned from building a production database

551 points by dangoldin 5y ago

deleted 5y ago
[deleted]
greenSunglass 5y ago
Is it what people do on a sabbatical?
nicoburns 5y ago
This is a really excellent list. Some of it is specific to the kind of low level work this person was doing, but a lot of it isn’t. I would love to work on a team with the author of this blog post.
krisrm 5y ago
Very good list. [22] Avoid using real-time for correctness guarantees or comparing clocks across machines unless you have (and understand) error bounds on the clock. I shudder to think of the hours spent head-scratching that went into this Lesson Learned :)
- dmoy 5y ago
  For [22] taken to an extreme level, read the original Spanner paper from 2012 (?): https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c... It goes into a lot of detail on this
turbocon 5y ago
I really think this is a great list however I think one of the major factors why this project succeeded isn't listed but is mentioned in the intro. > We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for project success in my experience, especially with complex projects like this. That small team can have a much tighter communication loop and all keep a single unified picture of what the end goal is. When the team grows this single unified picture is no longer feasible so it has to be broken down into sections, which is a good thing, but if the core isn't setup so that the sections are sufficiently decoupled things fall apart quickly.
- maheshba 5y ago
  That's a great point; I should definitely have included it in the list! I think new projects (especially ones that seek to innovate) have two phases: in the first phase, everyone's trying to kill it; in the second phase, everyone's trying to grow it. The kill-it phase is actually valuable for exactly the reason you outlined: it gives a small team the chance to create something highly coherent. (The grow-it phase is sometimes worse than the kill-it phase, but that's an entirely different story).
  - aurelius83 5y ago
    Can you explain more what you mean by the kill-it phase? Why is everyone trying to kill it?
    - BatteryMountain 5y ago
      Have you tried to build something (lets say, a new kind of storage engine that is not SQL-based) and bring it up to other developers or people at your workplace (never mind business guys)? What is/was their response? Perhaps kill it with fire. If not due to valid technical reasons, financial reasons might seal the deal for it. And I totally understand the impulse to kill it. Do not unleash it onto the world unless you have thought about your projects long term future (something php & js ignored to do at the time it seems). Thus you end up with tons of pet projects that are either sensibly kept private/ununleashed or because people fear the judgement of others (aka the kill it impulse). I'd say the nature of the beast is that entrenchment/reinforcement of existing tools/patterns and mental models tend to stick around as they are safe, so a ton of innovation never occurs as someone might not bother to follow a certain path. It's the same with water flowing into existing grooves/cracks in rocks, instead of making new grooves, unless forced to.
    - vishnugupta 5y ago
      I have been part of platform and product teams. So I think I sort of know what the author means by "kill-it" phase. When new platform is being worked upon it's extremely difficult to get product teams to adopt. The PMs on the product teams view a new platform as a distraction because it seemingly doesn't help them launch new features. Worse it's taking the engineers who could be working on features to integrate with this new platform. Typically during planning phase not too many oppose a new platform. However, when it comes to integrating (or adopting it) then everyone suddenly becomes conservative; no one wants to be the first mover, take the risk to be early adopter. It could also stem from product teams' prior experience of having burnt their fingers trying to integrate with a new platform. I have seen quite a few platform teams get abandoned after everyone signed up. The platform team need to put in an extra effort to get product teams adopt.
nzgrover 5y ago
Nowhere does it say what an "IC" is. Is it so hard to provided the Full Name(FN) and the abbrevation the first time you use it?
- otras 5y ago
  IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service). (Vouched for this flagged comment after seeing the other comment)
  - bebop 5y ago
    Thanks, I had never heard this term until I started working at my current employer. It seems like bay area lingo is assumed to be understood by everyone, even though it may not be used in the industry as widely as some folks assume.
  - Avalaxy 5y ago
    > IC is a commonly used acronym for Individual Contributor Its not though. Outside of hacker news I have literally never heard this term before, nor does it make any sense. If I'd start talking about "Individual Contributors" to my colleagues they would all be confused wth I'm talking about and wondering what's wrong with me. Who even coined this stupid term, and why do we pretend we need a new name for "engineers"?
    - notacoward 5y ago
      It has been around in industry for at least twenty years. Facebook is a bit unusual in using it as an actual job title, but it's by no means new. I remember talking about how DEC had a healthy IC track separate from going into management, and DEC hasn't even existed as a separate company since 1998.
      - 5y ago
tyingq 5y ago
There was a comment asking what an "IC" was, but the comment was flagged and now dead. Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".
- turbocon 5y ago
  Yes, it means "individual contributor", pretty common terminology at least in the FAANG world. He mentioned this was a project at Facebook/Meta.
- ajcp 5y ago
  "Interface directly with customer ICs" I've only ever seen it to mean "individual contributor", but then this would make it odd/bad advice. There should only be 1 "customer". Sure, that customer may have a whole team that actually needs the product, and may provide them for clarification or what not, but to avoid unrecorded/incorrect handshake agreements, specs, or pet requirements that have nothing to do with the initial product ordered there should only be 1 POC for the product unless otherwise specified.
  - Twisol 5y ago
    Despite all that, I still find it absolutely necessary to get in the same headspace with someone on the other "side". We can and should always float any outcomes to the primary points of contact on either side, but a POC is a primary point of contact, not the only point of contact. Put differently, interfacing directly with customer ICs is good and necessary, so long as it's done with the go-ahead and oversight of each POC.
  - dbt00 5y ago
    You should keep a record of what you're doing and why, but getting clear and accurate feedback from the real decision makers who have control over the technical success of your product is essential.
  - mst 5y ago
    When multiple teams are using the thing, talking to at least one developer on each of those teams is going to surface things you won't find out anywhere else. I agree that an individual user shouldn't be driving roadmap stuff directly, but getting their perspective, pet peeves, war stories etc. is still valuable advisory input to the overall process.
sjansen 5y ago
Although the author says his advice is for people "building new infra at large companies", there's certainly a number of points that apply to a larger audience. For example: > Measurement is a means, not an end. Reminds me of a lesson I learned early in my career. Bad things happen when you lose sight of your purpose and focus just on measurement. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
cmg 5y ago
Assuming I'm not the only one who's never heard of Delos or Cubby: https://engineering.fb.com/2019/06/06/data-center-engineering/delos/ https://engineering.fb.com/2019/06/06/data-center-engineerin...
- taddevries 5y ago
  Thanks for this link. I could not tell from the post what this product was or what issues it specifically solved.
- titanomachy 5y ago
  Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?
  - rad_gruchalski 5y ago
    ZooKeeper is modeled after Chubby.
  - boulos 5y ago
    The paper describing Chubby was published in 2006 [1]. [1] https://research.google/pubs/pub27897/ https://research.google/pubs/pub27897/
  - gravypod 5y ago
    From what I understand Chubby is basically etcd. So, Chubby, very few. The concepts behind it? Most of the people who work low level with kube.
  - mellavora 5y ago
    The author says [40] Keep track of research in your space. Soon you’ll have a shorthand with your ICs that enables super-fast communication: “what if we try that thing from projectX? And combine it with the technique in projectY?”. so by that learning it is not at all weird that they would refer to infra at an alternative company
throwthrow564 5y ago
> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations. I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.
- krschultz 5y ago
  We use the same principle on some of the frameworks I have worked on, it pushes you to make the API more explicit and keep the context localized. That makes codemods easier. This can sometimes lead to a more verbose / less magical API, but IMO at scale the tradeoff is worth it.
- scv119 5y ago
  The first engineer on the Delos team here. This design decision gave us an edge: we delivered the first production cluster in 8 months by layering over existing systems ; and we smoothly migrated all underlying implementations in the following 2 years without our customer aware of it (0 downtime). This design decision is well captured by the OSDI’20 Virtual Consensus in Delos paper.
  - throwthrow564 5y ago
    Very interesting. How did you deal with the issue where the old API had some feature X that wasn't supported in the new API but some customers depended on feature X?
    - scv119 5y ago
      I guess that a hard question for user facing API: you need to support both the old and the new features during the migration. In our case, this design principle mostly applied to our internal APIs. Particularly, we designed our internal sub components (consensus protocol, particularly) in a way we can swap it without any downtime.
bertr4nd 5y ago
I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations? And oh by the way shipping this major product in under a year?
- mym1990 5y ago
  I think you assess your skills and what you bring, figure out the deficiencies, and then find people who can fill in other gaps(and hopefully have some overlapping skills as well). At the end of the day having half of the time and intellect is a great place to be, that is much better than having none of the time and intellect.
- bigodanktime 5y ago
  I think you are assuming all these things are independent of one another. I think deep engagement with customers directs your problem statement. Build the smallest possible thing to test the idea you are proposing. If you are trying to publish than its a requirement to be reading papers in general, you need to be able to argue why this idea you are testing is orthogonal to other ideas being proposed. As time goes on you just scan important conferences in your field and get better at knowing which papers you need to skim versus deeply understand. Edit: Also intelligence is an over-rated metric, persistence is far more important when it comes to these hard problems.
  - rmbyrro 5y ago
    I agree, persistence, discipline and ordering of daily activities. It's very easy to get lost in day to day mess.
- physicles 5y ago
  Yeah I know what you mean, I'm in awe of the author's ability to create the perfect environment for his team to succeed. Part of the key, I think, is that he was dedicated to caring about every factor that might affect success (customers, project management, design, and the other categories from the list), which left his team free to execute on the technical aspects. I also think we might be underestimating how much one could do with one year, a relatively well-specified problem, and a team of three super motivated and intelligent coders.
scrubs 5y ago
Wow! One hell of a good list. And well explained. You bet I book marked it. Heck even the 1-connected comment was worth the read. Because it's true. I'm dealing with that right now. +5 and another reason I love HN. Now, if it you would have made it 43 items, a nice prime number, then I would be jealous to the point of angry (that's a riff on John Daily).
- macintux 5y ago
  Speaking as a John Daily, but clearly not the one you're referencing: who dat?
- vishnugupta 5y ago
  > if it you would have made it 43 items, a nice prime number, Something tells me the author deliberately stopped at 42nd item. https://hitchhikers.fandom.com/wiki/42 https://hitchhikers.fandom.com/wiki/42
throwaway984393 5y ago
"[1] Keep your customers happy" - It's hard to actually have a team that respects this lesson. Even if one or two people on the team believe in this rule, others won't, and if they have equal say in the final product, it means the customers will suffer. Dogfooding is really the only way to fight this. Become your own most demanding customer. "[4] But realize that customers may not express what they really need; don’t take requirements at face-value, instead spend the time to understand their use case in detail. Read their code." - Yep - Users Are Liars. They will lie right to your face, about requirements, deadlines, use cases, functionality, costs, etc, for no reason at all. Never believe anything a user says. And obviously, never tell them you think this. "[9] If you get good and/or aligned managers, be as understanding, supportive, and accommodating as you can. If you don’t get such managers…" - Become the shadow manager; Team Lead, Lead Developer, Delegate Architect, etc. Ask the manager if you can get "hands-on experience" with various tasks like: running ceremonies, taking meeting notes, coordinating work, planning, resolving blockers. Reach out to other teams in private to resolve problems without involving any management, and be careful that the people you reach out to don't widen scope to their managers. Hide problems and bubble up accomplishments to the bad manager. You'll get none of the credit, but the manager won't have the opportunity to make everyone's lives bad. "[10] Make your project robust to re-orgs." - Of these, the most resilient I've seen are the ones that promise the moon, and then deftly delay everyone's realization that it doesn't live up to the hype until they're all deeply bought in. Don't make the mistake of trying to get an executive to buy in, because they'll be gone in 3 years and everyone will ditch your project. "[15] Design APIs with migration to new implementations as a first-class consideration" - Yes, yes, yes!! Literally all technology is replaced eventually, most of it in 3-5 years. Plan the exit strategy at the outset. "[21] Have the right number of abstractions" - The correct number of abstractions is the minimum possible. Every abstraction should be a solution to a problem. You have to define the problem that the abstraction was meant to solve, and then ask yourself if there is a simpler way to solve that problem.
nyanpasu64 5y ago
Is a diff Facebook's equivalent of a feature branch or PR?
- gnulinux 5y ago
  A "diff" is the output of cmd "diff", also more famously "git diff". This is the same format you can apply diffs with "git apply x.diff"
- andrewmunsell 5y ago
  Isn't diff just a standard, generic term for a change difference? (Like in git: https://git-scm.com/docs/git-diff https://git-scm.com/docs/git-diff) Someone correct me if I'm wrong, but if my memory is correct, I was under the impression that a pull request was a term coined in the early Github days specific to their PR feature.
  - xyzzyz 5y ago
    Yes, diff is a generic name, after a Unix tool with the same name. However, it hadn’t been the most popular name for the self contained unit of changes, that you would send to others to review or apply — the “patch” was. In Linux git workflow, you would use the “git format-patch” command to generate patch to send to others for review and to apply it to their branch.
    - mst 5y ago
      However, even if you're using an email based workflow you're still making a request to somebody to pull your changes into their tree, and I'm reasonably sure 'pull request' as a term for that came from the linux workflow. GH more provided features to formalise a common concept (which, even where I've disagreed with them on the details of how they formalised it, has definitely been a net win overall).
- packetslave 5y ago
  A "diff" in Phabricator is roughly equivalent to a PR, yes. A single point-in-time collection of, well, diffs against head/master that get code-reviewed and committed as a unit. If you were at Google, you'd call it a "CL"
_pastel 5y ago
> [16] Design as a team; implement as individuals. This is pithy. I'm curious - does anyone here follow this philosophy? How do you actually design as a team? A meeting for every single design decision? I'm used to the team finding general alignment on the problem in meetings, but then a single person coming up with the design, writing a doc, and circulating it for comments.
- kumarvvr 5y ago
  I guess the intent is to say, decide on the API interfaces and endpoints beforehand and implement to that spec. Which is understandable, as you have to integrate anyway and doing it later is more painful than doing it earlier.
- Hermitian909 5y ago
  Within a high level design there's often a lot of room for variations in the implementation such as code structure; though this freedom decreases significantly if you have a wide enough audience to need to consider Hyrum's Law. Concretely, let's say I'm designing a term-rewriting system. The program will be inspecting an abstract syntax tree (AST) for certain patterns, and if it detects those patterns, rewriting them. (e.g. if we see the AST for "1 + 1" rewrite to "2"). I might write up a document on the architecture for the system, and a list of rewriting rules I need. How those rules are written can be left up to individuals on my team as long as it fits into the architecture, has the required functionality, and is of reasonable quality.
- notapenny 5y ago
  I think your example could be seen as designing as a team, its just one person who does the writing, but everyone who needed to be involved was involved. On a smaller scale I've worked in teams where when someone picked up a story, for certain cases we would first do a rough outline of the implementation and then take that to one or two devs from the team, just to talk about how to tackle the problem. Not a meeting, just five minutes of "here's what I think would work". Its hard but good alignment before writing code really helps with finding the errors in a design.
- bad_username 5y ago
swuecho 5y ago
Interface directly with customer ICs ==== what's IC?
- detaro 5y ago
  https://news.ycombinator.com/item?id=29324416 https://news.ycombinator.com/item?id=29324416
  - swuecho 5y ago
    thanks!
physicles 5y ago
> [10] Make your project robust to re-orgs. A company management hierarchy is inherently fragile (a tree is a 1-connected graph, after all); socialize the project continuously with managers who might take over in the future. Do whatever it takes to make sure that manager churn does not result in unfair career outcomes for ICs. > [17] For storage systems, bias heavily in the beginning towards consistency and durability rather than availability; these are harder to measure and harder to fix if broken. Because availability is easier to measure, there will be external pressure to prioritize it first; push back. It's kind of crazy how well the author is able to move from the very top to the very bottom of the technical project management stack.
- ketzo 5y ago
  This is, to me, the definition of a senior software engineer. Awareness across every single part of the process of making software. Not expertise, necessarily, but enough knowledge to know what to be on the lookout for.
  - polskibus 5y ago
    More of a team leader at least, non-IC. Some Senior ICs deliberately don't want any part in politics, just want a task and want to get it done (see for example advice on HN about avoiding meetings and politics at all costs).
    - penguin_booze 5y ago
      One ill effect of not deciding to participate is that you'll be subjected to rules that which you play no part in establishing. I think it's about the balance to strike between being well and content with what's going around, and needing to exercise control to change the environment.
      - machinesbuddy 5y ago
        Well said! Same as a country/society.
bbulkow 5y ago
There is a critical element the post doesn't state: a successful small project has (has) very limited scope of customers and use patterns. That is where complexity in writing databases arises: limited scope databases are far easier.
- vincnetas 5y ago
  Its on the list : [2] Be careful to have the right number of customers (in the beginning, just one) and the right customers (whose requirements allow you to build out key technology); and grow that number carefully.
vishnugupta 5y ago
Lot of good, actionable learnings there. I liked this one in particular. > [26] In a transparent codebase with quick review cycles, APIs will leak implementation details unless you gate-keep. It's tempting to make quick fixes to APIs but please take it slow. Think through existing customers as well as how could new ones will use it. Also, something I learned while building/maintaining public APIs was to always clearly document (along with examples) the context in which an API should (ideally) be used and how it combines with other APIs to enable common use cases.
vincnetas 5y ago
This list is amazing for most software that i worked on. Dropping "database" from list name would only remove couple of items from the list. Most of the items are applicable much broader than just for database projects. Favourited for sure for future references.
alex64 5y ago
This is a good list, but I'm trying to understand the following in better detail: > [12] Be conservative on APIs and liberal with implementations. > [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail). The word implementation is thrown out a lot. What does implementation mean in these quotes? The databases used (ie. dependency injection), the endpoints (ie. /user, /user/123), or something else? I'm trying to visualize this in my head and am having trouble.
- buro9 5y ago
  Once people use your APIs you cannot trivially modify the bits being used or take things away. A large API likely has complexity, security implications, messy transaction boundaries, etc... keep it small and simple, think of UX of an API, make it intuitive and capable. That's the conservative bit... now the liberal bit. Make clients, make one in Python and look at date parsing, make one in Go and consider struct re-use and how to deal with sets/arrays of things, make one in JavaScript and consider how easy it it to re-use parts of the code. If you have an endpoint that returns a list of things... is it supposed to be paginated over? How easy is that? Is it an infinite list? How easy is that? What would a client want to achieve? Is it possible? When you produce prototype applications in a variety of languages the majority of the small inconsistencies and issues are super obvious. And remember, once the API is out there it's hard to change it and you can't remove things easily... so discovering all of this early helps you get it more right than wrong the first time round.
- vishnugupta 5y ago
  To take Stripe's example, the API is as documented here[1]. Essentially a documented contract between a platform provider and their clients. Implementation is things like, as you noted, DB used, programming language, and so on. By "Be conservative on APIs" they mean be careful what you promise to your clients as communicated and documented through APIs. Once promised it's almost impossible to go back; you will lose your clients' trust and piss them off. In case of Stripe they are committing to support idempotent requests[2] which is sort of a big deal. And I don't expect them to drop that support in any of their future APIs because I imagine hundreds of millions of $$ business would be relying on that idempotency support. [1] https://stripe.com/docs/api https://stripe.com/docs/api [2] https://stripe.com/docs/api/idempotent_requests https://stripe.com/docs/api/idempotent_requests
dalanmiller 5y ago
> [12] Be conservative on APIs and liberal with implementations. Curious how people interpret this? Is it: * Attempt to keep our API small while also... * thinking widely about the way it's used? Or how does this work in practice?
- bottled_poe 5y ago
  I’d interpret this to mean keeping you API specification basic to minimise backwards compatibility problems and reduce future refactoring difficulty. Don’t have special methods for unique cases, stick to restful principles where possible, keep data structures simple, etc.
  - rmbyrro 5y ago
    I'd add give enough freedom for users to accomplish what they need from a small set of API abstractions. Many times, this shifts burden to users. It's always more convenient to have one API which does exactly what you need. But the idea here is to keep the API simple and stable in a phase where instability is already high. Perhaps, after it gets mature, it'll make sense to incorporate common use cases into the API structure for convenience.
lysecret 5y ago
The main thing i learned in my first production db work is that you pay for every bit of complexity you introduce. As a noob you are somehow much more afraid to not "model the data right" so you tend to create much to many tables much to many n to m relationships you make everything a string a uuid a relationship, never use bools etc. But then you have to iterate and live with those systems and then you pay and every bit of complexity makes everything harder. To give you some simple examples: 1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join. In the end, if you can keep it simple even if you don't model the data "perfectly". But of course sometimes the complexity is really necessary. To know when is being a good Backend / Db Engineer.
- epr 5y ago
  > 1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join. Whether or not these were good decisions depends on the use cases of course, but papering over db implementation details like this with a view is pretty straightforward, no?
- bob1029 5y ago
  This is the whole point of the database: model the data exactly the way the business needs it, so you can have easy 1:1 conversations. Then, build some convenience views or other mapping layers on top of this data for use in reporting systems, etc. Don't screw up the physical tables just because you don't like the knowledge that they contain in certain contexts. Facts are facts. If you don't like what you are seeing, change your query or filter the result set. If a customer record needs to be retained after deletion per biz requirements, then an IsDeleted fact is 100% the right path. Both of your points enumerated above are trivially solved with views.
Kalimoto 5y ago
So nothing about how to build a database. And someone who didn't mind ethically working for Facebook. It's not bad advice but nothing new either.
e12e 5y ago
In this kind of structure, what role does managers fill? It sounds like ICs can manage themselves, and the project manager manages the project... What's left for managers to do?
- ryathal 5y ago
  Manage bureaucracy around people: leave, hiring, firing, and promoting. They also generally need to be involved in things that may be more project management depending on the authority project mangers have in organizations: procuring new tools, updating senior management, escalation for cross team issues/communication, filter out noise from external forces, etc.

551 points by dangoldin 5y ago

deleted 5y ago

[deleted]

greenSunglass 5y ago

Is it what people do on a sabbatical?

nicoburns 5y ago

This is a really excellent list. Some of it is specific to the kind of low level work this person was doing, but a lot of it isn’t. I would love to work on a team with the author of this blog post.

krisrm 5y ago

Very good list. [22] Avoid using real-time for correctness guarantees or comparing clocks across machines unless you have (and understand) error bounds on the clock. I shudder to think of the hours spent head-scratching that went into this Lesson Learned :)

dmoy 5y ago
For [22] taken to an extreme level, read the original Spanner paper from 2012 (?): https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c... It goes into a lot of detail on this

turbocon 5y ago

I really think this is a great list however I think one of the major factors why this project succeeded isn't listed but is mentioned in the intro. > We hit production with a 3-person team in less than a year; and subsequently scaled the team to 30+ engineers spanning multiple sub-teams Starting with a small skilled team to build the core and actively plan for a larger team at some point is absolutely critical for project success in my experience, especially with complex projects like this. That small team can have a much tighter communication loop and all keep a single unified picture of what the end goal is. When the team grows this single unified picture is no longer feasible so it has to be broken down into sections, which is a good thing, but if the core isn't setup so that the sections are sufficiently decoupled things fall apart quickly.

maheshba 5y ago
That's a great point; I should definitely have included it in the list! I think new projects (especially ones that seek to innovate) have two phases: in the first phase, everyone's trying to kill it; in the second phase, everyone's trying to grow it. The kill-it phase is actually valuable for exactly the reason you outlined: it gives a small team the chance to create something highly coherent. (The grow-it phase is sometimes worse than the kill-it phase, but that's an entirely different story).
- aurelius83 5y ago
  Can you explain more what you mean by the kill-it phase? Why is everyone trying to kill it?
  - BatteryMountain 5y ago
    Have you tried to build something (lets say, a new kind of storage engine that is not SQL-based) and bring it up to other developers or people at your workplace (never mind business guys)? What is/was their response? Perhaps kill it with fire. If not due to valid technical reasons, financial reasons might seal the deal for it. And I totally understand the impulse to kill it. Do not unleash it onto the world unless you have thought about your projects long term future (something php & js ignored to do at the time it seems). Thus you end up with tons of pet projects that are either sensibly kept private/ununleashed or because people fear the judgement of others (aka the kill it impulse). I'd say the nature of the beast is that entrenchment/reinforcement of existing tools/patterns and mental models tend to stick around as they are safe, so a ton of innovation never occurs as someone might not bother to follow a certain path. It's the same with water flowing into existing grooves/cracks in rocks, instead of making new grooves, unless forced to.
  - vishnugupta 5y ago
    I have been part of platform and product teams. So I think I sort of know what the author means by "kill-it" phase. When new platform is being worked upon it's extremely difficult to get product teams to adopt. The PMs on the product teams view a new platform as a distraction because it seemingly doesn't help them launch new features. Worse it's taking the engineers who could be working on features to integrate with this new platform. Typically during planning phase not too many oppose a new platform. However, when it comes to integrating (or adopting it) then everyone suddenly becomes conservative; no one wants to be the first mover, take the risk to be early adopter. It could also stem from product teams' prior experience of having burnt their fingers trying to integrate with a new platform. I have seen quite a few platform teams get abandoned after everyone signed up. The platform team need to put in an extra effort to get product teams adopt.

nzgrover 5y ago

Nowhere does it say what an "IC" is. Is it so hard to provided the Full Name(FN) and the abbrevation the first time you use it?

otras 5y ago
IC is a commonly used acronym (at least at some of the big tech companies) for Individual Contributor. For instance, when the author says "customer ICs", I imagine that means the internal engineers that would be using the database they created (individual contributors on other teams who would be customers/clients of this service). (Vouched for this flagged comment after seeing the other comment)
- bebop 5y ago
  Thanks, I had never heard this term until I started working at my current employer. It seems like bay area lingo is assumed to be understood by everyone, even though it may not be used in the industry as widely as some folks assume.
- Avalaxy 5y ago
  > IC is a commonly used acronym for Individual Contributor Its not though. Outside of hacker news I have literally never heard this term before, nor does it make any sense. If I'd start talking about "Individual Contributors" to my colleagues they would all be confused wth I'm talking about and wondering what's wrong with me. Who even coined this stupid term, and why do we pretend we need a new name for "engineers"?
  - notacoward 5y ago
    It has been around in industry for at least twenty years. Facebook is a bit unusual in using it as an actual job title, but it's by no means new. I remember talking about how DEC had a healthy IC track separate from going into management, and DEC hasn't even existed as a separate company since 1998.
    - 5y ago

tyingq 5y ago

There was a comment asking what an "IC" was, but the comment was flagged and now dead. Guessing by the context, I'm pretty sure it means "individual contributor", or "person that is not a manager".

turbocon 5y ago
Yes, it means "individual contributor", pretty common terminology at least in the FAANG world. He mentioned this was a project at Facebook/Meta.
ajcp 5y ago
"Interface directly with customer ICs" I've only ever seen it to mean "individual contributor", but then this would make it odd/bad advice. There should only be 1 "customer". Sure, that customer may have a whole team that actually needs the product, and may provide them for clarification or what not, but to avoid unrecorded/incorrect handshake agreements, specs, or pet requirements that have nothing to do with the initial product ordered there should only be 1 POC for the product unless otherwise specified.
- Twisol 5y ago
  Despite all that, I still find it absolutely necessary to get in the same headspace with someone on the other "side". We can and should always float any outcomes to the primary points of contact on either side, but a POC is a primary point of contact, not the only point of contact. Put differently, interfacing directly with customer ICs is good and necessary, so long as it's done with the go-ahead and oversight of each POC.
- dbt00 5y ago
  You should keep a record of what you're doing and why, but getting clear and accurate feedback from the real decision makers who have control over the technical success of your product is essential.
- mst 5y ago
  When multiple teams are using the thing, talking to at least one developer on each of those teams is going to surface things you won't find out anywhere else. I agree that an individual user shouldn't be driving roadmap stuff directly, but getting their perspective, pet peeves, war stories etc. is still valuable advisory input to the overall process.

sjansen 5y ago

Although the author says his advice is for people "building new infra at large companies", there's certainly a number of points that apply to a larger audience. For example: > Measurement is a means, not an end. Reminds me of a lesson I learned early in my career. Bad things happen when you lose sight of your purpose and focus just on measurement. https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

cmg 5y ago

Assuming I'm not the only one who's never heard of Delos or Cubby: https://engineering.fb.com/2019/06/06/data-center-engineering/delos/ https://engineering.fb.com/2019/06/06/data-center-engineerin...

taddevries 5y ago
Thanks for this link. I could not tell from the post what this product was or what issues it specifically solved.
titanomachy 5y ago
Chubby, not cubby. And I also thought it was weird that the author referred to a piece of Google internal infrastructure to describe a piece of Facebook internal infrastructure... how many non-{X,G}ooglers know what Chubby is?
- rad_gruchalski 5y ago
  ZooKeeper is modeled after Chubby.
- boulos 5y ago
  The paper describing Chubby was published in 2006 [1]. [1] https://research.google/pubs/pub27897/ https://research.google/pubs/pub27897/
- gravypod 5y ago
  From what I understand Chubby is basically etcd. So, Chubby, very few. The concepts behind it? Most of the people who work low level with kube.
- mellavora 5y ago
  The author says [40] Keep track of research in your space. Soon you’ll have a shorthand with your ICs that enables super-fast communication: “what if we try that thing from projectX? And combine it with the technique in projectY?”. so by that learning it is not at all weird that they would refer to infra at an alternative company

throwthrow564 5y ago

> [15] Design APIs with migration to new implementations as a first-class consideration; custom migrations are huge time-sinks and sources of unreliability. Every major API should have a single CLI-driven call for switching implementations. I've never actually thought about doing this as an initial design principle. I wonder how it materially affects the design.

krschultz 5y ago
We use the same principle on some of the frameworks I have worked on, it pushes you to make the API more explicit and keep the context localized. That makes codemods easier. This can sometimes lead to a more verbose / less magical API, but IMO at scale the tradeoff is worth it.
scv119 5y ago
The first engineer on the Delos team here. This design decision gave us an edge: we delivered the first production cluster in 8 months by layering over existing systems ; and we smoothly migrated all underlying implementations in the following 2 years without our customer aware of it (0 downtime). This design decision is well captured by the OSDI’20 Virtual Consensus in Delos paper.
- throwthrow564 5y ago
  Very interesting. How did you deal with the issue where the old API had some feature X that wasn't supported in the new API but some customers depended on feature X?
  - scv119 5y ago
    I guess that a hard question for user facing API: you need to support both the old and the new features during the migration. In our case, this design principle mostly applied to our internal APIs. Particularly, we designed our internal sub components (consensus protocol, particularly) in a way we can swap it without any downtime.

bertr4nd 5y ago

I’m reading this list, and it’s a great list but it also makes me sort of depressed. I think I have the combination of time and intellect to achieve maybe half of what’s proposed here. What do you do when you can’t possibly achieve writing throwaway prototypes to get design input while engaging deeply with your customers and keeping up with the research in your field while designing your APIs for 2+ implementations? And oh by the way shipping this major product in under a year?

mym1990 5y ago
I think you assess your skills and what you bring, figure out the deficiencies, and then find people who can fill in other gaps(and hopefully have some overlapping skills as well). At the end of the day having half of the time and intellect is a great place to be, that is much better than having none of the time and intellect.
bigodanktime 5y ago
I think you are assuming all these things are independent of one another. I think deep engagement with customers directs your problem statement. Build the smallest possible thing to test the idea you are proposing. If you are trying to publish than its a requirement to be reading papers in general, you need to be able to argue why this idea you are testing is orthogonal to other ideas being proposed. As time goes on you just scan important conferences in your field and get better at knowing which papers you need to skim versus deeply understand. Edit: Also intelligence is an over-rated metric, persistence is far more important when it comes to these hard problems.
- rmbyrro 5y ago
  I agree, persistence, discipline and ordering of daily activities. It's very easy to get lost in day to day mess.
physicles 5y ago
Yeah I know what you mean, I'm in awe of the author's ability to create the perfect environment for his team to succeed. Part of the key, I think, is that he was dedicated to caring about every factor that might affect success (customers, project management, design, and the other categories from the list), which left his team free to execute on the technical aspects. I also think we might be underestimating how much one could do with one year, a relatively well-specified problem, and a team of three super motivated and intelligent coders.

scrubs 5y ago

Wow! One hell of a good list. And well explained. You bet I book marked it. Heck even the 1-connected comment was worth the read. Because it's true. I'm dealing with that right now. +5 and another reason I love HN. Now, if it you would have made it 43 items, a nice prime number, then I would be jealous to the point of angry (that's a riff on John Daily).

macintux 5y ago
Speaking as a John Daily, but clearly not the one you're referencing: who dat?
vishnugupta 5y ago
> if it you would have made it 43 items, a nice prime number, Something tells me the author deliberately stopped at 42nd item. https://hitchhikers.fandom.com/wiki/42 https://hitchhikers.fandom.com/wiki/42

throwaway984393 5y ago

"[1] Keep your customers happy" - It's hard to actually have a team that respects this lesson. Even if one or two people on the team believe in this rule, others won't, and if they have equal say in the final product, it means the customers will suffer. Dogfooding is really the only way to fight this. Become your own most demanding customer. "[4] But realize that customers may not express what they really need; don’t take requirements at face-value, instead spend the time to understand their use case in detail. Read their code." - Yep - Users Are Liars. They will lie right to your face, about requirements, deadlines, use cases, functionality, costs, etc, for no reason at all. Never believe anything a user says. And obviously, never tell them you think this. "[9] If you get good and/or aligned managers, be as understanding, supportive, and accommodating as you can. If you don’t get such managers…" - Become the shadow manager; Team Lead, Lead Developer, Delegate Architect, etc. Ask the manager if you can get "hands-on experience" with various tasks like: running ceremonies, taking meeting notes, coordinating work, planning, resolving blockers. Reach out to other teams in private to resolve problems without involving any management, and be careful that the people you reach out to don't widen scope to their managers. Hide problems and bubble up accomplishments to the bad manager. You'll get none of the credit, but the manager won't have the opportunity to make everyone's lives bad. "[10] Make your project robust to re-orgs." - Of these, the most resilient I've seen are the ones that promise the moon, and then deftly delay everyone's realization that it doesn't live up to the hype until they're all deeply bought in. Don't make the mistake of trying to get an executive to buy in, because they'll be gone in 3 years and everyone will ditch your project. "[15] Design APIs with migration to new implementations as a first-class consideration" - Yes, yes, yes!! Literally all technology is replaced eventually, most of it in 3-5 years. Plan the exit strategy at the outset. "[21] Have the right number of abstractions" - The correct number of abstractions is the minimum possible. Every abstraction should be a solution to a problem. You have to define the problem that the abstraction was meant to solve, and then ask yourself if there is a simpler way to solve that problem.

nyanpasu64 5y ago

Is a diff Facebook's equivalent of a feature branch or PR?

gnulinux 5y ago
A "diff" is the output of cmd "diff", also more famously "git diff". This is the same format you can apply diffs with "git apply x.diff"
andrewmunsell 5y ago
Isn't diff just a standard, generic term for a change difference? (Like in git: https://git-scm.com/docs/git-diff https://git-scm.com/docs/git-diff) Someone correct me if I'm wrong, but if my memory is correct, I was under the impression that a pull request was a term coined in the early Github days specific to their PR feature.
- xyzzyz 5y ago
  Yes, diff is a generic name, after a Unix tool with the same name. However, it hadn’t been the most popular name for the self contained unit of changes, that you would send to others to review or apply — the “patch” was. In Linux git workflow, you would use the “git format-patch” command to generate patch to send to others for review and to apply it to their branch.
  - mst 5y ago
    However, even if you're using an email based workflow you're still making a request to somebody to pull your changes into their tree, and I'm reasonably sure 'pull request' as a term for that came from the linux workflow. GH more provided features to formalise a common concept (which, even where I've disagreed with them on the details of how they formalised it, has definitely been a net win overall).
packetslave 5y ago
A "diff" in Phabricator is roughly equivalent to a PR, yes. A single point-in-time collection of, well, diffs against head/master that get code-reviewed and committed as a unit. If you were at Google, you'd call it a "CL"

_pastel 5y ago

> [16] Design as a team; implement as individuals. This is pithy. I'm curious - does anyone here follow this philosophy? How do you actually design as a team? A meeting for every single design decision? I'm used to the team finding general alignment on the problem in meetings, but then a single person coming up with the design, writing a doc, and circulating it for comments.

kumarvvr 5y ago
I guess the intent is to say, decide on the API interfaces and endpoints beforehand and implement to that spec. Which is understandable, as you have to integrate anyway and doing it later is more painful than doing it earlier.
Hermitian909 5y ago
Within a high level design there's often a lot of room for variations in the implementation such as code structure; though this freedom decreases significantly if you have a wide enough audience to need to consider Hyrum's Law. Concretely, let's say I'm designing a term-rewriting system. The program will be inspecting an abstract syntax tree (AST) for certain patterns, and if it detects those patterns, rewriting them. (e.g. if we see the AST for "1 + 1" rewrite to "2"). I might write up a document on the architecture for the system, and a list of rewriting rules I need. How those rules are written can be left up to individuals on my team as long as it fits into the architecture, has the required functionality, and is of reasonable quality.
notapenny 5y ago
I think your example could be seen as designing as a team, its just one person who does the writing, but everyone who needed to be involved was involved. On a smaller scale I've worked in teams where when someone picked up a story, for certain cases we would first do a rough outline of the implementation and then take that to one or two devs from the team, just to talk about how to tackle the problem. Not a meeting, just five minutes of "here's what I think would work". Its hard but good alignment before writing code really helps with finding the errors in a design.
bad_username 5y ago

swuecho 5y ago

Interface directly with customer ICs ==== what's IC?

detaro 5y ago
https://news.ycombinator.com/item?id=29324416 https://news.ycombinator.com/item?id=29324416
- swuecho 5y ago
  thanks!

physicles 5y ago

> [10] Make your project robust to re-orgs. A company management hierarchy is inherently fragile (a tree is a 1-connected graph, after all); socialize the project continuously with managers who might take over in the future. Do whatever it takes to make sure that manager churn does not result in unfair career outcomes for ICs. > [17] For storage systems, bias heavily in the beginning towards consistency and durability rather than availability; these are harder to measure and harder to fix if broken. Because availability is easier to measure, there will be external pressure to prioritize it first; push back. It's kind of crazy how well the author is able to move from the very top to the very bottom of the technical project management stack.

ketzo 5y ago
This is, to me, the definition of a senior software engineer. Awareness across every single part of the process of making software. Not expertise, necessarily, but enough knowledge to know what to be on the lookout for.
- polskibus 5y ago
  More of a team leader at least, non-IC. Some Senior ICs deliberately don't want any part in politics, just want a task and want to get it done (see for example advice on HN about avoiding meetings and politics at all costs).
  - penguin_booze 5y ago
    One ill effect of not deciding to participate is that you'll be subjected to rules that which you play no part in establishing. I think it's about the balance to strike between being well and content with what's going around, and needing to exercise control to change the environment.
    - machinesbuddy 5y ago
      Well said! Same as a country/society.

bbulkow 5y ago

There is a critical element the post doesn't state: a successful small project has (has) very limited scope of customers and use patterns. That is where complexity in writing databases arises: limited scope databases are far easier.

vincnetas 5y ago
Its on the list : [2] Be careful to have the right number of customers (in the beginning, just one) and the right customers (whose requirements allow you to build out key technology); and grow that number carefully.

vishnugupta 5y ago

Lot of good, actionable learnings there. I liked this one in particular. > [26] In a transparent codebase with quick review cycles, APIs will leak implementation details unless you gate-keep. It's tempting to make quick fixes to APIs but please take it slow. Think through existing customers as well as how could new ones will use it. Also, something I learned while building/maintaining public APIs was to always clearly document (along with examples) the context in which an API should (ideally) be used and how it combines with other APIs to enable common use cases.

vincnetas 5y ago

This list is amazing for most software that i worked on. Dropping "database" from list name would only remove couple of items from the list. Most of the items are applicable much broader than just for database projects. Favourited for sure for future references.

alex64 5y ago

This is a good list, but I'm trying to understand the following in better detail: > [12] Be conservative on APIs and liberal with implementations. > [18] Maintain multiple implementations in test for APIs; compare results between them. The cost is worth it (it will help with correctness, and also prevent leakage of implementation detail). The word implementation is thrown out a lot. What does implementation mean in these quotes? The databases used (ie. dependency injection), the endpoints (ie. /user, /user/123), or something else? I'm trying to visualize this in my head and am having trouble.

buro9 5y ago
Once people use your APIs you cannot trivially modify the bits being used or take things away. A large API likely has complexity, security implications, messy transaction boundaries, etc... keep it small and simple, think of UX of an API, make it intuitive and capable. That's the conservative bit... now the liberal bit. Make clients, make one in Python and look at date parsing, make one in Go and consider struct re-use and how to deal with sets/arrays of things, make one in JavaScript and consider how easy it it to re-use parts of the code. If you have an endpoint that returns a list of things... is it supposed to be paginated over? How easy is that? Is it an infinite list? How easy is that? What would a client want to achieve? Is it possible? When you produce prototype applications in a variety of languages the majority of the small inconsistencies and issues are super obvious. And remember, once the API is out there it's hard to change it and you can't remove things easily... so discovering all of this early helps you get it more right than wrong the first time round.
vishnugupta 5y ago
To take Stripe's example, the API is as documented here[1]. Essentially a documented contract between a platform provider and their clients. Implementation is things like, as you noted, DB used, programming language, and so on. By "Be conservative on APIs" they mean be careful what you promise to your clients as communicated and documented through APIs. Once promised it's almost impossible to go back; you will lose your clients' trust and piss them off. In case of Stripe they are committing to support idempotent requests[2] which is sort of a big deal. And I don't expect them to drop that support in any of their future APIs because I imagine hundreds of millions of $$ business would be relying on that idempotency support. [1] https://stripe.com/docs/api https://stripe.com/docs/api [2] https://stripe.com/docs/api/idempotent_requests https://stripe.com/docs/api/idempotent_requests

dalanmiller 5y ago

> [12] Be conservative on APIs and liberal with implementations. Curious how people interpret this? Is it: * Attempt to keep our API small while also... * thinking widely about the way it's used? Or how does this work in practice?

bottled_poe 5y ago
I’d interpret this to mean keeping you API specification basic to minimise backwards compatibility problems and reduce future refactoring difficulty. Don’t have special methods for unique cases, stick to restful principles where possible, keep data structures simple, etc.
- rmbyrro 5y ago
  I'd add give enough freedom for users to accomplish what they need from a small set of API abstractions. Many times, this shifts burden to users. It's always more convenient to have one API which does exactly what you need. But the idea here is to keep the API simple and stable in a phase where instability is already high. Perhaps, after it gets mature, it'll make sense to incorporate common use cases into the API structure for convenience.

lysecret 5y ago

The main thing i learned in my first production db work is that you pay for every bit of complexity you introduce. As a noob you are somehow much more afraid to not "model the data right" so you tend to create much to many tables much to many n to m relationships you make everything a string a uuid a relationship, never use bools etc. But then you have to iterate and live with those systems and then you pay and every bit of complexity makes everything harder. To give you some simple examples: 1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join. In the end, if you can keep it simple even if you don't model the data "perfectly". But of course sometimes the complexity is really necessary. To know when is being a good Backend / Db Engineer.

epr 5y ago
> 1. We don't want to delete old data so we just use a "deleted" column and do that instead. Now every single time you have to filter out those results. 2. You just made this thing a relationship now every time you have to join. Whether or not these were good decisions depends on the use cases of course, but papering over db implementation details like this with a view is pretty straightforward, no?
bob1029 5y ago
This is the whole point of the database: model the data exactly the way the business needs it, so you can have easy 1:1 conversations. Then, build some convenience views or other mapping layers on top of this data for use in reporting systems, etc. Don't screw up the physical tables just because you don't like the knowledge that they contain in certain contexts. Facts are facts. If you don't like what you are seeing, change your query or filter the result set. If a customer record needs to be retained after deletion per biz requirements, then an IsDeleted fact is 100% the right path. Both of your points enumerated above are trivially solved with views.

Kalimoto 5y ago

So nothing about how to build a database. And someone who didn't mind ethically working for Facebook. It's not bad advice but nothing new either.

e12e 5y ago

In this kind of structure, what role does managers fill? It sounds like ICs can manage themselves, and the project manager manages the project... What's left for managers to do?

ryathal 5y ago
Manage bureaucracy around people: leave, hiring, firing, and promoting. They also generally need to be involved in things that may be more project management depending on the authority project mangers have in organizations: procuring new tools, updating senior management, escalation for cross team issues/communication, filter out noise from external forces, etc.