4 ms·

> But know that making a real production database tends to be harder than it looks, and so tends to be all-consuming, and a team that depends on a custom databa

by rewmie 3y ago

> But know that making a real production database tends to be harder than it looks, and so tends to be all-consuming, and a team that depends on a custom database tends to become a database team rather than an anything else team. The original statement does not include any caveat or constraint, and is an absolute statement that does not depend on specifics. The statement is not "Never write a custom database as a subproject of any other project". The original statement is "Never write a database." Like in any conceivable scenario, including a project whose main goal is to write a database. I also feel that this sort of clickbait advice contrasts heavily with the history of SQLite.

ncphillips 3y ago
The tweet doesn’t but the context of the tweet is a conversation about building a custom database to power a business.
- coldtea 3y ago
  Well, the company I worked at did, and it worked out just fine. Not even that many people involved, like 3-4, the main architect and coder had never done a DB before (but is a genius anyway). For the scope and within that domain (not relational, heavy writes), the performance smokes anything out there. In production for over a decade now.
derefr 3y ago
It is a statement directed at a readership of — and therefore assuming it will only be read by — application developers, rather than systems programmers; and so the context of "Never write a custom database as a subproject of any other project" is implicit. Also, you're reacting to an incomplete quote. The tweet is: > "Never write a database. Even if you want to, even if you think you should. Resist. Never write a database. Unless you have to write a database. But you don't." There are precisely two cases where you "have to write a database": 1. when you want to learn how databases are architected, and educational material on the ground is wholly inadequate (which it is!); and 2. when you discover, under a production workload, that no existing DBMS sits in exactly the right part of DBMS configuration-space to meet your scaling needs — and nor can any existing DBMS even be modified to reach that point in DBMS configuration-space — and nor can you modify the design of your application to scale in a different way; such that the only possible way to address your scaling challenges is to write an entirely novel DBMS with an entirely novel architecture. I can only think of two times #2 has happened in recent memory: Amazon with Dynamo (the core of S3 and DynamoDB); and Google with Dremel (the core of BigQuery.) Both projects have resulted in many research papers, and later copycat DBMSes (Dynamo → Cassandra, Riak, BigTable, etc; Dremel → Redshift, Snowflake, etc.) If you don't think your DB architecture is novel enough to result in that kind of response, then it's probably not something that requires "writing a database" — it's probably instead something you can do as a hack on top of some existing database. (Think: the way Citus/Timescale/Greenplum build on Postgres.)
- benj111 3y ago
  >Unless you have to write a database. But you don't. The quote you quoted says you don't need to write a database, so if you want to agree, you also have to agree that Amazon and Google didn't need to write databases. And learning would seem to be covered by the "want to" or "should"
  - derefr 3y ago
    Again, pedantry: Amazon and Google aren't people; they aren't reading advice on the Internet. So the "you" in this statement will never apply to them. It only applies to individuals reading the statement. And, indeed, "you" do not need to write a database. Some company might very well need to write a database; and therefore coerce one or more of its employees into writing a database. But, like the statement says, the employees should resist. Very likely the company is wrong about needing to write a database, and resistance will help it discover that. However, every once in a while, even after everyone resists for months/years, it will turn out that the company really does need to write a database, and so — still mostly against their will — the individuals employed there will set out to write a database. (And once they've "written a database", and it turned out to actually be the best course of action for the company, then the advice no longer applies — because they're no longer considering "writing a database" at that point, but rather maintaining an existing database, that they happened to be the creators of.) > And learning would seem to be covered by the "want to" or "should" Instrumental vs. terminal goals. If I want to do X, then I need to do Y to achieve X. If I want to learn how databases work, then I need to write a database, because there's no other good way.
    - benj111 3y ago
      >If I want to do X, then I need to do Y to achieve X. And you accuse me of pedantry? This then comes under need then?!? >Amazon and Google aren't people No, but the people within them make decisions, ie decisions to write a database. So the 'you' would be directed at the person in the organisation who decided to write the database.
      - 3y ago

by rewmie 3y ago

ncphillips 3y ago

The tweet doesn’t but the context of the tweet is a conversation about building a custom database to power a business.

coldtea 3y ago
Well, the company I worked at did, and it worked out just fine. Not even that many people involved, like 3-4, the main architect and coder had never done a DB before (but is a genius anyway). For the scope and within that domain (not relational, heavy writes), the performance smokes anything out there. In production for over a decade now.

derefr 3y ago

It is a statement directed at a readership of — and therefore assuming it will only be read by — application developers, rather than systems programmers; and so the context of "Never write a custom database as a subproject of any other project" is implicit. Also, you're reacting to an incomplete quote. The tweet is: > "Never write a database. Even if you want to, even if you think you should. Resist. Never write a database. Unless you have to write a database. But you don't." There are precisely two cases where you "have to write a database": 1. when you want to learn how databases are architected, and educational material on the ground is wholly inadequate (which it is!); and 2. when you discover, under a production workload, that no existing DBMS sits in exactly the right part of DBMS configuration-space to meet your scaling needs — and nor can any existing DBMS even be modified to reach that point in DBMS configuration-space — and nor can you modify the design of your application to scale in a different way; such that the only possible way to address your scaling challenges is to write an entirely novel DBMS with an entirely novel architecture. I can only think of two times #2 has happened in recent memory: Amazon with Dynamo (the core of S3 and DynamoDB); and Google with Dremel (the core of BigQuery.) Both projects have resulted in many research papers, and later copycat DBMSes (Dynamo → Cassandra, Riak, BigTable, etc; Dremel → Redshift, Snowflake, etc.) If you don't think your DB architecture is novel enough to result in that kind of response, then it's probably not something that requires "writing a database" — it's probably instead something you can do as a hack on top of some existing database. (Think: the way Citus/Timescale/Greenplum build on Postgres.)

benj111 3y ago
>Unless you have to write a database. But you don't. The quote you quoted says you don't need to write a database, so if you want to agree, you also have to agree that Amazon and Google didn't need to write databases. And learning would seem to be covered by the "want to" or "should"
- derefr 3y ago
  Again, pedantry: Amazon and Google aren't people; they aren't reading advice on the Internet. So the "you" in this statement will never apply to them. It only applies to individuals reading the statement. And, indeed, "you" do not need to write a database. Some company might very well need to write a database; and therefore coerce one or more of its employees into writing a database. But, like the statement says, the employees should resist. Very likely the company is wrong about needing to write a database, and resistance will help it discover that. However, every once in a while, even after everyone resists for months/years, it will turn out that the company really does need to write a database, and so — still mostly against their will — the individuals employed there will set out to write a database. (And once they've "written a database", and it turned out to actually be the best course of action for the company, then the advice no longer applies — because they're no longer considering "writing a database" at that point, but rather maintaining an existing database, that they happened to be the creators of.) > And learning would seem to be covered by the "want to" or "should" Instrumental vs. terminal goals. If I want to do X, then I need to do Y to achieve X. If I want to learn how databases work, then I need to write a database, because there's no other good way.
  - benj111 3y ago
    >If I want to do X, then I need to do Y to achieve X. And you accuse me of pedantry? This then comes under need then?!? >Amazon and Google aren't people No, but the people within them make decisions, ie decisions to write a database. So the 'you' would be directed at the person in the organisation who decided to write the database.
    - 3y ago