3 ms·

It gets hairier than this. For example, in Postgres, the place where all the metadata is stored that the DB uses in internal business logic is not regular C st

It gets hairier than this. For example, in Postgres, the place where all the metadata is stored that the DB uses in internal business logic is not regular C structs, but as rows in special tables (relations). Like say you want to look up the physical file location of a table, using its ID (something Postgres functions have to do internally at various points). It does this by reading from a relation. It's sort of like a compiler that compiles itself. It can't always have been that way, so I have to wonder at what point the codebase swapped over to doing it this way.

derefr 4y ago
> It can't always have been that way Sure it could have. 1. Start with a library version of just the storage engine; 2. write some CLI tools that use that library to direct-manipulate table heap files, batch import/export between them and serialization formats, etc — y'know, like these ones for BDB (which was the SQLite of its day, and contemporary to Postgres at Berkeley) — https://docs.oracle.com/cd/E17076_05/html/api_reference/C/utilities.html https://docs.oracle.com/cd/E17076_05/html/api_reference/C/ut... ; 3. define the core system-schema tables for the DBMS, and the required static data to go in them; 4. write a script that uses the CLI tools to bootstrap a "database" (directory of loose heap files) using that schema — call it initdb(1); 5. and now start writing the rest of the DBMS, which can now take "there is an initialized cluster with a populated system schema" as an axiomatic assumption, opening it and reading important state and configuration from said schema files from almost the first line of code. (I have no special knowledge of PG's earliest versions, so this might be a just-so story; but today, initdb(1) is a separate tool with separate code that doesn't come from some library that also gets compiled into the PG postmaster. And that's not the sort of design you would move to, if you had started out with a design where the postmaster could bootstrap an empty cluster directory into existence on startup.)
- gavinray 4y ago
  Huh thanks for the explanation, I've always wondered how that worked.

derefr 4y ago

> It can't always have been that way Sure it could have. 1. Start with a library version of just the storage engine; 2. write some CLI tools that use that library to direct-manipulate table heap files, batch import/export between them and serialization formats, etc — y'know, like these ones for BDB (which was the SQLite of its day, and contemporary to Postgres at Berkeley) — https://docs.oracle.com/cd/E17076_05/html/api_reference/C/utilities.html https://docs.oracle.com/cd/E17076_05/html/api_reference/C/ut... ; 3. define the core system-schema tables for the DBMS, and the required static data to go in them; 4. write a script that uses the CLI tools to bootstrap a "database" (directory of loose heap files) using that schema — call it initdb(1); 5. and now start writing the rest of the DBMS, which can now take "there is an initialized cluster with a populated system schema" as an axiomatic assumption, opening it and reading important state and configuration from said schema files from almost the first line of code. (I have no special knowledge of PG's earliest versions, so this might be a just-so story; but today, initdb(1) is a separate tool with separate code that doesn't come from some library that also gets compiled into the PG postmaster. And that's not the sort of design you would move to, if you had started out with a design where the postmaster could bootstrap an empty cluster directory into existence on startup.)

gavinray 4y ago
Huh thanks for the explanation, I've always wondered how that worked.