7 ms·

I’m going to read the rest of this. I’m enjoying it. But, simultaneously, part II has me so triggered - it bears striking resemblance to repeated situations I’v

I’m going to read the rest of this. I’m enjoying it. But, simultaneously, part II has me so triggered - it bears striking resemblance to repeated situations I’ve encountered where the meaning and content of columns in a relational database were overloaded in varying degrees of heaviness (which is a practice I absolutely detest) - that I need to take a short break.

XorNot 2y ago
People do this with system hostnames a lot. And it's almost impossible to get them to stop: the hostname should either be a random UUID or a random name from a pronounceable list depending on scale (or a syllabic UUID thing). Because every other factor has one answer: you look up the other data you need in your CMDB. If that's too hard, you fix that so it's easy (DNS TXT records can be surprisingly useful here).
hermitcrab 2y ago
I manage a database for a small local charity. I have set it up so that only I can add, delete or change the column structure. If someone wants a change, they have to email me and convince me (they are fine about this BTW). I'm sure the database would be an utter disaster zone by now if everyone was allowed to change it.
- klysm 2y ago
  I think database schemas deserve to be protected with one’s life as the holy ground of the system. If the schema is fucked, everything else will be fucked too.
  - hermitcrab 2y ago
    With a database you can lock down the schema. In reality though, many data system are composed mainly of people emailing in Excel spreadsheets. Good luck enforcing any sort of schema there. My day job is writing a desktop/file-based ETL system. I have just added in a schema version feature to cover these sort of issues. It was one of the most requested features, because most people aren't able to control the schemas of the data they receive.
    - klysm 2y ago
      Yeah excel based data ingest is a pretty brutal problem to solve.
      - hermitcrab 2y ago
        We can automatically handle some schema drift if columns are renamed or reordered, or columns added or deleted. But if they are both renamed AND re-ordered, you are out of luck!
        bartread 2y ago
        If you detect a level of drift that you can't handle, this is the perfect opportunity to delegate that bit of work to an LLM, if it's a problem that you deal with regularly enough to feel the cost of it to your business. The latest generation of LLMs are pretty, actually very, good at this kind of situation, where somebody has renamed something - but kept some semblance of the meaning - and also moved it so a basic, or even a fuzzy, comparison might not be able to make a good match. But a model like GPT-4o-mini will eat a problem like this for breakfast, and it's now incredibly cheap to use it for this kind of thing as well.
  - chachacharge 2y ago
    Schemas require domain knowledge. When domain knowledge is unclear or lacks ownership, it can lead to a range of issues that impact both data integrity and system functionality. Things that screw this up in the financial world include: working in different countries, acquiring new branches, new hires, and leavers. And people who think they can insist the database schema be protected somehow. A manager told me to add the last reason, it wasnt my idea and makes little sense.

XorNot 2y ago

People do this with system hostnames a lot. And it's almost impossible to get them to stop: the hostname should either be a random UUID or a random name from a pronounceable list depending on scale (or a syllabic UUID thing). Because every other factor has one answer: you look up the other data you need in your CMDB. If that's too hard, you fix that so it's easy (DNS TXT records can be surprisingly useful here).

hermitcrab 2y ago

I manage a database for a small local charity. I have set it up so that only I can add, delete or change the column structure. If someone wants a change, they have to email me and convince me (they are fine about this BTW). I'm sure the database would be an utter disaster zone by now if everyone was allowed to change it.

klysm 2y ago
I think database schemas deserve to be protected with one’s life as the holy ground of the system. If the schema is fucked, everything else will be fucked too.
- hermitcrab 2y ago
  With a database you can lock down the schema. In reality though, many data system are composed mainly of people emailing in Excel spreadsheets. Good luck enforcing any sort of schema there. My day job is writing a desktop/file-based ETL system. I have just added in a schema version feature to cover these sort of issues. It was one of the most requested features, because most people aren't able to control the schemas of the data they receive.
  - klysm 2y ago
    Yeah excel based data ingest is a pretty brutal problem to solve.
    - hermitcrab 2y ago
      We can automatically handle some schema drift if columns are renamed or reordered, or columns added or deleted. But if they are both renamed AND re-ordered, you are out of luck!
      - bartread 2y ago
        If you detect a level of drift that you can't handle, this is the perfect opportunity to delegate that bit of work to an LLM, if it's a problem that you deal with regularly enough to feel the cost of it to your business. The latest generation of LLMs are pretty, actually very, good at this kind of situation, where somebody has renamed something - but kept some semblance of the meaning - and also moved it so a basic, or even a fuzzy, comparison might not be able to make a good match. But a model like GPT-4o-mini will eat a problem like this for breakfast, and it's now incredibly cheap to use it for this kind of thing as well.
- chachacharge 2y ago
  Schemas require domain knowledge. When domain knowledge is unclear or lacks ownership, it can lead to a range of issues that impact both data integrity and system functionality. Things that screw this up in the financial world include: working in different countries, acquiring new branches, new hires, and leavers. And people who think they can insist the database schema be protected somehow. A manager told me to add the last reason, it wasnt my idea and makes little sense.