14 ms·

The examples in the article are not wrong, but seem a bit simple. Rarely do I find the cost of the switch from not scalable to scalable to be close to zero. Mor

The examples in the article are not wrong, but seem a bit simple. Rarely do I find the cost of the switch from not scalable to scalable to be close to zero. More often, I find software built on a years-long pattern of these types of low-cost immediate-effect decisions. The result is something that can hardly be fixed at all.

wolletd 3y ago
To stay in the examples: when it came to the text file and database I thought. The issue usually isn't that you receive too many mails for the size of a text file, but that you probably want to store additional data soon, in a structured way. So you start hacking some field separators in your file, it's becoming a file format and switching to a database later is notably more work than it would have been at the start.
- citrin_ru 3y ago
  To store additional info one can use a tab separated file with filed names in a header - such file is easy to load into a database table later.
  - Karellen 3y ago
    I think that's part of the issue. Once you're using a text file, adding "one more feature" to the text file implementation is always easier than switching to a database. So you add your tab-separated value. Then you find out that in some cases the extra field you're adding can contain a tab. So now you need to be able to escape actual tabs in your tab-separated file - because once you have the tab-separated file, adding an escape is still easier than switching to a database. Cut to: 3 years later. You now have optional fields, a key-value implementation, locking, and two or three bits of code that work around the edges of the "proper" API to access the file directly, because at every stage that was easier than creating a "real" database backend. But now its harder to change than it would have been to do the right thing at the start, and the sunk cost fallacy means that everyone's brain is resistant to throwing away all the work that's been invested in the current solution even if they know it's the right thing to do, and even though whenever you onboard a new recruit they always fail a SAN check and take a couple of weeks to recover from the horror of your codebase before they can face it for reals. And implementing the next new feature in the text file still looks easier than replacing the whole implementation with a real database right now, so let's put that off for just one more release cycle, OK?
    - gregmac 3y ago
      What I've also experienced is a whole ecosystem has been built up in other departments in the organization (but outside of development) around the crazy CSV file formats, eg: Marketing uses it to trigger campaigns; Accounting uses it to reconcile live accounts; Support uses it to find accounts. So anything you do to make this better means you now have to deal with this ecosystem. At best, you're solving some pain point these groups have and they're happy about it, but in most cases you're probably forcing unexpected, extra work on them to "fix" something they perceive as not broken. At worst, you don't discover they're even doing this until after you've changed it, and now you just broke a production system you didn't even know existed. In some cases, the hack another department put in place will also have their own ecoystem built on top -- it's turtles all the way down. How bad this situation is directly corresponds to how your organization has grown and how siloed different departments are. And keep in mind we're talking about a fictional but easy-to-understand "emails in a CSV file" problem -- most of the real-world problems are significantly more complicated.
    - citrin_ru 3y ago
      I'm not saying that one should use a file as long as possible but a real DB from the start for anything the system writes is not always warranted. Yes, with a file you'll need to think about either escaping or data validation but there is an option to use json-per-line at a price of higher overhead (compare to TSV) and a JSON library will take care of serialization (including escaping). Locking is not required (on Linux/BSD) if the file is opened in the append mode and each line is written as a single write request (which is atomic). > But now its harder to change than it would have been to do the right thing at the start, and the sunk cost fallacy means that everyone's brain is resistant to throwing away all the work that's been invested With "proper" implementation suck cost fallacy can be even stronger because more effort was spend to make it right from the beginning. IMHO switching from a file to say ClickHouse would be as easy as switching from MySQL to ClickHouse (or my be even easier).
- lelanthran 3y ago
  I dunno about that. It's really easy to switch from a CSV format to a single table in a DB. Switching when your data is spread out over multiple CSV files is a lot more work (still less work than was expended into CRUD across multiple CSV files, though).

wolletd 3y ago

To stay in the examples: when it came to the text file and database I thought. The issue usually isn't that you receive too many mails for the size of a text file, but that you probably want to store additional data soon, in a structured way. So you start hacking some field separators in your file, it's becoming a file format and switching to a database later is notably more work than it would have been at the start.

citrin_ru 3y ago
To store additional info one can use a tab separated file with filed names in a header - such file is easy to load into a database table later.
- Karellen 3y ago
  I think that's part of the issue. Once you're using a text file, adding "one more feature" to the text file implementation is always easier than switching to a database. So you add your tab-separated value. Then you find out that in some cases the extra field you're adding can contain a tab. So now you need to be able to escape actual tabs in your tab-separated file - because once you have the tab-separated file, adding an escape is still easier than switching to a database. Cut to: 3 years later. You now have optional fields, a key-value implementation, locking, and two or three bits of code that work around the edges of the "proper" API to access the file directly, because at every stage that was easier than creating a "real" database backend. But now its harder to change than it would have been to do the right thing at the start, and the sunk cost fallacy means that everyone's brain is resistant to throwing away all the work that's been invested in the current solution even if they know it's the right thing to do, and even though whenever you onboard a new recruit they always fail a SAN check and take a couple of weeks to recover from the horror of your codebase before they can face it for reals. And implementing the next new feature in the text file still looks easier than replacing the whole implementation with a real database right now, so let's put that off for just one more release cycle, OK?
  - gregmac 3y ago
    What I've also experienced is a whole ecosystem has been built up in other departments in the organization (but outside of development) around the crazy CSV file formats, eg: Marketing uses it to trigger campaigns; Accounting uses it to reconcile live accounts; Support uses it to find accounts. So anything you do to make this better means you now have to deal with this ecosystem. At best, you're solving some pain point these groups have and they're happy about it, but in most cases you're probably forcing unexpected, extra work on them to "fix" something they perceive as not broken. At worst, you don't discover they're even doing this until after you've changed it, and now you just broke a production system you didn't even know existed. In some cases, the hack another department put in place will also have their own ecoystem built on top -- it's turtles all the way down. How bad this situation is directly corresponds to how your organization has grown and how siloed different departments are. And keep in mind we're talking about a fictional but easy-to-understand "emails in a CSV file" problem -- most of the real-world problems are significantly more complicated.
  - citrin_ru 3y ago
    I'm not saying that one should use a file as long as possible but a real DB from the start for anything the system writes is not always warranted. Yes, with a file you'll need to think about either escaping or data validation but there is an option to use json-per-line at a price of higher overhead (compare to TSV) and a JSON library will take care of serialization (including escaping). Locking is not required (on Linux/BSD) if the file is opened in the append mode and each line is written as a single write request (which is atomic). > But now its harder to change than it would have been to do the right thing at the start, and the sunk cost fallacy means that everyone's brain is resistant to throwing away all the work that's been invested With "proper" implementation suck cost fallacy can be even stronger because more effort was spend to make it right from the beginning. IMHO switching from a file to say ClickHouse would be as easy as switching from MySQL to ClickHouse (or my be even easier).
lelanthran 3y ago
I dunno about that. It's really easy to switch from a CSV format to a single table in a DB. Switching when your data is spread out over multiple CSV files is a lot more work (still less work than was expended into CRUD across multiple CSV files, though).