4 ms·
I haven't read the article (yet), but this is a problem that I've had to deal with several times now recently. The most recent causing me to work 12+ hour days
by ilitirit 4y ago
I haven't read the article (yet), but this is a problem that I've had to deal with several times now recently. The most recent causing me to work 12+ hour days over the holidays.
In this case, there's a table that an ETL job loads data into several times a day. The problem is the original dev(s) thought it would be a good idea to add status columns to this table that the application can write to when it has finished processing the data ("InProgress", "Finalising", "Complete" etc). What does this mean now? If the load process takes longer than a minute or so, user interaction is blocked and ultimately deadlocked. This is extremely poor database (and system) design. The application never needs to update the "loaded" columns. So all modifiable columns should be moved to a separate table and updates either need to be serialised or written at the appropriate isolation levels. Now try to get any of these changes through during the Xmas period...
This is somewhat related to a refactor of another system I performed last year. The system had one Auditing microservice that wrote to a massive table with some keys, identifiers and metadata. Ignoring the fact they needed to write an auditing microservice to begin with, the problem was every time a new part of the system needed auditing the table needed to be modified. And every time you did a query, 70% of the columns were null. Again, this is very poor design. I deleted most of the columns except the identifiers and keys, separated the metadata into their own tables with a foreign key to the Audit table, and left-joined everything. Same result, but far less of a hassle in terms of maintenance and scalability.
[EDIT] I should probably end with this: Make your tables as big or small as they need to be considering your use cases.