5 ms·

With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient me

With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be recorded, all migrations should be reversible and tested, and all data should be backed up at least nightly. Amazing how companies pulling in millions often won't do more than backup every week or so and say three hail Marys.

theLiminator 3y ago
And then gdpr fucks that up that nice clean concept completely
- TeMPOraL 3y ago
  GDPR only affects data you shouldn't have or keep in the first place.
  - detaro 3y ago
    No, it really doesn't. E.g. deletion of data about a contract or account that just expired (or expired <mandatory-retention-period months ago>) is data you were totally fine/required to have, but can't be deletion that can be rolled back long-term.
  - SgtBastard 3y ago
    Section 17 (Right to be forgotten) 1a and 1b both refer to situations where there was a legitimate need to process and/or keep a subjects data. https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/ Implementing this as a rollback-able delete will not be compliant.
- callalex 3y ago
  If it’s so hard to be a good steward of data, don’t collect it in the first place.
  - jimbokun 3y ago
    It’s not that GDPR is overly onerous to implement. It’s simply that GDPR is fundamentally incompatible with unlimited undo.
- brazzy 3y ago
  Seems to me that it shouldn't be too difficult to extend an "unlimited undo" system to have specific time-based limits for some data.
thesz 3y ago
Blockchains do not delete records, etc. Also, you have to pay pretty much real money to put your record there. So we can have a good aproximation of what can happen with your proposal even if you need to pay extra for it. [1] https://ycharts.com/indicators/ethereum_chain_full_sync_data_size https://ycharts.com/indicators/ethereum_chain_full_sync_data... Ethereum [1] routinely accumulates around 300G per year and routinely hits over one terabyte of data to sync. Remember, this is the size of the data to sync/transmit, not size of the data that is stored, which we may safely assume to be several times more, because of indices, etc. Also, your proposal makes two tier database system: one that maintains current consistent view of the state and another for log purposes. The logging system needs high throughput storage with key range read request, which makes it, well, another pretty much fully fledged database (SELECT...GROUP BY...ORDER BY is needed). The reason nobody does what you described because it is really prohibitive in storage space aspect and really is quite complex - a database on top of another database.

theLiminator 3y ago

And then gdpr fucks that up that nice clean concept completely

TeMPOraL 3y ago
GDPR only affects data you shouldn't have or keep in the first place.
- detaro 3y ago
  No, it really doesn't. E.g. deletion of data about a contract or account that just expired (or expired <mandatory-retention-period months ago>) is data you were totally fine/required to have, but can't be deletion that can be rolled back long-term.
- SgtBastard 3y ago
  Section 17 (Right to be forgotten) 1a and 1b both refer to situations where there was a legitimate need to process and/or keep a subjects data. https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/ Implementing this as a rollback-able delete will not be compliant.
callalex 3y ago
If it’s so hard to be a good steward of data, don’t collect it in the first place.
- jimbokun 3y ago
  It’s not that GDPR is overly onerous to implement. It’s simply that GDPR is fundamentally incompatible with unlimited undo.
brazzy 3y ago
Seems to me that it shouldn't be too difficult to extend an "unlimited undo" system to have specific time-based limits for some data.

thesz 3y ago

Blockchains do not delete records, etc. Also, you have to pay pretty much real money to put your record there. So we can have a good aproximation of what can happen with your proposal even if you need to pay extra for it. [1] https://ycharts.com/indicators/ethereum_chain_full_sync_data_size https://ycharts.com/indicators/ethereum_chain_full_sync_data... Ethereum [1] routinely accumulates around 300G per year and routinely hits over one terabyte of data to sync. Remember, this is the size of the data to sync/transmit, not size of the data that is stored, which we may safely assume to be several times more, because of indices, etc. Also, your proposal makes two tier database system: one that maintains current consistent view of the state and another for log purposes. The logging system needs high throughput storage with key range read request, which makes it, well, another pretty much fully fledged database (SELECT...GROUP BY...ORDER BY is needed). The reason nobody does what you described because it is really prohibitive in storage space aspect and really is quite complex - a database on top of another database.