4 ms·

Reading about their fix with a "magic" number it seems that when there was corruption - the file would never be deleted again since you've lost information at t

Reading about their fix with a "magic" number it seems that when there was corruption - the file would never be deleted again since you've lost information at this point. (As they claim, even if the magic number went back to "0" you would still need to keep the file.) Rather than this questionable solution, why not create an transaction-like system where a delete would result in a tombstone/marker placed on the email database with a rollback mechanism if one or both of the services crashed. For example, an index marking the email as deleted, then you delete the file, then if the file was successfully deleted actually clear the email record. So the email table would be: {id uint64, ... deleted bool} 1. If the file delete service failed, then unmark the deleted index (or try again). 2. If the marking the email as deleted failed, then you're back to square one with no change. 3. If the email is marked, the file is deleted, but actually deleting the email failed. Restart/Tryagain/etc until you get that pesky email record removed.

PSIAlt 10y ago
Thanks for your question. In your solution there is some consistency vulnerabilities: - (Step 1) If "delete service failed" - timeout is a fail? The decrement probably was done, but your app do not see response. In this case you dont know whatever decrement was done really or not. You cannot do re-try with this error (unless you use "magic" =)) - (Step 1) If you do "unmark deleted index" this does not mean it was fixed on disk forever. Your app or even server can shutdown/reboot before you ensure disk write. - (Step 2) If marking failed you generally cannot rollback. You can try to rollback the decrement, but operation can possibly fail too. On next delete iteration the counter in filedb will be corrupted in this case. - (Step 3) Indexes become exclusive data owner. If you lose indexes for 1 mailbox somehow - you cannot rely on counters in filedb anymore. So, you cannot do any decrements anymore until full double-side recheck. - Any bug in any app in this chain can destroy filedb consistency.
- zzzcpan 10y ago
  Your magic numbers don't seem to guarantee consistency either, multiple faulty exchanges and the file could be gone forever: maybe random numbers happened to be the same, or the sum of substractions during faulty exchanges got to 0, or there was an overflow, etc. Distributed algorithms are not easy, but for this case there are proven solutions, look up how Riak deletes records for example, with "tombstones" and "reaping". Basically you have to think in terms of CRDTs and have a log of operations, a versioning scheme, a synchronization, etc., all of which help you to make sure that no operation is applied twice anywhere and that in case of faulty communications nodes can synchronize and apply missing operations. This gives you consistency, eventually, and a reliable way to know when files can be physically deleted. Or, you know, just use Riak.
  - grogenaut 10y ago
    or dynamo, or cassandra, or several others that all exhibit good behavior around these things. Also all of the above (and riak) have event streams when tombstones go away so you can hook into this and reap the file.
- Xeoncross 10y ago
  There would be no rollback of the deleted marker since we don't want to undo the delete - we want to keep trying until it works. > timeout is a fail? No, a timeout would just issue a retry until we got a response back (or if there is a reader it would keep connecting until it got through). There is no danger about deleting a file multiple times since any additional unneeded deletes would report such. > this does not mean it was fixed on disk forever Then the system would keep seeing the marker (after it restarted from the crash), repeat the delete call, and then trying to delete the email for good. It can crash 100 times but eventually it would get both removed. > Indexes become exclusive data owner. If you lose indexes... Then no mater how you build this system you're toast and have to restore backups. No system can handle losing indexes (& the error correcting or checksum copies) whether they are btrees, file blocks, or memory addresses.
throwawayish 10y ago
Transactions are "hard and slow". Distributed transactions are hard-er and slow-er. Random 32 bit integer additions are likely "good enough" and are fast and simple. Also, assuming that their storage is directly some FS, you don't have access to the transaction. You can't do a distributed transaction against something that, as a black box, does not appear to do any transactions. In this case, doing a distributed transaction would mean to use an entirely different storage solution. A transactional one supporting TPC or similar. Magic counters sound good enough yet? ;)

PSIAlt 10y ago

Thanks for your question. In your solution there is some consistency vulnerabilities: - (Step 1) If "delete service failed" - timeout is a fail? The decrement probably was done, but your app do not see response. In this case you dont know whatever decrement was done really or not. You cannot do re-try with this error (unless you use "magic" =)) - (Step 1) If you do "unmark deleted index" this does not mean it was fixed on disk forever. Your app or even server can shutdown/reboot before you ensure disk write. - (Step 2) If marking failed you generally cannot rollback. You can try to rollback the decrement, but operation can possibly fail too. On next delete iteration the counter in filedb will be corrupted in this case. - (Step 3) Indexes become exclusive data owner. If you lose indexes for 1 mailbox somehow - you cannot rely on counters in filedb anymore. So, you cannot do any decrements anymore until full double-side recheck. - Any bug in any app in this chain can destroy filedb consistency.

zzzcpan 10y ago
Your magic numbers don't seem to guarantee consistency either, multiple faulty exchanges and the file could be gone forever: maybe random numbers happened to be the same, or the sum of substractions during faulty exchanges got to 0, or there was an overflow, etc. Distributed algorithms are not easy, but for this case there are proven solutions, look up how Riak deletes records for example, with "tombstones" and "reaping". Basically you have to think in terms of CRDTs and have a log of operations, a versioning scheme, a synchronization, etc., all of which help you to make sure that no operation is applied twice anywhere and that in case of faulty communications nodes can synchronize and apply missing operations. This gives you consistency, eventually, and a reliable way to know when files can be physically deleted. Or, you know, just use Riak.
- grogenaut 10y ago
  or dynamo, or cassandra, or several others that all exhibit good behavior around these things. Also all of the above (and riak) have event streams when tombstones go away so you can hook into this and reap the file.
Xeoncross 10y ago
There would be no rollback of the deleted marker since we don't want to undo the delete - we want to keep trying until it works. > timeout is a fail? No, a timeout would just issue a retry until we got a response back (or if there is a reader it would keep connecting until it got through). There is no danger about deleting a file multiple times since any additional unneeded deletes would report such. > this does not mean it was fixed on disk forever Then the system would keep seeing the marker (after it restarted from the crash), repeat the delete call, and then trying to delete the email for good. It can crash 100 times but eventually it would get both removed. > Indexes become exclusive data owner. If you lose indexes... Then no mater how you build this system you're toast and have to restore backups. No system can handle losing indexes (& the error correcting or checksum copies) whether they are btrees, file blocks, or memory addresses.

throwawayish 10y ago

Transactions are "hard and slow". Distributed transactions are hard-er and slow-er. Random 32 bit integer additions are likely "good enough" and are fast and simple. Also, assuming that their storage is directly some FS, you don't have access to the transaction. You can't do a distributed transaction against something that, as a black box, does not appear to do any transactions. In this case, doing a distributed transaction would mean to use an entirely different storage solution. A transactional one supporting TPC or similar. Magic counters sound good enough yet? ;)