12 ms·

>The root of both bugs was a design flaw: the decision that >git.kde.org was always to be considered the trusted, >canonical source. It seems that an even bigg

>The root of both bugs was a design flaw: the decision that >git.kde.org was always to be considered the trusted, >canonical source. It seems that an even bigger design flaw is that they (still) aren't doing regular backups. The mirroring of course provides some redundancy, similar to what raid does, but as they say: "raid is not a backup solution".

brokenparser 14y ago
I thought this meant they do have some sort of backup: However, while we planned for the case of the server losing a disk or entirely biting the dust, or the total loss of the VM’s filesystem...
- Nitramp 14y ago
  I read that as meaning that their mirroring strategy makes sure to detect these conditions and avoid clobbering the intact replicas in the event.
- kzrdude 14y ago
  They were autosyncing. They were prepared for not being able to sync (restore main system and sync back), but not for the sync to replicate a corrupt state. Any backup strategy with only one backup (instead of preserving historical snapshots) is vulnerable this way.
sho_hn 14y ago
Backups address the problem only to the degree that they give you an older revision to revert back to. The interesting thing that happened here is that corrupted data was propagated through the mirror network, which syncs more often than backups get made, and how to prevent that. Because while having a safety net is nice, avoiding developers being inconvenienced by a failure is the real challenge. Plus unchecked backups of corrupted data aren't worth a lot, and corruption-proof mirroring acts as further (and timely) backup.
- Nitramp 14y ago
  Having a hot standby/failover is nice, but of course only the icing on top of your backup strategy. I read the article same as the OP, it doesn't mention any backup system, only the mirroring. In fact, they did restore from a replica that was out of date in the end (from projects.kde.org). Had they had actual backups, the article would surely mention they only used projects.kde.org because it was somewhat more recent than their last backup? The story about planning to do regular ZFS snapshots hints to the same, if they had a backup system they wouldn't need that. edit: sorry, is that your post/do you have more insight? In that case I'm sure you know better than me speculating on what the author meant ;-)
  - sho_hn 14y ago
    > edit: sorry, is that your post/do you have more insight? In that case I'm sure you know better than me speculating on what the author meant ;-) Nah, I'm not Jeff. I have some general insight because I was involved with setting up our git infrastructure in its early days, but I haven't worked on the mirroring code, and I've been out of the loop on day-to-day admin operations for a while, so I can't comment on the backup schemes that may or may not be in action on the servers right now.
- rmc 14y ago
  Sure, if you restore from a backup, you lose some work, ie you restore from last week's backup, and you lose 1 weeks worth of work. But loosing one weeks worth of work is much better than loosing 10 years worth of work!

brokenparser 14y ago

I thought this meant they do have some sort of backup: However, while we planned for the case of the server losing a disk or entirely biting the dust, or the total loss of the VM’s filesystem...

Nitramp 14y ago
I read that as meaning that their mirroring strategy makes sure to detect these conditions and avoid clobbering the intact replicas in the event.
kzrdude 14y ago
They were autosyncing. They were prepared for not being able to sync (restore main system and sync back), but not for the sync to replicate a corrupt state. Any backup strategy with only one backup (instead of preserving historical snapshots) is vulnerable this way.

sho_hn 14y ago

Backups address the problem only to the degree that they give you an older revision to revert back to. The interesting thing that happened here is that corrupted data was propagated through the mirror network, which syncs more often than backups get made, and how to prevent that. Because while having a safety net is nice, avoiding developers being inconvenienced by a failure is the real challenge. Plus unchecked backups of corrupted data aren't worth a lot, and corruption-proof mirroring acts as further (and timely) backup.

Nitramp 14y ago
Having a hot standby/failover is nice, but of course only the icing on top of your backup strategy. I read the article same as the OP, it doesn't mention any backup system, only the mirroring. In fact, they did restore from a replica that was out of date in the end (from projects.kde.org). Had they had actual backups, the article would surely mention they only used projects.kde.org because it was somewhat more recent than their last backup? The story about planning to do regular ZFS snapshots hints to the same, if they had a backup system they wouldn't need that. edit: sorry, is that your post/do you have more insight? In that case I'm sure you know better than me speculating on what the author meant ;-)
- sho_hn 14y ago
  > edit: sorry, is that your post/do you have more insight? In that case I'm sure you know better than me speculating on what the author meant ;-) Nah, I'm not Jeff. I have some general insight because I was involved with setting up our git infrastructure in its early days, but I haven't worked on the mirroring code, and I've been out of the loop on day-to-day admin operations for a while, so I can't comment on the backup schemes that may or may not be in action on the servers right now.
rmc 14y ago
Sure, if you restore from a backup, you lose some work, ie you restore from last week's backup, and you lose 1 weeks worth of work. But loosing one weeks worth of work is much better than loosing 10 years worth of work!