3 ms·
Last I looked into this, if you had a lot of data to back up (Say 1 Gbyte per second, continuous, with a retention time of 1 year), it was still far cheaper to
by londons_explore 6y ago
Last I looked into this, if you had a lot of data to back up (Say 1 Gbyte per second, continuous, with a retention time of 1 year), it was still far cheaper to simply use hard drives. One employee can keep up with all drive replacements, hardware setup, etc with time to spare. Drives aren't super power hungry, so any old office building is suitable. Encrypt the drive contents on another site so you don't need 24/7 security. Total system cost was sub $1M with a running cost of $500k/year and storage of 50PB. Bargain.
Now that GDPR applies, most companies need to rewrite backups every 30 days anyway to remove data where a GDPR deletion request applies. That tips the scale further in the direction of always spinning hard drives. Just hook up 64 drives to each machine, make sure you only do streaming writes of 1Gb+ files, do some ZFS raid-like scheme, and away you go.
- jeffbee 6y agoThat didn’t add up. 1GB per second for a month is 2.6PB. Do you have a source of 40TB disk drives?
- londons_explore 6y ago1GB/second for a year... Plus spare drives for redundancy, hotspotting, a test/training setup, and unplanned for increases in storage requirements.
- kaslai 6y ago"to each machine" implies that there's more than one machine with 64 drives attached.
- pessimizer 6y agoAnd I'm pretty sure "any old office building" means buying a large office or small office building and filling it up with them.
- Znafon 6y agoYou don't have to do it this way, you can just use encryption at rest with a different key for each user and throw the key when a user ask for deletion of their data. No need to get all the backup back to scrub them one by one.
- londons_explore 6y agoOur lawyers didn't think that sufficient to cover ourselves. Upon finding out it wasn't going to cost many millions to simply scrub the actual data rather than the keys, they came back that it was money well spent to delete the actual data. It was mostly because a user might share data with another user, for example two users at the same postal address. Our fraud team needs to be able to look stuff like that up, so there need to be database keys on stuff like that, both in the backups and in production. If one of the users with a specific address has a GDPR deletion, we need to delete that users data, but if another user has the same address, we still need to keep the address itself. Yet if both users have a GDPR deletion apply to them, as well as deleting the address, we need to delete the fact both deleted users had the same address (even if we don't know the address, because the pattern of which deleted users shared info with other deleted users could identify them) See... it's complex! Simplest solution properly delete the data and rewrite the backups!
- pestaa 6y agoIANAL but you do not need to rewrite backups to brute force compliance. You need to inform your customers what your retention policy is though. I've seen large enterprises communicating backup lifetime up to 6 months after a deletion request.
- seg_lol 6y agocryptoshredding