4 ms·
> Has Backblaze honestly ever had this problem, where 2GB of RAM was not enough to reliably find a customer's backup state? I'm the author of the article and w
by brianwski 9y ago
> Has Backblaze honestly ever had this problem, where 2GB of RAM was not enough to reliably find a customer's backup state?
I'm the author of the article and wrote the Backblaze Windows desktop client.
The problem isn't "finding" the customer's backup state, it IS the customer backup state. The "backup state" is a list of every filename that is backed up, with the size of the file, a SHA-1 of the file, and the location of that file in the Backblaze datacenter.
This is a bit over simplified, but while your computer is backing up, as a new file is transmitted from your laptop to the Backblaze datacenter we append one line to the "Backup State". That's EASY to do and takes a very small amount of RAM and Backblaze is efficient.
The first problem occurs when a customer purchases a new laptop and doesn't want to re-upload all their files from scratch, they want to "Inherit the Backup State". All of it at once. Backblaze was originally written 10 years ago, so for an average customers this all fit inside of a "reasonable" amount of RAM, let's say 50K of RAM. But the Inherit Backup feature itself caused an interesting problem. You are now carrying around all of the "baggage" of the previous file locations from all of your laptops going back 10 years. The logs only get longer, and we need all the logs to replay back everything that has occurred. So in the first year it is 50K. Then after 10 years of Inheriting it 5 laptops and 10 years later it is up to 500K of RAM. Still just fine.
The second problem is that the amount of data in the world has skyrocketed in the last 10 years, and some of our customers have 50 TBytes of data backed up with Backblaze (for $5/month). Again, all fine and all works, but when that customer goes to Inherit the Backup state we find it has passed 2 GBytes in size. Again, COMPLETELY FINE and the same original code still works -> but only on a 64 bit newer laptop which describes 98% of our customers.
I could rewrite the code, but I have to choose what to work on each day. Rewriting this code to work with 2% of our customers means that some OTHER feature does not get built.
- billforsternz 9y agoI don't understand. It goes from 50k to 500k, sure. But then suddenly 2g plus? (growth factor is 10 in stage 1 then 4000 in stage 2)
- brianwski 9y agoIt's 500K or less for "most customers", but then for a few of our largest data customers it 2 GByte. Think of a bell curve where the average is 500K but the top 10% are above 2Gbytes. A randomly interesting part is it isn't really the size of the backup, it is the NUMBER OF FILES in the backup. So a customer who has 1 billion 1 byte files is only storing 1 GByte of data, but the "Backup State" is more like 40 GBytes (for each of the 1 byte files we store a 40 byte SHA1).
- deleted 9y ago[deleted]