Hi Folks, Just wanted to clarify the issue for anyone who didn't have the time to read the full gist. When we first started DigitalOcean we occasionally recei

Hi Folks, Just wanted to clarify the issue for anyone who didn't have the time to read the full gist. When we first started DigitalOcean we occasionally received tickets from customers about recovering a droplet that they had destroyed. Unfortunately when a droplet was destroyed it was gone from the system and it wasn't possible to recover. To help our customers we decided that it was a good idea to take a temporary snapshot of the droplet after the destroy was issued that would automatically expire. This way if someone mistakenly destroyed a droplet they could still recreate it. This proved to be a lifesaver for many customers of DigitalOcean when a third party company that provided a provisioning service that integrated with DO, AWS, Rackspace, etc. was compromised and the attacker issued a delete to all customers and all instances. Because this mechanism was in place we were able to recover almost everyone's droplets. We ran into an issue with securely scrubbing data which was publicized on HN and we implemented a fix immediately with a scrub flag. Unfortunately we made a mistaken and made the default setting false. Most customers often click the default, and I myself do the same thing, since I assume that the default is the best course of action and this led to this issue resurfacing. This also was posted to HN and we immediately decided that the default behavior should be to scrub. Prior to this when a customer selected scrub securely because they were taking two actions, issuing a destroy, and setting a flag, it was safe to assume that they indeed want the data completely destroyed. However when we had to reverse the default we were left in a situation where the default would not create a temporary snapshot if we used the secure destroy flag as the indicator for whether or not a temporary snapshot should be created. Since we've implemented the temporary snapshot feature we have had 1154 droplets that have been restored after a destroy from different 752 customers. That's 752 customers that were elated to find out that they could recover a droplet that was mistakenly destroyed so obviously this is a very beneficial feature since each time one of those customers recovered a droplet it was a huge win for them. We assumed that since the temporary snapshots are automatically destroyed this would not be an issue. In fact in the control panel we provided an additional feature which would make the snapshot permanent otherwise that snapshot is deleted. I think the issue that is brought up here is definitely worth a discussion and we treat security very highly. Since we had the prior HN post regarding changing our default behavior we have been working behind the scenes to ensure that would be the default behavior so that the scrub flag could be removed entirely and that all destroys regardless of how they would be issued would be secure. That behind the scenes work is almost entirely done so this discussion of the temporary snapshot is great because it allows us to revisit this issue once again. We have not had any other customer complaints that during a secure destroy the droplet and the backups and the snapshots were not immediately destroyed. So it was great to engage in a conversation with the customer to understand their view on how they wanted these commands to functions. We'll be engaging with engineering tomorrow to see if it's time for us to begin to phase out the scrub data flag and instead perhaps open up a new flag which would create a temporary snapshot. For the UX/UI of the control panel we would make the default behavior of the destroy create a temporary snapshot and we would have to discuss to see if the API should behave the same way. Often API customers are creating and destroying many servers so it may be safe to assume that they do not want a temporary snapshot, though having default behaviors differ between the control panel and the API is generally not a good idea. I think in general this highlights an issue that all startups deal with. That is as the product grows and matures and as features are added there are often unintended cascading consequences. In this case we have done our best to do right by the largest number of customers to ensure that data is safely and securely destroyed while still providing a default behavior that would protect customers against accidental destroys whether they be self-initiated or otherwise. If anyone has any questions regarding this issue or anything else always please feel free to email me directly, my first name, Moisey, at DO (expand that) . com. Thanks, Moisey Cofounder DigitalOcean