11 ms·
I used to work at Tumblr, the entirety of their user content is stored in a single multi-petabyte AWS S3 bucket, in a single AWS account, no backup, no MFA dele
by zxcvbn4038 8y ago
I used to work at Tumblr, the entirety of their user content is stored in a single multi-petabyte AWS S3 bucket, in a single AWS account, no backup, no MFA delete, no object versioning. It is all one fat finger away from oblivion.
- labster 8y agoI'm surprisingly okay with this. Well, I guess I'd miss McMansion Hell.
- ivm 8y agoThousands of skilled artists use Tumblr as their main publishing platform.
- lostlogin 8y agoPicasso (supposedly) drew on a napkin, and Banksy draws on derelict walls or sticks his work through a shredder. The medium doesn’t need to be lasting. Edit: The potentially short-lived medium was chosen by the above artists. Tumblr users many not be too happy if work is lost.
- buboard 8y agobanksy's walls are sold though; and he is still kind of the exception because of his art format. Not everything needs to be lasting but 100% temporary art is not common.
- whoami_whereami 8y agoOh, it's more common than you think, only it being highly valued is rare. That doodle you drew while having a conversation on the phone? That's throwaway art, even if you don't consider yourself an artist.
- benjaminikuta 8y agoHi, I'm new here. Why is your name green?
- criddell 8y agoHow many do you think they would be willing to pay some small monthly fee? I'm guessing most of them think their work is worth at least $5/month, right? Maybe Tumblr should become a paid service and ditch the advertising model. That way they could be more relaxed about what types of content they are willing to host.
- kevinmchugh 8y agoMcMansion Hell is now archived by the Library of Congress. Don't be too concerned. http://mcmansionhell.com/post/181936133241/what-level-of-posting-is-this http://mcmansionhell.com/post/181936133241/what-level-of-pos...
- ummonk 8y agoWhat the hell. It is so easy to configure multi-region glacier backups, mfa delete, etc. for a single S3 bucket. Took me like a couple hours to setup versioning and backups, and a few days to setup mfa for admin actions. Why would they not set this stuff up?
- gregrata 8y agoThe key words you probably need to look at are "multi-petabyte". Not saying they shouldn't be doing something but it all costs - and at multi-petabytes, it cooooosts 1 Petabyte (and they have multiple) S3 - $30,000 a month, $360,000 a year S3 - reduced redundancy - $24,000 a month, $288,000 a year S3 - infrequent access - $13,100 a month, $157,000 a year Glacier - $7340 a month - $88,000 a year
- PostOnce 8y agoWell, the business was acquired for $500,000,000 and a single employee probably costs what backing up two petabytes of data for a year (on glacier) does. They could also always use tapes, for something as critical as the data that is the blood of your business. Imagine if facebook lost everyones' contact lists, how bad would that be for their business? Backups are cheap insurance.
- antt 8y agoThey are expensive until the business goes bankrupt.
- Bartweiss 8y agoI wouldn't be surprised if this was actually the rationale for not having backups. Tumblr is apparently fragile and tech-debt laden on engineering side, stagnant on users, and unprofitable. At a certain point, it's a coherent decision to just say "a few days of downtime would seal our fate, the business can only be saved if everything goes right", and not spend any money on mitigation.
- 8y ago
- scarface74 8y agoI’m not saying it isn’t dumb, but that one fat finger would have to be aws s3 rm bucket —-recursive It won’t let you just go into the console or delete the stack that made it if the bucket isn’t empty.
- VWWHFSfQ 8y agothere was a S3 sync client that some people used that did: aws s3 sync --delete ./ s3://your-bucket/ The delete flag was added by just a very innocuous checkbox in the UI. The result is that it removes anything not in the source directory. Kaboom. Everything's gone. The point is you have no idea what stuff is going to do even if you think it's obvious.
- PetahNZ 8y agoThis would take so many hours to actually run though, probably weeks for that amount of data.
- itronitron 8y agomaybe someone at Tumblr can test this...
- dahfizz 8y agoMaybe the moral is that you shouldn't rely on third party clients for mission critical stuff if you dont know what they do.
- VWWHFSfQ 8y agoand also have backups like a normal competent person/organization does
- foxtrottbravo 8y agoI know in the particular example that is something that's good advise and more or less easily done. Do you think it would be good to extend said argument to say scp / ftp clients?
- 8y ago
- kirillzubovsky 8y agoI’ve heard from Amazon friend that AWS as a whole is like that, one click away from a total meltdown. Probably true.
- dodobirdlord 8y agoFor most AWS services it would be fairly difficult to cause multi-region damage by mistake.
- stone-monkey 8y agoThat's basically what happened with S3 a couple years back. Mistyped command caused an outage for large parts of the internet in the US. Now, I dunno if they could make a big enough mistake that would bring down the whole company, but certainly it's been proven that a single mistake can affect major portions of the internet.
- antt 8y agoI always find it funny how I'm designing with best practices in mind on top of infrastructure someone out of university build as their first project.
- nostrebored 8y agoThis is not the case with S3 and not the case with that incident.
- soup10 8y agoi do this too with my data on a smaller scale, but i'm suprised tumblr does this because even with only a few million files s3 buckets that big are awkward to work with
- cortesoft 8y agoWhen was this? Being owned by Yahoo, I am surprised they don't use NetApp.
- zxcvbn4038 8y agoTumblr rejected all things Yahoo, except the money, so the answer to just about anything Yahoo asked was either “no”, “get stuffed”, or silence and a note to David that he needed to escalate to Marissa. On the other side the Yahoo services were so heavily integrated that it was hard to carve out any piece of them, and the few times we tried it was a slow and painful process because Yahoo’s piece was glitchey and unreliable outside of it’s home turf and the Tumblr engineers defensive and argumentative about everything and not willing to help.
- zimpenfish 8y ago> Tumblr rejected all things Yahoo Having worked at Yahoo, I understand this stance.
- aasasd 8y agoThat's exactly how I imagined Tumblr's design and development, based on my multiple unsuccessful attempts, over the years, to find any useful navigation between blogs, or the function of reading comments.
- johnvanommen 8y ago> When was this? Being owned by Yahoo, I am surprised they don't use NetApp. Dell used to offer an online backup service. It wasn't even running on Dell equipment! Basically they acquired a company that offered the service, and while it would be "nice" if a Dell company ran on Dell gear, a lot of the time it's simply impractical/expensive to overhaul things.
- Reedx 8y agoDid anyone in the company make a big deal out of this?
- aiven 8y agoAfter porn ban they probably have only ~one petabyte.
- leowoo91 8y agoI guess your statement is a bit beyond NDA, but thank you for sharing.
- dev_dull 8y agoBorderline whistleblowing.
- SmellyGeekBoy 8y agoIt's not covered by NDA if it's made up.
- heinstrom 8y agoIndeed, wild what people will say on the inter webs.
- alekratz 8y agoThis is fascinating. Are there any other crazy "wtf, how has this site not died yet" stories from the inside?
- Bartweiss 8y agoThere's an awful lot of less-critical stuff that users have tracked down themselves. A few random highlights: - The mobile and desktop sites are completely separate products with vastly different behavior. Some privacy features (relevant to both) can only be accessed on one, some on the other. Tags are rendered in all-lowercase on mobile, but as written on desktop. Block quotes on desktop render as enlarged-font cursive on mobile, for some awful reason. - Tumblr support(s/ed) font coloring, with no documentation of that fact. You enable it by using the HTML editor and picking among color tags with Friends-themed names like "Monica Pizazz Orange". Oh, and the preview feature won't honor the tags, but actually posting will. - NSFW content is flagged even in drafts, but if that content is reviewed and approved, it's automatically posted publicly, not returned to drafts where it started. - Tumblr's desktop sign up page use(s/d) semi-random images from the site as backgrounds. Yes, they did serve cartoon porn to people trying to make accounts. - Certain posts were impossible to view. Tumblr accounts can have their own themed pages, or simply be popup sidebars over the main news feed. Tumblr "read more" content hiders took users from the news feed to the poster's account - if that account was in popup format, a readmore opened from the wrong location would simply force a circular redirect. - All Tumblr links are actually pushed through a site-specific forwarding system to track users. As a result, Twitter and many other sites are inaccessible because they view all link clicks as bot traffic from a "single source".
- nightpool 8y agoyour info is somewhat put of daye. aside from #1, these are all bugs that have been fixed. #2 was only before the feature was officially launched. #3 was fixed within a few days. #4 wasn't a bug, serving artistic nudity was intentional and part of tumblrs brand (just like an art museum would). #5 was a bug for a while and it sucked. I've never heard of #6 being an issue—its true that they use a link tracking system but I've never heard of it causing "bot traffic" issues, respectfully, that sounds like bullshit—while I hate it for privacy reasons, lots of sites use link tracking, like Google and Facebook.
- kizm 8y agoWhat is Tumblr?
- johnvanommen 8y ago> I used to work at Tumblr, the entirety of their user content is stored in a single multi-petabyte AWS S3 bucket, in a single AWS account, no backup, no MFA delete, no object versioning. It is all one fat finger away from oblivion. Remember when Microsoft lost all of the data for their Sidekick users? Basically they were upgrading their SAN and things went badly.
- tazjin 8y agoMy experience with Tumblr was generally that a large part of the content, especially larger media content like videos, failed to load most of the time. Makes me wonder if that's related ...
- xvllltbsgxkskk 8y agoKill them
- supermanlocal 8y agoAnd I know being an auditor, how the controls in the SOC audits are designed around to miss the pressing issues of cloud!!!