13 ms·

>Someday, this will happen to Facebook, Instagram, Tumblr, etc. Don't trust the platforms to archive your data. This also goes for Google Drive, Dropbox, and m

• zxcvbn4038 8y ago
  I used to work at Tumblr, the entirety of their user content is stored in a single multi-petabyte AWS S3 bucket, in a single AWS account, no backup, no MFA delete, no object versioning. It is all one fat finger away from oblivion.
  • labster 8y ago
    I'm surprisingly okay with this. Well, I guess I'd miss McMansion Hell.
    • ivm 8y ago
      Thousands of skilled artists use Tumblr as their main publishing platform.
      • lostlogin 8y ago
        Picasso (supposedly) drew on a napkin, and Banksy draws on derelict walls or sticks his work through a shredder. The medium doesn’t need to be lasting. Edit: The potentially short-lived medium was chosen by the above artists. Tumblr users many not be too happy if work is lost.
        • buboard 8y ago
          banksy's walls are sold though; and he is still kind of the exception because of his art format. Not everything needs to be lasting but 100% temporary art is not common.
          • whoami_whereami 8y ago
            Oh, it's more common than you think, only it being highly valued is rare. That doodle you drew while having a conversation on the phone? That's throwaway art, even if you don't consider yourself an artist.
            • benjaminikuta 8y ago
              Hi, I'm new here. Why is your name green?
      • criddell 8y ago
        How many do you think they would be willing to pay some small monthly fee? I'm guessing most of them think their work is worth at least $5/month, right? Maybe Tumblr should become a paid service and ditch the advertising model. That way they could be more relaxed about what types of content they are willing to host.
    • kevinmchugh 8y ago
      McMansion Hell is now archived by the Library of Congress. Don't be too concerned. http://mcmansionhell.com/post/181936133241/what-level-of-posting-is-this http://mcmansionhell.com/post/181936133241/what-level-of-pos...
  • ummonk 8y ago
    What the hell. It is so easy to configure multi-region glacier backups, mfa delete, etc. for a single S3 bucket. Took me like a couple hours to setup versioning and backups, and a few days to setup mfa for admin actions. Why would they not set this stuff up?
    • gregrata 8y ago
      The key words you probably need to look at are "multi-petabyte". Not saying they shouldn't be doing something but it all costs - and at multi-petabytes, it cooooosts 1 Petabyte (and they have multiple) S3 - $30,000 a month, $360,000 a year S3 - reduced redundancy - $24,000 a month, $288,000 a year S3 - infrequent access - $13,100 a month, $157,000 a year Glacier - $7340 a month - $88,000 a year
      • PostOnce 8y ago
        Well, the business was acquired for $500,000,000 and a single employee probably costs what backing up two petabytes of data for a year (on glacier) does. They could also always use tapes, for something as critical as the data that is the blood of your business. Imagine if facebook lost everyones' contact lists, how bad would that be for their business? Backups are cheap insurance.
        • antt 8y ago
          They are expensive until the business goes bankrupt.
          • Bartweiss 8y ago
            I wouldn't be surprised if this was actually the rationale for not having backups. Tumblr is apparently fragile and tech-debt laden on engineering side, stagnant on users, and unprofitable. At a certain point, it's a coherent decision to just say "a few days of downtime would seal our fate, the business can only be saved if everything goes right", and not spend any money on mitigation.
        • 8y ago
  • scarface74 8y ago
    I’m not saying it isn’t dumb, but that one fat finger would have to be aws s3 rm bucket —-recursive It won’t let you just go into the console or delete the stack that made it if the bucket isn’t empty.
    • VWWHFSfQ 8y ago
      there was a S3 sync client that some people used that did: aws s3 sync --delete ./ s3://your-bucket/ The delete flag was added by just a very innocuous checkbox in the UI. The result is that it removes anything not in the source directory. Kaboom. Everything's gone. The point is you have no idea what stuff is going to do even if you think it's obvious.
      • PetahNZ 8y ago
        This would take so many hours to actually run though, probably weeks for that amount of data.
      • itronitron 8y ago
        maybe someone at Tumblr can test this...
      • dahfizz 8y ago
        Maybe the moral is that you shouldn't rely on third party clients for mission critical stuff if you dont know what they do.
        • VWWHFSfQ 8y ago
          and also have backups like a normal competent person/organization does
        • foxtrottbravo 8y ago
          I know in the particular example that is something that's good advise and more or less easily done. Do you think it would be good to extend said argument to say scp / ftp clients?
        • 8y ago
  • kirillzubovsky 8y ago
    I’ve heard from Amazon friend that AWS as a whole is like that, one click away from a total meltdown. Probably true.
    • dodobirdlord 8y ago
      For most AWS services it would be fairly difficult to cause multi-region damage by mistake.
    • stone-monkey 8y ago
      That's basically what happened with S3 a couple years back. Mistyped command caused an outage for large parts of the internet in the US. Now, I dunno if they could make a big enough mistake that would bring down the whole company, but certainly it's been proven that a single mistake can affect major portions of the internet.
      • antt 8y ago
        I always find it funny how I'm designing with best practices in mind on top of infrastructure someone out of university build as their first project.
        • nostrebored 8y ago
          This is not the case with S3 and not the case with that incident.
          • lugg 8y ago
            Pretty sure there are first year grads who have worked on S3 as their first project.
            • StavrosK 8y ago
              So what? You're saying it as if they gave them root access to the servers and went "go nuts".
  • soup10 8y ago
    i do this too with my data on a smaller scale, but i'm suprised tumblr does this because even with only a few million files s3 buckets that big are awkward to work with
  • cortesoft 8y ago
    When was this? Being owned by Yahoo, I am surprised they don't use NetApp.
    • zxcvbn4038 8y ago
      Tumblr rejected all things Yahoo, except the money, so the answer to just about anything Yahoo asked was either “no”, “get stuffed”, or silence and a note to David that he needed to escalate to Marissa. On the other side the Yahoo services were so heavily integrated that it was hard to carve out any piece of them, and the few times we tried it was a slow and painful process because Yahoo’s piece was glitchey and unreliable outside of it’s home turf and the Tumblr engineers defensive and argumentative about everything and not willing to help.
      • zimpenfish 8y ago
        > Tumblr rejected all things Yahoo Having worked at Yahoo, I understand this stance.
      • aasasd 8y ago
        That's exactly how I imagined Tumblr's design and development, based on my multiple unsuccessful attempts, over the years, to find any useful navigation between blogs, or the function of reading comments.
    • johnvanommen 8y ago
      > When was this? Being owned by Yahoo, I am surprised they don't use NetApp. Dell used to offer an online backup service. It wasn't even running on Dell equipment! Basically they acquired a company that offered the service, and while it would be "nice" if a Dell company ran on Dell gear, a lot of the time it's simply impractical/expensive to overhaul things.
  • Reedx 8y ago
    Did anyone in the company make a big deal out of this?
  • aiven 8y ago
    After porn ban they probably have only ~one petabyte.
  • leowoo91 8y ago
    I guess your statement is a bit beyond NDA, but thank you for sharing.
    • dev_dull 8y ago
      Borderline whistleblowing.
    • SmellyGeekBoy 8y ago
      It's not covered by NDA if it's made up.
      • heinstrom 8y ago
        Indeed, wild what people will say on the inter webs.
  • alekratz 8y ago
    This is fascinating. Are there any other crazy "wtf, how has this site not died yet" stories from the inside?
    • Bartweiss 8y ago
      There's an awful lot of less-critical stuff that users have tracked down themselves. A few random highlights: - The mobile and desktop sites are completely separate products with vastly different behavior. Some privacy features (relevant to both) can only be accessed on one, some on the other. Tags are rendered in all-lowercase on mobile, but as written on desktop. Block quotes on desktop render as enlarged-font cursive on mobile, for some awful reason. - Tumblr support(s/ed) font coloring, with no documentation of that fact. You enable it by using the HTML editor and picking among color tags with Friends-themed names like "Monica Pizazz Orange". Oh, and the preview feature won't honor the tags, but actually posting will. - NSFW content is flagged even in drafts, but if that content is reviewed and approved, it's automatically posted publicly, not returned to drafts where it started. - Tumblr's desktop sign up page use(s/d) semi-random images from the site as backgrounds. Yes, they did serve cartoon porn to people trying to make accounts. - Certain posts were impossible to view. Tumblr accounts can have their own themed pages, or simply be popup sidebars over the main news feed. Tumblr "read more" content hiders took users from the news feed to the poster's account - if that account was in popup format, a readmore opened from the wrong location would simply force a circular redirect. - All Tumblr links are actually pushed through a site-specific forwarding system to track users. As a result, Twitter and many other sites are inaccessible because they view all link clicks as bot traffic from a "single source".
      • nightpool 8y ago
        your info is somewhat put of daye. aside from #1, these are all bugs that have been fixed. #2 was only before the feature was officially launched. #3 was fixed within a few days. #4 wasn't a bug, serving artistic nudity was intentional and part of tumblrs brand (just like an art museum would). #5 was a bug for a while and it sucked. I've never heard of #6 being an issue—its true that they use a link tracking system but I've never heard of it causing "bot traffic" issues, respectfully, that sounds like bullshit—while I hate it for privacy reasons, lots of sites use link tracking, like Google and Facebook.
  • kizm 8y ago
    What is Tumblr?
  • johnvanommen 8y ago
    > I used to work at Tumblr, the entirety of their user content is stored in a single multi-petabyte AWS S3 bucket, in a single AWS account, no backup, no MFA delete, no object versioning. It is all one fat finger away from oblivion. Remember when Microsoft lost all of the data for their Sidekick users? Basically they were upgrading their SAN and things went badly.
  • tazjin 8y ago
    My experience with Tumblr was generally that a large part of the content, especially larger media content like videos, failed to load most of the time. Makes me wonder if that's related ...
  • xvllltbsgxkskk 8y ago
    Kill them
  • supermanlocal 8y ago
    And I know being an auditor, how the controls in the SOC audits are designed around to miss the pressing issues of cloud!!!
• gdulli 8y ago
  I've seen occasional attachments to years-old gmail messages get irrevocably lost. Well, irrevocably given the likelihood of ever getting human attention on it.
• mholt 8y ago
  This is why I wrote Timeliner: https://github.com/mholt/timeliner https://github.com/mholt/timeliner - a tool to download all my content from essential cloud services (like Google Photos, etc) -- I don't like to trust the cloud as a master copy of my data.
  • dredmorbius 8y ago
    Very interesting. I've had a thought about a monthly "data preservation day" concept, (on the 11th, because: https://xkcd.com/1140/ https://xkcd.com/1140/, and yes, I know: https://drhagen.com/blog/the-missing-11th-of-the-month/ https://drhagen.com/blog/the-missing-11th-of-the-month/) A multiplatform archival tool would be useful.
  • zimpenfish 8y ago
    I use(d) Perkeep (Camlistore) for this but the current version is broken wrt Pleroma (which might be Pleroma's fault) and Pinboard. I'll definitely give Timeliner (and possibly a Pinboard datasource) to fill in the gaps.
  • hiei 8y ago
    This tool appears to be great! One question though, what does your storing solution look like?
    • mholt 8y ago
      It's just a SQLite database, combined with a folder on the file system.
      • hiei 8y ago
        Forgive me - I meant personal storage solutions.
  • nitemice 8y ago
    I've been amassing a collection of scripts to basically do the same thing over the last couple of months, just running them all once a month with cron. None of the services I've been backing up (Goodreads, Trakt, DeviantArt, Tumblr) are currently covered by Timeliner, but the extra twist of assembling all your data into a single timeline sounds kinda cool, so maybe it's worth contributing a few data sources.
• kyrra 8y ago
  In the Google case, I don't believe the files were ever deleted. While the article says they were unable to download it again, my understanding is that Google never deletes or prevents access to your own files on Drive (in cases like this). So maybe something else was going on here I'm unfamiliar with. Google was preventing the sharing of the files, but that should have been it. (I'm a googler, but don't have any direct info on this specific incident)
• kerneltime 8y ago
  At one point Yahoo was invincible.. I really need a way to find an alternative to gmail for email where I am in control and can replicate it when I like.
  • OrgNet 8y ago
    The only reason i still use gmail is because of the search engine... im about to implement my own using existing techs so that I can move away from Google... why does search sucks so much in open source programs.. on a side note, Myspace is still active?
    • zimpenfish 8y ago
      > why does search sucks so much in open source programs.. If you have/can get command line access to your mailboxes, mairix[1] is pretty good at indexing and searching. [1] https://github.com/vandry/mairix https://github.com/vandry/mairix
  • asteli 8y ago
    you can always set up an email account on another service and have the mail accessed through google's interfaces. i wouldn't recommend setting up your own email server unless you really like securing servers though
  • SpaethCo 8y ago
    You can be in control of your actual data pretty easily with gmail. One of the better options I’ve found for maintaining a local archive is the Got Your Back[0] script. It maintains an SQLite DB with all of the message tag associations, and stores all your emails in standard .msg format text files. If you want additional peace of mind, you can spend a few pennies and use restic[1] to do regular incremental backups of the archive to Backblaze B2 to add yet another storage location and version index. [0] https://github.com/jay0lee/got-your-back/wiki https://github.com/jay0lee/got-your-back/wiki [1] https://restic.net https://restic.net
• ksec 8y ago
  I want a simple to use NAS that has NAND + 2x 2.5" HDD Mixed with ZFS. And Backup Option to likely B2 or other Services. Preferably coming from Apple as new Time Capsule. But All these vendor want is Cloud Services Revenue. Interestingly There are a few Android handset maker with Wireless + Backup devices. That backup while recharging your battery.
• Hoasi 8y ago
  > This also goes for Google Drive, Dropbox, and many other websites (if not all) Of course, but that is not a bad thing per se. Digital data archiving itself is unknown territory. What should happen to the data you put online is uncertain, even in the short term. The analogy of a "cloud" is revealing. A cloud is fleeting by nature. Nothing last forever online, no matter what they say. And that is not mentioning compatibility issues or reading old files made with outdated apps. Some files that are just above 10 years old can be hard to retrieve today.
• alkonaut 8y ago
  I might not trust them 100%, but I trust them around 1000x more than I trust myself. I have had several data loss incidents and it was invariably my own fault. I simply cannot be trusted with my own data. If one doesn't trust Google/Dropbox et.al. to handle the data, then just use two of them, or use a personal solution and one of them. Just don't use your home rolled backup system only because you don't trust any of the online storage providers.
• djsumdog 8y ago
  It's a little different for Dropbox, MEGA and similar services because they sync your machines. So long as you have at least one machine with those tools installed and running, you have a copy (unless someone finds an exploit in their servers and deletes all their customer data ... so yea .. still keep backups).
• johnchristopher 8y ago
  No. It goes against the whole "what you post online will stay online forever". Which is it ?
  • jraph 8y ago
    If you hope it remains private or it gets forgotten, you should assume that it may go public / remain online forever. And archives exist for many pieces of public content. If you hope it stays forever, you should assume it may vanish in a few seconds.
• josefresco 8y ago
  Anecdotally my experience has been the inverse. Number of my files lost by cloud drive providers: 0 Number of my files lost by me because I'm a dumbass: Tens of thousands, including some of most cherished family photos