11 ms·
A Dropbox account gave me stomach ulcers
- ermir 7y agoWhat a complete nightmare! I bet that most of that data is useless, and the Dropbox account is just used as a dumping ground for everything and there's no tagging, cataloging, or any sort of data structures.
- Iv 7y agoMy assumption, given the storage requirements for laptops, is that the actual database is 2-3TB is size and that the rest is archived snapshots of the DB.
- ceezuns 7y agoAssuming that this database has been building up over the past decade, I'm in awe over how much money they must have spent on Laptop's in the early part of this decade.
- bsaul 7y agoi had trouble understanding the laptop part of the story. You can’t sync 450 terabytes to a 4terabytes laptop anyway. So why the big storage ?
- jdbernard 7y agoIt's probably as poster above says: each snapshot (sounds like it's DB + app instance or something, maybe a VM snapshot?) is probably > 1TB in size. If you have a 1.5TB image it would make sense to want 4TB storage so that you could have two versions locally at a time. Also explains how long it takes them to do things. Every time they switch versions they have to swap out TBs of data to pull latest. Because it also sounds like they don't have a way to grab just the diffs. They grab whole images at a time.
- Rotten194 7y agoDropbox selective sync that ignores the archived versions maybe?
- _bxg1 7y agoI remember as a freshman in college using Dropbox as version control for a 4-person group project. Even then it was a hurdle. Even with everyone sitting in the same room literally saying "okay I'm working on X nobody touch that file for a couple minutes". Jesus.
- sombremesa 7y agoI was imagining (wishful thinking, I'm sure) that they were pushing to a Git repo hosted on Dropbox, which actually works remarkably well.
- tigershark 7y agoNo, he said that they had someone to manually download the last version of each file. If they were really using git hosted over Dropbox they could have simply cloned the repo.
- lazylizard 7y agoIs this a problem with tech? I imagine the users who decided to put sensitive data on dropbox are personally liable? Legal might have work to do? But i fail to see the tech challenge?
- robjan 7y agoIt seems that Dropbox was hosting all of the production data and being used as some kind of document store / db system in this story.
- lazylizard 7y agoCan tech fix it by 1. Disabling the dropbox account? 2. Stopping dropbox urls from resolving in their lan if need be? Or blocking dropbox ips? 3. Buying a md1280 or similar storage quickly n put a freenas or similar on it? 4. Put syncthing or similar on all the user computers n this new nas?
- lazylizard 7y agoActually why is dropbox involved if its a git directory???
- lazylizard 7y agoGit annex? Sparkleshare? Can tech just stop the dropbox bit n just set up sparkleshare for everyone if git is holding up? If git is less than efficient..then syncthing, seafile would all work for their file sharing needs n r trivial to setup?
- djrogers 7y agoI think you missed the part where it’s not Dropbox as a file sharing solution that’s the only problem - their entire web app runs via Dropbox API calls from Heroku!
- justinclift 7y ago
- farrelmahaztra 7y agoThe best worst thing I’ve ever read. No idea how they pulled that off for so long.
- allthecybers 7y agoReminds me of a firm I worked for. They decided to adopt Google Drive for the team of about 20 employees. Every couple weeks an employee would mistake the synced folder for their local disk and rearrange files or delete files and then the files would disappear for the rest of the company. Then we’d have to go into the backups and restore the files and send a message out to everyone to hold the files they were working on before saving them to the drive again.
- blfr 7y agoHow come? Google Drive has fairly granular access control access control.
- Iv 7y ago> 5. Audit has some explaining todo. Yes, that's a total fuck up on their side as well. That an incompetent company dies while doing stupid stunts like that is understandable and how the market filters out stupidity. That another company looks at it and thinks "Oh that looks fine", that's a total failure at their work.
- kyberias 7y agoWhat "audit" are we talking about here, though?
- DoingIsLearning 7y agoUsually when you acquire a company you sign of some intent lettet that shows you are serious about the purchase provided they can pass your smell test. Then after NDA's and a few other docs if a company accepts being purchased they have to give access to their internal docs. Finance books, IT, etc. If during this "due diligence" period the buyers find a major red flag they might pull out, revise the price down, etc.
- distant_hat 7y agoHaving seen a couple of acquisitions what I see is that things go forward because the top execs are close and went to the same school and so on. The technical details shared are light and often don't matter until things are settled and the plebs have to merge everything.
- swarnie_ 7y agoIt sounds like he's referring to the due diligence as part of the M&A. Personally I've never seen one of these discovery phases bring back anything remotely close to real life. No random IT manager alive is going to throw his dirty laundry out to a potential buyer and potentially torpedo a deal. Some of the things I've seen/found during my company buying spree since 2015 have been magical.
- ryandrake 7y agoI hate to be That Guy but how on earth is this not discovered during due diligence before “insane asylum” Company was acquired?
- InvaderFizz 7y agoThat's easy. Lies, misdirection, and the person doing the technical due diligence being a C-Level/Director with zero IT background but plenty of project management experience. They know just enough to be dangerous and frustratingly useless getting useful information from. So your integration team has to do all the planning with only the foggiest idea of what they are getting into. They then spend the entire first day on the ground just getting the lay of the land. No time to schedule local cable contractors for the really needed cable runs. Let's hope that the existing CAT5 runs work fine on gigabit for the next few weeks till we can get those contractors in to replace all the wiring. Worse case, we'll have to limit the port speeds to 100Mbps at the new switch if retransmits become an issue. Even more fun when the next project is even less detail but the entity you are acquiring is on another continent instead of a few states over.
- foobiekr 7y agoHave been there for an acquisition that did not go as planned. Essentially on the small side you are very dependent on the disclosure of the parties involved; short of truly burdensome diligence processes you are sure to miss things if the company being acquired is not up front about things.
- bakul 7y agoReminds me of HP's 11.1 Billion dollar blunder of buying Autonomy.... https://www.nytimes.com/2012/12/01/business/hps-autonomy-blunder-might-be-one-for-the-record-books.html https://www.nytimes.com/2012/12/01/business/hps-autonomy-blu...
- HeWhoLurksLate 7y agoThat article was worth clearing cookies for. Thank you.
- 7y ago
- eschneider 7y agoAnd this, kids, is why companies do a technical due diligence during mergers. :0
- fortran77 7y agoIt doesn't matter. The board and CEO will _never_ listen to the CIO and technical team. If they want to do an acquisition, they'll do it. I've seen this at least a dozen times in my 36 year career.
- Gibbon1 7y agoI keep coming back to from the board and CEO's perspective they have a choice, spend a lot of their time and effort to gain market share by investing in their own company. Or acquiring it through an acquisition. Seems like an acquisition is a lot less hassle and risk. And can be done quickly. Most 'problems' can be made to go away by throwing money and fixers at them.
- guitarbill 7y agoThis doesn't always work, see HPE vs Autonomy. That case is incredibly interesting for an insight into how idiotic and lazy execs can be. Usually though, it works out for them, probably by luck. Would be, err, "interesting" if there were more Mike Lynch around to provide similar insight (Autonomy's CEO). Then again, the case is interesting as HPE was willing to litigate, and a certain British tech news outlet picked it up.
- thendrill 7y agoIt is called a "formality"
- doctor_eval 7y agoWhat I find interesting is that the “insane asylum” apparently knew that it was wrong. e.g. when asked why they need so much space on their laptops, they were evasive. To my mind this makes it much worse. Not only were they doing something dumb, but they knew it was dumb, and they hid it instead of fixing it. It certainly invites speculation about the company culture in the acquired business.
- nikanj 7y agoBased on my experience: IT was well aware it was insane. Alas, IT got absolutely no budget for fixing it from top brass, as kicking the can down the road always made more sense short-term
- Thorrez 7y agoIT of the insane company, or the acquiring company? Because the post says IT of the acquiring company didn't know until the problem blew up.
- swarnie_ 7y agoDD failed to find it during the M&A, issue didn't become apparent until the purchased company lost multiple employees who kept the plates spinning. It's a failure all round but i would say acquiring companies IT team are least to blame here.
- vezycash 7y agoFrom the article, it doesn't seem to be the fault of the IT guys. It's more like some rich guy who goes out buys a shiny used plane and hands over to the pilot to use.
- mannykannot 7y agoAccording to OP, the plate-spinning employees were 'let go' at the time of the acquisition - presumably a decision by the acquiring company.
- amarshall 7y agoIt should be noted that the evidence that stress is a primary cause of gastric ulcers has been fairly weak since the discovery of H. pylori, though there is some evidence stress may be a contributing factor.
- semi-extrinsic 7y agoFWIW stress is a contributing factor in pretty much all diseases we know. Particularly those where the immune system is active.
- im3w1l 7y agoMaybe you could see stress as taking on "health" debt.
- vezycash 7y agoI've got ulcer. My last ulcer attack, 5 years had me rolling on the floor. Since then, I avoided stress. Ate at the slightest hint of hunger, even as I grew a pot belly. Also, I avoided situps, cos the muscle pains felt like ulcer attacks. There's this specific food that I love. It's eaten RAW but often prepared in a not too hygienic manner. Since I banned myself from eating it, I've slowly become able to stay a few hours without food and can handle more stress.
- doctor_eval 7y agoI actually have had the reverse experience. My company was acquired 18 months ago. We were the smaller company, about 1/4 the size, and as the founder/CTO, I expected the acquiring company to be all over us. I mean, we had a pretty tight operation but there was only so much 16 people could achieve. There were plenty of things we could improve, and I was looking forward to the resources and experience of this bigger company to help us. Was I in for a shock! While their software dev practices were OK (certainly no better than ours), what hit me hardest was their secops. It was a nightmare. And these guys service some brand name clients. As an example, not only was the corporate wifi password memorable - not actually “P455w0rd” but similarly bad - but it was written on a whiteboard in view of the whole office! Service passwords are still stored in clear text in the corporate wiki, and few people understand why this is terrible. It’s like a 90s IT shop, where people used to ask me “why would anyone care about our password?” Kicking the can down the road is fine for a while, but without strong technical leadership, kicking the can becomes how things get done, and that’s been my experience here. I’m not long for this place.
- n00b123 7y ago> Service passwords are still stored in clear text in the corporate wiki. I know this painfully well :(. Can you recommend some good solution? Ideally open source and self hosted, instead of cloud password manager.
- Operyl 7y agoWhile I’ve not personally used it, I’m happy with 1Password Teams, bitwarden has a pretty well received project. If you want self hosted teams, though, you will need to purchase a license (it’s one of their money makers, and I think that is completely fair). Most of their product is otherwise open, afaik.
- tmikaeld 7y agoBitwarden is fully open source and audited. You can self host the server and client if you want without additional cost (they charge small per user).
- eismcc 7y agoWhile it all feels insane from one perspective, it’s not hard to imagine a different narrative where the “founders of ‘insane asylum’ hacked the startup system, were product first, got acquired, and used their new home to pay down tech debt.”
- malux85 7y agoand made everyone else suffer? Also - they are NOT paying down the tech debt. The programmers and sysadmin staff doing overtime mentioned are.
- andrewfong 7y agoUh, as much as I enjoy reading this, posting this on Reddit seems particularly unwise for job security purposes. This seems to be an ongoing incident and there's more than enough info shared to de-anonymize.
- Operyl 7y agoI’ve come to the assumption that either this happened already months ago, or the much more likely scenario that it’s probably largely fake (if based on a true story)
- Thorrez 7y agoIt mentions December, so either that's a lie, it's recent, or it was December in some previous year.
- fierarul 7y agoThere's enough info if you are are somewhat related to that incident. For the rest of the world, no.
- kstenerud 7y agoThat was a cool nightmare scenario story, but holy mother of mercy was it ever tedious to read! I can tolerate some bad grammar, but somehow the combination of incorrect articles, words auto-spellchecked to something completely different, run-on sentences, and lack of punctuation, made it a real chore to read through without getting lost...
- caymanjim 7y agoI couldn't make it past the first block of text. I wonder if people who write like that realize how it makes them look? I'd never hire them.
- lucasverra 7y agoPersonally I found this piece pretty fun to read. The guy is ventilating some stress through prose, not applying to work with you.
- makarhum 7y agoI enjoyed it too. It reminds me of The Cuckoo's Egg by Cliff Stoll. Both were written in very similar style.
- caymanjim 7y agoI don't doubt that there's an interesting story in there. I tried to hang in long enough to find out. I can only take so much, though.
- andai 7y agoHang in there, man.
- johnchristopher 7y agoi en joyed it but s true that lack of pucntuation and grammatical errors made it harder to read then necessary. /s
- donio 7y agoI don't use Dropbox so I am wondering, is this fundamentally different from storing everything in S3? Is it like a single S3 bucket? Or multiple buckets on the same account?
- FascistDonut 7y agoDropbox is essentially just a cloud synced folder. Anyone who had the shared login credentials had access to the entire production server and fragments of it could be left on any computers that were synced to it. Anyone could also delete or modify anything and it would propagate the changes to everyone else's copies with very little logging or version control. You can restore a previous version of the file, but it isn't like git or anything... just more like recently saved versions.
- donio 7y agoIs it shared credentials even for business accounts? Surely they must have some kind of team based shared access option for that?
- FascistDonut 7y agoThe post said they had an enterprise account, which I assume could limit access via separate logins, but many of the users were sharing the same single set of credentials (e.g. everyone logging in with the same account), so there was no real access control or knowledge of who all had access using that account.
- planetjones 7y agoDropbox would really let you get to half a Petabyte ? I know it says as much space as your team needs, but wow. Do Dropbox start to charge per data stored when you take out an enterprise account ?
- useful 7y agohaha this is the kind of shit that happens when the engineering team cant get anything from the either the sysadmins or leadership. Can't run an application? Can't afford a license? Use the network share. No network share? Use dropbox.
- danielovichdk 7y agoWhen Danske Bank aquired Sampo Bank in Estland, Danske Bank spend one million hours on moving and lifting Sampo Bank's IT into Danske Bank's systems. That didn't work out. And we all know what happened then. When the board and ceo want's to aquired a business, IT is just something they believe can follow along easily. Most developers are straight up lazy and uprofessional to do a job, and mixing in security is just scary as hell.
- hsivonen 7y agoDanske had some _really bad_ technical ideas: https://hsivonen.fi/sampo-epic-multifail/ https://hsivonen.fi/sampo-epic-multifail/
- thendrill 7y agoOmg. It is not like he watched his buddy lose a finger trying to hang a bridge. It fucking just data. It is just made up numbers. There is a reason why higher up mangers and bosses don't go e shit about code or understanding technology. Coz in the end it doesn't matter. There is always a role playing nerd that would love to slay the dragon. So much nerds that get promoted because they belive in Dragons and such.
- pbourke 7y agoExcept when the made up numbers are the whole damned business. Or not (waves at Experian)
- thendrill 7y agoAhh well if we gonna focus on the numbers only. You are right. But then people should shut up about all this environmental and economical crisis. My numbers are fine screw you if your numbers are bad. Why should that be my problem?
- im3w1l 7y agoWhat are the advantages / disadvantages of going panic mode like this? Would just increasing the size of the dropbox to keep things muddling along and then trying to figure out a migration strategy come Monday morning be impossible for legal reasons?
- geuis 7y agoSounds like the big issues are at the compliance level. They’re in a time of the year where lots of companies have much more limited availability of engineering, management, and legal staff. Sure they could easily eat the cost of temporarily upgrading the Dropbox account. But as OP describes, there are huge implications in terms of security. They didn’t find out until today that a number of senior staff were laid off during the acquisition. Lots of room there for bad feelings. That opens them up to all kinds of vulnerabilities from disgruntled ex employees. Imagine the repercussions come post-holidays if something bad happens during the next few weeks before everyone is back on board and there’s a security breach over the meanwhile.
- Tistron 7y agoI worked in a software company once where we acquired a product to integrate into ours. It was developed by a self taught programmer in his 50ies that did versioning by making copies of the whole directory when he had something that worked, the way I also worked before university. Since he was the only developer and he was pretty good this mostly worked. His first name was/is Leif so from then on me and my friends speak of this style of "version control" as "using Leif Points". I guess because it is phonetically somewhat close to Save Points like in some computer games. - Are we using git or Leif Points for this project? - eh, how about git? This story sounds like they also used Leif Points, but for their database :O
- narag 7y agoI won't name the offender, but a well-known company I worked for did something funnier: zipping entire directory and checking the zip files in Visual Source Safe.
- masklinn 7y agoGiven how broken VSS is, that might have been safer than using VSS as if it worked properly.
- narag 7y agoLOL, I used it in 1996-97 and it worked OK for a modest setup with only six users and blocking at file level. Later heard some horror stories from people working at big shops, but no worse than svn stories, anyway I haven't seen any of those.
- mikestew 7y agoAbout your time of using VSS, I was working at Microsoft in developer support (working on COM data access, and not VSS, thank $DEITY). It was fine for small internal stuff we’d do, the stories I heard from industrial-strength use did not instill confidence for anything more than a half-dozen devs. OTOH, we shipped small components for a larger product using VSS, and I don’t recal any issues. Hated the workflow and interface, but it worked
- deleted 7y ago[deleted]
- someonehere 7y agoI am not a fan of Dropbox. Stories like this are why I don’t use it. A couple of years ago I inherited managing Dropbox for a company I was at. I’d never really used Dropbox in a more daily use company. I discovered Dropbox wasn’t very IT admin friendly. If you share out a personal folder as your team folder, IT can’t find your folder to share it to new members of your team. The person who owns the folder has to share it out. So when new employees ask why IT can’t access and share the folder, we had to explain that’s how Dropbox works. The other crappy thing I discovered about Dropbox from that time. I have no way to audit or tell who has what folder. If someone was sharing something illegal from their folder, I had to turn on the auditing capability and download logs as they were generated. It was confusing because if someone were sharing something and I was the admin of that account, I should have access to all files and accounts. Nope. So we dumped Dropbox before I left to avoid scrutiny down the road for any acquisitions of the company. Dropbox is still a consumer product trying to be a workplace tool IMO. Maybe others have a different opinion. This is a good story to tell on Monday.
- plg 7y agowhat do you suggest as an alternative. asking for a friend
- usaar333 7y ago> If you share out a personal folder as your team folder, IT can’t find your folder to share it to new members of your team. The person who owns the folder has to share it out. > It was confusing because if someone were sharing something and I was the admin of that account, I should have access to all files and accounts. Nope. In both cases, you should be able to assume the shared folder owner (via the admin console) and modify their permissions. Did this not work for you? (note: I'm an ex-Dropbox eng that worked on a lot of the sharing/team management systems)
- jborichevskiy 7y ago> The other crappy thing I discovered about Dropbox from that time. I have no way to audit or tell who has what folder. Have felt similar pain with G Suite Drive. Perhaps I simply didn’t find or didn’t have the proper permissions (I wasn’t the top level admin on the org) but at some point I wrote a Python script to output a list of users who had access to each file in our organization. I saw no way of generating this automatically.
- z3t4 7y agoThis story has many angles, the developer who built the software, the founder who took all the money, then sold the software and fired the developer. Then Dropbox who we all laughed at, is now used for distributed databases.
- manceraio 7y agoThis looks like how no-code projects will end up in the future.
- alimiracle 7y agoI had fun reading this, up to the point where I don't even care if it's real or not, it was entertaining.