6 ms·
> The vendor product stored every customer transaction as a json record in a giant json document.... Adding a new transaction involved reading the entire json d
by irjustin 2y ago
> The vendor product stored every customer transaction as a json record in a giant json document.... Adding a new transaction involved reading the entire json document out of the database, then appending the new record to the end.
Does this sound insane to you? Obviously it should, but in the vein of "insane things" I was helping do tech diligence of a potential investment for a fund. The startup's users table also contained the "tickets/booking" data. Each ticket was a column. So if your most active user had 5 tickets in total history, you needed 5 columns. These guys had 500+ columns by the time I reviewed it, and were looking for an investment to "scale".
Of course, it's a solvable problem, but as you can imagine everything was designed in some backwards, janky way. That was just the most obvious "wtf" moment.
They did not get the investment.
- prakhar897 2y agoIt’s genius. Schmooze the execs with fancy dinners and trips. Deliver a horrible product so they need you for the foreseeable future. The story itself says the CTO is unaware of everything. From his perspective, everything worked out fine in the end. Win for everyone except for the devs who put in 80 hour weeks.
- osigurdson 2y agoYou missed the part where they say they felt like a rock star. Different people are motivated by different things. The key is, if you are very hard working try to channel it in a way that will actually benefit you.
- t43562 2y agoYes, don't let other people channel it towards themselves! You're in a company, - they will fire you when it becomes convenient to do so - don't burn yourself out for them!
- dvaun 2y agoThey read that column-oriented DBs delivered great performance.
- shagie 2y agoThat sounds like some Bad CaRMa that I read about once... https://www.red-gate.com/simple-talk/opinion/opinion-pieces/bad-carma/ https://www.red-gate.com/simple-talk/opinion/opinion-pieces/... ... which while there were other red-gate posts out there, I can't find that one ever submitted as a dup to link the comments about it (this shall be rectified).
- datadrivenangel 2y agoWriting "RUN LIKE HELL" on a white board is insane, and should not be ignored.
- labster 2y agoYeah, column-oriented works best for musical performance because it makes it easy to switch from Ionian to Dorian mode.
- deleted 2y ago[deleted]
- CoolCold 2y agothanks for explaining on what are column-oriented DBs are in a ELI5 way
- yjftsjthsd-h 2y ago... In case this is serious, the joke is that that is absolutely not what that term means.
- tangjurine 2y agoHow about reading the entire Json document out of the database, making a copy of that document, updating the copy, and saving that copy to the database... Is how someone on my old team designed an internal tool
- danielheath 2y agoDepending on load patterns, for an internal tool that might be perfectly reasonable; you can trivially look at the history, for one thing.
- Spivak 2y agoIs this not how you update a JSON column on a db that doesn't have (or you choose to not use for reasons) a native JSON type with a sprinkling of data should be immutable?
- dude187 2y agoWell without some form of locking you have a race condition. Two updates done at the same time can have the overwrite the other
- dikei 2y agoOne team in my company insists on storing everything related to a target in an MB-size Elasticsearch document and then do all the aggregation client-side, because they already use ES for everything else and don't know how to use a relational database.
- klabb3 2y agoI don’t understand the confidence that leads to such misuse. I’m NOT a db guy by any means, and yet when I need one I’m carefully looking up things like “what’s the space complexity of compound indices”, at a development stage. Defensive and paranoid. I guess there are a lot of cowboys out there.
- hobs 2y agoMost people who use an index wrong dont understand space and time tradeoffs at all, how to measure if there is a problem, or how to identify what a thing should be in the first place. Most people who at least think a bit get pretty far before the vendor specific gotchas start biting.
- el_benhameen 2y agoMan, I’ve had one of those “I’m an idiot and only an idiot would rely on me to build things” kind of days. This made me feel better.
- summerlight 2y ago> Each ticket was a column. So if your most active user had 5 tickets in total history, you needed 5 columns. I was thinking like, "Oh yeah, they have 5 tickets, so need 5 rows. What's wrong?". And I read it again and found it's the word "column". It must be a big fun to write a query for such tables.
- emrah 2y agoHonestly this is not necessarily a good reason not to invest. Sure this needs to be fixed, plus whoever allowed this to happen should be replaced but analyzing the business as an investment opportunity and passing because of this is just as insane in my opinion
- nicolas_t 2y agoExecution is what matters in startups, if they either can't find developers who are halfway decent or if a cofounder wrote this, that's a major red flag that they don't know how to execute.
- SkyPuncher 2y agoThis is something I’d do it a ship it tomorrow, trash it the day after MVP. In a consultant project with nearly a year to deliver, no way.
- luhn 2y agoWould it help, though, even in the short term? Making an actual `ticket` table really isn't that much work. If for some reason it is, you can always use an array type, or even shove JSON into a text column. Even if you're not thinking further than an MVP—hell, further than 5pm—one column=one ticket seems the most awkward and difficult solution I can think of.
- albert_e 2y agothe most far-fetched justification I can muster up for this: the UI/user requried a tabular display of all tickets as columns with one row per user ... and they had a UI that only displays DB table contents as-is. Nah, even for a 5pm demo that just sounds absurd (with up to 500 tickets per user).
- hansvm 2y agoMy best guess is it started as a single ticket column, then maybe a manual migration to quickly add a second when a rat's nest of code was built around the single column already.
- melozo 2y agoI worked at <large tech company everyone hates> and we had this exact architecture. The senior engineer (who went to Stanford and was proud of it) designed this system. I had long arguments with him about how this system wouldn't scale when we launched it, and used real production data to prove that it would happen. No one listened to me, it launched and imploded within a few weeks, and soon after I transferred off the team. The worst part is, that senior engineer was eventually promoted and that system was passed off to an entirely new team to battle with. That entire system was terribly designed, and I think you can guess why.
- beachy 2y agoShow me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975) It astonished me that people can ever think this is a good idea (everything in a single json document) or that anyone who thought it was could ever get funding or even get through any basic customer due diligence.
- cess11 2y agoOne part of my work entails entering companies on their death bed and going through their software development and related investments. I see a lot of JSON-storage in the initially well funded ones. They cough up some crap on Firebase, get some tens of millions of dollars in venture funding, burn through, and never leave Firebase. It's really, really hard to sell systems implemented with these sexylicious clown technologies. Like, extremely hard. Apparently no sane person with money wants to pay a non-trivial amount for a pile of locked in JSON and Python scripts, and I haven't managed to add any insane ones to my network yet. The small companies without funding that topple over and end up in my way, more commonly they instead put some stuff on a VPS or two, and typically the market values them quite a bit higher. Fashionable technology choices burn through a lot of investment money. Seems to me like maybe a sign that money has been a bit too cheap a bit too long.
- itronitron 2y agoSo which is preferable storage from an eventual valuation perspective, organized files with a set of scripts, or a relational database?
- nicolas_t 2y ago> Does this sound insane to you? Early in my career, a client came to us with their internally developed perl web app that was very slow and asked us to help them. They were storing all of the information as csv files and would do exactly that, read the entire file, add a new row etc... We explained to them that we should rewrite it for them :)
- knodi123 2y ago> Does this sound insane to you? Obviously it should, I dealt with the exact same thing in my 2nd ever job. Entire customer and product database (with plaintext passwords) stored in a multi-meg public-facing .js flat file that had to finish loading before the app could do anything (with early-2000s internet). And to top it off, the app was a single monolithic file, in a directory with names like index.1.js, index.final.js, index.newest.js, index.45.js, etc etc. I had enough experience with best practices to go to our CEO and get the CTO fired, and went about rewriting it with git, mysql, and server-side logic with an actual architecture. And then, the windows server it was all running on got pwned into a porn server, and somehow it was my responsibility despite me never having seen this server and not even having admin permissions. Man, my first few jobs were educational!
- itronitron 2y agoI heard rumor back in the late `90's of an e-commerce site that stored the prices for it's products in the browser cookie. Presumably a motivated buyer could go in and edit the cookie before checkout, although I don't know if that ever actually happened.
- kstrauser 2y agoThe one I’m thinking of stored prices in the HTML form as a hidden field. It trusted the client-submitted value when deciding how much to charge you. The justification I heard for the ethics of enjoying the home-rolled discount was that it was similar to haggling. The store chose to accept the offer. Not sure I totally accepted that, but appreciated the creative thinking.
- AntonyGarand 2y agoThis brings back memory: This was the case for a gold-buying website for the Runescape game in the 2000s. You could edit your cookies or other front-end facing information to change the price of items in your cart, so you could buy gold or items for much cheaper than the market rate. At some point, while the vulnerability remained, they started cancelling orders abusing this and manually checking the orders. I think you could still find some old youtube videos or threads on obscure forums with enough digging about that topic, that's how I learned of it initially. So this was a real thing!
- mrweasel 2y agoThose types of design, especially the giant json document, has to be part of the reason some companies pushed pretty hard against the GDPR. Imagine trying to remove personal data from such a system.
- karmajunkie 2y agoI worked very briefly for a company (that I won't name) that did this with elasticsearch. Every customer basically had a customized form for data entry, and every field on that form became a field in ES, even if that same field name and type was on every other form in the system. Eventually they had to contract with another company to run a customized build of ES on a bespoke cluster because they blew past the default 10k limit on the number of fields in ES. When it became clear that they had zero interest in fixing this, I tendered my resignation and quickly found a new gig with sane architectural principals.
- CRConrad 2y agoHopefully those sane principals implemented sane principles.
- karmajunkie 2y agotouché…