4 ms·

I once worked on an application that integrated with a third party api. The way it did this was with a large and horrible client library that used a separate db

by sfn42 2y ago

I once worked on an application that integrated with a third party api. The way it did this was with a large and horrible client library that used a separate db to cache the data. The data was then fetched from the main application and used to rebuild the pages (in the main db) based on this data once a day. The library had lots of problems, and one day it stopped working. I was tasked with fixing it - we had the source code, it was purchased from someone and copied into the repo. I spent most of the week if not more trying to figure out what was wrong, but I couldn't. What I did learn was that this library was some of the worst most pointless code I'd ever seen. So I told the team that I think I can rebuild this thing from scratch faster than I can fix this bug. The intermediate db is pointless and most of the library code is dead, the rest is garbage. I can make a simple little thing that does exactly what we need better, faster and easier. Nope. No bueno, fix what we have. So I spent a few hours over the weekend, less than a workday, building the new solution. Come Monday I had it pretty much working, a few things needed to be done but it already supported the use case. The pages were built correctly, they had the necessary content but some things were a bit messed up, nothing difficult to fix. Showed it to the team, said I want to use this and delete the old stuff - nope. The only half-decent explanation I got was that the client had paid a way too high amount of money for this garbage library and I guess the team lead didn't want to tell them we wanted to throw it out or something like that.

skydhash 2y ago
As soon as the stakeholders have a say in the technical implementation (other than financial and time constraints), you’re in a world of hurt.
deleted 2y ago
[deleted]
kstrauser 2y ago
Sigh. I worked at a shop that was spending months waterfalling a frontend to some background API calls. I finally got annoyed enough to spend a weekend actually implementing the thing as a Django app. There. Done. I got my ass handed to me by management for not going through the proper processes. I learned something that day: I never want to work somewhere that engineers serve the processes and not the reverse. There are some that are good and necessary: like “thou shalt deploy via CD and not SSH into prod to edit code”. There are others that only exist to serve bureaucracy, and those try my patience.
- hansvm 2y ago
  > thou shalt not live-edit prod Even then, there's some nuance. During an outage, cowboy coding can get you back up and running much faster.
  - kstrauser 2y ago
    Yeah, depending on the particulars of a system. If you're at a startup and report to the CTO, that might be perfectly fine in an emergency. At a company with a few million users, almost certainly not. There's a spectrum of possibilities.
    - hansvm 2y ago
      In an emergency, that sort of thing even happens at Google (more for their smaller services, and almost always in the form of auto-LGTM hot-fixes bypassing the normal checks rather than actual live-editing of a script or binary, but even that latter thing happens occasionally). There are checks and controls, but an emergency for a billion users is a big deal.
titusjohnson 2y ago
I've been on both sides of this table multiple times, as the IC and as the Manager of an eager IC. Here's a list of all the reasons why I as your manager would also flat-out say No to this situation. (These are of course heavily tainted by my own recent experience of trying to coach a mid-level dev through a very similar problem) - "Pretty much working" means all the fun stuff is done and the actual hard thing is left to wrap up. It's a useless estimate that only accounts for your coding work, which is usually the smallest amount of work performed on an integration feature like this. - It's a rewrite so we've gotta do a full regression test on every piece of data that thing pulls back. Since it's old functionality it's not fully covered by our automated tests, so this goes to QA. Our QA team is overloaded so this unauthorized, not on the roadmap project now needs to jockey for priority with things that Marketing is literally making artifacts for _today_. - "It's already built" isn't really a justification for a priority change, so now I'm in the awkward position of changing priorities for a non-roadmap task and justifying this to every single stakeholder who is respecting the process, or telling you it'll be 2 months minimum before QA can even think about it. Either way no one is happy and now I have to worry about you going rogue again and trying to work channels around me to get this thing shipped out of band. - It's a full rewrite and going through manual QA, so it's nearly guaranteed that critical, but undocumented business rule fixes were missed. Somewhere in that library is a weird function holding up the world, but it was "obviously cruft" and left out. There's a good chance we won't find the issue until it has already polluted a ton of Prod data. That's why I won't let you do Developer QA. You've only been here a year and this service predates you, me, and the rest of the team, we literally have no context. - If the client finds out we did a full rewrite, they too are going to do a full regression test on their end. Do you know the size of the shitstorm this is going to bring on us? Every single question, problem, feature change, bug, enhancement, communication, _everything_ we went through over the last XX years since we built this integration is going to resurface. I get re-litigate every. single. thing. "Since you're working on our integration can we get XX, XX, and XXXX?" (each is a sprints worth of dev time minimum), "YYY isn't working, did you guys break it again?" (it's always been broken but now someone gets to spend 3 hours in Datadog pulling logs to prove this). - I've been using the "Rewrite This Library" and "Refactor That Service" projects as leverage to negotiate for more budget to bring on 2 more headcount so that we could actually do those rewrites with proper time and space. You talking about getting 80% done over a weekend has completely undermined the work I've put into this effort, and at the same time didn't remove the Refactor issue from my backlog. Now I will essentially have to shit-talk you in my own 1-on-1s in order to regain lost ground. "sfn42 is a decent developer but he just doesn't have a lot of context to what's happening outside his role. Needs more time in the oven before he gets the bump to Sr. Maybe I can pull him into more planning meetings so he can start growing in this area" -- congrats you just got invited to 6 hours of meetings a month regarding work you won't perform. - In 6 months when our team is planning out some future work that's just way too much for the headcount & timeline we have, and you bring up "we could really use another Sr. Dev or two, any word on our headcount request?", I might reply politely with a "still no word if we can pull that off this quarter", but internally I'm wondering if the pain of bringing a new dev up to speed is less than the pain of working with you. - Lastly, the most petulant reason, you were told No last week. I'm sorry you lost a weekend to this, but a No is a No and I need you to understand that. Other things are happening at this company outside the scope of your purview. Again, this is all drawn from my own experience. I had a mid-level dev show me a huge refactor he started on the weekend. He was convinced it was almost done, "just a few small things left" is an exact quote. However I knew that this part was literally the smallest bit of the effort. I was seeing at least 3 months of work across 4 departments before it would actually be Done, in Production, and working to our satisfaction. If I had the space I would normally be just fine letting the young fella just experience that pain. Make him do the scheduling, put him on point for everything, and just let him spin on it for a month or so. I did not have that time and space, so instead we spent a few hours white boarding out the rest of what needed to happen, and thankfully he mothballed his project of his own volition.
rqtwteye 2y ago
"I spent most of the week if not more trying to figure out what was wrong, but I couldn't. What I did learn was that this library was some of the worst most pointless code I'd ever seen." I would probably be skeptical if somebody made these statements. You don't know what's wrong and you declare the code to be pointless. Maybe you put a good effort into it but I have heard it too many times that somebody declared "this is all crap. we need a rewrite". Most times they just didn't put the effort into understanding the current code. And usually the time to get to "pretty much working" is often only a fraction of what it takes to "totally done".
- sfn42 2y ago
  The problem was not that I did not understand the code. I understood it just fine, it wasn't complicated it was just bad and old. All it did was get some data from an api, change it somewhat, store it in a db. Then a scheduled job would call a method which would get that data, change it a bit more and return it where it would be changed yet a bit more and stored as pages for the main web app. There was no reason all these data mutations couldn't have been in one place instead of all over the place. There was no reason to store it in one db then get it from that one just to store it in another db. Someone said the third party api was slow and unreliable but I don't see how that's relevant - if the api is down then you don't get updated data, it doesn't matter if we have outdated data in an intermediate db. We already have that outdated data in our main db and we'll get updates when the api starts working again. During testing I had absolutely no issues with the performance of the api, it transferred all the data we needed in a completely acceptable amount of time, and this was just in a nightly scheduled job anyway so if it had taken a minute that would have been fine as well. But it didn't, it responded in milliseconds. I never noticed any unreliability on their side either, but if it had been unreliable that would have been totally fine. The app just wouldn't have gotten updates until it started responding again. Nothing can solve that problem. I honestly can't remember what the actual problem was or how I fixed it in the end. The code had been in production for years and only received the minimum necessary amount of changes. Some dependency or something probably broke from years of nobody wanting to touch that huge piece of crap. But that's not why I say it was bad and pointless. It was bad because whoever wrote it didn't know about libraries for xml parsing and had implemented all of the parsing from scratch with string operations. We're not talking about real parsing here with lexers and tokenizers and stuff. We're talking about what you might expect if you gave a mediocre first year CS student the task of parsing some specific xml. The db interaction was similarly overcomplicated and outdated, and the code itself was sloppy and full of old messes nobody had bothered to clean up. All it did was get some data, store it and make it available through some method calls, and for that there were like 50k loc most of which was dead and most of the code still in use was that monstrosity of a homerolled xml parser. The things left to do on my new solution were trivial. Some of the columns had html tags and stuff like that in them, it just needed to be cleaned out where necessary. Some other stuff needed to be modified a bit. I did not skip it because it was hard, I skipped it because it was tedious and I didn't want to spend all the effort before I got the green light, which turned out to be a great decision because it didn't get the green light. And they probably still to this day waste man-hours on keeping that piece of crap running.

by sfn42 2y ago

skydhash 2y ago

As soon as the stakeholders have a say in the technical implementation (other than financial and time constraints), you’re in a world of hurt.

deleted 2y ago

[deleted]

kstrauser 2y ago

Sigh. I worked at a shop that was spending months waterfalling a frontend to some background API calls. I finally got annoyed enough to spend a weekend actually implementing the thing as a Django app. There. Done. I got my ass handed to me by management for not going through the proper processes. I learned something that day: I never want to work somewhere that engineers serve the processes and not the reverse. There are some that are good and necessary: like “thou shalt deploy via CD and not SSH into prod to edit code”. There are others that only exist to serve bureaucracy, and those try my patience.

hansvm 2y ago
> thou shalt not live-edit prod Even then, there's some nuance. During an outage, cowboy coding can get you back up and running much faster.
- kstrauser 2y ago
  Yeah, depending on the particulars of a system. If you're at a startup and report to the CTO, that might be perfectly fine in an emergency. At a company with a few million users, almost certainly not. There's a spectrum of possibilities.
  - hansvm 2y ago
    In an emergency, that sort of thing even happens at Google (more for their smaller services, and almost always in the form of auto-LGTM hot-fixes bypassing the normal checks rather than actual live-editing of a script or binary, but even that latter thing happens occasionally). There are checks and controls, but an emergency for a billion users is a big deal.

titusjohnson 2y ago

I've been on both sides of this table multiple times, as the IC and as the Manager of an eager IC. Here's a list of all the reasons why I as your manager would also flat-out say No to this situation. (These are of course heavily tainted by my own recent experience of trying to coach a mid-level dev through a very similar problem) - "Pretty much working" means all the fun stuff is done and the actual hard thing is left to wrap up. It's a useless estimate that only accounts for your coding work, which is usually the smallest amount of work performed on an integration feature like this. - It's a rewrite so we've gotta do a full regression test on every piece of data that thing pulls back. Since it's old functionality it's not fully covered by our automated tests, so this goes to QA. Our QA team is overloaded so this unauthorized, not on the roadmap project now needs to jockey for priority with things that Marketing is literally making artifacts for _today_. - "It's already built" isn't really a justification for a priority change, so now I'm in the awkward position of changing priorities for a non-roadmap task and justifying this to every single stakeholder who is respecting the process, or telling you it'll be 2 months minimum before QA can even think about it. Either way no one is happy and now I have to worry about you going rogue again and trying to work channels around me to get this thing shipped out of band. - It's a full rewrite and going through manual QA, so it's nearly guaranteed that critical, but undocumented business rule fixes were missed. Somewhere in that library is a weird function holding up the world, but it was "obviously cruft" and left out. There's a good chance we won't find the issue until it has already polluted a ton of Prod data. That's why I won't let you do Developer QA. You've only been here a year and this service predates you, me, and the rest of the team, we literally have no context. - If the client finds out we did a full rewrite, they too are going to do a full regression test on their end. Do you know the size of the shitstorm this is going to bring on us? Every single question, problem, feature change, bug, enhancement, communication, _everything_ we went through over the last XX years since we built this integration is going to resurface. I get re-litigate every. single. thing. "Since you're working on our integration can we get XX, XX, and XXXX?" (each is a sprints worth of dev time minimum), "YYY isn't working, did you guys break it again?" (it's always been broken but now someone gets to spend 3 hours in Datadog pulling logs to prove this). - I've been using the "Rewrite This Library" and "Refactor That Service" projects as leverage to negotiate for more budget to bring on 2 more headcount so that we could actually do those rewrites with proper time and space. You talking about getting 80% done over a weekend has completely undermined the work I've put into this effort, and at the same time didn't remove the Refactor issue from my backlog. Now I will essentially have to shit-talk you in my own 1-on-1s in order to regain lost ground. "sfn42 is a decent developer but he just doesn't have a lot of context to what's happening outside his role. Needs more time in the oven before he gets the bump to Sr. Maybe I can pull him into more planning meetings so he can start growing in this area" -- congrats you just got invited to 6 hours of meetings a month regarding work you won't perform. - In 6 months when our team is planning out some future work that's just way too much for the headcount & timeline we have, and you bring up "we could really use another Sr. Dev or two, any word on our headcount request?", I might reply politely with a "still no word if we can pull that off this quarter", but internally I'm wondering if the pain of bringing a new dev up to speed is less than the pain of working with you. - Lastly, the most petulant reason, you were told No last week. I'm sorry you lost a weekend to this, but a No is a No and I need you to understand that. Other things are happening at this company outside the scope of your purview. Again, this is all drawn from my own experience. I had a mid-level dev show me a huge refactor he started on the weekend. He was convinced it was almost done, "just a few small things left" is an exact quote. However I knew that this part was literally the smallest bit of the effort. I was seeing at least 3 months of work across 4 departments before it would actually be Done, in Production, and working to our satisfaction. If I had the space I would normally be just fine letting the young fella just experience that pain. Make him do the scheduling, put him on point for everything, and just let him spin on it for a month or so. I did not have that time and space, so instead we spent a few hours white boarding out the rest of what needed to happen, and thankfully he mothballed his project of his own volition.

rqtwteye 2y ago

"I spent most of the week if not more trying to figure out what was wrong, but I couldn't. What I did learn was that this library was some of the worst most pointless code I'd ever seen." I would probably be skeptical if somebody made these statements. You don't know what's wrong and you declare the code to be pointless. Maybe you put a good effort into it but I have heard it too many times that somebody declared "this is all crap. we need a rewrite". Most times they just didn't put the effort into understanding the current code. And usually the time to get to "pretty much working" is often only a fraction of what it takes to "totally done".

sfn42 2y ago
The problem was not that I did not understand the code. I understood it just fine, it wasn't complicated it was just bad and old. All it did was get some data from an api, change it somewhat, store it in a db. Then a scheduled job would call a method which would get that data, change it a bit more and return it where it would be changed yet a bit more and stored as pages for the main web app. There was no reason all these data mutations couldn't have been in one place instead of all over the place. There was no reason to store it in one db then get it from that one just to store it in another db. Someone said the third party api was slow and unreliable but I don't see how that's relevant - if the api is down then you don't get updated data, it doesn't matter if we have outdated data in an intermediate db. We already have that outdated data in our main db and we'll get updates when the api starts working again. During testing I had absolutely no issues with the performance of the api, it transferred all the data we needed in a completely acceptable amount of time, and this was just in a nightly scheduled job anyway so if it had taken a minute that would have been fine as well. But it didn't, it responded in milliseconds. I never noticed any unreliability on their side either, but if it had been unreliable that would have been totally fine. The app just wouldn't have gotten updates until it started responding again. Nothing can solve that problem. I honestly can't remember what the actual problem was or how I fixed it in the end. The code had been in production for years and only received the minimum necessary amount of changes. Some dependency or something probably broke from years of nobody wanting to touch that huge piece of crap. But that's not why I say it was bad and pointless. It was bad because whoever wrote it didn't know about libraries for xml parsing and had implemented all of the parsing from scratch with string operations. We're not talking about real parsing here with lexers and tokenizers and stuff. We're talking about what you might expect if you gave a mediocre first year CS student the task of parsing some specific xml. The db interaction was similarly overcomplicated and outdated, and the code itself was sloppy and full of old messes nobody had bothered to clean up. All it did was get some data, store it and make it available through some method calls, and for that there were like 50k loc most of which was dead and most of the code still in use was that monstrosity of a homerolled xml parser. The things left to do on my new solution were trivial. Some of the columns had html tags and stuff like that in them, it just needed to be cleaned out where necessary. Some other stuff needed to be modified a bit. I did not skip it because it was hard, I skipped it because it was tedious and I didn't want to spend all the effort before I got the green light, which turned out to be a great decision because it didn't get the green light. And they probably still to this day waste man-hours on keeping that piece of crap running.