3 ms·
This is a common problem I see across many things. We had a guy who spent two weeks writing a report script that collated data uploaded into S3 and wrote out a
by hvgk 5y ago
This is a common problem I see across many things.
We had a guy who spent two weeks writing a report script that collated data uploaded into S3 and wrote out another data lump into S3 after some post processing then sent this data lump to a guy via email. This entire thing had a bunch of lambda layers to pull in the pipeline, a build pipeline in jenkins, terraform to deploy it. The python script itself was about 200 lines of the equivalent of spreadsheet cell manipulation basically.
Only after it was implemented we found out it was only run every 3 months and took the original dude 5 mins to paste it into his existing excel workbook and the data popped out in another sheet instantly. This was of course a “massive business risk” and justified a project to replace it. What it should have been was a one page thing in confluence with the business process in it and the sheet attached. But no it now needs a GitHub repo full of shite and 2 separate people to understand every facet of it who require 3x the salary of the original dude each. And every 3 months, something is broken or it pukes an error which requires someone 3 hours of debugging and reverse engineering the stack to work out why.
Fundamentally, event sourcing, as the above, is the outcome of seeing something shiny and covered in marketing and wanting to play with it on any available business case rather than doing a rational evaluation of suitability. There are use cases for this stuff, but sometimes people don’t know when to apply the knowledge they have.
As for ES itself, I have resisted this one big time, settling on CQRS class functionality which is quite frankly no difference to query/mutate database APIs using stored procedures in PL/sql I dealt with in the dark ages. People also don’t have the architectural skills or conceptual ability generally to rationalise the change on data consistency on top of that or the recovery concerns. I would rather find another way to skin the scalability cat than use that architecture.
- larve 5y agoI wouldn't use the overengineering of a single script influence your view of the technology you named (S3 data laking, lambda pipelines, build in jenkins, terraform to deploy). Those are well understood concepts and technologies to do reporting. The problem here is that these tools were not in use at the company. Once you have 50 scripts run by different people on different schedules in different excel version, having everything go through S3 (immutable, cheap), ingested by lambdas (scalable, well versioned, monitored), built and deployed in jenkins using terraform (vs say, aws command line scripts) significantly reduces the complexity of the problem.
- tsss 5y agoYou don't give people enough credit. I know damn well that the "complicated" solution is at best a monumental waste of money but doing the same trivial CRUD shit for years on end is neither intellectually stimulating nor good for my career. Until "the business" finds a way to change that I will use every available opportunity for "resume-driven development". Really, I don't give a crap if the shareholders make money or not. I only care about my own paycheck.
- hvgk 5y agoOh yes same here. I wrote a code generator that wrote all the code in the end. Now I maintain a shitty code generator. You can’t win. I think you eventually run out of fucks. I have.
- speed_spread 5y agoThat's when you write a fuck generator, so fucks can be given automatically on your behalf while you fuck off somewhere else.
- hvgk 5y agoNow that’s genius automation :)
- deleted 5y ago[deleted]