6 ms·

> The entire thing is stitched together by spreadsheets that are parsed by Python, dropped into S3, parsed by Lambdas into more S3, the S3 files are picked up b

by neilv 3y ago

> The entire thing is stitched together by spreadsheets that are parsed by Python, dropped into S3, parsed by Lambdas into more S3, the S3 files are picked up by MongoDB, then MongoDB records are passed by another Lambda into S3, the S3 files are pulled into Snowflake via Snowpipe, the new Snowflake data is pivoted by a Javascript stored procedure into a relational format... and that's how you edit someone's database access. That whole process is to upload like a 2KB CSV to a database that has people's database roles in it. Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.

syntaxing 3y ago
I’m genuinely curious what a unit test for something would look like.
- jacquesm 3y ago
  Checksum on the resulting csv with a parallel implementation of the whole pipeline ;)
  - syntaxing 3y ago
    I don’t work with large databases so pardon my ignorance. Is there typically a “unit test” bucket you run it on or do you just put in test entries on a production bucket?
    - hightrix 3y ago
      We tend to only test what is being tested. So, most DB calls are mocked in our unit tests. For stored procs or other tests that need to be run on a DB, we use a test DB that is setup to mirror production. I'd bet there are a 100 different answers to your question though. This is the way we handle it.
    - jacquesm 3y ago
      Normally you'd fire up a separate environment, mock the process and see if it produces the expected results. By the time you put 'test entries in a production bucket' there are so many lines crossed that it likely won't end well even if the tests do pass.
- yonixw 3y ago
  // TODO
  - ak39 3y ago
    LOL. What did this comment (no pun!) do for your HN karma points?
- tommek4077 3y ago
  That is probably quite straight forward and of course they have 100% coverage.
- FartyMcFarter 3y ago
  That would be an integration test, not a unit test.
- wredue 3y ago
  We usually use very complicated UIPath flows to “test” these things. If that doesn’t exist yet (it usually doesn’t), we test manually, but only core workflows.
topaz0 3y ago
Of course the answers on stack overflow are partly a result of resume-driven answerers.
baz00 3y ago
Everything I look at these days looks like this. And most of the time it doesn't even solve the initial problem statement but everyone is too naive to even realise that. The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it. The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%
- foobiekr 3y ago
  I know of an engineer who built a work queue by having a chain from an app to kafka to a processor to Kafka to a database writer. Literally instead of a table. This stuff is everywhere. Microservices made it worse and half legitimized it.
  - JohnMakin 3y ago
    This must be common or we worked at the same company, seen this exact pattern. It went like: App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
  - Groxx 3y ago
    I've seen so many people spend weeks if not months "working" to avoid doing a trivial database migration. Database fear is overwhelmingly powerful in a lot of people it seems.
oooyay 3y ago
You can actually make a very comfortable career of Senior and Staff by learning to identify this kind of work/system and proposing ways to simplify it. These kinds of systems, as the author pointed out, are incredibly expensive and inefficient, but look readable on an architecture diagram.
- lainga 3y ago
  > You can actually make a very comfortable career of Senior and Staff by learning to identify this kind of work/system and proposing ways to simplify it. Where?
  - tomrod 3y ago
    IT departments, typically, though occasionally there are whole companies that work in "technology" where this type of work can be found. I said the above as a jest, but seriously, simplification of complex stacks has been a good consulting gig.
  - foobiekr 3y ago
    Most large companies. There is a stark difference between the distinguished engineers and the tier below them in terms of asking people to stop doing things badly.
  - oooyay 3y ago
    I've typically worked in SRE and platform engineering work and that's where I've gotten exposed to these kinds of Rube Goldberg machines. Make a short list of them when you find them and then use them as a hit list during "cost cutting". Most people don't want to touch these systems because they look big and expansive and generally "work". They're just very poorly optimized. Dare I say, any time I see a function as a service my brain immediately drifts to inspecting the cost implications of said process.
- mring33621 3y ago
  As opposed to all those people that make similarly comfortable careers in middle and upper mgmt by identifying simple systems and complicating them beyond recognition?
foobiekr 3y ago
I see stuff like this every day. It is a natural consequence of people who only “develop” by gluing things together. God help them if they’d actually have to write some core function themselves.
- pphysch 3y ago
  The worst kind of "DevOps engineer" that doesn't really understand development, operations, or engineering. But hey, they can run some docker and git commands and piddle around an AWS GUI, which means they are highly technical.
  - candiodari 3y ago
    You don't give enough credit to organization chart and project driven engineering. When developing anything: 1) you don't get to touch anyone else's code. And another department's code? Something another manager's team manages ... that amounts to treason. Never for any reason. MAYBE if they've totally abandoned it and you absolutely need it (but only during unpaid overtime) 2) you don't get to spend ANY time on anything outside of the current project or JIRA ticket. Any time at all. So really, NOT optimizing anything is faster and cheaper. Just look at all the spreadsheets made!
  - wredue 3y ago
    >piddle around in an AWS GUI I’ve had enough calls with the “Senior/Technical Lead Azure Cloud Engineers” telling them exactly what they need to do that me and them really really don’t get along. I don’t do any of that shit and even I can muddy my way through it, but these people cannot. The real kicker of it is how much these people make. And you know how there are those people who, every time you need to work with them, they answer a teams call and then “need to get to my computer, give me 5” and their status is perpetually set to away? I don’t want to RTO at all, but dealing with this team almost makes me think I’m wrong about that.
- vsareto 3y ago
  >It is a natural consequence of people who only “develop” by gluing things together. God help them if they’d actually have to write some core function themselves. That's on the industry for not training and gating well. It would be nice to have glue/plumber positions so expectations are not out of line too.
latexr 3y ago
That reads like the KRAZAM Microservices sketch. https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
- neilv 3y ago
  I've used that video to explain to business people. It's watchable, and communicates important ideas of what a poopshow this can easily become, without having to talk about real partners/teams close to home as problems.
- Lacerda69 3y ago
  KRAZAM is a prophet and must be protected at all cost
- ludicity 3y ago
  At my workplace (the one in the post), whenever one of the good engineers asks about how something works and it's one of these spaghetti-balls, we chorus "It's the design of our backend, okay?"
gruturo 3y ago
:O And I thought my unholy xmllint -xpath (bad stuff, lots of slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/ -e /ohyeahthistoo/somethingelse/ | grep something | while read AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort was as bad as things get. I need to get my horror code game up. I mean, not only is the code awful, its very purpose is horrifying (XML to CSV with some transformations, bit of math, all without being able to use any external sources due to security, only what's in a baseline RHEL7 (soon 8, yay!) ). I promise I'll rewrite it in python at some point.
tshaddox 3y ago
I think it's "iterative-StackOverflow-driven development" most of the time, and that actually causes the increased popularity of those resume keywords.
kmfrk 3y ago
... And a partridge in a pear tree.
feoren 3y ago
This sounds like absolute hell. This is everything I hate about modern software development.
deleted 3y ago
[deleted]
crooked-v 3y ago
The more experience I have, the more I start to think the Omnigres people are right about "just put literally everything into Postgres".
- codrilya 3y ago
  [dead]
toasted-subs 3y ago
You have no idea how unbelievably annoying it is to work in a company that doesn't a well defined architecture. Every "buzz word" service should be easily justified. This is why I hate recruiters, I can't even tell you how many times I've had a recruiter call me saying they are looking for service XYZ. The same concept rephrased in my resume. I have to rewrite my resume just to satisfy these people? No thanks.
- recfab 3y ago
  I had recruiter pull that in my most recent job search. Has to stick "C#/..." in front of everything because they didn't understand that ASP.NET, WPF, WCF, WinForms and several other C#-specific tech had anything to do with .NET.
neycoda 3y ago
So, what tech service can I add to that bloated pipeline as a middle-man to get a fraction of a penny per transaction?
Vicinity9635 3y ago
I feel like this might be case of data engineers. They're not usually software engineers. They're tool users not tool makers. So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.
- entropicdrifter 3y ago
  Agreed. I've been "the backend engineer who works with the data engineers" for several years now and I've seen their general trend of re-inventing the wheel the hard way a number of times. I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.
- scruple 3y ago
  I stepped into a Data Engineering Lead role in 2019. Stepped out of it in 2021. My team was the first in the org to really approach data engineering and we were all software engineers. I'm told that the systems we built have largely been replaced by Rube Goldberg machines pieced together by the folks who came after us. Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!
Agentlien 3y ago
I feel like I've never seen anything even reminiscent of this bad in the twelve years I've worked as a software engineer. I really want to believe this pipeline as described is satire. Yet, somehow, it does not quite seem that way. This scares me. But also somehow explains why some companies contain so incredibly much more engineering staff than I can possibly explain looking at their output.
BonoboIO 3y ago
Resume Driven development - RDR Never heard that before, but that’s so on point.

by neilv 3y ago

syntaxing 3y ago

I’m genuinely curious what a unit test for something would look like.

jacquesm 3y ago
Checksum on the resulting csv with a parallel implementation of the whole pipeline ;)
- syntaxing 3y ago
  I don’t work with large databases so pardon my ignorance. Is there typically a “unit test” bucket you run it on or do you just put in test entries on a production bucket?
  - hightrix 3y ago
    We tend to only test what is being tested. So, most DB calls are mocked in our unit tests. For stored procs or other tests that need to be run on a DB, we use a test DB that is setup to mirror production. I'd bet there are a 100 different answers to your question though. This is the way we handle it.
  - jacquesm 3y ago
    Normally you'd fire up a separate environment, mock the process and see if it produces the expected results. By the time you put 'test entries in a production bucket' there are so many lines crossed that it likely won't end well even if the tests do pass.
yonixw 3y ago
// TODO
- ak39 3y ago
  LOL. What did this comment (no pun!) do for your HN karma points?
tommek4077 3y ago
That is probably quite straight forward and of course they have 100% coverage.
FartyMcFarter 3y ago
That would be an integration test, not a unit test.
wredue 3y ago
We usually use very complicated UIPath flows to “test” these things. If that doesn’t exist yet (it usually doesn’t), we test manually, but only core workflows.

topaz0 3y ago

Of course the answers on stack overflow are partly a result of resume-driven answerers.

baz00 3y ago

Everything I look at these days looks like this. And most of the time it doesn't even solve the initial problem statement but everyone is too naive to even realise that. The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it. The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%

foobiekr 3y ago
I know of an engineer who built a work queue by having a chain from an app to kafka to a processor to Kafka to a database writer. Literally instead of a table. This stuff is everywhere. Microservices made it worse and half legitimized it.
- JohnMakin 3y ago
  This must be common or we worked at the same company, seen this exact pattern. It went like: App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
- Groxx 3y ago
  I've seen so many people spend weeks if not months "working" to avoid doing a trivial database migration. Database fear is overwhelmingly powerful in a lot of people it seems.

oooyay 3y ago

You can actually make a very comfortable career of Senior and Staff by learning to identify this kind of work/system and proposing ways to simplify it. These kinds of systems, as the author pointed out, are incredibly expensive and inefficient, but look readable on an architecture diagram.

lainga 3y ago
> You can actually make a very comfortable career of Senior and Staff by learning to identify this kind of work/system and proposing ways to simplify it. Where?
- tomrod 3y ago
  IT departments, typically, though occasionally there are whole companies that work in "technology" where this type of work can be found. I said the above as a jest, but seriously, simplification of complex stacks has been a good consulting gig.
- foobiekr 3y ago
  Most large companies. There is a stark difference between the distinguished engineers and the tier below them in terms of asking people to stop doing things badly.
- oooyay 3y ago
  I've typically worked in SRE and platform engineering work and that's where I've gotten exposed to these kinds of Rube Goldberg machines. Make a short list of them when you find them and then use them as a hit list during "cost cutting". Most people don't want to touch these systems because they look big and expansive and generally "work". They're just very poorly optimized. Dare I say, any time I see a function as a service my brain immediately drifts to inspecting the cost implications of said process.
mring33621 3y ago
As opposed to all those people that make similarly comfortable careers in middle and upper mgmt by identifying simple systems and complicating them beyond recognition?

foobiekr 3y ago

I see stuff like this every day. It is a natural consequence of people who only “develop” by gluing things together. God help them if they’d actually have to write some core function themselves.

pphysch 3y ago
The worst kind of "DevOps engineer" that doesn't really understand development, operations, or engineering. But hey, they can run some docker and git commands and piddle around an AWS GUI, which means they are highly technical.
- candiodari 3y ago
  You don't give enough credit to organization chart and project driven engineering. When developing anything: 1) you don't get to touch anyone else's code. And another department's code? Something another manager's team manages ... that amounts to treason. Never for any reason. MAYBE if they've totally abandoned it and you absolutely need it (but only during unpaid overtime) 2) you don't get to spend ANY time on anything outside of the current project or JIRA ticket. Any time at all. So really, NOT optimizing anything is faster and cheaper. Just look at all the spreadsheets made!
- wredue 3y ago
  >piddle around in an AWS GUI I’ve had enough calls with the “Senior/Technical Lead Azure Cloud Engineers” telling them exactly what they need to do that me and them really really don’t get along. I don’t do any of that shit and even I can muddy my way through it, but these people cannot. The real kicker of it is how much these people make. And you know how there are those people who, every time you need to work with them, they answer a teams call and then “need to get to my computer, give me 5” and their status is perpetually set to away? I don’t want to RTO at all, but dealing with this team almost makes me think I’m wrong about that.
vsareto 3y ago
>It is a natural consequence of people who only “develop” by gluing things together. God help them if they’d actually have to write some core function themselves. That's on the industry for not training and gating well. It would be nice to have glue/plumber positions so expectations are not out of line too.

latexr 3y ago

That reads like the KRAZAM Microservices sketch. https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ

neilv 3y ago
I've used that video to explain to business people. It's watchable, and communicates important ideas of what a poopshow this can easily become, without having to talk about real partners/teams close to home as problems.
Lacerda69 3y ago
KRAZAM is a prophet and must be protected at all cost
ludicity 3y ago
At my workplace (the one in the post), whenever one of the good engineers asks about how something works and it's one of these spaghetti-balls, we chorus "It's the design of our backend, okay?"

gruturo 3y ago

:O And I thought my unholy xmllint -xpath (bad stuff, lots of slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/ -e /ohyeahthistoo/somethingelse/ | grep something | while read AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort was as bad as things get. I need to get my horror code game up. I mean, not only is the code awful, its very purpose is horrifying (XML to CSV with some transformations, bit of math, all without being able to use any external sources due to security, only what's in a baseline RHEL7 (soon 8, yay!) ). I promise I'll rewrite it in python at some point.

tshaddox 3y ago

I think it's "iterative-StackOverflow-driven development" most of the time, and that actually causes the increased popularity of those resume keywords.

kmfrk 3y ago

... And a partridge in a pear tree.

feoren 3y ago

This sounds like absolute hell. This is everything I hate about modern software development.

deleted 3y ago

[deleted]

crooked-v 3y ago

The more experience I have, the more I start to think the Omnigres people are right about "just put literally everything into Postgres".

codrilya 3y ago
[dead]

toasted-subs 3y ago

You have no idea how unbelievably annoying it is to work in a company that doesn't a well defined architecture. Every "buzz word" service should be easily justified. This is why I hate recruiters, I can't even tell you how many times I've had a recruiter call me saying they are looking for service XYZ. The same concept rephrased in my resume. I have to rewrite my resume just to satisfy these people? No thanks.

recfab 3y ago
I had recruiter pull that in my most recent job search. Has to stick "C#/..." in front of everything because they didn't understand that ASP.NET, WPF, WCF, WinForms and several other C#-specific tech had anything to do with .NET.

neycoda 3y ago

So, what tech service can I add to that bloated pipeline as a middle-man to get a fraction of a penny per transaction?

Vicinity9635 3y ago

I feel like this might be case of data engineers. They're not usually software engineers. They're tool users not tool makers. So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.

entropicdrifter 3y ago
Agreed. I've been "the backend engineer who works with the data engineers" for several years now and I've seen their general trend of re-inventing the wheel the hard way a number of times. I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.
scruple 3y ago
I stepped into a Data Engineering Lead role in 2019. Stepped out of it in 2021. My team was the first in the org to really approach data engineering and we were all software engineers. I'm told that the systems we built have largely been replaced by Rube Goldberg machines pieced together by the folks who came after us. Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!

Agentlien 3y ago

I feel like I've never seen anything even reminiscent of this bad in the twelve years I've worked as a software engineer. I really want to believe this pipeline as described is satire. Yet, somehow, it does not quite seem that way. This scares me. But also somehow explains why some companies contain so incredibly much more engineering staff than I can possibly explain looking at their output.

BonoboIO 3y ago

Resume Driven development - RDR Never heard that before, but that’s so on point.