7 ms·

Everything I look at these days looks like this. And most of the time it doesn't even solve the initial problem statement but everyone is too naive to even real

by baz00 3y ago

Everything I look at these days looks like this. And most of the time it doesn't even solve the initial problem statement but everyone is too naive to even realise that. The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it. The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%

foobiekr 3y ago
I know of an engineer who built a work queue by having a chain from an app to kafka to a processor to Kafka to a database writer. Literally instead of a table. This stuff is everywhere. Microservices made it worse and half legitimized it.
- JohnMakin 3y ago
  This must be common or we worked at the same company, seen this exact pattern. It went like: App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
- Groxx 3y ago
  I've seen so many people spend weeks if not months "working" to avoid doing a trivial database migration. Database fear is overwhelmingly powerful in a lot of people it seems.
  - znpy 3y ago
    > Database fear is overwhelmingly powerful in a lot of people it seems. Database are still fairly poorly documented when it comes to administrative work. There is an incredible amount of tutorial, books and courses on how to write sql queries and stuff... But there is almost zero content on how to properly administer a database. I mean, from novice admin to DBA-level capabilities. I said all this before and i'm ready to write this again: i think there's a good market space for dba-style courses.
    - foobiekr 3y ago
      This does not honestly seem true for the main sql databases. For everything else, yes, but if people were actually learning databases instead of hiding from them most of the uptake there wouldn’t exist anyway.
- KMag 3y ago
  But, going the other way, I worked for over a decade on Goldman Sach's SecDB system. It's a quirky steampunk alternative future that branched from our light cone around 1995. There's a globally distributed eventually consistent NoSQL database tightly integrated with a data-flow gradually-typed scripting language (and a very 1990s feel 16 color IDE). I'm sure in the late 1990s/early 2000s (before globally distributed NoSQL was popular and before gradual/dynamic typing had a resurgence) it was more like discovered alien technology than steampunk alternative future. (Also, with source code being executed from globally distributed immutable database snapshots, deployment is much nicer than anything else I've used to date. After release testing, set a database key to point to the latest snapshot, and you're deployed.) There's a service that watches the transaction log of your regional replica so that you can make long-poll HTTP requests that return when any change matching your filter is committed. (Edit: usually the HTTP result handler is used to invalidate specific memoized results in the data flow graph, letting lazy re-evaluation re-fetch the database records as needed.) It makes a lot of sense for a financial risk system, where you end up calculating millions of slight variations on a scenario. The data flow model with aggressive memoization makes this sort of thing much cheaper. However, I saw plenty of systems written where you'd attempt to write your request to the next key matching some regex (and retry with the next key if it already existed), where your request would contain some parameters and the database key and/or filesystem path where results should be written. Under-experience with databases easily results in rewriting a database using message queue/bus. Under-experience with message queues/busses easily results in rewriting a message queue/bus using a database.
  - kwillets 3y ago
    Message queueing triggers a host of psychological needs. Synchronous jobs rarely need monitoring and management features, but move the same work to a queue, and everybody loses their minds.
- ljm 3y ago
  I think I've seen enough complexity created by engineering teams given total autonomy, with hands-off leadership, that I'd prefer a much more constrained approach. There should still be autonomy, of course, but proposals for new tech, languages and paradigms should only be considered with due diligence. The most unpleasant codebases I've dealt with are ones that have suffered from a lack of strong leadership, and they are almost uniquely microservice setups that pull in everything but the kitchen sink, usually because it's just trendy to use it. Monoliths can get pretty damn ugly too but at least it's contained in one single codebase.
- deleted 3y ago
  [deleted]
specialist 3y ago
Repeating myself... Just two examples from my prior gig (fashion e-commerce). #1 Our hottest dataset (db of current products) stored in DynamoDB. Core dependency for all our code. Easily fits in < 1Gb of RAM. OMG, just make a hashmap. Over a year, I managed to persuade the team to start transition from DynamoDB to Redis. #2 Tiny (vs micro) service that munged some URLs. Blocker for an important campaign. Prior team of 4 churned for a year, was no closer to delivery. Spring, ORMs, CI/CD pipelines, the works. I spent a week unraveling the requirements (repeated facepalm). A second week banging out a trivial nodejs thing. (My team preferred nodejs, which was their prerogative.) Really trivial. I felt so bad for the biz dev people who'd been dying to get this functionality for so long.
dmd 3y ago
I am currently working with a US government system for downloading public scientific data. You select some data you want to download and add it to a shopping cart. Check out, and select 'create database'. This generates your own copy of an Oracle database, with your own credentials and hostname and db name. Connect to that and construct a query against a table that has some metadata about studies you're interested in. Using the identifiers from that table, join with a LIKE against another table for s3:// URLs. (There are no primary keys and the other table's column is not exactly the same; you need to use a LIKE. This is all documented.) Those s3 URLs point to a CSV which contains another identifier which you use to download manifests which contains links to a web page created on-the-fly which contains to the s3 files to download. By the time you've done all this, your access has likely expired and you must start over from scratch.
- alliao 3y ago
  voted for the Kafkaesque of 2023
- dmd 3y ago
  (I assume this is all a reflection of https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law )
- blantonl 3y ago
  I'm going to take a deep breath, and move on to the other stories further down. So far I'm feeling pretty good about what I've developed over the years.

by baz00 3y ago

foobiekr 3y ago

I know of an engineer who built a work queue by having a chain from an app to kafka to a processor to Kafka to a database writer. Literally instead of a table. This stuff is everywhere. Microservices made it worse and half legitimized it.

JohnMakin 3y ago
This must be common or we worked at the same company, seen this exact pattern. It went like: App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
Groxx 3y ago
I've seen so many people spend weeks if not months "working" to avoid doing a trivial database migration. Database fear is overwhelmingly powerful in a lot of people it seems.
- znpy 3y ago
  > Database fear is overwhelmingly powerful in a lot of people it seems. Database are still fairly poorly documented when it comes to administrative work. There is an incredible amount of tutorial, books and courses on how to write sql queries and stuff... But there is almost zero content on how to properly administer a database. I mean, from novice admin to DBA-level capabilities. I said all this before and i'm ready to write this again: i think there's a good market space for dba-style courses.
  - foobiekr 3y ago
    This does not honestly seem true for the main sql databases. For everything else, yes, but if people were actually learning databases instead of hiding from them most of the uptake there wouldn’t exist anyway.
KMag 3y ago
But, going the other way, I worked for over a decade on Goldman Sach's SecDB system. It's a quirky steampunk alternative future that branched from our light cone around 1995. There's a globally distributed eventually consistent NoSQL database tightly integrated with a data-flow gradually-typed scripting language (and a very 1990s feel 16 color IDE). I'm sure in the late 1990s/early 2000s (before globally distributed NoSQL was popular and before gradual/dynamic typing had a resurgence) it was more like discovered alien technology than steampunk alternative future. (Also, with source code being executed from globally distributed immutable database snapshots, deployment is much nicer than anything else I've used to date. After release testing, set a database key to point to the latest snapshot, and you're deployed.) There's a service that watches the transaction log of your regional replica so that you can make long-poll HTTP requests that return when any change matching your filter is committed. (Edit: usually the HTTP result handler is used to invalidate specific memoized results in the data flow graph, letting lazy re-evaluation re-fetch the database records as needed.) It makes a lot of sense for a financial risk system, where you end up calculating millions of slight variations on a scenario. The data flow model with aggressive memoization makes this sort of thing much cheaper. However, I saw plenty of systems written where you'd attempt to write your request to the next key matching some regex (and retry with the next key if it already existed), where your request would contain some parameters and the database key and/or filesystem path where results should be written. Under-experience with databases easily results in rewriting a database using message queue/bus. Under-experience with message queues/busses easily results in rewriting a message queue/bus using a database.
- kwillets 3y ago
  Message queueing triggers a host of psychological needs. Synchronous jobs rarely need monitoring and management features, but move the same work to a queue, and everybody loses their minds.
ljm 3y ago
I think I've seen enough complexity created by engineering teams given total autonomy, with hands-off leadership, that I'd prefer a much more constrained approach. There should still be autonomy, of course, but proposals for new tech, languages and paradigms should only be considered with due diligence. The most unpleasant codebases I've dealt with are ones that have suffered from a lack of strong leadership, and they are almost uniquely microservice setups that pull in everything but the kitchen sink, usually because it's just trendy to use it. Monoliths can get pretty damn ugly too but at least it's contained in one single codebase.
deleted 3y ago
[deleted]

specialist 3y ago

Repeating myself... Just two examples from my prior gig (fashion e-commerce). #1 Our hottest dataset (db of current products) stored in DynamoDB. Core dependency for all our code. Easily fits in < 1Gb of RAM. OMG, just make a hashmap. Over a year, I managed to persuade the team to start transition from DynamoDB to Redis. #2 Tiny (vs micro) service that munged some URLs. Blocker for an important campaign. Prior team of 4 churned for a year, was no closer to delivery. Spring, ORMs, CI/CD pipelines, the works. I spent a week unraveling the requirements (repeated facepalm). A second week banging out a trivial nodejs thing. (My team preferred nodejs, which was their prerogative.) Really trivial. I felt so bad for the biz dev people who'd been dying to get this functionality for so long.

dmd 3y ago

I am currently working with a US government system for downloading public scientific data. You select some data you want to download and add it to a shopping cart. Check out, and select 'create database'. This generates your own copy of an Oracle database, with your own credentials and hostname and db name. Connect to that and construct a query against a table that has some metadata about studies you're interested in. Using the identifiers from that table, join with a LIKE against another table for s3:// URLs. (There are no primary keys and the other table's column is not exactly the same; you need to use a LIKE. This is all documented.) Those s3 URLs point to a CSV which contains another identifier which you use to download manifests which contains links to a web page created on-the-fly which contains to the s3 files to download. By the time you've done all this, your access has likely expired and you must start over from scratch.

alliao 3y ago
voted for the Kafkaesque of 2023
dmd 3y ago
(I assume this is all a reflection of https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law )
blantonl 3y ago
I'm going to take a deep breath, and move on to the other stories further down. So far I'm feeling pretty good about what I've developed over the years.