4 ms·
I once worked on a payment processing system. Files came in with about a thousand payments, in a textual format, 10-20 short lines of text each. The geniuses w
by twic 3y ago
I once worked on a payment processing system. Files came in with about a thousand payments, in a textual format, 10-20 short lines of text each.
The geniuses who led the project before me decided that the first step was to split the file into single payment chunks, then put each one into RabbitMQ, from where a different service would pick them up and parse them - then encode the parsed payments in JSON, and put them back in RabbitMQ, for a third process to pick up and handle. This was all for performance, of course.
I tried to explain that just parsing the whole file in one go would be much faster than splitting it up, writing it to a queue, and reading it back in. That it probably wouldn't be much slower than the splitting. But nobody was interested. Obviously distributing the work would be much faster.
It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.
- tuyiown 3y agoSince many have been told repeatedly that rails is slow, maybe some just stopped caring if the code actually got slow.
- deleted 3y ago[deleted]
- renegade-otter 3y agoThat intuitive feel of "what's faster" has been lost ever since computing power exploded. Back when the resources were at a premium, we were just better engineers. Not all of us had the command of Leetcode algorithm problems, but we rarely missed a database index and knew to stay close to the metal as much as possible, avoiding network trips, avoiding database trips. It still holds true today, and software has become bloated and slower because these simple rules are being ignored. It's all in the cloud, it's all distributed, it's all "free". As for Rails - it's the same problem but it's different. If you stay too high up in the abstraction orbit, it's hard to think of the nitty gritty of performance. What is the cloud doing? What is the ORM doing? Once it's all "magic", you just keep piling on non-performant crap until it becomes an emergency. Then you just add more cache :)
- SketchySeaBeast 3y agoI don't think it's necessarily abstraction, it's just complexity. We've been given a thousand footguns and we're expected to know how to use each one. You didn't consider microservices and lambda twenty years ago because they didn't exist. But you also didn't have to build an app that worked everywhere and looked good on every screen.
- deleted 3y ago[deleted]
- stabamole 3y agoThere’s also the fact that we shouldn’t be thinking about problems the way they used to be solved - not that we shouldn’t be considering database calls and network, but because compute/memory/storage are cheaper than they used to be, we’re able to make more tradeoffs to save developer time or to improve experience. Back when the option was to further optimize how many registers your assembly is using or to leave an application in a virtually unusable state, you didn’t really have a choice. You optimized the small details
- renegade-otter 3y agoOh, compute and storage are cheap alright, until they are not. At many jobs I had to deal with the all-hands-on-deck, code red emergency "our AWS bill got too damn high, what can we cut" situation. I can guarantee you a ton of companies are going through this exercise right now. Some software engineers purposefully develop using fewer resources to force themselves to reason about efficiency. Abundance leads to complacency and laziness, and those resources are no longer cheap if you are abusing them.
- asimpletune 3y agoI saw this exact same pattern before. I think it's actually common, because I've also heard people in favor of micro services point to this design as a point in their favor. One reason always given is the ability to replay chunks if the system goes down, at least for Kafka.
- deleted 3y ago[deleted]
- giovannibonetti 3y ago> It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails. Same experience here. I feel like I'm always repeating myself when explaining to other developers why we should avoid moving data around unnecessary and prefer to operate directly in the database when necessary, specially when it comes to large data migrations. Also important is why it is bad to do a bunch of small operations instead of batching them in chunks closer to the page size (8 kB).
- themerone 3y agoIt sounds like somebody read that RabitMQ was the hot new thing, and then came up with a design to justify using it.
- RetpolineDrama 3y ago>My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, Not surprised. Rails seems to encourage the worst behavior in developers.
- ljm 3y agoI don't think that's on Rails, more that people with less experience tend to use more indirection in their solutions because each step of the problem is seen as independent and breaking everything down into its own class or method enhances 'readability'. - Step one: put file in queue - Step two: break down file and insert rows - Step three: convert to json - Step four: compute results - etc... Once your experience improves you start to understand the cost of such indirection and realise that the problem could have been solved in a much more straightforward manner.
- poloniculmov 3y agoThat would be pretty stupid to do with Rails, there's a pretty high cost to starting jobs, so it makes more sense to batch everything.