3 ms·
My kneejerk reaction to the headline is ‘why would it?’. It’s unsurprising to me that an AI company appears to have chosen exactly the wrong tool for the job.
by callamdelaney 1y ago
My kneejerk reaction to the headline is ‘why would it?’.
It’s unsurprising to me that an AI company appears to have chosen exactly the wrong tool for the job.
- bravesoul2 1y agoYeah I have no idea whether it would. But I'd load test it if it needed to scale. SQS may have been a good "boring" choice for this?
- kristianc 1y agoSounds like a deliberate attempt to avoid spinning up Redis, Kafka, or an outbox system early on.. and then underestimated how quickly their scale would make it blow up. Story as old as time.
- j16sdiz 1y agoKafka head of line blocking sucks.
- chrnola 1y agoGuaranteeing order has its tradeoffs. There is work happening currently to make Kafka behave more like a queue: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...
- LgWoodenBadger 1y agoIsn't this one of the things partitioning is meant to ameliorate? Either through partitions themselves, or through an appropriate partitioning strategy?
- const_cast 1y agoI find the opposite story more true: additional complexity in the form of caching early, for a scale that never comes. I've worked on one too many sprawling, distributed systems with too little users to justify it.
- physix 1y ago"Sprawling distributed systems". I like that. Sounds like a synonym for "Platform Engineering". :-) I remember being amazed that lambda architecture was considered a kind of reference, when it looked to me more like a workaround. We like to build IT cathedrals, until we have to run them.
- const_cast 1y agoIf there's one thing I took away from school, it's that distributed systems are hard. More failure points and much more communication hops. Serialization into deserialization into serialization again over network hops.
- v5v3 1y agoBetter to be successful with simple tech and have a minor 'blow up', then over engineer and go bust.
- oulipo 1y agoNot sure I get it... how would you replicate this functionality with Kafka? You'd still need to have the database LISTEN to changes and push it to Kafka no?
- TheTaytay 1y agoBecause documentation doesn’t warn about this well-loved feature effectively ruins the ability to perform parallel writes, and because everything else in Postgres scales well. I think it’s a reasonable assumption. Based on the second half of your comment, you clearly don’t think highly of “AI companies,” but I think that’s a separate issue.