3 ms·
I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The
by malfist 3y ago
I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes.
The amount of complexity the architecture has because of those constraints is insane.
When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to building for more scale than you need. It's not as simple as just tweaking a few things.
To me there's a couple of big breakpoints in scale:
* When you can run on a single server
* When you need to run on a single server, but with HA redundancies
* When you have to scale beyond a single server
* When you have to adapt your scale to deal with the limits of a distributed system, i.e. designing for DyanmoDB's partition limits.
Each step in that chain add irrevocable complexity, adds to OE, adds to cost to run and cost to build. Be sure you have to take those steps before you decide too.
- disqard 3y agoI'm trying to guess what "OE" stands for... over engineering? operating expenditure? I'd love to know what you meant :)
- kuschku 3y agoMaybe I'm misunderstanding something, but that's about 2700 a second. Or about 3Mbps. Even a very unoptimized application running on a dev laptop can serve 1Gbps nowadays without issues. So what are the constraints that demand a complex architecture?
- rdoherty 3y agoI'm not the OP but a few things: * Reading/fetching the data - usernames, phone number, message, etc. * Generating the content for each message - it might be custom per person * This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open. * Retries on errors, rescheduling, backoffs * At least once or at most once sends? Each has tradeoffs * Stopping/starting that many messages at any time * Rate limits on some services you might be using alongside your service (network gateway, database, etc) * Recordkeeping - did the message send? When?
- joshstrange 3y agoI literally spent the last week speccing out a system just like this and you are completely correct. You’ve touched on almost every single thing we ran into.
- kuschku 3y agoOh, I absolutely agree that the complexity is in these topics. I'm just sceptic that they're enough to turn a task that could run on a laptop into one that requires an entire cluster of machines. The third party API is the part that has the potential to turn this straightforward task into a byzantine mess, though, so I suspect that's the missing piece of information. I'm comparing this to my own experience with IRC, where handling the same or larger streams of messages is common. And that's with receiving this in real time, storing the messages, matching and potentially reacting to them, and doing all that while running on a raspberry pi.
- goguy 3y agoThat really doesn't require that much complexity. I used to send something like 250k a minute complete with delivery report processing from a single machine running a bunch of other services like 10 years ago.
- nine_k 3y agoNice. But average latency is not the whole picture; tail latency is. For good tail latency and handling of spikes, you have to have a sizable "untapped" reserve of performance.