6 ms·

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so m

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

Repulsion9513 3y ago
A LOT... especially here.
malfist 3y ago
I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The amount of complexity the architecture has because of those constraints is insane. When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to building for more scale than you need. It's not as simple as just tweaking a few things. To me there's a couple of big breakpoints in scale: * When you can run on a single server * When you need to run on a single server, but with HA redundancies * When you have to scale beyond a single server * When you have to adapt your scale to deal with the limits of a distributed system, i.e. designing for DyanmoDB's partition limits. Each step in that chain add irrevocable complexity, adds to OE, adds to cost to run and cost to build. Be sure you have to take those steps before you decide too.
- disqard 3y ago
  I'm trying to guess what "OE" stands for... over engineering? operating expenditure? I'd love to know what you meant :)
  - madisp 3y ago
    probably operating expenses
  - malfist 3y ago
    Sorry, thought it was a common term. Operational Excellence. All the effort and time it takes to keep a service online, on call included
- kuschku 3y ago
  Maybe I'm misunderstanding something, but that's about 2700 a second. Or about 3Mbps. Even a very unoptimized application running on a dev laptop can serve 1Gbps nowadays without issues. So what are the constraints that demand a complex architecture?
  - rdoherty 3y ago
    I'm not the OP but a few things: * Reading/fetching the data - usernames, phone number, message, etc. * Generating the content for each message - it might be custom per person * This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open. * Retries on errors, rescheduling, backoffs * At least once or at most once sends? Each has tradeoffs * Stopping/starting that many messages at any time * Rate limits on some services you might be using alongside your service (network gateway, database, etc) * Recordkeeping - did the message send? When?
    - joshstrange 3y ago
      I literally spent the last week speccing out a system just like this and you are completely correct. You’ve touched on almost every single thing we ran into.
    - kuschku 3y ago
      Oh, I absolutely agree that the complexity is in these topics. I'm just sceptic that they're enough to turn a task that could run on a laptop into one that requires an entire cluster of machines. The third party API is the part that has the potential to turn this straightforward task into a byzantine mess, though, so I suspect that's the missing piece of information. I'm comparing this to my own experience with IRC, where handling the same or larger streams of messages is common. And that's with receiving this in real time, storing the messages, matching and potentially reacting to them, and doing all that while running on a raspberry pi.
- goguy 3y ago
  That really doesn't require that much complexity. I used to send something like 250k a minute complete with delivery report processing from a single machine running a bunch of other services like 10 years ago.
  - nine_k 3y ago
    Nice. But average latency is not the whole picture; tail latency is. For good tail latency and handling of spikes, you have to have a sizable "untapped" reserve of performance.
icedchai 3y ago
Many. I regularly see systems built for "big data", built for scale using "serverless" and some proprietary cloud database (like DynamoDB), storing a few hundred megabytes total. 20 years ago we would've built this on PHP and MySQL and called it a day.
Szpadel 3y ago
In may day job I often see systems that have the opposite. Especially for database queries, developers tested on local machine with 100s of records and everything was quick and snappy and on production with mere millions of records I often see queries taking minutes up to a hour just because some developer didn't see need for creating indexes or created query in a way there is no way to even create any index that would work
- layer8 3y ago
  That’s true, but has little to do with distributed cloud architecture vs. single local instance.
- pooper 3y ago
  This is a different topic and not always a skills issue. The stupid match for "productivity" and "velocity" means you have to cut corners. Also Sometimes, it is poor communication. Just yesterday I saw some code that requests auth token before every request even though each bearer token comes with expires in (about twelve hours).
kristopolous 3y ago
You're not considered serious if you don't. Kinda stupid.
- nine_k 3y ago
  In the startup world, this is correct. The success that VCs are after is when your customer base doubles every month. Better yet, every week. Having a reasonably scalable infra at the start ensures that a success won't kill you. Of course, the chances of a runaway success like this are slim, so 99% or more startups overbuild, given their resulting customer base. But it's like 99% or more pilots who put on a parachute don't end up using it; the whole point is the small minority who do, and you never know. For a stable, predictable, medium-scale business it may make total sense to have a few dedicated physical boxes and run their whole operation from them comfortably, for a fraction of cloud costs. But starting with it is more expensive than starting with a cloud, because you immediately need an SRE, or two.
  - kristopolous 3y ago
    You aren't going to get there. The risks and complexity of a startup are high to begin with. Adding artificial roadblocks because of aspirational fantasies is going to hold you back. Look at the big successes such as youtube, twitter, facebook, airbnb, lyft, google, yahoo - exactly zero of them did this preventatively. Even altavista and babelfish, done by DEC and running on Alphas, which they had plenty of, had to be redone multiple times due to growth. Heck, look at the first 5 years of Amazon. AWS was initially ideated in a contract job for Target. Address the immediate and real needs and business cases, not pie in the sky aspirations of global dominance - wait until it becomes a need and then do it. The chances of getting there are only reasonable if you move instead of plan, otherwise you'll miss the window and product opportunity. I know it ruffles your engineering feathers - that's one of the reasons most attempts at building these things fails. The best ways feel wrong, are counterintuitive and are incidentally often executed by young college kids who don't know any better. It's why successful tech founders tend to be inexperienced; it can actually be advantageous if they make the right "mistakes". Forget about any supposedly inevitable disaster until it's actually affecting your numbers. I know it's hard but the most controllable difference between success and failure in the startup space is in the behavioral patterns of the stakeholders.

Repulsion9513 3y ago

A LOT... especially here.

malfist 3y ago

I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The amount of complexity the architecture has because of those constraints is insane. When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to building for more scale than you need. It's not as simple as just tweaking a few things. To me there's a couple of big breakpoints in scale: * When you can run on a single server * When you need to run on a single server, but with HA redundancies * When you have to scale beyond a single server * When you have to adapt your scale to deal with the limits of a distributed system, i.e. designing for DyanmoDB's partition limits. Each step in that chain add irrevocable complexity, adds to OE, adds to cost to run and cost to build. Be sure you have to take those steps before you decide too.

disqard 3y ago
I'm trying to guess what "OE" stands for... over engineering? operating expenditure? I'd love to know what you meant :)
- madisp 3y ago
  probably operating expenses
- malfist 3y ago
  Sorry, thought it was a common term. Operational Excellence. All the effort and time it takes to keep a service online, on call included
kuschku 3y ago
Maybe I'm misunderstanding something, but that's about 2700 a second. Or about 3Mbps. Even a very unoptimized application running on a dev laptop can serve 1Gbps nowadays without issues. So what are the constraints that demand a complex architecture?
- rdoherty 3y ago
  I'm not the OP but a few things: * Reading/fetching the data - usernames, phone number, message, etc. * Generating the content for each message - it might be custom per person * This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open. * Retries on errors, rescheduling, backoffs * At least once or at most once sends? Each has tradeoffs * Stopping/starting that many messages at any time * Rate limits on some services you might be using alongside your service (network gateway, database, etc) * Recordkeeping - did the message send? When?
  - joshstrange 3y ago
    I literally spent the last week speccing out a system just like this and you are completely correct. You’ve touched on almost every single thing we ran into.
  - kuschku 3y ago
    Oh, I absolutely agree that the complexity is in these topics. I'm just sceptic that they're enough to turn a task that could run on a laptop into one that requires an entire cluster of machines. The third party API is the part that has the potential to turn this straightforward task into a byzantine mess, though, so I suspect that's the missing piece of information. I'm comparing this to my own experience with IRC, where handling the same or larger streams of messages is common. And that's with receiving this in real time, storing the messages, matching and potentially reacting to them, and doing all that while running on a raspberry pi.
goguy 3y ago
That really doesn't require that much complexity. I used to send something like 250k a minute complete with delivery report processing from a single machine running a bunch of other services like 10 years ago.
- nine_k 3y ago
  Nice. But average latency is not the whole picture; tail latency is. For good tail latency and handling of spikes, you have to have a sizable "untapped" reserve of performance.

icedchai 3y ago

Many. I regularly see systems built for "big data", built for scale using "serverless" and some proprietary cloud database (like DynamoDB), storing a few hundred megabytes total. 20 years ago we would've built this on PHP and MySQL and called it a day.

Szpadel 3y ago

In may day job I often see systems that have the opposite. Especially for database queries, developers tested on local machine with 100s of records and everything was quick and snappy and on production with mere millions of records I often see queries taking minutes up to a hour just because some developer didn't see need for creating indexes or created query in a way there is no way to even create any index that would work

layer8 3y ago
That’s true, but has little to do with distributed cloud architecture vs. single local instance.
pooper 3y ago
This is a different topic and not always a skills issue. The stupid match for "productivity" and "velocity" means you have to cut corners. Also Sometimes, it is poor communication. Just yesterday I saw some code that requests auth token before every request even though each bearer token comes with expires in (about twelve hours).

kristopolous 3y ago

You're not considered serious if you don't. Kinda stupid.

nine_k 3y ago
In the startup world, this is correct. The success that VCs are after is when your customer base doubles every month. Better yet, every week. Having a reasonably scalable infra at the start ensures that a success won't kill you. Of course, the chances of a runaway success like this are slim, so 99% or more startups overbuild, given their resulting customer base. But it's like 99% or more pilots who put on a parachute don't end up using it; the whole point is the small minority who do, and you never know. For a stable, predictable, medium-scale business it may make total sense to have a few dedicated physical boxes and run their whole operation from them comfortably, for a fraction of cloud costs. But starting with it is more expensive than starting with a cloud, because you immediately need an SRE, or two.
- kristopolous 3y ago
  You aren't going to get there. The risks and complexity of a startup are high to begin with. Adding artificial roadblocks because of aspirational fantasies is going to hold you back. Look at the big successes such as youtube, twitter, facebook, airbnb, lyft, google, yahoo - exactly zero of them did this preventatively. Even altavista and babelfish, done by DEC and running on Alphas, which they had plenty of, had to be redone multiple times due to growth. Heck, look at the first 5 years of Amazon. AWS was initially ideated in a contract job for Target. Address the immediate and real needs and business cases, not pie in the sky aspirations of global dominance - wait until it becomes a need and then do it. The chances of getting there are only reasonable if you move instead of plan, otherwise you'll miss the window and product opportunity. I know it ruffles your engineering feathers - that's one of the reasons most attempts at building these things fails. The best ways feel wrong, are counterintuitive and are incidentally often executed by young college kids who don't know any better. It's why successful tech founders tend to be inexperienced; it can actually be advantageous if they make the right "mistakes". Forget about any supposedly inevitable disaster until it's actually affecting your numbers. I know it's hard but the most controllable difference between success and failure in the startup space is in the behavioral patterns of the stakeholders.