5 ms·

How do you design a system that is barely used most of the year and then experiences one huge spike only one day? Do they have tons of capacity sitting around m

How do you design a system that is barely used most of the year and then experiences one huge spike only one day? Do they have tons of capacity sitting around most of the year?

deleted 9y ago
[deleted]
ben_jones 9y ago
Well they could have a sophisticated auto-scaling system that only pays for capacity as needed. Or they could have some lone IT person turn on a bunch of servers, configure a load balancer, then cross their fingers, the morning of.
- newbie912 9y ago
  Does it really need to auto scale for a well known and understood usage? If dominos can do it for the super bowl surely the irs can provision for annual fillings
  - billhendricksjr 9y ago
    Yeah, the first peak around Feb 1 is almost a mirror image of tax day. You could load up for first peak, ramp down for the trough, then load back up again for the “deadline” (not actually a deadline). But you’re taking about 100s of times more scale, not 10s.
wizardforhire 9y ago
I ask the same question when I look out at parking lots. Yet there they are; wasting resources, destroying habitats, soaking up heat, and suppressing carbon absorption.
- organsnyder 9y ago
  At least computing resources can be repurposed more easily. That can be done with parking lots (park-and-rides, carnivals, etc.) but the physical proximity makes that much harder.
- hinkley 9y ago
  There are people pushing for box stores to use permeable pavement and rain gardens in the back half of their lots for this reason. </tangent>
raverbashing 9y ago
To be fair "one huge spike" is not as big of a load as a lot of internet services How big is a tax return? Some XML 10k/100k in size? What do you need to do: save the file, maybe do some basic sanity checking and store the SSN of a person on a db. It is complicated, but not impossible.
- vorpalhex 9y ago
  But, you're going from maybe 10k * X requests a day, to several million. Any system is vulnerable in that kind of very aggressive scaling situation - see every single game launch with a multiplayer component since the dawn of time. Obviously you can scale systems ahead of time if you know, but you still run into issues because you probably haven't scaled the entire system and don't know the likely bottlenecks since it's hard to generate that kind of traffic in a QA environment. Think of the scale of facebook, and then realize most of their payloads are <1k json/protobuf. It's not the size of the payload, it's the delta in quantity.
  - khedoros1 9y ago
    But they also know roughly when the load is going to hit, and how many requests they'll receive. It's not a one-time event like a game launch. They've also been using the same system for decades and ought to be pretty familiar with the various bottlenecks.
    - billhendricksjr 9y ago
      MeF (“Modern” eFile) is only 5ish years old. The earlier version was EDI based I believe.
codemac 9y ago
Modeling & simulation[0] is a very standard way of handling this. It is not that difficult also to rent servers as opposed to having them sit around. Note: just about every online store must do the same type of thing around specific holidays. [0]: https://en.wikipedia.org/wiki/Modeling_and_simulation https://en.wikipedia.org/wiki/Modeling_and_simulation
kristianc 9y ago
This is the archetypal use case for cloud services - most retailers (now at least, they didn't use to), have some kind of autoscaling switched on and I'd imagine this is the same.
- maxxxxx 9y ago
  I have never dealt with cloud services autoscaling. Does it really work seamlessly or will it take time to scale up? Let's say I have a system that has 10 users/day and suddenly I have 10 million for some reason. Does AWS handle this (as long as my credit card can handle it) or will there be outages while AWS is scaling up?
  - billhendricksjr 9y ago
    I’m not an engineer, but the best ones I’ve worked with have stated that autoscaling ain’t easy to architect right.
  - kristianc 9y ago
    Not easy to architect right but the savings in license from not having to run at peak capacity all year often justify it.
ateesdalejr 9y ago
I read an article that says the IRS tax infrastructure is in assembly and runs on a batch system [0]. [0] - https://federalnewsradio.com/tom-temin-commentary/2018/01/irs-clutches-its-modernization-holy-grail/ https://federalnewsradio.com/tom-temin-commentary/2018/01/ir...
- hinkley 9y ago
  I always figured the real reason, the dark truth, for why we have all these reliable queuing services was to feed these old battle axe batch systems, throttled to a rate they can handle.
hinkley 9y ago
The Cloud wants you to answer this a different way today, but there’s a stupid simple trick that some medium sized companies and MMOs have used. Schedule your hardware upgrade cycle and lower priority workloads around the event so that you can get your hands on several times more hardware for a month or two.
maerF0x0 9y ago
You enqueue them all and ensure the queue is empty by next January.

How do you design a system that is barely used most of the year and then experiences one huge spike only one day? Do they have tons of capacity sitting around most of the year?

deleted 9y ago

[deleted]

ben_jones 9y ago

Well they could have a sophisticated auto-scaling system that only pays for capacity as needed. Or they could have some lone IT person turn on a bunch of servers, configure a load balancer, then cross their fingers, the morning of.

newbie912 9y ago
Does it really need to auto scale for a well known and understood usage? If dominos can do it for the super bowl surely the irs can provision for annual fillings
- billhendricksjr 9y ago
  Yeah, the first peak around Feb 1 is almost a mirror image of tax day. You could load up for first peak, ramp down for the trough, then load back up again for the “deadline” (not actually a deadline). But you’re taking about 100s of times more scale, not 10s.

wizardforhire 9y ago

I ask the same question when I look out at parking lots. Yet there they are; wasting resources, destroying habitats, soaking up heat, and suppressing carbon absorption.

organsnyder 9y ago
At least computing resources can be repurposed more easily. That can be done with parking lots (park-and-rides, carnivals, etc.) but the physical proximity makes that much harder.
hinkley 9y ago
There are people pushing for box stores to use permeable pavement and rain gardens in the back half of their lots for this reason. </tangent>

raverbashing 9y ago

To be fair "one huge spike" is not as big of a load as a lot of internet services How big is a tax return? Some XML 10k/100k in size? What do you need to do: save the file, maybe do some basic sanity checking and store the SSN of a person on a db. It is complicated, but not impossible.

vorpalhex 9y ago
But, you're going from maybe 10k * X requests a day, to several million. Any system is vulnerable in that kind of very aggressive scaling situation - see every single game launch with a multiplayer component since the dawn of time. Obviously you can scale systems ahead of time if you know, but you still run into issues because you probably haven't scaled the entire system and don't know the likely bottlenecks since it's hard to generate that kind of traffic in a QA environment. Think of the scale of facebook, and then realize most of their payloads are <1k json/protobuf. It's not the size of the payload, it's the delta in quantity.
- khedoros1 9y ago
  But they also know roughly when the load is going to hit, and how many requests they'll receive. It's not a one-time event like a game launch. They've also been using the same system for decades and ought to be pretty familiar with the various bottlenecks.
  - billhendricksjr 9y ago
    MeF (“Modern” eFile) is only 5ish years old. The earlier version was EDI based I believe.

codemac 9y ago

Modeling & simulation[0] is a very standard way of handling this. It is not that difficult also to rent servers as opposed to having them sit around. Note: just about every online store must do the same type of thing around specific holidays. [0]: https://en.wikipedia.org/wiki/Modeling_and_simulation https://en.wikipedia.org/wiki/Modeling_and_simulation

kristianc 9y ago

This is the archetypal use case for cloud services - most retailers (now at least, they didn't use to), have some kind of autoscaling switched on and I'd imagine this is the same.

maxxxxx 9y ago
I have never dealt with cloud services autoscaling. Does it really work seamlessly or will it take time to scale up? Let's say I have a system that has 10 users/day and suddenly I have 10 million for some reason. Does AWS handle this (as long as my credit card can handle it) or will there be outages while AWS is scaling up?
- billhendricksjr 9y ago
  I’m not an engineer, but the best ones I’ve worked with have stated that autoscaling ain’t easy to architect right.
- kristianc 9y ago
  Not easy to architect right but the savings in license from not having to run at peak capacity all year often justify it.

ateesdalejr 9y ago

I read an article that says the IRS tax infrastructure is in assembly and runs on a batch system [0]. [0] - https://federalnewsradio.com/tom-temin-commentary/2018/01/irs-clutches-its-modernization-holy-grail/ https://federalnewsradio.com/tom-temin-commentary/2018/01/ir...

hinkley 9y ago
I always figured the real reason, the dark truth, for why we have all these reliable queuing services was to feed these old battle axe batch systems, throttled to a rate they can handle.

hinkley 9y ago

The Cloud wants you to answer this a different way today, but there’s a stupid simple trick that some medium sized companies and MMOs have used. Schedule your hardware upgrade cycle and lower priority workloads around the event so that you can get your hands on several times more hardware for a month or two.

maerF0x0 9y ago

You enqueue them all and ensure the queue is empty by next January.