7 ms·

As a developer I immediately though of the power of queues. Twenty people trying to submit same form does not work for everyone, but a queue processing one pers

As a developer I immediately though of the power of queues. Twenty people trying to submit same form does not work for everyone, but a queue processing one person at a time might allow the twenty people to submit within a short time. It is flattening the curve! If I was contracted to fix this ASAP, I would set up an nginx front-end proxy config that doesn't allow more than X sessions and suggest a time in the future when they could try again.

pjc50 7y ago
The matching UK system has a (huge) queue in it: https://www.computerweekly.com/news/252480546/Huge-queues-for-Universal-Credit-online-applications-due-to-coronavirus-surge https://www.computerweekly.com/news/252480546/Huge-queues-fo...
- Demiurge 7y ago
  Is there a human factor in processing these?
jkaljundi 7y ago
In related news on Queue-it: https://tech.eu/brief/queue-it-funding/ https://tech.eu/brief/queue-it-funding/
- _curious_ 7y ago
  Solid company
JMTQp8lwXL 7y ago
Put the web form (plain static assets JS/CSS/HTML) on a globally accessible CDN. Then use SQS intake for each unemployment application form. Then firehouse it out, wherever it needs to go, at a rate which you can realistically deal with it. Queuing access to the form itself and telling someone to wake up at 4:52 AM so they can then merely access the static assets is a less-than-desirable user experience.
- Demiurge 7y ago
  It is more desirable than 504, and first thing I would do in 15 minutes with zero context. If I can get more context, of course something like your solution is more desirable, depending on the issue. It would take some time to figure whether it is necessary to bring in AWS or just database connection pooler, or whatever.
- judge2020 7y ago
  Even typeform/Google Forms would be better suited for the task.
  - alasdair_ 7y ago
    >Even typeform/Google Forms would be better suited for the task. And now you've given a private company access to market-moving unemployment data. And a million other issues, especially legal ones. The technology part in and of itself isn't that difficult, it's all of the constraints (and, often, mountains of laws) that are the bigger issue.
driverdan 7y ago
Having worked on this type of application in the past they should find a new company to work with if they can't handle this traffic. We were handling hundreds of requests per second with ease 10 years ago. That was with MySQL and the app running on the same server. It doesn't take many resources to show the user a form, validate it, and save to a DB.
- projektfu 7y ago
  Usually it's not so much the form that causes things to fall down but some validation step that they are trying to do synchronously, that might have to access an IBM mainframe, and things time out. When you're getting a few an hour, it's not a big deal.
- Demiurge 7y ago
  Completely agree with the sentiment. I think most often it is inadequate default configuration that bottle-necks somewhere, that never got tested with more than a handful of users at a time. Going to a hundred highlights some bugs. going to 1000 others. On the other hand, I have worked on a project for USDA and they had 10 year old servers running 15 year old software and did not allow any system administration, while the system admins were some unknown government employees completely inaccessible. I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.
- onion2k 7y ago
  It doesn't take many resources to show the user a form, validate it, and save to a DB. I bet that's what the previous developers thought. What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike? What if your database is rejecting transactions occasionally? What happens when your backup process locks all the database tables? How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions? What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic? What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926 https://news.ycombinator.com/item?id=22691926)? You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.
commandlinefan 7y ago
> If I was contracted to fix this ASAP You’re assuming that the people who built it in the first place (or the people that may or may not be contracted to fix it later) know or care. Remember, this is government contracting we’re talking about - lowest bidder wins. How do you win the lowest bid? By doing it as cheap and quick as you can. That means hiring inexperienced/cheap developers who can build something that looks like it will work for far less money than you can build something that actually will.
OJFord 7y ago
Ocado (the IaaS for online supermarkets company, and, in the UK, online-only supermarket itself) has done this in response to the increased demand, and makes you wait in a 'virtual queue' (virtual relative to what in America you call a 'line-up', but we call a 'queue', at a physical supermarket) before you can place or edit your order.

pjc50 7y ago

The matching UK system has a (huge) queue in it: https://www.computerweekly.com/news/252480546/Huge-queues-for-Universal-Credit-online-applications-due-to-coronavirus-surge https://www.computerweekly.com/news/252480546/Huge-queues-fo...

Demiurge 7y ago
Is there a human factor in processing these?

jkaljundi 7y ago

In related news on Queue-it: https://tech.eu/brief/queue-it-funding/ https://tech.eu/brief/queue-it-funding/

_curious_ 7y ago
Solid company

JMTQp8lwXL 7y ago

Put the web form (plain static assets JS/CSS/HTML) on a globally accessible CDN. Then use SQS intake for each unemployment application form. Then firehouse it out, wherever it needs to go, at a rate which you can realistically deal with it. Queuing access to the form itself and telling someone to wake up at 4:52 AM so they can then merely access the static assets is a less-than-desirable user experience.

Demiurge 7y ago
It is more desirable than 504, and first thing I would do in 15 minutes with zero context. If I can get more context, of course something like your solution is more desirable, depending on the issue. It would take some time to figure whether it is necessary to bring in AWS or just database connection pooler, or whatever.
judge2020 7y ago
Even typeform/Google Forms would be better suited for the task.
- alasdair_ 7y ago
  >Even typeform/Google Forms would be better suited for the task. And now you've given a private company access to market-moving unemployment data. And a million other issues, especially legal ones. The technology part in and of itself isn't that difficult, it's all of the constraints (and, often, mountains of laws) that are the bigger issue.

driverdan 7y ago

Having worked on this type of application in the past they should find a new company to work with if they can't handle this traffic. We were handling hundreds of requests per second with ease 10 years ago. That was with MySQL and the app running on the same server. It doesn't take many resources to show the user a form, validate it, and save to a DB.

projektfu 7y ago
Usually it's not so much the form that causes things to fall down but some validation step that they are trying to do synchronously, that might have to access an IBM mainframe, and things time out. When you're getting a few an hour, it's not a big deal.
Demiurge 7y ago
Completely agree with the sentiment. I think most often it is inadequate default configuration that bottle-necks somewhere, that never got tested with more than a handful of users at a time. Going to a hundred highlights some bugs. going to 1000 others. On the other hand, I have worked on a project for USDA and they had 10 year old servers running 15 year old software and did not allow any system administration, while the system admins were some unknown government employees completely inaccessible. I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.
onion2k 7y ago
It doesn't take many resources to show the user a form, validate it, and save to a DB. I bet that's what the previous developers thought. What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike? What if your database is rejecting transactions occasionally? What happens when your backup process locks all the database tables? How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions? What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic? What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926 https://news.ycombinator.com/item?id=22691926)? You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.

commandlinefan 7y ago

> If I was contracted to fix this ASAP You’re assuming that the people who built it in the first place (or the people that may or may not be contracted to fix it later) know or care. Remember, this is government contracting we’re talking about - lowest bidder wins. How do you win the lowest bid? By doing it as cheap and quick as you can. That means hiring inexperienced/cheap developers who can build something that looks like it will work for far less money than you can build something that actually will.

OJFord 7y ago

Ocado (the IaaS for online supermarkets company, and, in the UK, online-only supermarket itself) has done this in response to the increased demand, and makes you wait in a 'virtual queue' (virtual relative to what in America you call a 'line-up', but we call a 'queue', at a physical supermarket) before you can place or edit your order.