14 ms·

Further, unemployment benefits are managed by the states, and those states are running web services which typically see a few hundred hits a day. They are now

by luma 7y ago

Further, unemployment benefits are managed by the states, and those states are running web services which typically see a few hundred hits a day. They are now trying to process tens of thousands of new records each day, and at least in MI the service is absolutely not up to the task. My wife managed to get her filing completed a little after 1am this morning. She was the only one of her 20 coworkers to successfully file, the rest are continuing to attempt to get the state web site to work today, while more people pile in. These numbers are going to get much, much worse.

alexpetralia 7y ago
I confirmed that the state's unemployment website has slowed to a crawl (if it is working at all).
Demiurge 7y ago
As a developer I immediately though of the power of queues. Twenty people trying to submit same form does not work for everyone, but a queue processing one person at a time might allow the twenty people to submit within a short time. It is flattening the curve! If I was contracted to fix this ASAP, I would set up an nginx front-end proxy config that doesn't allow more than X sessions and suggest a time in the future when they could try again.
- pjc50 7y ago
  The matching UK system has a (huge) queue in it: https://www.computerweekly.com/news/252480546/Huge-queues-for-Universal-Credit-online-applications-due-to-coronavirus-surge https://www.computerweekly.com/news/252480546/Huge-queues-fo...
  - Demiurge 7y ago
    Is there a human factor in processing these?
- jkaljundi 7y ago
  In related news on Queue-it: https://tech.eu/brief/queue-it-funding/ https://tech.eu/brief/queue-it-funding/
  - _curious_ 7y ago
    Solid company
- JMTQp8lwXL 7y ago
  Put the web form (plain static assets JS/CSS/HTML) on a globally accessible CDN. Then use SQS intake for each unemployment application form. Then firehouse it out, wherever it needs to go, at a rate which you can realistically deal with it. Queuing access to the form itself and telling someone to wake up at 4:52 AM so they can then merely access the static assets is a less-than-desirable user experience.
  - Demiurge 7y ago
    It is more desirable than 504, and first thing I would do in 15 minutes with zero context. If I can get more context, of course something like your solution is more desirable, depending on the issue. It would take some time to figure whether it is necessary to bring in AWS or just database connection pooler, or whatever.
  - judge2020 7y ago
    Even typeform/Google Forms would be better suited for the task.
    - alasdair_ 7y ago
      >Even typeform/Google Forms would be better suited for the task. And now you've given a private company access to market-moving unemployment data. And a million other issues, especially legal ones. The technology part in and of itself isn't that difficult, it's all of the constraints (and, often, mountains of laws) that are the bigger issue.
- driverdan 7y ago
  Having worked on this type of application in the past they should find a new company to work with if they can't handle this traffic. We were handling hundreds of requests per second with ease 10 years ago. That was with MySQL and the app running on the same server. It doesn't take many resources to show the user a form, validate it, and save to a DB.
  - projektfu 7y ago
    Usually it's not so much the form that causes things to fall down but some validation step that they are trying to do synchronously, that might have to access an IBM mainframe, and things time out. When you're getting a few an hour, it's not a big deal.
  - Demiurge 7y ago
    Completely agree with the sentiment. I think most often it is inadequate default configuration that bottle-necks somewhere, that never got tested with more than a handful of users at a time. Going to a hundred highlights some bugs. going to 1000 others. On the other hand, I have worked on a project for USDA and they had 10 year old servers running 15 year old software and did not allow any system administration, while the system admins were some unknown government employees completely inaccessible. I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.
  - onion2k 7y ago
    It doesn't take many resources to show the user a form, validate it, and save to a DB. I bet that's what the previous developers thought. What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike? What if your database is rejecting transactions occasionally? What happens when your backup process locks all the database tables? How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions? What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic? What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926 https://news.ycombinator.com/item?id=22691926)? You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.
- commandlinefan 7y ago
  > If I was contracted to fix this ASAP You’re assuming that the people who built it in the first place (or the people that may or may not be contracted to fix it later) know or care. Remember, this is government contracting we’re talking about - lowest bidder wins. How do you win the lowest bid? By doing it as cheap and quick as you can. That means hiring inexperienced/cheap developers who can build something that looks like it will work for far less money than you can build something that actually will.
- OJFord 7y ago
  Ocado (the IaaS for online supermarkets company, and, in the UK, online-only supermarket itself) has done this in response to the increased demand, and makes you wait in a 'virtual queue' (virtual relative to what in America you call a 'line-up', but we call a 'queue', at a physical supermarket) before you can place or edit your order.
enraged_camel 7y ago
I mean, you can tell the numbers are extremely inaccurate via just a simple, cursory glance at the report. Pennsylvania reported 378k claims. California reported... 186k claims. Yesterday, California's governor said they've received more than 1 million claims since March 13th (so, over a 12 day period from the 13th to the 25th). This DOL report covers March 14th through the 21st. Are we to believe that the remaining 800k+ people all filed on March 13th, or March 22nd through the 25th? But there's more. Utah reported an increase of only 9 claims compared to the week before. They went from 1,305 to 1,314. Then, New York, where more than half of Covid-19 cases in the US are, reported only 80k?
- JMTQp8lwXL 7y ago
  As of 3/21. Lockdown in California began in the evening of 3/19. Still not enough time for the numbers to react.
  - _curious_ 7y ago
    "Are we to believe that the remaining 800k+ people all filed on March 13th, or March 22nd through the 25th?" I could believe it over the 22-25 stretch.
    - SpicyLemonZest 7y ago
      Especially since "filed" may here mean when the paperwork was finally able to snake its way into some particular system.
taylodl 7y ago
Somewhere there is an architect saying "I told you so!" I can almost guarantee the requirement was to handle several hundred requests per day, an architect pointed out if we get deluged then we won't be able to handle it, so maybe they were able to get them to allow for one or two thousand requests per day. Now of course we don't know what the architecture of this system is and what the deltas in cost would have been to allow this to scale-out more - but I do know that all too often the more robust solution giving you much greater protection and lower cost down the road is often discarded if it costs even just 5%-10% more. Then the day comes when the people making these decisions get caught flat-footed and they try to blame everyone but themselves. It doesn't always happen like this - but it happens a lot.
- jsight 7y ago
  I think that is small-think. The technical solution is only part of the problem and scaling up all systems to meet the .1% case seldom makes sense. They were smart to save 5-10%.
  - sushisource 7y ago
    Eh.... On the flip side, processing and storing some simple text forms should be able to handle 1000s of simultaneous users on one box. So, probably like most software of this nature, the reason it's not scaling is simply because the people who made it probably weren't the greatest engineers on the block.
    - chaz 7y ago
      These are the same kinds of assumptions that lead engineers to think they can build a [any product] clone in a weekend. It's unlikely that the problem or constraints are nearly as simple as one may think. Consider: single auth across all the state's services, external APIs, identity verification, address verification, employer ID verification, federal/military ID verification, income/tax verification, phone verification, bank account information, translation into multiple languages, accessibility features, etc. Also, there's probably a lot of legacy infrastructure and process. Also, if "ability to burst to 10x normal filings per week that might happen once every 40 years" wasn't in the spec, I think they were right not to engineer for it.
fivre 7y ago
They also tend to have some... interesting features dictated by the state UI office. When I'd applied in Wisconsin about 6 years back the site stopped accepting form submissions outside business hours. I assume some less computer-literate higher up thought that someone needed to be around to actually accept the form, same as in-person submissions.
murph-almighty 7y ago
Secondhand story: I briefly interned with a state judiciary's IT department around 2015 and got to get lunch with the CIO. He described to me how most court filings in the state had been manual prior to 2008 when the mortgage crisis hit and judges in the tax courts got _slammed_ with cases surrounding foreclosures. This , in turn, drove a need to develop a platform to automate the process of filing a case. It started with the tax court and gradually expanded to automate filings for other court divisions as well (e.g. Family, Civil). I wouldn't be shocked if the revelation of "holy shit no one can file for unemployment" drove such an investment. I honestly think the next generation of politicians should take a page from product owners by isolate some shitty process that they'd have jurisdiction over, and finding some way to automate it. Bonus points if it's right before a watershed moment- imagine if someone had considered the problem you described prior to the coronavirus epidemic.
MattSayar 7y ago
Wow, just found what my state (CO) is doing to help manage the influx. Talk about a low-tech workaround. >IMPORTANT NOTICE: Because of the high volume of claims, we are asking that you help us help you and our greater community. >If you need to file an unemployment claim and your last name begins with the letter A - M, file a claim on Sunday, Tuesday, Thursday, or after 12 noon on Saturday. >If you need to file an unemployment claim and your last name begins with the letter N - Z, file a claim on Monday, Wednesday, Friday or before 12 noon on Saturday.
- josephorjoe 7y ago
  Ooooh, like gas rationing in the 70s.
  - Melting_Harps 7y ago
    > Wow, just found what my state (CO) is doing to help manage the influx. Talk about a low-tech workaround. > Ooooh, like gas rationing in the 70s. I wasn't born back then, but I heard about that being based on License plates at a few car meetups by the older guys in the group and I had the same thought when I heard that on CPR. Odd, but it could work if you have total compliance; lets see how that pans out.
- billyhoffman 7y ago
  Distributed load balancing!
- TremendousJudge 7y ago
  well as long as it works...

by luma 7y ago

alexpetralia 7y ago

I confirmed that the state's unemployment website has slowed to a crawl (if it is working at all).

Demiurge 7y ago

As a developer I immediately though of the power of queues. Twenty people trying to submit same form does not work for everyone, but a queue processing one person at a time might allow the twenty people to submit within a short time. It is flattening the curve! If I was contracted to fix this ASAP, I would set up an nginx front-end proxy config that doesn't allow more than X sessions and suggest a time in the future when they could try again.

pjc50 7y ago
The matching UK system has a (huge) queue in it: https://www.computerweekly.com/news/252480546/Huge-queues-for-Universal-Credit-online-applications-due-to-coronavirus-surge https://www.computerweekly.com/news/252480546/Huge-queues-fo...
- Demiurge 7y ago
  Is there a human factor in processing these?
jkaljundi 7y ago
In related news on Queue-it: https://tech.eu/brief/queue-it-funding/ https://tech.eu/brief/queue-it-funding/
- _curious_ 7y ago
  Solid company
JMTQp8lwXL 7y ago
Put the web form (plain static assets JS/CSS/HTML) on a globally accessible CDN. Then use SQS intake for each unemployment application form. Then firehouse it out, wherever it needs to go, at a rate which you can realistically deal with it. Queuing access to the form itself and telling someone to wake up at 4:52 AM so they can then merely access the static assets is a less-than-desirable user experience.
- Demiurge 7y ago
  It is more desirable than 504, and first thing I would do in 15 minutes with zero context. If I can get more context, of course something like your solution is more desirable, depending on the issue. It would take some time to figure whether it is necessary to bring in AWS or just database connection pooler, or whatever.
- judge2020 7y ago
  Even typeform/Google Forms would be better suited for the task.
  - alasdair_ 7y ago
    >Even typeform/Google Forms would be better suited for the task. And now you've given a private company access to market-moving unemployment data. And a million other issues, especially legal ones. The technology part in and of itself isn't that difficult, it's all of the constraints (and, often, mountains of laws) that are the bigger issue.
driverdan 7y ago
Having worked on this type of application in the past they should find a new company to work with if they can't handle this traffic. We were handling hundreds of requests per second with ease 10 years ago. That was with MySQL and the app running on the same server. It doesn't take many resources to show the user a form, validate it, and save to a DB.
- projektfu 7y ago
  Usually it's not so much the form that causes things to fall down but some validation step that they are trying to do synchronously, that might have to access an IBM mainframe, and things time out. When you're getting a few an hour, it's not a big deal.
- Demiurge 7y ago
  Completely agree with the sentiment. I think most often it is inadequate default configuration that bottle-necks somewhere, that never got tested with more than a handful of users at a time. Going to a hundred highlights some bugs. going to 1000 others. On the other hand, I have worked on a project for USDA and they had 10 year old servers running 15 year old software and did not allow any system administration, while the system admins were some unknown government employees completely inaccessible. I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.
- onion2k 7y ago
  It doesn't take many resources to show the user a form, validate it, and save to a DB. I bet that's what the previous developers thought. What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike? What if your database is rejecting transactions occasionally? What happens when your backup process locks all the database tables? How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions? What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic? What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926 https://news.ycombinator.com/item?id=22691926)? You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.
commandlinefan 7y ago
> If I was contracted to fix this ASAP You’re assuming that the people who built it in the first place (or the people that may or may not be contracted to fix it later) know or care. Remember, this is government contracting we’re talking about - lowest bidder wins. How do you win the lowest bid? By doing it as cheap and quick as you can. That means hiring inexperienced/cheap developers who can build something that looks like it will work for far less money than you can build something that actually will.
OJFord 7y ago
Ocado (the IaaS for online supermarkets company, and, in the UK, online-only supermarket itself) has done this in response to the increased demand, and makes you wait in a 'virtual queue' (virtual relative to what in America you call a 'line-up', but we call a 'queue', at a physical supermarket) before you can place or edit your order.

enraged_camel 7y ago

I mean, you can tell the numbers are extremely inaccurate via just a simple, cursory glance at the report. Pennsylvania reported 378k claims. California reported... 186k claims. Yesterday, California's governor said they've received more than 1 million claims since March 13th (so, over a 12 day period from the 13th to the 25th). This DOL report covers March 14th through the 21st. Are we to believe that the remaining 800k+ people all filed on March 13th, or March 22nd through the 25th? But there's more. Utah reported an increase of only 9 claims compared to the week before. They went from 1,305 to 1,314. Then, New York, where more than half of Covid-19 cases in the US are, reported only 80k?

JMTQp8lwXL 7y ago
As of 3/21. Lockdown in California began in the evening of 3/19. Still not enough time for the numbers to react.
- _curious_ 7y ago
  "Are we to believe that the remaining 800k+ people all filed on March 13th, or March 22nd through the 25th?" I could believe it over the 22-25 stretch.
  - SpicyLemonZest 7y ago
    Especially since "filed" may here mean when the paperwork was finally able to snake its way into some particular system.

taylodl 7y ago

Somewhere there is an architect saying "I told you so!" I can almost guarantee the requirement was to handle several hundred requests per day, an architect pointed out if we get deluged then we won't be able to handle it, so maybe they were able to get them to allow for one or two thousand requests per day. Now of course we don't know what the architecture of this system is and what the deltas in cost would have been to allow this to scale-out more - but I do know that all too often the more robust solution giving you much greater protection and lower cost down the road is often discarded if it costs even just 5%-10% more. Then the day comes when the people making these decisions get caught flat-footed and they try to blame everyone but themselves. It doesn't always happen like this - but it happens a lot.

jsight 7y ago
I think that is small-think. The technical solution is only part of the problem and scaling up all systems to meet the .1% case seldom makes sense. They were smart to save 5-10%.
- sushisource 7y ago
  Eh.... On the flip side, processing and storing some simple text forms should be able to handle 1000s of simultaneous users on one box. So, probably like most software of this nature, the reason it's not scaling is simply because the people who made it probably weren't the greatest engineers on the block.
  - chaz 7y ago
    These are the same kinds of assumptions that lead engineers to think they can build a [any product] clone in a weekend. It's unlikely that the problem or constraints are nearly as simple as one may think. Consider: single auth across all the state's services, external APIs, identity verification, address verification, employer ID verification, federal/military ID verification, income/tax verification, phone verification, bank account information, translation into multiple languages, accessibility features, etc. Also, there's probably a lot of legacy infrastructure and process. Also, if "ability to burst to 10x normal filings per week that might happen once every 40 years" wasn't in the spec, I think they were right not to engineer for it.

fivre 7y ago

They also tend to have some... interesting features dictated by the state UI office. When I'd applied in Wisconsin about 6 years back the site stopped accepting form submissions outside business hours. I assume some less computer-literate higher up thought that someone needed to be around to actually accept the form, same as in-person submissions.

murph-almighty 7y ago

Secondhand story: I briefly interned with a state judiciary's IT department around 2015 and got to get lunch with the CIO. He described to me how most court filings in the state had been manual prior to 2008 when the mortgage crisis hit and judges in the tax courts got _slammed_ with cases surrounding foreclosures. This , in turn, drove a need to develop a platform to automate the process of filing a case. It started with the tax court and gradually expanded to automate filings for other court divisions as well (e.g. Family, Civil). I wouldn't be shocked if the revelation of "holy shit no one can file for unemployment" drove such an investment. I honestly think the next generation of politicians should take a page from product owners by isolate some shitty process that they'd have jurisdiction over, and finding some way to automate it. Bonus points if it's right before a watershed moment- imagine if someone had considered the problem you described prior to the coronavirus epidemic.

MattSayar 7y ago

Wow, just found what my state (CO) is doing to help manage the influx. Talk about a low-tech workaround. >IMPORTANT NOTICE: Because of the high volume of claims, we are asking that you help us help you and our greater community. >If you need to file an unemployment claim and your last name begins with the letter A - M, file a claim on Sunday, Tuesday, Thursday, or after 12 noon on Saturday. >If you need to file an unemployment claim and your last name begins with the letter N - Z, file a claim on Monday, Wednesday, Friday or before 12 noon on Saturday.

josephorjoe 7y ago
Ooooh, like gas rationing in the 70s.
- Melting_Harps 7y ago
  > Wow, just found what my state (CO) is doing to help manage the influx. Talk about a low-tech workaround. > Ooooh, like gas rationing in the 70s. I wasn't born back then, but I heard about that being based on License plates at a few car meetups by the older guys in the group and I had the same thought when I heard that on CPR. Odd, but it could work if you have total compliance; lets see how that pans out.
billyhoffman 7y ago
Distributed load balancing!
TremendousJudge 7y ago
well as long as it works...