7 ms·

Having worked on this type of application in the past they should find a new company to work with if they can't handle this traffic. We were handling hundreds o

Having worked on this type of application in the past they should find a new company to work with if they can't handle this traffic. We were handling hundreds of requests per second with ease 10 years ago. That was with MySQL and the app running on the same server. It doesn't take many resources to show the user a form, validate it, and save to a DB.

projektfu 7y ago
Usually it's not so much the form that causes things to fall down but some validation step that they are trying to do synchronously, that might have to access an IBM mainframe, and things time out. When you're getting a few an hour, it's not a big deal.
Demiurge 7y ago
Completely agree with the sentiment. I think most often it is inadequate default configuration that bottle-necks somewhere, that never got tested with more than a handful of users at a time. Going to a hundred highlights some bugs. going to 1000 others. On the other hand, I have worked on a project for USDA and they had 10 year old servers running 15 year old software and did not allow any system administration, while the system admins were some unknown government employees completely inaccessible. I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.
onion2k 7y ago
It doesn't take many resources to show the user a form, validate it, and save to a DB. I bet that's what the previous developers thought. What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike? What if your database is rejecting transactions occasionally? What happens when your backup process locks all the database tables? How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions? What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic? What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926 https://news.ycombinator.com/item?id=22691926)? You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.
- Dylan16807 7y ago
  Or it's a much simpler problem that they didn't make it semi-fast because it didn't need to be semi-fast. When "hundreds of times the usual level" is still only 50 page loads per second, and 10 milliseconds of CPU per page would be extreme overkill for anything written in a reasonable way, it actually is straightforward.
  - GordonS 7y ago
    It's not just CPU though, but IO - I've worked with horrible enterprise systems before that had response times measured in seconds.
    - Dylan16807 7y ago
      Even 5 seconds will work if the actions can overlap. If it can't do things in parallel then we have issues much more fundamental than "performance", and there's no defending it as a competent system. (That is not to say it's necessarily the devs' fault.)
      - GordonS 7y ago
        I don't mean to defend it too much, because realistically it should be possible with relative ease to handle much more traffic than that - but my point is that in the enterprise and government worlds, things are often not as simple as you think. Aside from potentially having to interface with dozens of unreliable, painfully slow SOAP-based web services, everything is often hosted on creaking, over-subscribed VMWare hosts, in VMs that would be under-specced regardless. There is also often a "governing body" that severely restricts your tech stack choices. Want to use Postgres? Nope, our standard is SQL Server - 2008 edition, actually! Want to use Python/Ruby/Elixir/Clojure/Kotlin? None of that hipster nonsense here, we use good ole Java/VB.NET here! Message queue, you say? It's Windows Message Queue with distributed COM all the way down here! "Containers"? What's a one of those? You'll get a crappy VM with 1 vCPU and 1GB of RAM, and you'll thank me for it! etc... As a dev, it's horrible and soul-destroying to work under such limitations, but if you have no choice...
dubcanada 7y ago
A bunch of armchair developers seem to have been summoned to tell the Federal government how to handle form submissions for an extremely security and privacy intense application using their fancy modern techniques. You are talking about comparing a basic web form with an application for unemployment benefits which must go into a federal tax database and be processed using a what I assume is a garbage mainframe system. It not only needs to be validated, it needs to securely store records, be able to compare them, and hook up to the system that handles payments, etc. They can't just circumvent it and dump it into some silly Amazon or MySQL database and call it a day. That would require the employees to basically copy and paste that data into the actual warehouse and considering they have 3+ million to go through as it is making it easy for them to process is just as important as allowing people to submit. For the time being the correct response is a queue gate. Stop being silly.
- Demiurge 7y ago
  I also completely agree with this sentiment. A gov't form could be an unsightly complex beast that can't be re-architected, sometimes, ever.
  - deleted 7y ago
    [deleted]
  - vsareto 7y ago
    They could, they just don't want to pay for it. The government has no interest in being known for easily handling a huge spike of traffic during a crisis. They can just take the lower road and get by with less and saying 'try again later'. There's no repercussions here because it's the government. Hence mainframe maintainers should really move to charging $1 million/year in a decade or two.
    - seph-reed 7y ago
      So, legitimately, how to we make it so the government does have repercussions? I see a lot of people making jokes about guillotines and nooses, but is there no better way?
noelsusman 7y ago
At this point introducing a new company could cause more problems than it solves, and I think it's understandable to not be prepared for a volume of jobless claims that is almost an order of magnitude more than at any point in US history.

projektfu 7y ago

Usually it's not so much the form that causes things to fall down but some validation step that they are trying to do synchronously, that might have to access an IBM mainframe, and things time out. When you're getting a few an hour, it's not a big deal.

Demiurge 7y ago

Completely agree with the sentiment. I think most often it is inadequate default configuration that bottle-necks somewhere, that never got tested with more than a handful of users at a time. Going to a hundred highlights some bugs. going to 1000 others. On the other hand, I have worked on a project for USDA and they had 10 year old servers running 15 year old software and did not allow any system administration, while the system admins were some unknown government employees completely inaccessible. I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.

onion2k 7y ago

It doesn't take many resources to show the user a form, validate it, and save to a DB. I bet that's what the previous developers thought. What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike? What if your database is rejecting transactions occasionally? What happens when your backup process locks all the database tables? How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions? What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic? What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926 https://news.ycombinator.com/item?id=22691926)? You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.

Dylan16807 7y ago
Or it's a much simpler problem that they didn't make it semi-fast because it didn't need to be semi-fast. When "hundreds of times the usual level" is still only 50 page loads per second, and 10 milliseconds of CPU per page would be extreme overkill for anything written in a reasonable way, it actually is straightforward.
- GordonS 7y ago
  It's not just CPU though, but IO - I've worked with horrible enterprise systems before that had response times measured in seconds.
  - Dylan16807 7y ago
    Even 5 seconds will work if the actions can overlap. If it can't do things in parallel then we have issues much more fundamental than "performance", and there's no defending it as a competent system. (That is not to say it's necessarily the devs' fault.)
    - GordonS 7y ago
      I don't mean to defend it too much, because realistically it should be possible with relative ease to handle much more traffic than that - but my point is that in the enterprise and government worlds, things are often not as simple as you think. Aside from potentially having to interface with dozens of unreliable, painfully slow SOAP-based web services, everything is often hosted on creaking, over-subscribed VMWare hosts, in VMs that would be under-specced regardless. There is also often a "governing body" that severely restricts your tech stack choices. Want to use Postgres? Nope, our standard is SQL Server - 2008 edition, actually! Want to use Python/Ruby/Elixir/Clojure/Kotlin? None of that hipster nonsense here, we use good ole Java/VB.NET here! Message queue, you say? It's Windows Message Queue with distributed COM all the way down here! "Containers"? What's a one of those? You'll get a crappy VM with 1 vCPU and 1GB of RAM, and you'll thank me for it! etc... As a dev, it's horrible and soul-destroying to work under such limitations, but if you have no choice...

dubcanada 7y ago

A bunch of armchair developers seem to have been summoned to tell the Federal government how to handle form submissions for an extremely security and privacy intense application using their fancy modern techniques. You are talking about comparing a basic web form with an application for unemployment benefits which must go into a federal tax database and be processed using a what I assume is a garbage mainframe system. It not only needs to be validated, it needs to securely store records, be able to compare them, and hook up to the system that handles payments, etc. They can't just circumvent it and dump it into some silly Amazon or MySQL database and call it a day. That would require the employees to basically copy and paste that data into the actual warehouse and considering they have 3+ million to go through as it is making it easy for them to process is just as important as allowing people to submit. For the time being the correct response is a queue gate. Stop being silly.

Demiurge 7y ago
I also completely agree with this sentiment. A gov't form could be an unsightly complex beast that can't be re-architected, sometimes, ever.
- deleted 7y ago
  [deleted]
- vsareto 7y ago
  They could, they just don't want to pay for it. The government has no interest in being known for easily handling a huge spike of traffic during a crisis. They can just take the lower road and get by with less and saying 'try again later'. There's no repercussions here because it's the government. Hence mainframe maintainers should really move to charging $1 million/year in a decade or two.
  - seph-reed 7y ago
    So, legitimately, how to we make it so the government does have repercussions? I see a lot of people making jokes about guillotines and nooses, but is there no better way?

noelsusman 7y ago

At this point introducing a new company could cause more problems than it solves, and I think it's understandable to not be prepared for a volume of jobless claims that is almost an order of magnitude more than at any point in US history.