16 ms·
Examining the covidtests.gov architecture
- johnbatch 5y agoI just discover that https://analytics.usa.gov https://analytics.usa.gov exists to show the number of users on the site and other participating .gov sites in real time.
- IAmGraydon 5y agoAny guesses as to why Graceville, FL (with a population under 3,000) ranks under Washington and above Chicago for traffic?
- tyingq 5y agoI'm guessing some shady entrepreneur that has somehow compiled a bunch of unique shipping names and addresses to stockpile covid tests and try to sell them.
- jjeaff 5y agoNo doubt this is happening to some extent. But it does appear that the USPS requires valid residential addresses in order to send a mask. The USPS does have a database of actual addresses that they validate against. You can't simply send to something that looks valid. Perhaps some good citizen hacker created a bot that automatically submitted a request for every address in town.
- hoppyhoppy2 5y agoa minor correction just in case anyone's confused: this site is for ordering free covid tests to be delivered to your home; the free masks will be distributed starting next week at locations like pharmacies and health clinics
- johnbatch 5y agoIts also the top 3 on other non covid website like the DEP of commerce - https://analytics.usa.gov/commerce/ https://analytics.usa.gov/commerce/ You can get a JSON of the data also and see they have “ZZ” and “not set” as two cities excluded from the front dashboard. https://analytics.usa.gov/data/live/top-cities-realtime.json https://analytics.usa.gov/data/live/top-cities-realtime.json
- smarx007 5y agoMost likely a similar reason as to why Potwin got so much attention: https://theweek.com/articles/624040/how-internet-mapping-glitch-turned-kansas-farm-into-digital-hell https://theweek.com/articles/624040/how-internet-mapping-gli...
- perihelions 5y agoWhy's the #3 most active city in the US listed as Graceville, a Florida village with a population of 2,153? Is that a VPN exit point for some large provider? Google isn't helpful. I assume Ashburn traffic (#8) comes from AWS us-east-1. edit: And another interesting thing, there are no EU countries in the geolocation table.
- rojaz 5y agoI was wondering the same thing. The town has a total population of like 2200. I can’t find any data centers near there. Maybe it’s just a flaw in their metrics.
- samprotas 5y agoI’ve worked on systems in the past that had different levels of ip geolocation data precision. When the precision was low, they would pick the midpoint of the known area (maybe a whole state) to “fill in” more precise data (ex: the city). Since even “nowhere” is typically part of some town limits, we’d see otherwise tiny towns show up way too frequently. My guess is that’s what’s happening here.
- alksjdalkj 5y agoGraceville isn't really the midpoint of anywhere so I'd still be curious how it ended up being picked as the location.
- smarx007 5y agoCase in point: https://theweek.com/articles/624040/how-internet-mapping-glitch-turned-kansas-farm-into-digital-hell https://theweek.com/articles/624040/how-internet-mapping-gli...
- lmkg 5y agoAccording to their documentation, DAP is built on-top of Google Analytics, and also IP Anonymization is enabled by default. As a Google Analytics power-user, I happen to know two relevants facts: 1) Geography is inferred from IP Address 2) IP Anonymization happens before geo look-ups. So, what's probably happening is that some IP address in Graceville happens to end in the octect 0. And a bunch of other high-volume addresses, when you drop their last octect because of IP Anonymization, look like the IP address from Graceville. As a more general note, the accuracy of geography reports in GA are suspect to begin with (Amazon and Google only agree at the state level about 75% of the time), and with IP Anonymization enabled it takes a substantial nosedive.
- roughly 5y agoBoy, a look at that site and you see why this was hosted by USPS - almost 200M hits to USPS properties in the last 7 days. I’d been tangentially aware of USPS’s pretty solid improvements to their digital architecture/offerings, but I didn’t realize they’d grown that much.
- deleted 5y ago[deleted]
- cguess 5y agoGiven that essentially every American in the country interacts with USPS tangentially every day except Sundays, it seems reasonable. I get an email every day with an image of all mail coming to my mailbox, that probably counts when I open up the mail.
- np_tedious 5y agoThis is a wonderful service. I did it when I moved and set up forwarding, but continue see it every morning. USPS Informed Delivery / Daily Digest is awesome and underappreciated
- adanto6840 5y agoAnd is one of the few remaining [government] things that fails (for me, at least) to auto-verify for those of us who have an apostrophe (or suffix) in our name. I keep meaning to get signed up but, like the TSA quick-pass programs, it's always low on my list and I never get around to going in to "verify my identity". The auto-verify fails with any/all cards I've tried tried to use all the auto methods they provide (and address is owned & occupied single-family home/very 'standard'). Anyone that works on this -- please try to recognize those of us with "unusual" characters or a suffix as our given name! =D
- rootsudo 5y agoMeanwhile, chilling effect - government knows who sends you mail and can actively build a list of whom sends you mail and whom you send mail too.
- divbzero 5y agoI was looking at this yesterday too. Using USPS as the back end for COVIDtests.gov was particularly apt because (a) USPS is typically the US government website with the highest traffic and (b) verifying and storing addresses is their bread and butter.
- mlac 5y agoAnd I think (hope) they will ship it.
- TheSoftwareGuy 5y agoOmg it’s so snappy
- Axsuul 5y agoThat's so cool! It really feels like tech best practices have made lots of headway in the past 5 years within the U.S. government. Looks like this analytics frontend was brought to us by 18F, with the codebase even hosted on GitHub[0]. Other governments in the US also seem to be using this like the city of Los Angeles[1]. 0. https://github.com/18F/analytics.usa.gov https://github.com/18F/analytics.usa.gov 1. http://analyticsdash-46.s3-website-us-east-1.amazonaws.com http://analyticsdash-46.s3-website-us-east-1.amazonaws.com
- alpb 5y agoBackend for that seems to be Google Analytics, I wish the government didn't have to add such trackers shared with third-parties.
- deleted 5y ago[deleted]
- kingcharles 5y agoI wish it worked for me. My home has three separate addresses (it's a corner property), but USPS won't accept any of them. One it says is a commercial property and they won't deliver the COVID tests to there, and the other two it says don't exist. Despite the fact I get mail at all three addresses every day. I can't leave my home to get a COVID test because I am on house arrest, so I guess I am just out of luck.
- wmichelin 5y agoI'm half trolling here, but if you can't leave your home, why would you need a COVID test?
- toolz 5y agoI don't think that's a troll question, but I could imagine a situation where you're a high risk demographic you'd want to know early so you can get early treatment like monoclonal antibodies, paxlovid or ivermectin.
- jaredwiener 5y agoI know this is not the main point of this thread, but please do not use ivermectin for Covid. https://www.fda.gov/consumers/consumer-updates/why-you-should-not-use-ivermectin-treat-or-prevent-covid-19 https://www.fda.gov/consumers/consumer-updates/why-you-shoul...
- toolz 5y ago
- deleted 5y ago[deleted]
- tomrod 5y agoPlease don't spout trolling medical advice. Deworming medicine like Ivermectin does not treat COVID-19. https://academic.oup.com/qjmed/article-abstract/114/10/721/6375958 https://academic.oup.com/qjmed/article-abstract/114/10/721/6... > Based on very low to moderate quality of evidence, ivermectin was not efficacious at managing COVID-19. Its safety profile permits its use in trial settings to further clarify its role in COVID-19 treatment.
- blakesterz 5y ago"There is no excuse for getting this wrong. It’s heartening to see that the team seems to have gotten it right." It is amazing that every single thing I've read said more or less the same thing.
- bloodyplonker22 5y agoIt's amazing that this has become the low-bar standard for government websites -- simply to get basic things right is "heartening".
- perihelions 5y agoIt astonished me that the people who wrote the CDC covid tracker made it refuse Firefox visitors through User Agent checks. Mozilla had to add an intervention to spoof UA's, specifically on that site: https://bugzilla.mozilla.org/show_bug.cgi?id=1719846 https://bugzilla.mozilla.org/show_bug.cgi?id=1719846 ("Bug 1719846: Add UA override for https://covid.cdc.gov/covid-data-tracker/ https://covid.cdc.gov/covid-data-tracker/")
- jackson1442 5y agoWow. Really the only excuse to restrict a mainstream browser like that would be for a complex internal webapp where you can mandate browsers through organization policy. I remember I had to install a UA spoofer addon when I used FF to access some sites. Absolutely obnoxious.
- nevir 5y agoTBH it's a low bar for any software company – in general, we (developers) tend to dramatically over-engineer quite a few things.
- Tempest1981 5y agoEngineers have long memories. Get some thing wrong once, and it'll take years to overcome. They need to stay perfect for 3 to 5 years before "heartening" becomes "normal".
- willis936 5y agoOnly one test per address. Multi-family homes are sol. Top on their FAQ is "sorry to people living in apartments. Put in a request."
- davidw 5y agoIs it better to get tests to a bunch of people ASAP or wait until all the bugs have been worked out?
- willis936 5y agoI am not expecting perfection, but I am expecting the people who need the resources the most actually get access to it. Sfh in sprawlland benefit from some at-home tests that would cost them a negligible amount?
- davidw 5y agoAs a YIMBY, I'm very cognizant of the benefits of multifamily housing. But apparently their system has some problems with it. Would it be better to wait until the system is perfect for all addresses or go with what is functioning now and get tests to people?
- ahahahahah 5y agoBut again, your entire premise is wrong. It's not 1 request per physical building. People in apartments are able to get free tests from this just fine. You've made up a totally wrong hypothetical scenario and gotten all upset about what happens to people in your imaginary world.
- willis936 5y agoHow is my premise wrong? This isn't reddit, I'm going to need a bit more. People in mfh have more hiccups than those in sfh. How does this not advantage the advantaged?
- 5y ago
- flatiron 5y agoMan, are they going to blow through their free tier lambdas real quick! /s S3 is nasty expensive for outbound bandwidth. I’m surprised the usps landing page is there. No reason that couldn’t have been on lambda as well.
- pkulak 5y agoIt's fronted by CloudFront though. I really don't think it's appropriate to render static HTML from an AWS lambda.
- daveevad 5y agoAnyone up for guessing how much the AWS bill would be for this specific application? I'm guessing it's less than $100/month.
- mike_d 5y agoAny AWS product at scale gets real expensive real quick. A general rule of thumb is CloudFront is $2500/mo for every 100 Mbps of traffic you have, not counting origin or per request fees. I would expect they are pushing multiple Gbps with every household in the US trying to order tests.
- Tempest1981 5y agoCan you re-state the cost in terms of F-35 fighter jets?
- throwanem 5y agoAbout 3.1 Tbps per aircraft, if we generously assume the $78M flyaway cost is accurate.
- daveevad 5y agoI had missed the CloudFront part, once I saw Akamai my mind must have skipped it.
- teptoria 5y agoClicking "Order Free At-Home Tests" gives an HTTP error... bit rough @ US gov devs EDIT: US Postal Service's fault not gov
- CamperBob2 5y agoThe USPS is governed by, well, the government. The Republicans are actively looking to kneecap it, and as usual, the Democrats are doing basically nothing to stop them. So, calibrate your expectations accordingly.
- pjscott 5y agoIn my experience government services in areas completely controlled by Democrats don't seem better-run than ones in mixed-party or Republican-controlled areas. The "Starve the Beast" hypothesis doesn't seem to have much predictive value at the state and municipal levels, though I admit I haven't disproven it at the federal level.
- wslack 5y agoTry getting off VPN if you're on one.
- georgesequeira 5y ago"There is still no turnkey solution for UX" What would that even look like? Do they expect a unified UX language from the Gov t?
- shotgun 5y agoPerhaps like USWDS? https://designsystem.digital.gov/ https://designsystem.digital.gov/
- dylanz 5y agoIt definitely looks like USWDS, which happens to have accessibility well baked in.
- mxuribe 5y agoIt has been years since i have had to manage web design stuff...but as i recall, section 508 specifically calls for federal-centric web properties to - by law - include accessibility features to accomodate the American Disabilities Act (the ADA). So, yeah, it better damn well have accessibility baked in, else it would be illegal. :-) To be fair, the little documentation and blog posts here and there that i have read on the US Digital Service is that those folks reaslly are attentive to design considerations for always including accessbility...which is awesome!!
- llimllib 5y agoThis site uses the USWDS
- Androider 5y agoThe UK has https://www.gov.uk/service-toolkit https://www.gov.uk/service-toolkit and https://design-system.service.gov.uk/ https://design-system.service.gov.uk/
- deleted 5y ago[deleted]
- testplzignore 5y agoInteresting thing: Some of the assets actually live on www.usps.com, like https://www.usps.com/assets/script/lib/jquery.special.js https://www.usps.com/assets/script/lib/jquery.special.js, which itself seems to be (based on the HTTP response headers) served from EdgeCast sitting in front of Azure.
- gazby 5y agoAzure offers three CDNs directly, one of which is EC (shown as Verizon due to the acquisition). https://azure.microsoft.com/en-us/pricing/details/cdn/#pricing https://azure.microsoft.com/en-us/pricing/details/cdn/#prici...
- EE84M3i 5y agoThere is approximately 0% chance that this is using Akamai NetStorage. No one uses this. The Akamai portion is probably just hosted on S3.
- bloodyplonker22 5y agoWith Cloudflare being so cheap and easy to use, I am also always astonished when I hear of people still using Akamai.
- 8192kjshad09- 5y agoThey and their salespeople manage to get their meaty claws in the door at most large organizations.
- dallasgutauckis 5y agoCareful. Missouri Governor, Mike Parson, would consider this hacking.
- danhab99 5y agoGuys let's print out Missouri's website and snail mail it to them. They'll get so scared of hackers trying to shut their internet down, maybe even secede.
- deleted 5y ago[deleted]
- mrintellectual 5y ago> But scalable web services are, in 2022, a commodity available to all. A quick aside - I feel most folks on HN can say that they have been, in some way, shape, or form, a part of this journey for the industry. Take pride in being part of a community that has made this a possibility.
- tonymet 5y ago"scalable" is just a euphemism for inefficient. With the hardware available today, 8m reads / day is easily served on 1 small instance. Until we measure our services in transactions / kW, things will continue to get more redundant.
- switchbak 5y agoPower efficiency is definitely a concern at places like Google. Hopefully it'll catch on more, and encourage the use of productive AND efficient approaches.
- tonymet 5y agoit’s a concern for anyone with “engineer” in their title
- asutekku 5y agoTheoretically yes, in reality sadly no.
- astrange 5y agoIt's a concern for people who can't afford their power bills. It's not really a concern for Google, who have infinite money and could save power by not running 20 different messenger services if they really wanted to.
- orf 5y agoIs it power efficient to have 1 (small|medium|large) instance per workload? Or more realistically >=2 instances for redundancy. No? Maybe we could get cost, energy and other efficiencies by aggregating disparate smaller and larger workloads over a system that scales up and down with demand, so every instance is processing X million TPS. Cool idea. We could label that scalable, because it’s vastly more efficient!
- lelandfe 5y agoIt's really fast too! They're preloading fonts! I'm so proud.
- dbbk 5y agoI'm surprised the US government design system even recommends custom fonts. Surely the native stack would be sufficient?
- lelandfe 5y agoWell, it depends. Perhaps they want a multilingual font that looks good? Perhaps they have dyslexia requirements around the characters? Those are both things I would consider if making a design system for the government. That being said, while this site is already extremely fast, they could cut their first paint times by a further 90% by just reducing their font file sizes. Since this is a static page, and we want things really fast for all Americans (99 percentile?), maybe subset characters to what's shown above the fold, base64 encode that subsetted font into the document and load the remaining characters async as a fast follow. Bang, great CLS and ~0s FCP.
- encryptluks2 5y agoWhen they fail to deliver a majority of tests kits by early February, at least they can say the website was working (for the most part).
- deleted 5y ago[deleted]
- erlkjlkjlkjth 5y ago[flagged]
- babbledabbler 5y ago"Effective, mission-critical software at scale need not be complicated." Amen.
- bijoo 5y agoit looks like submitting the form gets stuck now
- toomuchtodo 5y agoUSDS is crushing it. Really well executed. Thank you govtech folks.
- tonymet 5y agothey don't get any pats on the back until we see the invoice. The original healthcare.gov was about $300m . I'm guessing this glamorous system had a similar budget. Wed be better off letting people buy masks on amazon and deducting on their 1040
- devman0 5y agohealthcare.gov launched before USDS was established. I would argue that part of the reason USDS exists is that fiasco which if I recall correctly was mostly CGI Federal.
- VaxWithSex 5y agoThat is exactly the reason.
- throwawaygh 5y agoUsds is internal, so highly unlikely. The healthcare.gov fiasco was the result of government NOT doing things and instead subcontracting everything.
- tonymet 5y agowhy downvote? does anyone know how much this service cost? You like paying for it?
- VaxWithSex 5y agoOMG, IT IS EVEN BETTER THAN YOU THINK, if you look at the form endpoint mentioned: https://special.usps.com/testkits https://special.usps.com/testkits without submitting anything, it returns this error code: QWNjZXNzIG1haW4gcHJvZ3JhbS4gQWNjZXNzIG1haW4gc2VjdXJpdHkuIEFjY2VzcyBtYWluIHByb2dyYW0gZ3JpZC4 which base64 translates to: "Access main program. Access main security. Access main program grid." Which is a JURASSIC PARK reference: https://jurassicsystems.com/ https://jurassicsystems.com/
- killjoywashere 5y agoI've worked with DDS, USDS's Defense child org, and the author's description sounds exactly like what we did.
- ehsankia 5y agoI'm curious how the mechanics of this work. Can someone just put in orders for every single address in the US, even those that they don't live at? Also, I assume a bunch of people will get these and resell them, which I assume is very illegal, is there something to stop that?
- chithanh 5y agoIn my limited understanding so far: You cannot order to any address, only to residential addresses that USPS ships to (as indicated by their Residential Delivery Indicator). Specifically, you cannot order to commercial addresses and PO boxes. Resale is not prevented but (intentionally or not) limited by the distribution of only 4 tests per home, regardless of how many people live there. This means that poor folks where typically many people share a household get fewer tests per person to begin with, while more affluent folks (who are less likely to resell the tests) get more per person.
- xeromal 5y agoJust like with a lot of government programs, the cost to manage preventing fraud is usually more expensive than to just let the fraud take place.
- sethetter 5y agoNot to mention it prevents people trying to legitimately access these services from doing so by putting extra barriers in place, defeating the purpose of the assistance program in the first place. It's a tricky balance to strike!
- tjbiddle 5y agoSo - It's a landing page with a CTA that goes to the USPS website.
- solatic 5y agoWhat's interesting about this is the conscious decision to throw money at the government IT problem. Meaning, services like AWS Lambda get very, very expensive at scale. Nationwide government-scale is probably pretty high scale, and I shudder to think what the AWS bill would be. Yet, most government IT projects run very high costs to begin with anyway. Indeed, for all we know, they probably saved money, even with the high costs of running this at scale on Lambda. And it'll be worth it if the service is actually functional. Just interesting to think about the cost of introducing stability as a standard before eventually the pendulum will swing back and get people to think about cost optimization.
- dclowd9901 5y agoGiven what they’d probably spend to a contractor like General Dynamics to completely flub this, I’m glad for them to foot a massive AWS bill instead.
- koblas 5y agoIf you go with the authors original description that the lambda function for handling orders is small and fast. Probably a bunch of server side validation before stuffing in DynamoDB or maybe into a queue (Message Queues / Kafka). We can assume that the program is small and fast. If you saw 300,000,000 requests in the month on a 256MB lambda function and 500ms of runtime. It would only cost $678 for the order capture. Maybe my math is wrong here, but not huge in my book.
- interblag 5y ago> I suspect what’s happening from here is that API Gateway is in front of a Lambda function (indeed, this is a common design pattern that AWS documents) that does minimal-to-no processing of the JSON blob and puts it in a database, likely DynamoDB given the overall managed services flavor of this implementation. At that point, backend processes can take over... A bit of an aside, but since this is an article about architecture, wouldn't the more common pattern here be APIGateway->Lambda->MessageQueue->BackendServices rather than APIGateway->Lambda->Database->BackendServices? Or does DynamoDB have something like a queue that backend processes can subscribe to? (Non-AWS user here).
- Cyberdogs7 5y agoReally either would work, depending on your specific use case. DynamoBD can emit an event stream based on writes/updates, which would be published to an SQS queue.
- dpcx 5y agoDynamoDB has DynamoDB Streams[0] and Kinesis Streams[1] which it can hook in to. 0: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... 1: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
- jacobr 5y agoIt’s interesting that this is a government service in the US. That would probably not be politically possible in Sweden - there are already online stores where you can buy tests, the government would be competing with them. We also don’t have a national postal service, it’s a deregulated market. There’s an operator owned by the state, but only 50%, the rest is owned by Denmark, so it’s unlikely it could participate in something like this without it being an open contract for all delivery companies.
- petarb 5y agoIt was simple and boring. But boring is good because it worked
- johnthrwy 5y agocan someone explain why it is so convoluted? are all sites so convoluted? is it security by obfucstion going on?