4 ms·
Serving a billion web requests with boring code
- deleted 2y ago[deleted]
- aegypti 2y agoI wonder if it’s possible for this to get resubmitted, or given another chance. We often see these “boring” tech write ups, but very rarely applied to something like healthcare.gov. With hindsight, I’m curious which of the other choices made would have changed the most with a multi page architecture, and have you discovered a cleaner way of avoiding it? Struggling with something similar at the moment.
- gregors 2y agoTalks about boring code, then immediately introduces a completely modern stack that most developers would love to use including grpc, gateways, rds, s3, ec2, testing, separated services & databases, mocking frameworks... There's nothing boring about this, it's unfortunate that they took to name this "boring code". Aside from this, the actual implementation details are quite interesting and worth a read. A better title would be "lessons learned using a modern stack for healthcare.gov"
- yellowapple 2y agoEC2/S3/RDS is about as boring of a cloud stack as it gets.
- ramses0 2y agoI think they're saying "boring" in that there were no surprising or unusual choices. Go. Test. Mock. Three Postgres Databases, mostly immutable, partitioned by data patterns / access patterns. Crontabs and shell scripts for data loading. No microservices, no multiple repo's, no strange/innovative technology. Pre-computing a bunch of stuff. 250-line query-builder instead of using GraphQL. vis: "The Complicators Gloves" https://thedailywtf.com/articles/the_complicator_0x27_s_gloves https://thedailywtf.com/articles/the_complicator_0x27_s_glov... ... I can imagine co-workers (past and present) who would have tried to add more complexity. `grpc` is a bit fancy (compared to `curl *.json`), which he admits to, but is also fairly battle-tested / boring, and gets some extra benefit of schema + performance. Maybe swagger or something would have been more "boring", but also veers a little away from "the default golang stack".
- llimllib 2y ago(Kind of a side note: grpc-gateway can generate swagger for you, which we did use for building a documentation site. Just thought that was neat)
- laurent_du 2y agoI think it's boring in the sense that none of it is shiny and new. It's all well-known and battle-tested. I agree that the article is very much worth a read.
- llimllib 2y agoI think it's important that to figure out what boring code is, you need to know the context of the organization creating it. At ad hoc in 2018, using go/react/rds/ec2 was a pretty standard stack that I could be sure my coworkers would have no trouble getting up to speed with.
- pizzafeelsright 2y agoHow is that a billion? 5MM a day is 150MM a month. Is this a year? That's 1.3B I generally measure the per second rate. Per millisecond can be mathed quick enough for conversational calculations.
- llimllib 2y agoI meant it in that we served at least a billion requests, which I just think is pretty neat. 5 million a day was a normal weekday non-open-enrollment request number; it was much higher during open enrollment but I no longer have the stats.
- frays 2y agoA better title would be "lessons learned using a modern stack for healthcare.gov" Very interesting post, thanks for sharing.
- fatnoah 2y agoIn the late 2010's, my startup built a messaging app with millions of daily users sending and receiving >1B messages a month, (somewhere north of 300 messages/sec, IIRC) with a trailing 12 month average uptime of our systems of 99.995% When we were going through due diligence as part of being acquired by a large tech company, I had to answer a lot of questions about our infrastructure and to "prove" that our boring tech stack could actually achieve this. That "stack" was 3 pods (2 in one AWS region, 1 in another) comprised of 4 Linux boxes and 3 Windows boxes. In a stack, Linux boxes were 2 monitoring servers and 2 app servers running an OSS SW package, while the Windows boxes were two app servers running our .NET APIs, and a SQL server. Fail over was handled by SQL cluster at DB level and ELB at the app level. While the stack and code were boring, a lot of work went into performance, reliability, and observability. After we were acquired, my new infra budget was based on their own modeling for scale, resulting in a monthly AWS budget that was 10X my previous annual budget.
- tiffanyh 2y agoUsing such new technology seems anything but “boring” to me. React was only 5-months old, and Go was only 4-years old, when healthcare.gov was launched. First released: - Go (Nov 2009) - React (May 2013) - Healthcare.gov (Oct 2013) https://en.m.wikipedia.org/wiki/Go_(programming_language) https://en.m.wikipedia.org/wiki/Go_(programming_language) https://en.m.wikipedia.org/wiki/React_(JavaScript_library) https://en.m.wikipedia.org/wiki/React_(JavaScript_library) https://en.m.wikipedia.org/wiki/HealthCare.gov https://en.m.wikipedia.org/wiki/HealthCare.gov EDIT: I missed this in the article: > There are many valid criticisms of react; this piece is an example, and I was aware of the issues already in 2018 when I was building the site.
- llimllib 2y agoThis work was begun in June of 2018. Go in the healthcare.gov context that Paul Smith used it was indeed an innovation token; they were in a dire situation at that point.