11 ms·
How Gov.uk Notify reliably sends text messages to users
- ohlookabird 7y agoIt is a really useful service! I used their email service to get updates on potential travel destinations in the past months when Covid-19 started to be a thing.
- blntechie 7y agoNot to diminish anything but 100 to 200k messages per day is not really huge in relative scale of things. Several private services and government orgs in China and India easily send 10x or more of that number.
- robin_reala 7y agoTo be fair, both China and India have over 15x the population of the UK.
- SparkyMcUnicorn 7y agoThey never said 100-200k was the limit, but rather that's how many they're usually sending out per day. It could be a limit, but I don't see anything that indicates there's a correlation.
- petepete 7y agoDefinitely not a limit, on March 28th they sent 637k SMS messages. https://www.gov.uk/performance/govuk-notify https://www.gov.uk/performance/govuk-notify I've used Notify on several projects over the last couple of years, it's a really nice service and has never caused us any problem whosoever.
- adrianmonk 7y agoIf the available providers' services start bogging down when you try to actually do it, then it can be considered huge relative to their capacity.
- dirtydroog 7y agoThat surprised me too. We handle that amount of HTTP requests per second, and you don't really need a lot of infrastructure to do it. Something is wrong here. I bet it's all synchronous code and capacity is being wasted on waiting. Also, I doubt they're tailoring each individual SMS so there are likely to be efficiencies to be had here too.
- adwww 7y agoInstead of betting, you could read the code and tell us. https://github.com/alphagov/notifications-api https://github.com/alphagov/notifications-api
- dirtydroog 7y agoIs it this bit? https://github.com/alphagov/notifications-api/blob/e386d2ac38b44ecb341599189befd450def3d73f/app/clients/sms/firetext.py#L85 https://github.com/alphagov/notifications-api/blob/e386d2ac3... I'd like to clarify that the remark about load handling was aimed at your SMS gateway providers.
- mjw1007 7y agoI don't think the post was intended to be a boast about how many messages they send. I think it was intended to be a description of how it's sensible to set things up at the scale they happen to have.
- rospaya 7y agoThe company I work for (one of the largest providers in the world) handles 7+ billion interactions a month. Without checking our systems (and my NDA) I would bet we also provide for a lot of .gov.uk services.
- toomuchtodo 7y agoTake note USDS/18F! Something to consider instead of Govdelivery.
- SparkyMcUnicorn 7y agoWhat's interesting is that this looks like a service anyone can use. API and everything. https://www.notifications.service.gov.uk/ https://www.notifications.service.gov.uk/
- 867-5309 7y ago"if you work in central government, a local authority, or the NHS."
- toomuchtodo 7y agoReusable open source components all governments can use is still a huge improvement over the status quo, where each government overpays contractors to build suboptimal solutions that remain closed source.
- daguar 7y agoIt’s open source, and Australia and Canada have both deployed instances. Would love to see a US (or state run) instance.
- anticensor 7y agoUnited States Alert Message Service as a branch of USPS?
- toomuchtodo 7y agoYou’d want it under the authority of GSA (like Login.gov), as a foundational service (messaging.gov).
- 867-5309 7y agoI thought this might have explained the mechanics behind the recent GOV_UK CORONAVIRUS ALERT, but alas. It didn't even namedrop the SMS providers
- benbristow 7y agoFrom what I've heard (could be wrong) the government literally just said to all the networks 'send this to everyone' and let the individual networks handle it. Apparently they could've set up a system like in Japan where your phone gets emergency alerts (which I actually experienced when I visited on vacation a few years ago on my iPhone from the UK) and are handled specifically by the mobile operating system but the government were too cheap to set it up. https://www.theguardian.com/world/2020/mar/23/government-ignored-advice-set-up-uk-emergency-alert-system https://www.theguardian.com/world/2020/mar/23/government-ign...
- frutiger 7y agoSimilar to Japan, the US has https://www.fcc.gov/consumers/guides/wireless-emergency-alerts-wea https://www.fcc.gov/consumers/guides/wireless-emergency-aler....
- saluki 7y agoWe were in Target one night and a flash flood warning came through on everyone's phone, felt like a movie hearing the alerts cascade across everyones phones and see everyone stopping and viewing the alert.
- Symbiote 7y agoI think this is just GSM Cell Broadcast [1]. It seems strange that it would be complicated or expensive to set up. I've used cheap, aimed-at-tourist SIMs in developing countries where cell broadcast was used to send news and adverts, which is very annoying. [1] https://en.wikipedia.org/wiki/Cell_Broadcast https://en.wikipedia.org/wiki/Cell_Broadcast
- 7y ago
- jtchang 7y agoOne of the most important parts about this is that they applied backpressure to the load balancing process: > We also decided that if a provider is slow to deliver messages, measured in the same way as before, we would reduce their share of the load by 10 percentage points When doing systems design this is a critical piece to include in almost all load balancing aspects. Because a ton of time you will start with 100% of traffic load balanced between two boxes but what happens when one box fails or slows down? You don't want to overwhelm anything which leads to cascading failures.
- duxup 7y agoLoad balancers rule the world.
- oakesm9 7y agoHere's the full source code if anyone wants to take a look: https://github.com/alphagov/notifications-api https://github.com/alphagov/notifications-api
- hn_throwaway_99 7y agoI'm glad that they improved their delivery, but one thing I find frustrating about our industry is how often we all seem to be reinventing the wheel. I mean, there are tons of well-described load balancing algorithms with various pros/cons. From the article it sounds like they just figured out their load balancing algorithm through trial and error, rather than researching load balancing algorithms first, then tweaking them based on real world performance.
- Swizec 7y agoYeah but what looks better on your CV? “Developed and researched a novel algorithm to reliably send messages under intense load” Or “Tried 5 off the shelf solutions, picked 1 that seemed okay, and moved on with my life” That’s why we keep reinventing the wheel. Also it’s fun and we all think we’re the smartest.
- rootusrootus 7y agoAs a hiring manager, I know which one I'd rather see on a resume. Though I understand why an individual would choose the former.
- cosmodisk 7y agoThis is the stuff most of us have to deal on daily basis. Do I just google the it and move on, or maybe I should try to come up with it my self? Every time I google, I don't feel any good at all.In fact,I don't feel anything.When I come up with some solution myself,It elevates my motivation and I always learn something new. As a manager though,I sometimes allow to do this stuff,while on other occasions I specifically tell not to spend time on some creative stuff and just get something off the shelve.
- thanksforfish 7y ago> Yeah but what looks better on your CV? Neigher. Focus on achieved business outcomes instead. "Scaled GOV.UK Notify to 15 messages per second, ensuring reliable and timely delivery of 2FA codes and flood emergency warnings." "Developed and researched" tells me you did something interesting but I can't tell if it was resume padding or real work. If you can't articulate why, that's a red flag. A good hiring manager should be able to detect resume padding and avoid hiring people that waste company resources on pet projects.
- londons_explore 7y agoI'm not sure I'd advocate for this design... The traffic sharing and backoff seems crude... The "10% per minute" doesn't prevent sudden increases in load killing both providers. I would design it like this: Put all requests into a distributed queue, for example persistent pubsub. Have workers take work from that queue. Each worker should send a new request to a provider if the rate of requests sent in the past minute is < 2x the rate of requests in the previous minute, and the number of in flight requests is < 10x the average of the past minute, and the rate of errors, including timeouts, is <1%. If both providers are eligible, send to whichever has had the fewest requests in 24h. This prevents flooding/DoSing a badly configured provider (a well configured provider would have ingress ratelimiting, and you could do away with all the above logic). Have alerting on the age of the oldest item in the queue, and a monitoring dashboard showing dispatch rate to each provider, with response error codes. All the state is local to the worker, and doesn't need persisting. If a worker crashes, the item doesn't get acknowledged to pubsub, and will be retried. If you like, you can autoscale the number of workers based on their cpu utilization. I'd expect the above to scale to 10k qps per worker, and 5Mqps for 1000 workers before needing a redesign.
- sergiomattei 7y agoOn a side note, I'm a huge fan of the standardized design language of all UK Government sites. They really nailed the aesthetic and the consistency is unmatched. Edit: I'm impressed at this tooling.[1] 1: https://www.gov.uk/service-toolkit#gov-uk-services https://www.gov.uk/service-toolkit#gov-uk-services
- robin_reala 7y agoThat’s the GOV.UK Design System in play: https://design-system.service.gov.uk/ https://design-system.service.gov.uk/ (I helped to start the current incarnation of that, and it’s probably the thing in my career that I’m most proud of.)
- zouhair 7y agoSo that's where Canada.ca aesthetic comes from.
- robin_reala 7y agoAlso https://www.govt.nz/ https://www.govt.nz/ and https://www.gov.au/ https://www.gov.au/.
- frosted-flakes 7y agoAnd https://www.ontario.ca https://www.ontario.ca.
- sergiomattei 7y agoIncredible work. I don't think I've seen another government that has a design system so cohesive and well thought out.
- xiaq 7y agoI cannot praise the UI design on gov.uk more. It is a beacon of light in this dark age of web UI usability and I am not at all surprised that they have a coherent design system.
- 7y ago
- the_arun 7y agoWhat if aws was govt? - https://www.cloud.service.gov.uk/ https://www.cloud.service.gov.uk/
- CaciaraAsAServi 7y agoIs there a particular reason why, judging from the graphs, the rate suddenly drops every 6 minutes or so? Batching or something like it?
- ddddddj 7y agoHey, I'm the person who wrote the blog. I was slightly interested in the pattern too but didn't take the time to look into it. I assumed it was either something to do with a service sending us traffic in that pattern or maybe just something to do with Grafana. It could also be some unexplained behaviour in our system, maybe something to do with how we are pulling items off the queue. If I find some time next week I might take a proper look though!
- CaciaraAsAServi 7y agoCool! I was just curious :) as for the rest, fine article :)
- danpalmer 7y agoIf you’re using Prometheus and restarting servers regularly (a default behaviour for gunicorn for example) then you’ll lose a little data, up to your Prometheus scrape period, ever restart. My team found this pretty annoying for monitoring a Django site so we’ve ended up moving to a statsd push-based metrics approach and are finding numbers generally easier to trust and reason about.
- flibble 7y agoWe are facing the issue of SMS not getting delivered and have implemented failovers (Messagebird, Nexmo, Twilio) when we detect delivery problems. However we have a problem that frequently we get sent positive delivery receipts despite the SMS not being delivered. This makes it hard to know if we should fail over. Has anyone good solutions fur this?
- xeromal 7y agoWho do you get false positives from the most?
- dsincl12 7y agoFor me personally it was Messagebird. Two failed demos because of this and the response from them where non-existent. I was even contacted by one of their sales people after I wrote my last "angry I'm leaving" email asking me if I was interested in using their services.
- theblueprint 7y agoYes, similar position with Messagebird. Lack of transparency in their reporting/insights vs Twiliio. That said, our Twilio pricing is 2-3x Messagebird.
- aeonsky 7y agoIf its something requested, the user can always re-request it. If it not something requested like news, use two different providers to send 2 messages. One the actual content, and then a minute later, a different smaller message saying "This message brought to you buy MyCompany. Please text back STOP to stop messages or AGAIN to get the latest alert again." Some end users might get better delivery results with a specific texting service, so it might be useful to set service affinities on a user specific basis.
- john_minsk 7y agoYou must be kidding. This never happened to me so far, but if it did, I would be so unhappy with YourCompany that probably would never buy anything from you again
- harel 7y agoAs a user of Notify, I can vouch for the reliability of the service. It's rock solid.
- robga 7y agoIt is interesting the article didn’t mention the incident 2 weeks ago when it had problems with a 7-fold increase in volume. Though it did well under the strain. https://status.notifications.service.gov.uk/incidents/jpwxytmst2tx https://status.notifications.service.gov.uk/incidents/jpwxyt...
- ddddddj 7y agoHey, I'm the one who wrote the blog post. This was originally a talk a colleague on my team gave internally about 2 months ago and then I wrote it up as a blog post a few weeks back, before we had that big incident so there wasn't any particular thought on not mentioning it. As the postmortem mentioned, it wasn't related to any of this load balancing work or our providers, it was us running into trouble with a different part of our system. That was a busy week (both in terms of numbers as you can see on https://www.gov.uk/performance/govuk-notify/notifications-by-type https://www.gov.uk/performance/govuk-notify/notifications-by... but also in terms of us fighting fires).
- geospeck 7y agoOne interesting thing that I read the other day from a lead developer at Gov.uk was about how fast they managed to built a service within a couple of hours[1] What I also really like about Gov.uk is that they seem to have their apps open source[2] [1]https://twitter.com/RichardTowers/status/1243904365506760709 https://twitter.com/RichardTowers/status/1243904365506760709 [2]https://github.com/alphagov https://github.com/alphagov
- robin_reala 7y agohttps://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable https://www.gov.uk/service-manual/technology/making-source-c... When you create new source code, you must make it open so that other developers (including those outside government) can: - benefit from your work and build on it - learn from your experiences - find uses for your code which you hadn’t found
- Angostura 7y agoBefore we get too carried away; its worth seeing how the system coped with unprecedented pressure: https://www.bbc.co.uk/news/technology-52037573 https://www.bbc.co.uk/news/technology-52037573 "Millions of mobile users in the UK have yet to receive the government's text message alert about coronavirus. The SMS - telling people to stay at home - began being sent early on Tuesday morning. But Vodafone has confirmed it only expects to complete the process later this Wednesday...."
- ddddddj 7y agoJust in case it wasn't clear from discussion elsewhere on this post, Notify didn't send out the text message mentioned in the news. That message was sent directly by the networks at request of the government.
- leegraham 7y agoIs the BBC article talking about the same thing? The OP blog post says > we have 2 different [text message] providers but the BBC article says > Vodafone is the only one of the UK's big four networks that has not finished the task. If the two articles are talking about the same thing then it’s worth noting that the other major providers seemingly had no problems, and Vodafone chose to not send the messages at night time to avoid waking people up.
- rstuart4133 7y agoI see everyone is focusing on 3rd party providers. Here is another solution: smstools (https://packages.debian.org/buster/smstools https://packages.debian.org/buster/smstools), and a bunch of SMS modems pluggined into a USB hob. A SMS modem can send SMS in under 5 seconds. They say 200,000 a day max, so lets say you want to cope with sending 200,000 in 4 hours. That's a little under 14 per second, so lets say 15 per second. You need 75 modems[0] to do that, or about AUD$4,000 worth of modems. Sorry for the AUD$ - I'm Australian. You will also need SIM's - or about AUD$1,500 worth. Don't worry about having 75 modems in one spot, the mobile phone network is designed to cope with a stadium of people all sending at the same time. Perhaps you want to shard it for reliability - maybe 5 machines, so add another AUD$5000 for a NUC or similar all with a minimal Debian install + a web server or whatever for whatever delivery mechanism you are going to use to get the SMS's to the servers. That's AUD$10.5K total. Write some glue code - which is a week tops and job done. The one question I'd be asking is how does that compare to using the cloud. Third party providers charge around AUD$0.05 per SMS. They say a minimum of 100,000 SMS's per day - or AUD$150K / month. The cost for the non cloud solution is AUD$10.5K for the first month, then $1,500 / month after that for the pre-paid SIMS. Downsides: when it breaks (and it will), you will have to diagnose what's going on. That can be hard if the cause is a welding shop starting up next door. You are also going have to deal with the telco's screwing up their SMS infrastructure which seems to happen in Australia every 12 months or so. But you can fight that to some extent by geographically distributing your NUC's and using several different telco's servicing each NUC. That way it becomes more obvious what failed. Finally, instead of NUC's use industrial rated PC [1] to get your reliability up. [0] https://www.ebay.co.uk/itm/283828240407 https://www.ebay.co.uk/itm/283828240407 You need the version with an 'S' (for serial interface) suffix, although you can often just change the firmware. [1] https://fit-iot.com/web/products/fitlet2/ https://fit-iot.com/web/products/fitlet2/, industrial temperature rated. So, no stinking air conditioning required. :D
- ryanlol 7y agoWhy not just buy a sim box instead?
- rstuart4133 7y ago
- jimmySixDOF 7y agoThis topic reminds me of the story in Hawaii when they sent out a broadcast false alarm SMS alert to everyone about North Korean missiles If I remember correctly it was due to a poorly designed drop down menu & missing confirmation challenge box. "Test Send" and "Send Send" were right on top of each other in a 10point font lol. [1] https://en.wikipedia.org/wiki/2018_Hawaii_false_missile_alert https://en.wikipedia.org/wiki/2018_Hawaii_false_missile_aler...
- deleted 7y ago[deleted]