6 ms·

>To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time. Something of an understatement. T

>To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time. Something of an understatement. The project was red-flagged as'undeliverable' in 2019, after spending £154m https://www.theregister.com/2019/07/18/verify_to_be_flagged_undeliverable_by_gov_projects_watchdog/ https://www.theregister.com/2019/07/18/verify_to_be_flagged_... By 2020, and now at £200m sunk: https://www.computerweekly.com/blog/Computer-Weekly-Editors-Blog/Even-GDS-is-telling-GDS-to-shut-down-Verify https://www.computerweekly.com/blog/Computer-Weekly-Editors-... So now in 2021, we're back to informal tests and Post-It notes again?

EGreg 5y ago
Um ridiculous government waste! We could have done it all in $300K and so could have many other startups
- stuaxo 5y ago
  I don't know what they were covering, but there is a multitude of different systems from local councils up to central Gov. GDS is full of mostly django based systems, some legacy ruby and that's just the stuff from the last few years. Going back further there are various C# systems, the amount of separate systems is staggering, it's not too surprising costs add up.
  - deleted 5y ago
    [deleted]
  - rjzzleep 5y ago
    The problem in these environments is often not technical, but human, full of egos, people without competence wanting to be involved in every discussion and generally people talking and evaluating systems based on specs for a year without actually testing things. For every one of the systems you mentioned there is a fairly straightforward solution. And even if you can't solve them all you could gradually migrate everything. Oftentimes because of management wanting to push liability to a private party, they don't want to invest into local dev teams, but instead want to buy a big enterprise solution. The VA, NHS and the German systems all have these issues, but it's not because they're public institutions, it's because they work like enterprise.
  - petepete 5y ago
    Definitely not mostly-Django. There are probably more new Rails than Django projects these days.
- harel 5y ago
  Sounds like quite a naive statement. The scope of this project is huge. there are so many different systems at play here. Some are so legacy they go back to Tudor times.
  - tshaddox 5y ago
    Still, I could have attained an “undeliverable” state for far less money.
- dboreham 5y ago
  $300K doesn't get you much. Perhaps $10M though..
  - Aeolun 5y ago
    Hmm, it gets you me for 3 years or so. I can do quite a lot in 3 years if I have zero communications overhead.
- onion2k 5y ago
  You could deliver a single sign on system for 300k based on your requirements. Unfortunately you have to deliver it to the customer's requirements. If you go around quoting for projects without seeing the specifications outside of HN comments then your business is doomed.
  - Aeolun 5y ago
    Government projects always go to the lowest bidder! You’d just fail faster.
  - EGreg 5y ago
    That’s not how it works. If you make a system that is secure and flexible enough, you then work with other stakeholders to integrate with it over time. They each come up with a roadmap to migrate their systems to support your protocol. The key is to skillfully negotiate the budgets and politics between teams, so as not to get stuck building the next healthcare.gov.
smnrchrds 5y ago
Only £200m is impressively low for this type of failure. Canada's federal government is on track to waste more than 2.2 billion dollars on a pay system that does not work and needs to be replaced with another billion dollar system ASAP. https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system
- vletal 5y ago
  > provided by IBM in June 2011 In Czechia we have a system where all sellers have to log bills to prevent tax evasion. IBM also provided the system. It cost ~ 15m$ to set up and ~15m$/y to run. Given the rate of ~millions of bills submitted per day, of which the government only gets the total price and taxes, you have like ~hundred bills per second. I could scale a gunicorn+sqlite on my 2014 MacBook Pro to be able to receive this rate of requests with a payload which could fit inside a single TCP packet. Sure, there is auth/backups/analytics... etc. Yet, I still just do not see how could IBM charge the extra 14m$ to set that up...
  - soco 5y ago
    For one, it pays an army of "analysts" creating all kind of "documentation" nobody ever reads.
  - chupchap 5y ago
    Based on my limited experience in govt projects, there is a lot more to such implementations than what meets the eye. The number of backend systems this application would need to integrate with, the long list of compliance requirements that adds complexity to development process and setup, redundancies that need to be setup at different levels, odd technical requirements mentioned in the RFP to name a few are things that can bloat up the cost and implementation timeline. Now add dependency on any parallel development or upgrades to the mix and you have a very tough project to execute.
  - Aeolun 5y ago
    I can guarantee you that there’s like 1000 different idiotic rules that are applied to each of those items. They probably need to be stored on servers stored behind balistic glass, and cooled with helium. I’m not saying IBM is not still charging the government 14x more than necessary, but at least some of the expense is likely their own fault for having ridiculous requirements.
KronisLV 5y ago
Here's a thought experiment: suppose that you have a fledgling government somewhere that needs to digitize and can start fresh with the modern day technologies, as opposed to having their hands tied with millions of man hours that have been previously spent creating legacy code. Suppose that they decide that every single governmental system will use the same tech stack: - front end: Vue and JavaScript, or whatever's popular and easy to use - back end: Java with Dropwizard, or whatever has decent performance, decent maintainability (static types) and isn't too slow to write code for by the average developer (as opposed to something like Rust or C++) - database: PostgreSQL, or whatever is suitable for its needs (though open source, so otherwise MariaDB could be considered) - infrastructure: x86 servers, all running something like Ubuntu LTS or another popular, relatively stable distro - communication: REST everywhere with OpenAPI, due to their abundance In addition, perhaps there are some demands for the processes themselves: - all of the code must be open source - all of the discussions about the code and requirements must be public (a la GitLab's approach) - all of the services must follow 12 Factor App principles, have fully automated CI and must run in containers (say, Docker or anything OCI compatible, with a simple orchestrator, like Nomad or K3s) - all of the non-trivial services (e.g. files, reports etc.) must be separate, so not exactly monoliths, but not necessarily Netflix level of microservices - all of the services must have at least 75% method code coverage, separation of concerns, all methods longer than X lines must have comments explaining what they're doing and *why* - all of the services must use dependencies that are not older than X months, checked weekly - all of the above can be checked by a bunch of shell scripts and the CI pipelines will fail if the goals are not met So, with all of that present, how could any of that be better than our current approach of outsourcing, developing closed source projects poorly, not knowing who to know accountable and not learning anything from these due to a lack of post mortems? Would a strictly defined and consistent tech stack be a good choice or a bad choice? What about the job market and running a single stack in prod, across numerous systems? What about mandating low level technological decisions, as opposed to trying to sell everyone on some abstract business framework that has nothing to do with the end product? Short of powers that be wanting to line their pockets, for what other reasons would the above fail? Why wouldn't open source and strict limitations of what can be developed and how it must be developed work? Context: In Latvia, we have a system called "E-Health", there have been around 14-15 millions spent on developing it so far, however the project has been widely regarded as a failure: https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e-veselibas-galveno-diagnozi-atzist-slikto-planosanu-un-vadisanu.a350191/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=lt&_x_tr_pto=nui https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e... Now, they're planning on writing a new system instead, however it feels like they'll probably make the same mistakes, due to numerous companies having had their hand in the previous system's development, with no good leadership and bad technical implementation: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/e-veselibu-radis-no-jauna-izmaksas-meramas-miljonos.a396672/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=lt&_x_tr_pto=nui https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/... Surely there are both social and technical decisions that can be made make projects more successful?
quietbritishjim 5y ago
> To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time. This is a misleading paraphrasing of the article. With the full context, it becomes clear that it's specifically inclusiveness / accessibility that they're saying they didn't get right: > Inclusion is a hugely important part of our work, because anyone should be able to prove their identity to access government services, and it’s often the most vulnerable people who are at most risk of being excluded. To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time. This means that the rest of your comment doesn't follow: > Something of an understatement. The project was red-flagged as'undeliverable' in 2019, after spending £154m The second sentence could be right or wrong, I don't know. But the first half doesn't make any sense in the (actual) context of what you're quoting. If it's understatement then that means that they got accessibility very wrong.

EGreg 5y ago

Um ridiculous government waste! We could have done it all in $300K and so could have many other startups

stuaxo 5y ago
I don't know what they were covering, but there is a multitude of different systems from local councils up to central Gov. GDS is full of mostly django based systems, some legacy ruby and that's just the stuff from the last few years. Going back further there are various C# systems, the amount of separate systems is staggering, it's not too surprising costs add up.
- deleted 5y ago
  [deleted]
- rjzzleep 5y ago
  The problem in these environments is often not technical, but human, full of egos, people without competence wanting to be involved in every discussion and generally people talking and evaluating systems based on specs for a year without actually testing things. For every one of the systems you mentioned there is a fairly straightforward solution. And even if you can't solve them all you could gradually migrate everything. Oftentimes because of management wanting to push liability to a private party, they don't want to invest into local dev teams, but instead want to buy a big enterprise solution. The VA, NHS and the German systems all have these issues, but it's not because they're public institutions, it's because they work like enterprise.
- petepete 5y ago
  Definitely not mostly-Django. There are probably more new Rails than Django projects these days.
harel 5y ago
Sounds like quite a naive statement. The scope of this project is huge. there are so many different systems at play here. Some are so legacy they go back to Tudor times.
- tshaddox 5y ago
  Still, I could have attained an “undeliverable” state for far less money.
dboreham 5y ago
$300K doesn't get you much. Perhaps $10M though..
- Aeolun 5y ago
  Hmm, it gets you me for 3 years or so. I can do quite a lot in 3 years if I have zero communications overhead.
onion2k 5y ago
You could deliver a single sign on system for 300k based on your requirements. Unfortunately you have to deliver it to the customer's requirements. If you go around quoting for projects without seeing the specifications outside of HN comments then your business is doomed.
- Aeolun 5y ago
  Government projects always go to the lowest bidder! You’d just fail faster.
- EGreg 5y ago
  That’s not how it works. If you make a system that is secure and flexible enough, you then work with other stakeholders to integrate with it over time. They each come up with a roadmap to migrate their systems to support your protocol. The key is to skillfully negotiate the budgets and politics between teams, so as not to get stuck building the next healthcare.gov.

smnrchrds 5y ago

Only £200m is impressively low for this type of failure. Canada's federal government is on track to waste more than 2.2 billion dollars on a pay system that does not work and needs to be replaced with another billion dollar system ASAP. https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system

vletal 5y ago
> provided by IBM in June 2011 In Czechia we have a system where all sellers have to log bills to prevent tax evasion. IBM also provided the system. It cost ~ 15m$ to set up and ~15m$/y to run. Given the rate of ~millions of bills submitted per day, of which the government only gets the total price and taxes, you have like ~hundred bills per second. I could scale a gunicorn+sqlite on my 2014 MacBook Pro to be able to receive this rate of requests with a payload which could fit inside a single TCP packet. Sure, there is auth/backups/analytics... etc. Yet, I still just do not see how could IBM charge the extra 14m$ to set that up...
- soco 5y ago
  For one, it pays an army of "analysts" creating all kind of "documentation" nobody ever reads.
- chupchap 5y ago
  Based on my limited experience in govt projects, there is a lot more to such implementations than what meets the eye. The number of backend systems this application would need to integrate with, the long list of compliance requirements that adds complexity to development process and setup, redundancies that need to be setup at different levels, odd technical requirements mentioned in the RFP to name a few are things that can bloat up the cost and implementation timeline. Now add dependency on any parallel development or upgrades to the mix and you have a very tough project to execute.
- Aeolun 5y ago
  I can guarantee you that there’s like 1000 different idiotic rules that are applied to each of those items. They probably need to be stored on servers stored behind balistic glass, and cooled with helium. I’m not saying IBM is not still charging the government 14x more than necessary, but at least some of the expense is likely their own fault for having ridiculous requirements.

KronisLV 5y ago

Here's a thought experiment: suppose that you have a fledgling government somewhere that needs to digitize and can start fresh with the modern day technologies, as opposed to having their hands tied with millions of man hours that have been previously spent creating legacy code. Suppose that they decide that every single governmental system will use the same tech stack: - front end: Vue and JavaScript, or whatever's popular and easy to use - back end: Java with Dropwizard, or whatever has decent performance, decent maintainability (static types) and isn't too slow to write code for by the average developer (as opposed to something like Rust or C++) - database: PostgreSQL, or whatever is suitable for its needs (though open source, so otherwise MariaDB could be considered) - infrastructure: x86 servers, all running something like Ubuntu LTS or another popular, relatively stable distro - communication: REST everywhere with OpenAPI, due to their abundance In addition, perhaps there are some demands for the processes themselves: - all of the code must be open source - all of the discussions about the code and requirements must be public (a la GitLab's approach) - all of the services must follow 12 Factor App principles, have fully automated CI and must run in containers (say, Docker or anything OCI compatible, with a simple orchestrator, like Nomad or K3s) - all of the non-trivial services (e.g. files, reports etc.) must be separate, so not exactly monoliths, but not necessarily Netflix level of microservices - all of the services must have at least 75% method code coverage, separation of concerns, all methods longer than X lines must have comments explaining what they're doing and *why* - all of the services must use dependencies that are not older than X months, checked weekly - all of the above can be checked by a bunch of shell scripts and the CI pipelines will fail if the goals are not met So, with all of that present, how could any of that be better than our current approach of outsourcing, developing closed source projects poorly, not knowing who to know accountable and not learning anything from these due to a lack of post mortems? Would a strictly defined and consistent tech stack be a good choice or a bad choice? What about the job market and running a single stack in prod, across numerous systems? What about mandating low level technological decisions, as opposed to trying to sell everyone on some abstract business framework that has nothing to do with the end product? Short of powers that be wanting to line their pockets, for what other reasons would the above fail? Why wouldn't open source and strict limitations of what can be developed and how it must be developed work? Context: In Latvia, we have a system called "E-Health", there have been around 14-15 millions spent on developing it so far, however the project has been widely regarded as a failure: https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e-veselibas-galveno-diagnozi-atzist-slikto-planosanu-un-vadisanu.a350191/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=lt&_x_tr_pto=nui https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e... Now, they're planning on writing a new system instead, however it feels like they'll probably make the same mistakes, due to numerous companies having had their hand in the previous system's development, with no good leadership and bad technical implementation: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/e-veselibu-radis-no-jauna-izmaksas-meramas-miljonos.a396672/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=lt&_x_tr_pto=nui https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/... Surely there are both social and technical decisions that can be made make projects more successful?

quietbritishjim 5y ago

> To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time. This is a misleading paraphrasing of the article. With the full context, it becomes clear that it's specifically inclusiveness / accessibility that they're saying they didn't get right: > Inclusion is a hugely important part of our work, because anyone should be able to prove their identity to access government services, and it’s often the most vulnerable people who are at most risk of being excluded. To be honest we didn’t get this right with GOV.UK Verify. We have an opportunity and obligation to do much better this time. This means that the rest of your comment doesn't follow: > Something of an understatement. The project was red-flagged as'undeliverable' in 2019, after spending £154m The second sentence could be right or wrong, I don't know. But the first half doesn't make any sense in the (actual) context of what you're quoting. If it's understatement then that means that they got accessibility very wrong.