6 ms·

> This project was implemented by the same agencies - MCI together with Caracal. I suspect that this is the root cause of this and for many other systems faili

by 2rsf 3y ago

> This project was implemented by the same agencies - MCI together with Caracal. I suspect that this is the root cause of this and for many other systems failing. When a project is created by the lowest bidder, as a one time effort with fluffy requirements why would they invest in proper architecture, planning or testing? Why would they invest in securing resources when they are paid anyway?

pjc50 3y ago
This is where quite a lot of people would insert a rant about "state capacity": the ability of the state to actually do things it wants and intends to do. Which requires people to do those things, trained with appropriate skills. The peak of "state capacity" was undoubtedly WW2, when governments bypassed market mechanisms and became command economies. Out of necessity - war is the one venture in which failed state capacity can end the state itself, and the personal privileges of those running it and the elite around them. It's not a coincidence that the centralized socialist institutions of the UK, the NHS and state education, date from that period. Heck, the state commissioned the invention and building of cutting-edge computer technology! But since that no longer matters, there's little to no will to build state capacity in computing.
- afavour 3y ago
  It’s not WW2 mobilisation but I’ve always thought the UK’s Government Digital Service is a wonderful example of what government can achieve in tech: https://www.gov.uk/government/organisations/government-digital-service https://www.gov.uk/government/organisations/government-digit... As I understand it they’re effectively a central dev shop for other government agencies. It’s worth their time investing in good practises because they’re going to use them over and over again. And from the user perspective you get a very consistent, reliable set of tools for interacting with government. A win win in my book.
  - franga2000 3y ago
    This is something we desperately need more of in other countries. We've found something like a dozen breaches of similar severity in the last 6 years and they all came from systems developed through public tenders by companies that either aggressively under-priced or used other (legal or illegal) dirty tactics to win them. The very few things that were developed in-house have proven to be far more reliable and secure, not to mention developing them was far cheaper and the UX is better and more consistent between them.
    - sfn42 3y ago
      As a software "consultant" (meaning i do what everyone else does but get paid more) the consultancy business is fucking bullshit. Just padding resumes and counting billable hours, nobody gives a fuck about quality or maintainability or anything. Just money.
  - FourthProtocol 3y ago
    GDS did a great job building gov.uk, but everythig else they touched was an abject disaster. From the Diabetes project at the NHS, the fiasco that was the Office of the Public Guardian, the even bigger fiasco that happened at Border Force, the NHS, DWP... Sure, when they were building web sites they delivered stellar stuff. Agile, break things and all that. But when you had real complexity they just... couldn't... The Government Gateway is a prime example - single citizen login for ALL government services. It ran well, super robust and mature enough to have ironed out virtually all issues. Then GDS decided that because the Government Gateway was based on a Microsoft stack, it needed to be re-done. The tech lead didn't understand the concept of Identity Federation, let alone SAML tokens, and that you just! can't! do secure code using agile (2-week sprint no good for meaningful security testing...). I spent two long years at GDS banging my head against a wall. And then I left. And unsurprisingly the Microsoft-based Government gateway was never replaced, still going strong.
- veltas 3y ago
  It's a shame that (from my perspective anyway) a lot of that state capacity seems to have degassed for the NHS. Right now it's impossible to get a doctor's appointment where I live. If you call any local surgery within a split second of 8:00am then you have a very low chance of getting an appointment, every time I've tried the line's busy or I'm number 60 in the queue and after a 40 minute wait all the appointments are gone. They are also fully booked for advance appointments every day (they limit the time in advance you can book, probably to prevent having e.g. a 2 year wait time on appointments, because it 'looks better' if nobody can go than if there's literally a 2 year advance appointment wait). The only way to see a doctor is to go to A&E, or convince NHS 111 to give you an "emergency appointment" of some kind. All of this drains emergency resources and are not an option due to the time investment for average people with precarious employment. Unfortunately many people won't believe these facts, because depending on which area you live in there's always plentiful appointments, in advance or on the day! Where I last lived, getting an appointment was easy, the difference in outcomes based on how wealthy or urban your area is leaves a bad taste in my mouth. Briefly, we had a new surgery open that offered more appointments, at more convenient times of day, and we could actually see the doctor. All the other GP's started losing their patients to them. Then within a few months they were shut down by the local NHS trust, under multiple investigations (one of these investigations was regarding an offensive Facebook post by the surgery's chief, I kid you not). They then later reopened with normal appointment times and no free spaces, like all the other surgeries. I agree with what you say re the command economy of WW2 allowing the creation of the NHS. But it was not this country that founded the NHS: it is some ancient, lost nation that seems utterly alien to me today. I don't believe we could achieve even a small version of what WW2 Britain did anymore, if our survival depended on it. I cite the UK's response to COVID19 as evidence.
  - Scarblac 3y ago
    Same as in the US -- right wing politicians who believe the state is bad at doing things remove the funding that allowed the state to do it. Then when that means those government provided services become worse, they use that to show that the state is bad at doing those things, and therefore even less funding should go there, et cetera.
  - 3y ago
- marcosdumay 3y ago
  Governments do in-house development just as well as any company with a large development department. Governments also do development off-shoring just as well as any company with a large off-shored project. The kind of project people are talking about here never works. It doesn't matter who is doing it.
nness 3y ago
It's a fallacy to believe that all projects are just sold to the lowest bidder. There are probably a dozen reasons why something like this might have occurred, and not giving the vendors a free pass, but assuming that a more expensive vendor would do a better job with security and reviews is just as likely to be a mistaken belief. If a project is too expensive for a client to do well, they should not be doing that work in the first place.
- boringg 3y ago
  Its actually probably more of a situation that a client cant discern quality. Its impossible to tell if the most expensive or least expensive if the best option. How does a non technical/semi technical actually grade this stuff appropriately?
  - davemp 3y ago
    This is a problem traditionally solved by the professional engineering licensing system. Most engineering curriculum in the USA involve an Engineering Ethics course that goes over such issues. We're quite far from implementing such a system for software "engineers".
    - comboy 3y ago
      Just hire pentesters along with the development team. Make sure they are not affiliates. You can put in contract that as long as security issues are present they need to fix them before getting paid, which seems like a reasonable expectation. Even the best make mistakes, so let's at least leave those which are not trivial. We don't trust building ethics, independent inspector comes and checks if everything is as it should be before it can be used by the public.
      - otherme123 3y ago
        This can't happen more often than not, because the client has no idea what a pentest is. They don't even know what a is XSS means, or even API. These things are negotiated by people that only can see the frontend, and if it looks great, _snappy_, _flashy_, with random animations and following current trends it's OK. The contractors know and can easily detect this, so they focus on frontend and don't waste their time in behind-the-scenes polish. You don't polish the security or find a costly query that could bring the site to their knees, but you add a scroll-spy that brings some images from nowhere.
Joeri 3y ago
As this is a government contract, and there are strict public transparency rules on government contracts, I went digging. Here's the call for tenders: https://etendering.ted.europa.eu/cft/cft-display.html?cftId=8749 https://etendering.ted.europa.eu/cft/cft-display.html?cftId=... And here's the award: https://ted.europa.eu/udl?uri=TED:NOTICE:120998-2022:TEXT:EN:HTML&tabId=1 https://ted.europa.eu/udl?uri=TED:NOTICE:120998-2022:TEXT:EN... Some interesting things: 1. This is a broad framework contract for marketing, for the eye-watering amount of 300 million euro. The title is "Belgium-Brussels: Framework Service Contract for the Organisation of Large-scale Travels of Participants in the Context of Erasmus+/DiscoverEU" The reason they do these kinds of framework contracts is because the legally required tendering procedures surrounding government contracts are so onerous that it's better to do a broad contract once and bundle a lot of projects inside of them, than to have one contract per project. 2. The executing party (Caracal) is nowhere to be seen, instead the contract is awarded to EURail and MCI. EURail is the intermediary I suspect, responsible for navigating the wild world of government contracting, and MCI is a marketing agency. They have no doubt built up years of expertise in how to successfully navigate these kinds of tendering procedures, and they probably are not the lowest bidder. Caracal is no doubt subcontracted by MCI, but as MCI is a private company we cannot see how much they were paid or how they were selected. So much for transparency. In my own experience in government contracting, price is a factor but usually not the largest factor. There's a large set of requirements (which you can read through if you follow the first link), and the ability to prove that you will be able to meet them is mostly what determines who wins the contract. However, because it is so difficult to know how to do that, only a few parties will have submitted a tender, and the best of a poor batch may still not be very good. Personally I think this kind of public procurement legislation is well intended but ultimately flawed. It does not result in lower costs, faster turnaround, better transparency, or overall better government. I'm in favor of transparency rules, but they need to be a lot more thorough and they need to cover subcontracting as well. I'm against public tendering legislation, as I think it prevents the government from being efficient. (By the way, how awful is that public tendering website? It's like a flashback to 2003. No doubt built under one of those big framework contracts.)
- germanier 3y ago
  The linked tender is mostly not about marketing. They tendered basically a pretty large travel agency. Eurail is surely not just a intermediary here. They are in the business of selling train passes and operate lots of the services tendered for decades (e.g. an online train booking platform)

by 2rsf 3y ago

pjc50 3y ago

This is where quite a lot of people would insert a rant about "state capacity": the ability of the state to actually do things it wants and intends to do. Which requires people to do those things, trained with appropriate skills. The peak of "state capacity" was undoubtedly WW2, when governments bypassed market mechanisms and became command economies. Out of necessity - war is the one venture in which failed state capacity can end the state itself, and the personal privileges of those running it and the elite around them. It's not a coincidence that the centralized socialist institutions of the UK, the NHS and state education, date from that period. Heck, the state commissioned the invention and building of cutting-edge computer technology! But since that no longer matters, there's little to no will to build state capacity in computing.

afavour 3y ago
It’s not WW2 mobilisation but I’ve always thought the UK’s Government Digital Service is a wonderful example of what government can achieve in tech: https://www.gov.uk/government/organisations/government-digital-service https://www.gov.uk/government/organisations/government-digit... As I understand it they’re effectively a central dev shop for other government agencies. It’s worth their time investing in good practises because they’re going to use them over and over again. And from the user perspective you get a very consistent, reliable set of tools for interacting with government. A win win in my book.
- franga2000 3y ago
  This is something we desperately need more of in other countries. We've found something like a dozen breaches of similar severity in the last 6 years and they all came from systems developed through public tenders by companies that either aggressively under-priced or used other (legal or illegal) dirty tactics to win them. The very few things that were developed in-house have proven to be far more reliable and secure, not to mention developing them was far cheaper and the UX is better and more consistent between them.
  - sfn42 3y ago
    As a software "consultant" (meaning i do what everyone else does but get paid more) the consultancy business is fucking bullshit. Just padding resumes and counting billable hours, nobody gives a fuck about quality or maintainability or anything. Just money.
- FourthProtocol 3y ago
  GDS did a great job building gov.uk, but everythig else they touched was an abject disaster. From the Diabetes project at the NHS, the fiasco that was the Office of the Public Guardian, the even bigger fiasco that happened at Border Force, the NHS, DWP... Sure, when they were building web sites they delivered stellar stuff. Agile, break things and all that. But when you had real complexity they just... couldn't... The Government Gateway is a prime example - single citizen login for ALL government services. It ran well, super robust and mature enough to have ironed out virtually all issues. Then GDS decided that because the Government Gateway was based on a Microsoft stack, it needed to be re-done. The tech lead didn't understand the concept of Identity Federation, let alone SAML tokens, and that you just! can't! do secure code using agile (2-week sprint no good for meaningful security testing...). I spent two long years at GDS banging my head against a wall. And then I left. And unsurprisingly the Microsoft-based Government gateway was never replaced, still going strong.
veltas 3y ago
It's a shame that (from my perspective anyway) a lot of that state capacity seems to have degassed for the NHS. Right now it's impossible to get a doctor's appointment where I live. If you call any local surgery within a split second of 8:00am then you have a very low chance of getting an appointment, every time I've tried the line's busy or I'm number 60 in the queue and after a 40 minute wait all the appointments are gone. They are also fully booked for advance appointments every day (they limit the time in advance you can book, probably to prevent having e.g. a 2 year wait time on appointments, because it 'looks better' if nobody can go than if there's literally a 2 year advance appointment wait). The only way to see a doctor is to go to A&E, or convince NHS 111 to give you an "emergency appointment" of some kind. All of this drains emergency resources and are not an option due to the time investment for average people with precarious employment. Unfortunately many people won't believe these facts, because depending on which area you live in there's always plentiful appointments, in advance or on the day! Where I last lived, getting an appointment was easy, the difference in outcomes based on how wealthy or urban your area is leaves a bad taste in my mouth. Briefly, we had a new surgery open that offered more appointments, at more convenient times of day, and we could actually see the doctor. All the other GP's started losing their patients to them. Then within a few months they were shut down by the local NHS trust, under multiple investigations (one of these investigations was regarding an offensive Facebook post by the surgery's chief, I kid you not). They then later reopened with normal appointment times and no free spaces, like all the other surgeries. I agree with what you say re the command economy of WW2 allowing the creation of the NHS. But it was not this country that founded the NHS: it is some ancient, lost nation that seems utterly alien to me today. I don't believe we could achieve even a small version of what WW2 Britain did anymore, if our survival depended on it. I cite the UK's response to COVID19 as evidence.
- Scarblac 3y ago
  Same as in the US -- right wing politicians who believe the state is bad at doing things remove the funding that allowed the state to do it. Then when that means those government provided services become worse, they use that to show that the state is bad at doing those things, and therefore even less funding should go there, et cetera.
- 3y ago
marcosdumay 3y ago
Governments do in-house development just as well as any company with a large development department. Governments also do development off-shoring just as well as any company with a large off-shored project. The kind of project people are talking about here never works. It doesn't matter who is doing it.

nness 3y ago

It's a fallacy to believe that all projects are just sold to the lowest bidder. There are probably a dozen reasons why something like this might have occurred, and not giving the vendors a free pass, but assuming that a more expensive vendor would do a better job with security and reviews is just as likely to be a mistaken belief. If a project is too expensive for a client to do well, they should not be doing that work in the first place.

boringg 3y ago
Its actually probably more of a situation that a client cant discern quality. Its impossible to tell if the most expensive or least expensive if the best option. How does a non technical/semi technical actually grade this stuff appropriately?
- davemp 3y ago
  This is a problem traditionally solved by the professional engineering licensing system. Most engineering curriculum in the USA involve an Engineering Ethics course that goes over such issues. We're quite far from implementing such a system for software "engineers".
  - comboy 3y ago
    Just hire pentesters along with the development team. Make sure they are not affiliates. You can put in contract that as long as security issues are present they need to fix them before getting paid, which seems like a reasonable expectation. Even the best make mistakes, so let's at least leave those which are not trivial. We don't trust building ethics, independent inspector comes and checks if everything is as it should be before it can be used by the public.
    - otherme123 3y ago
      This can't happen more often than not, because the client has no idea what a pentest is. They don't even know what a is XSS means, or even API. These things are negotiated by people that only can see the frontend, and if it looks great, _snappy_, _flashy_, with random animations and following current trends it's OK. The contractors know and can easily detect this, so they focus on frontend and don't waste their time in behind-the-scenes polish. You don't polish the security or find a costly query that could bring the site to their knees, but you add a scroll-spy that brings some images from nowhere.

Joeri 3y ago

As this is a government contract, and there are strict public transparency rules on government contracts, I went digging. Here's the call for tenders: https://etendering.ted.europa.eu/cft/cft-display.html?cftId=8749 https://etendering.ted.europa.eu/cft/cft-display.html?cftId=... And here's the award: https://ted.europa.eu/udl?uri=TED:NOTICE:120998-2022:TEXT:EN:HTML&tabId=1 https://ted.europa.eu/udl?uri=TED:NOTICE:120998-2022:TEXT:EN... Some interesting things: 1. This is a broad framework contract for marketing, for the eye-watering amount of 300 million euro. The title is "Belgium-Brussels: Framework Service Contract for the Organisation of Large-scale Travels of Participants in the Context of Erasmus+/DiscoverEU" The reason they do these kinds of framework contracts is because the legally required tendering procedures surrounding government contracts are so onerous that it's better to do a broad contract once and bundle a lot of projects inside of them, than to have one contract per project. 2. The executing party (Caracal) is nowhere to be seen, instead the contract is awarded to EURail and MCI. EURail is the intermediary I suspect, responsible for navigating the wild world of government contracting, and MCI is a marketing agency. They have no doubt built up years of expertise in how to successfully navigate these kinds of tendering procedures, and they probably are not the lowest bidder. Caracal is no doubt subcontracted by MCI, but as MCI is a private company we cannot see how much they were paid or how they were selected. So much for transparency. In my own experience in government contracting, price is a factor but usually not the largest factor. There's a large set of requirements (which you can read through if you follow the first link), and the ability to prove that you will be able to meet them is mostly what determines who wins the contract. However, because it is so difficult to know how to do that, only a few parties will have submitted a tender, and the best of a poor batch may still not be very good. Personally I think this kind of public procurement legislation is well intended but ultimately flawed. It does not result in lower costs, faster turnaround, better transparency, or overall better government. I'm in favor of transparency rules, but they need to be a lot more thorough and they need to cover subcontracting as well. I'm against public tendering legislation, as I think it prevents the government from being efficient. (By the way, how awful is that public tendering website? It's like a flashback to 2003. No doubt built under one of those big framework contracts.)

germanier 3y ago
The linked tender is mostly not about marketing. They tendered basically a pretty large travel agency. Eurail is surely not just a intermediary here. They are in the business of selling train passes and operate lots of the services tendered for decades (e.g. an online train booking platform)