7 ms·

In Germany the app was developed by research institutes. Then canceled. Then SAP picked it up and charged 10 million Euros for what is essentially a few 1000 li

by quonn 6y ago

In Germany the app was developed by research institutes. Then canceled. Then SAP picked it up and charged 10 million Euros for what is essentially a few 1000 lines of UI code. The German Telekom got the same amount, just „because“ and will additionally charge 2-3 million per month mostly for the hotline (with limited capacity). So the total cost will be 50-100 millions. Despite the fact that most technical work was already done by Apple and Google for free. I‘m not even talking about the fact that my dad got a new phone to run it - sold by T-Mobile (Telekom) of course. And hardly anyone complained.

johannes1234321 6y ago
Nope, it's not 1000 lines of UI code. It is a project with lots of pressure (timeline, politics, ...) and two years of maintenance. Including two apps (iOS and Android) multiple Backend systems and a concept, project management (coordination between government, testing laboratories, ...) looking at the team size (educated guess based on GitHub stats) the hourly rate is on the higher side, but it's still low for a government contract and in case it has effect (which is still not proven) peanuts compared to the financial cost of COVID.
- quonn 6y ago
  I said a few 1000 and I actually browsed the code on Github when it came out. There were just six weeks to develop it. And the concepts where developed before by public research institutes for a tiny fraction of that price (not included). (If you insist, here is a small calculation. Let‘s say we charge a whopping 500 Euros per hour per engineer. In other words an engineer that costs 1 million per year. And we have 6 weeks time which was all they had. And let’s take 20 people for that. And let‘s add 500.000 for some overhead. Then that‘s still just about 3 million. And I‘ve used insane numbers for a product that trivial.)
  - tlarkworthy 6y ago
    20M for the boost of Germany's worldwide status is a drop in the ocean. The money was well spent, Germany did not embarrass themselves by penny pinching. It needed to work, and it appears to. Compare this to the UK who is still flapping around.
    - quonn 6y ago
      It doesn‘t work great. I actually run it on my phone. It has silly bugs, despite the simplicity. Additionally, for that price you could have easily have 3 independent teams develop it and take the best one and still be cheaper and better. I hope someone will challenge this in court. I don‘t think they can get away without a call for bids which should have been done (across Europe, as the law requires).
      - johannes1234321 6y ago
        Yes it has bugs, but most bugs are from the system framework ... they could have handled those better, but the cause is often out of their control
        rkachowski 6y ago
        In what sense are most of the bugs from the "system framework"?
        johannes1234321 6y ago
        There is this contact tracing framework in iOS and Google Play Services and that causes a few errors. Some users get a message like "not possible in your region" and some get some random error codes about rate limits and such errors, which bubble up from those frameworks and can only be fixed there.
        quonn 6y ago
        Yeah. It‘s the framework which does pretty much all of the interesting technical work, if I may say so. It does the communication with other phones and determines if there was a contact and generates, stores and maintains the tokens and given the server data figures out matches and so on. It has a tiny, simple to use API on top. It was jointly developed by Apple and Google. For free.
        rkachowski 6y ago
  - johannes1234321 6y ago
    First mistake: you calculate 6 weeks of dev time ignoring that SAP got 11.5M including maintenance for 2 years. Certainly amount of work in two years time is lower than initial dev work, but not zero. Secondly you are only taking in "development work", but such a contract entails other things - certainly there is a penalty if they deliver too late - there is legal risk to cover, if they deliver crap in some legal sense they are liable - probably SAP had other contracts which they had to postpone - count in all the lawyers writing the contract :-D And yes, they certainly made a profit out of this and yes a proper biding procedure might be better, but given the timelines and the transparency I think it's ok.
    - quonn 6y ago
      Okay, and the additional 1.5 million should suffice to „maintain“ an app that will not need or get any new features. Let‘s assume another million for lawyers „writing the contract“. It‘s still far off. Additionally let‘s remember the 20 million is just for the development. It does not include running the thing. That‘s what the other 50+ million are for. I don‘t agree. I run a business myself. I charge plenty. I know how much DAX size companies charge as well. The numbers don‘t add up, and this is especially true for something that could have been done pro bono by SAP in the current situation where pretty much everyone tries to do whatever they can.
      - johannes1234321 6y ago
        Operations for two years are actually not 20 but 40M. But this includes two phone hotlines, thus actual labor cost. (And by then the development costs are close to irrelevant already ...)
      - marvion 6y ago
        The German Spiegel listed the 20M cost somewhat differently: - 9.5M development (SAP) - 7.8M setup/operations (Telekom) - 2M Support(SAP) + Tax So, in the books, development is 10M, not 20M (german source, un-paywalled: https://outline.com/UYqb5b https://outline.com/UYqb5b)
      - quonn 6y ago
- quonn 6y ago
  Peanuts compared to the cost of COVID? How is that a valid measure?
  - number6 6y ago
    I think a lot of people could have made a good working app for a lot less. But on the other side: if SAP and Telekom had failed no one would have blamed the government. And if the both had failed their reputation would have suffered a lot. Well it's not the best as it is, so well. They have a top team and delivered on time. Thats rare in software development. And 20 million are nothing compared to the tax break which costs about 3 trillion. For the government it's money well spend in two german flagship companies. My guess most of it was insurance money in case something went wrong, so these companies would have taken the fall.
    - deleted 6y ago
      [deleted]
  - rsynnott 6y ago
    If one of these apps leads to avoidance of a handful of serious cases (by identifying clusters earlier and thus reducing spread) it has paid for itself. An ICU bed costs _at least_ 2000 euro a day (that's for a normal one, ongoing operational costs only; rush-job new ICU capacity is presumably more expensive); it doesn't take too many averted hospitalisations to pay off one of these apps. Obviously, it would have been better if the German app has been cheaper. But rush jobs are expensive, and the cost involved is pretty minimal relative to the cost of managing the pandemic writ large. If it's even slightly effective, it was worth it.
- somethoughts 6y ago
  Reminds me of the following article discussed on HN previously [1]: I could do that in a weekend! http://danluu.com/sounds-easy/ http://danluu.com/sounds-easy/ [1] https://news.ycombinator.com/item?id=12626314 https://news.ycombinator.com/item?id=12626314
  - quonn 6y ago
    The article was about companies actually shipping complex products over a long time. All of that is fully justified. Not someone taking over a fairly small project, finishing it in 6 weeks and then charging 20 millions for the job.
tannhaeuser 6y ago
Hey developers of German Corona Warn-App. Just wanted to say even I have installed it, not because I'm convinced it'll help too much (the verdict is still open I guess) but because it's a good project result given the constraints, and there's no doubt an effort was made to deliver the best possible privacy, so I didn't want to be the one boycotting it or something.
denzil_correa 6y ago
> In Germany the app was developed by research institutes. Then canceled. Why was it canceled?
- quonn 6y ago
  Because it was a mess. A privacy mess. Also because the fact that supporting iOS well requires using the Apple framework was ignored. And also lot‘s of politics.
- skwb 6y ago
  I cannot speak for Germany, but I have a bit of experience in how public health projects are funded in the US. Usually, the set of incentives are misaligned for maintaining products. Research institutes are typically funded through contract grants, where you as a principle investigator are supposed to be using the money for a research project. Obviously there are exceptions, but if you want to make a large name for yourself in academia, doing maintenance projects isn't the way to get yourself promoted; all the flashy rewards are in publishing novel cutting edge ideas in high ranking journal. I am not an expert on how things are in Germany, but if they're anything similar to the US experience, I'm not surprised at all that they would fail due to the misaligned incentives.
skrause 6y ago
The total cost of the app (over a 2-year timeline) is about the same as 1-2 days of Covid testing in Germany. If it helps to reduce the spread of the pandemic it will be absolutely worth it.
- quonn 6y ago
  The cost of the app over 2 years is 50-100 million as you can check. That‘s like the cost of 1-2 weeks of testing. But it‘s silly to argue that just because something necessary costs a lot we can disregard all other costs. For 50 million you can build a school. Sure, Germany can afford it. But the cost is ridiculously high.
denzil_correa 6y ago
In case, someone asks a source for this number. > The German government says its app cost 20 million euros ($22.7 million) to develop and will require 2.5 million to 3.5 million euros per month to operate. It’s available in German and English, with Turkish and other languages to follow. https://apnews.com/fd6cc8a04ae77185110cf8d70cf585a1 https://apnews.com/fd6cc8a04ae77185110cf8d70cf585a1
dividuum 6y ago
Not sure where you get the 1000 lines from. That's ridiculous. `cloc` counts 16k lines of Kotlin for the Android app and 22k lines of Swift for the iOS app. That does not even include other aspects like the server side code or support documentation. Also it's not only the app that had to be build: It required all the notification infrastructure and coordination to deliver test result to the app. This is really one of the very few big government projects where I can't really complain about the costs. There was only one attempt at getting this done right and in time. They managed both.
- quonn 6y ago
  Plural „a few 1000“ When I checked the actual Swift code was just a bit less than 10k. The backend apps are fairly small Spring boot apps. There was a messy attempt before (not finished in time at all) and they fixed and finished that attempt.
- kugelblitz 6y ago
  Yep, am positively surprised how the product turned out. Plus the launch also went well I think, including a comic manual and even an AMA on reddit; I didn't read of any glitches or anything bad either.
  - quonn 6y ago
    I can‘t help but wonder if anyone has actually read the specs published by Apple and Google. If anyone has actually shipped real iOS and Android apps. I have followed this project closely. Checked the protocols. The APIs. And what‘s built on top. The only complex thing I can think of is the way cases are confirmed. That‘s a bit involved. But it was worked out in advance.
- tpush 6y ago
  If only the app were available on the US App Store. Can't use it here in Germany because of my US Apple ID.
  - dividuum 6y ago
tmpz22 6y ago
Don't bother, HN users LOVE expensive government IT projects because it artificially inflates their own self worth. Remember those 20 lines of python you wrote before lunch was really hard and worth millions. You are a ROCKSTAR!
mercer 6y ago
I think what's worse is that I suspect the whole project won't be all that useful, and a drop in the bucket compared to society-wide (enforced) change in behavior. From everything I've read, this kind of app 1) is only really useful if enough people actively use it, and 2) have serious issues with false positives because of inherent positional inaccuracy (or inability to be accurate without sucking battery). In regards to 1, in my country at least, unless using the app would be mandatory, I know many more people who will refuse to use it or be too lazy to use it. And I'm pretty sure hell will freeze over before the government can make usage (and owning a smartphone) mandatory.
- rsynnott 6y ago
  So, false positives may not matter in some conditions. Ireland is currently reporting anything from 5 to 30 cases per day, doing anything from 3000 to 10000 tests per day, and has capacity for at least 15000 tests per day. If the app doubles the number of tests being done and finds a few new cases, that's fine; it's still a lot better than NOT finding those cases. These apps would, granted, be far less useful in places with major ongoing outbreaks and inadequate testing infrastructure; lockdowns should be used to bring numbers down to the level where this sort of thing is workable. (1) is probably a greater problem. Ireland has about 30% uptake, which probably isn't enough.
  - mercer 6y ago
    Yeah, after I wrote my comment I figured that perhaps I overstated how useless the app would be. I imagine even with relatively low uptake it'll have /some/ effect. Even a little would help, I suppose.
kanox 6y ago
> 50-100 millions Relative to other COVID-related economic damage this is peanuts. It's acceptable to throw a few tens of millions at a wall and hope that something sticks.
Tainnor 6y ago
You can follow this request for comment: https://fragdenstaat.de/anfrage/zusammensetzung-der-kosten-fur-die-entwicklung-der-corona-warn-app/ https://fragdenstaat.de/anfrage/zusammensetzung-der-kosten-f... Legally I believe the government is required to answer that, even though it already seems they're trying to weasel themselves out of it.

by quonn 6y ago

johannes1234321 6y ago

Nope, it's not 1000 lines of UI code. It is a project with lots of pressure (timeline, politics, ...) and two years of maintenance. Including two apps (iOS and Android) multiple Backend systems and a concept, project management (coordination between government, testing laboratories, ...) looking at the team size (educated guess based on GitHub stats) the hourly rate is on the higher side, but it's still low for a government contract and in case it has effect (which is still not proven) peanuts compared to the financial cost of COVID.

quonn 6y ago
I said a few 1000 and I actually browsed the code on Github when it came out. There were just six weeks to develop it. And the concepts where developed before by public research institutes for a tiny fraction of that price (not included). (If you insist, here is a small calculation. Let‘s say we charge a whopping 500 Euros per hour per engineer. In other words an engineer that costs 1 million per year. And we have 6 weeks time which was all they had. And let’s take 20 people for that. And let‘s add 500.000 for some overhead. Then that‘s still just about 3 million. And I‘ve used insane numbers for a product that trivial.)
- tlarkworthy 6y ago
  20M for the boost of Germany's worldwide status is a drop in the ocean. The money was well spent, Germany did not embarrass themselves by penny pinching. It needed to work, and it appears to. Compare this to the UK who is still flapping around.
  - quonn 6y ago
    It doesn‘t work great. I actually run it on my phone. It has silly bugs, despite the simplicity. Additionally, for that price you could have easily have 3 independent teams develop it and take the best one and still be cheaper and better. I hope someone will challenge this in court. I don‘t think they can get away without a call for bids which should have been done (across Europe, as the law requires).
    - johannes1234321 6y ago
      Yes it has bugs, but most bugs are from the system framework ... they could have handled those better, but the cause is often out of their control
      - rkachowski 6y ago
        In what sense are most of the bugs from the "system framework"?
        johannes1234321 6y ago
        There is this contact tracing framework in iOS and Google Play Services and that causes a few errors. Some users get a message like "not possible in your region" and some get some random error codes about rate limits and such errors, which bubble up from those frameworks and can only be fixed there.
        quonn 6y ago
        Yeah. It‘s the framework which does pretty much all of the interesting technical work, if I may say so. It does the communication with other phones and determines if there was a contact and generates, stores and maintains the tokens and given the server data figures out matches and so on. It has a tiny, simple to use API on top. It was jointly developed by Apple and Google. For free.
        rkachowski 6y ago
- johannes1234321 6y ago
  First mistake: you calculate 6 weeks of dev time ignoring that SAP got 11.5M including maintenance for 2 years. Certainly amount of work in two years time is lower than initial dev work, but not zero. Secondly you are only taking in "development work", but such a contract entails other things - certainly there is a penalty if they deliver too late - there is legal risk to cover, if they deliver crap in some legal sense they are liable - probably SAP had other contracts which they had to postpone - count in all the lawyers writing the contract :-D And yes, they certainly made a profit out of this and yes a proper biding procedure might be better, but given the timelines and the transparency I think it's ok.
  - quonn 6y ago
    Okay, and the additional 1.5 million should suffice to „maintain“ an app that will not need or get any new features. Let‘s assume another million for lawyers „writing the contract“. It‘s still far off. Additionally let‘s remember the 20 million is just for the development. It does not include running the thing. That‘s what the other 50+ million are for. I don‘t agree. I run a business myself. I charge plenty. I know how much DAX size companies charge as well. The numbers don‘t add up, and this is especially true for something that could have been done pro bono by SAP in the current situation where pretty much everyone tries to do whatever they can.
    - johannes1234321 6y ago
      Operations for two years are actually not 20 but 40M. But this includes two phone hotlines, thus actual labor cost. (And by then the development costs are close to irrelevant already ...)
    - marvion 6y ago
      The German Spiegel listed the 20M cost somewhat differently: - 9.5M development (SAP) - 7.8M setup/operations (Telekom) - 2M Support(SAP) + Tax So, in the books, development is 10M, not 20M (german source, un-paywalled: https://outline.com/UYqb5b https://outline.com/UYqb5b)
    - quonn 6y ago
quonn 6y ago
Peanuts compared to the cost of COVID? How is that a valid measure?
- number6 6y ago
  I think a lot of people could have made a good working app for a lot less. But on the other side: if SAP and Telekom had failed no one would have blamed the government. And if the both had failed their reputation would have suffered a lot. Well it's not the best as it is, so well. They have a top team and delivered on time. Thats rare in software development. And 20 million are nothing compared to the tax break which costs about 3 trillion. For the government it's money well spend in two german flagship companies. My guess most of it was insurance money in case something went wrong, so these companies would have taken the fall.
  - deleted 6y ago
    [deleted]
- rsynnott 6y ago
  If one of these apps leads to avoidance of a handful of serious cases (by identifying clusters earlier and thus reducing spread) it has paid for itself. An ICU bed costs _at least_ 2000 euro a day (that's for a normal one, ongoing operational costs only; rush-job new ICU capacity is presumably more expensive); it doesn't take too many averted hospitalisations to pay off one of these apps. Obviously, it would have been better if the German app has been cheaper. But rush jobs are expensive, and the cost involved is pretty minimal relative to the cost of managing the pandemic writ large. If it's even slightly effective, it was worth it.
somethoughts 6y ago
Reminds me of the following article discussed on HN previously [1]: I could do that in a weekend! http://danluu.com/sounds-easy/ http://danluu.com/sounds-easy/ [1] https://news.ycombinator.com/item?id=12626314 https://news.ycombinator.com/item?id=12626314
- quonn 6y ago
  The article was about companies actually shipping complex products over a long time. All of that is fully justified. Not someone taking over a fairly small project, finishing it in 6 weeks and then charging 20 millions for the job.

tannhaeuser 6y ago

Hey developers of German Corona Warn-App. Just wanted to say even I have installed it, not because I'm convinced it'll help too much (the verdict is still open I guess) but because it's a good project result given the constraints, and there's no doubt an effort was made to deliver the best possible privacy, so I didn't want to be the one boycotting it or something.

denzil_correa 6y ago

> In Germany the app was developed by research institutes. Then canceled. Why was it canceled?

quonn 6y ago
Because it was a mess. A privacy mess. Also because the fact that supporting iOS well requires using the Apple framework was ignored. And also lot‘s of politics.
skwb 6y ago
I cannot speak for Germany, but I have a bit of experience in how public health projects are funded in the US. Usually, the set of incentives are misaligned for maintaining products. Research institutes are typically funded through contract grants, where you as a principle investigator are supposed to be using the money for a research project. Obviously there are exceptions, but if you want to make a large name for yourself in academia, doing maintenance projects isn't the way to get yourself promoted; all the flashy rewards are in publishing novel cutting edge ideas in high ranking journal. I am not an expert on how things are in Germany, but if they're anything similar to the US experience, I'm not surprised at all that they would fail due to the misaligned incentives.

skrause 6y ago

The total cost of the app (over a 2-year timeline) is about the same as 1-2 days of Covid testing in Germany. If it helps to reduce the spread of the pandemic it will be absolutely worth it.

quonn 6y ago
The cost of the app over 2 years is 50-100 million as you can check. That‘s like the cost of 1-2 weeks of testing. But it‘s silly to argue that just because something necessary costs a lot we can disregard all other costs. For 50 million you can build a school. Sure, Germany can afford it. But the cost is ridiculously high.

denzil_correa 6y ago

In case, someone asks a source for this number. > The German government says its app cost 20 million euros ($22.7 million) to develop and will require 2.5 million to 3.5 million euros per month to operate. It’s available in German and English, with Turkish and other languages to follow. https://apnews.com/fd6cc8a04ae77185110cf8d70cf585a1 https://apnews.com/fd6cc8a04ae77185110cf8d70cf585a1

dividuum 6y ago

Not sure where you get the 1000 lines from. That's ridiculous. `cloc` counts 16k lines of Kotlin for the Android app and 22k lines of Swift for the iOS app. That does not even include other aspects like the server side code or support documentation. Also it's not only the app that had to be build: It required all the notification infrastructure and coordination to deliver test result to the app. This is really one of the very few big government projects where I can't really complain about the costs. There was only one attempt at getting this done right and in time. They managed both.

quonn 6y ago
Plural „a few 1000“ When I checked the actual Swift code was just a bit less than 10k. The backend apps are fairly small Spring boot apps. There was a messy attempt before (not finished in time at all) and they fixed and finished that attempt.
kugelblitz 6y ago
Yep, am positively surprised how the product turned out. Plus the launch also went well I think, including a comic manual and even an AMA on reddit; I didn't read of any glitches or anything bad either.
- quonn 6y ago
  I can‘t help but wonder if anyone has actually read the specs published by Apple and Google. If anyone has actually shipped real iOS and Android apps. I have followed this project closely. Checked the protocols. The APIs. And what‘s built on top. The only complex thing I can think of is the way cases are confirmed. That‘s a bit involved. But it was worked out in advance.
tpush 6y ago
If only the app were available on the US App Store. Can't use it here in Germany because of my US Apple ID.
- dividuum 6y ago

tmpz22 6y ago

Don't bother, HN users LOVE expensive government IT projects because it artificially inflates their own self worth. Remember those 20 lines of python you wrote before lunch was really hard and worth millions. You are a ROCKSTAR!

mercer 6y ago

I think what's worse is that I suspect the whole project won't be all that useful, and a drop in the bucket compared to society-wide (enforced) change in behavior. From everything I've read, this kind of app 1) is only really useful if enough people actively use it, and 2) have serious issues with false positives because of inherent positional inaccuracy (or inability to be accurate without sucking battery). In regards to 1, in my country at least, unless using the app would be mandatory, I know many more people who will refuse to use it or be too lazy to use it. And I'm pretty sure hell will freeze over before the government can make usage (and owning a smartphone) mandatory.

rsynnott 6y ago
So, false positives may not matter in some conditions. Ireland is currently reporting anything from 5 to 30 cases per day, doing anything from 3000 to 10000 tests per day, and has capacity for at least 15000 tests per day. If the app doubles the number of tests being done and finds a few new cases, that's fine; it's still a lot better than NOT finding those cases. These apps would, granted, be far less useful in places with major ongoing outbreaks and inadequate testing infrastructure; lockdowns should be used to bring numbers down to the level where this sort of thing is workable. (1) is probably a greater problem. Ireland has about 30% uptake, which probably isn't enough.
- mercer 6y ago
  Yeah, after I wrote my comment I figured that perhaps I overstated how useless the app would be. I imagine even with relatively low uptake it'll have /some/ effect. Even a little would help, I suppose.

kanox 6y ago

> 50-100 millions Relative to other COVID-related economic damage this is peanuts. It's acceptable to throw a few tens of millions at a wall and hope that something sticks.

Tainnor 6y ago

You can follow this request for comment: https://fragdenstaat.de/anfrage/zusammensetzung-der-kosten-fur-die-entwicklung-der-corona-warn-app/ https://fragdenstaat.de/anfrage/zusammensetzung-der-kosten-f... Legally I believe the government is required to answer that, even though it already seems they're trying to weasel themselves out of it.