15 ms·
The BBC's hi-tech failure: Don't Mention It
- _pmf_ 13y agoI vaguely remember reading uplifting, grandiose articles and presentations about domain driven design at the BBC; search for "BBC domain driven design". From this article, I cannot deduce whether this failed CMS is the same system that has been glorified in these presentations or something else; maybe someone can shed some light on this.
- rplacd 13y agoA side matter, but handing the BBC's governance over to Ofcom? They haven't held the Independent Television programme contractors to their remits since 2001 under the threat of any sanction - or, for that matter, put their contracts up to tender (not that it worked or ever will work if we want a One Nation Tory-style public broadcasting system again - and not that I remain suspicious of their inconsistent singling out of Thames Television) since 1993. The BBC Trust, at the very least, has an image to mantain.
- arethuza 13y agoI did a quick check to see how Wikipedia defined a "One Nation Tory" - after all they have been pretty thin on the ground this last 30 years (especially here in Scotland!). I'm somewhat amused to see that "One nation conservatism" appears to now be the policy of the Labour Party: http://en.wikipedia.org/wiki/One-nation_conservatism http://en.wikipedia.org/wiki/One-nation_conservatism
- JamisonM 13y agoReal headline: Publicly funded broadcaster makes same mistake most large private enterprises make in more or less the same way. It seems to me that months or years of auditing and hearings and cries for heads to roll are probably wasted. Just like a private company a few mangers should probably get tossed or reassigned and the second version of the same project should get started to address the businesses needs the first one was supposed to.
- macspoofing 13y ago>Publicly funded broadcaster makes same mistake most large private enterprises make in more or less the same way. Actually large private enterprises don't usually make this kind of mistake. This is a typical big government project. Software development is hard. Building large, complex, poorly defined software systems is hard. //EDIT I should clarify that. I don't think a private enterprise would engage in a project that would have it build an internal system, at that scale ($100 million), having so little impact on it's primary revenue driver, over so many years! Big enterprises fail with big software, that's a fact. But they wouldn't fail in this specific way. This is a quintessential big government software project failure.
- Retric 13y agoThere are a lot of numbers tossed around without a lot of hard data backing them up, but it's generally accepted that over half of all large software projects 'fail' at both private companies and government institutions.
- jerf 13y ago"Actually large private enterprises don't usually make this kind of mistake." Really? You sure about that? Because hard numbers are hard to come by, but I'm pretty sure they do with some frequency. It's notoriously common for large consultancies not working for the government to have the same outcome too. It's possible private enterprise makes this mistake at a different rate than the government, but I'd despair of trying to prove that claim, or even give it a solid definition. In general large software initiatives seem to fail, regardless of whose bright idea they are.
- JonnieCache 13y agoNote that the content in this CMS is hours and hours of raw video footage rather than the usual snippets of text.
- true_religion 13y agoDuely noted, but I don't think it ought to make a difference in the price tag unless the CMS was doing conversion and distribution.
- untog 13y ago...which it was. One of the big features was the ability to edit video in a lower bitrate (i.e. faster to work with) then send it off to a server to process on the full HD video. Great idea, but apparently it was a total mess to work with.
- archivator 13y agoIIRC, their CMS was supposed to be the Holy Grail of video management. Centralized and decentralized storage, processing farms, search-by-image, advanced tagging, etc. Sounded quite impressive but from what I understand, there are now commercial solutions that have some of those feature sets. The way they explain it, when they started no one had these features.
- rurounijones 13y agohours? This is the BBC. It is probably months, possibly years worth of material.
- Suitov 13y agoVery definitely years. It wasn't just for finished programmes (i.e. ~ number-of-channels*24 hours of footage/day): it was for all rushes. All the footage that was filmed in the course of making the final programme. We're talking stupid quantities of data here and I've never seen a detailed answer to how it was all supposed to be stored.
- amalag 13y agoIn the US I think noone even blinks an eye at $150 million anymore
- dopamean 13y agoIt's funny you mention that. My first thought was, "wow, that's a lot of money." But then I thought, "but how much is it really?" I actually have no idea how much it should cost for them to do what they were trying to do though I suspect it is probably a fraction of that $150MM.
- mikeash 13y agoFor a more concrete perspective, if you assume that a good developer costs about $150,000/year, then this is one hundred years of development time for a team of 10 good developers, which is quite a lot. I don't know just what this thing was supposed to do, so I don't know what the scope is.
- realtalker 13y agono one working for the BBC in the UK would make that sort of money as a developer
- matthewmacleod 13y agoI don't know about that. I've seen a few tech jobs around the £70-80k mark at the BBC, which is about $120k. Throw in the additional benefits and it's probably not far off.
- Aqueous 13y agoArticle doesn't actually tell us what it is, beyond a traditional CMS.
- taylorbuley 13y agoMore important, what is in it. A large difficulty with content management systems are the data types; pairing down or normalizing metadata while trying to map old to new.
- kailuowang 13y ago$150 million roughly equals to 1000 developer years.
- recpixe 13y agoMore like 100, because 10% of that would have been spent on devs, and the rest on hiring meeting rooms, biscuits, project managers, external agencies, etc etc etc. trust me, i'm at the bbc...
- justinph 13y agoFrom what I understand (and I work in the public media sphere), this was a monster CMS project. A way to integrate audio, video, web content, reporters notes, etc. Is it any wonder a project like that failed?
- nslocum 13y agoAs a big fan of the Economist, I'm surprised they didn't catch the typo in the name of the project they're reporting on. "Digitial Media Initiative (DMI)" Especially since a rudimentary spell-checker would have caught it.
- shitgoose 13y agoI guess Economist's CMS is not as sophisticated, as one at BBC
- squidi 13y agoPrevious discussion, including info from someone close to the project, here: https://news.ycombinator.com/item?id=5762116 https://news.ycombinator.com/item?id=5762116
- DrinkWater 13y agoI had the chance to see some of the work Adobe is planning to do with their CMS (some actual demos from the "Adobe Labs"), and i cant deny the resemblance to BBCs case. I am wondering whether Adobe was involved...
- astrodust 13y agoThe same organization that couldn't be arsed to host some of their historical web properties due to "high costs": http://853blog.com/2011/01/25/pulling-the-plug-on-the-bbcs-internet-history/ http://853blog.com/2011/01/25/pulling-the-plug-on-the-bbcs-i...
- MrDOS 13y agoIt strikes me that the BBC is hilariously bad at archival. They scrapped dozens of Doctor Who episodes in the name of saving space (http://en.wikipedia.org/wiki/Doctor_Who_missing_episodes http://en.wikipedia.org/wiki/Doctor_Who_missing_episodes).
- objclxt 13y agoThe BBC is, today, very good at archiving programming material. All output is kept for at least five years, and the overwhelming majority is kept permanently (exceptions are things like long-running quiz shows, where samples of material are kept post five years, although any disposed material is offered to the BFI and other third parties before being junked). Doctor Who wasn't scrapped in the name of saving space, it was scrapped because nobody thought it was worth keeping. The shows couldn't be repeated (because at the time, Equity had negotiated limits on repeat rights), and quad tape was very expensive and easily re-used. More importantly, nobody really thought that Quad was an appropriate archiving format. Programs would be tele-recorded onto 16mm for overseas sale and archive. This is not, in of itself, a crazy idea - try finding a quad tape player today vs a 16mm projector. The problem at the time was there was no mandate for archival, so the engineering department were junking the tapes, and the commercial department (who were selling Doctor Who on to other broadcasters) weren't retaining the 16mm reels. Even in the 1970s, many people were still not convinced of the value of archiving TV output. It's not totally dissimilar to the early days of the web - it was seen as ephemeral. This was happening at pretty much every broadcaster around the world - Wikipedia has a pretty big list of lost TV programs, most of which were wiped for space.
- KaiserPro 13y agoIn the name of saving cost. Tapes were (and still are) really expensive. at the time the BBC were really strapped for cash (as ITV were not long after.)
- dmragone 13y agoI'm surprised more public projects don't run as bounty-style "build x and get $1 million" type systems. Rather than picking who's going to do what (or worse, building all in-house), having many different small times competing for one pot seems like it might be a better way to get a good starting foundation. Or perhaps that's a crazy idea for something as big as this.
- jiggy2011 13y agoDo you think any decent consultancy is going to be willing to invest millions of £ into building a system where they will most likely be able to recover none of it? OTOH , more uk government code is being released as open source. So in theory any developer could study the source and offer to fix bugs/add features and give a more accurate estimate. This might increase competition.
- k2enemy 13y agoIt might not lead to savings though. Suppose that the normal contract price is $1,000,000. If you have a bunch of small firms try to build the system, each on only gets the reward with probability p = Prob(they succeed and they get picked out of all the successful firms). A firm would only participate if the reward were greater than $1,000,000/p.
- jasondemeuse 13y agoBecause the best aren't going to put good work into a project that will probably just get thrown away. If you entered a bounty project with two hundred other entries (all of which have systems similar to yours), would you put hundreds of hours and all of your effort into the project in spite of the fact that there's only a 0.5% chance you'll get paid? Not if you want to stay in business. The size of the project is barely even relevant, take a look at bounty programs that are up now for areas like design (99designs, etc) and how they are chastised for pumping out bad products from designers who are only putting a fraction of effort into the projects. This type of system sounds great for those looking for cheap labor, but in practice it falls apart.
- Finster 13y agoThat's spec work.
- tehwalrus 13y agoNowhere near as much as the £12Bn the NHS spent on an impossible moving-spec IT system that they also scrapped(1). Oh wait, it's a tenth of the amount. (1) http://en.wikipedia.org/wiki/NHS_Connecting_for_Health http://en.wikipedia.org/wiki/NHS_Connecting_for_Health
- tubs 13y agoA tenth of 12B would be 1.2B. This is a hundredth.
- tehwalrus 13y agoApologies, physicists are often off by an order of magnitude ;)
- lsd5you 13y agoThis is widely reported size of the waste, but as I understand it, because the program was cancelled 'only' £2bn was wasted.
- rch 13y agoThe US has contributed some nice failures as well. For example, caBIG ($300M+): https://www.google.com/search?q=cabig+failure https://www.google.com/search?q=cabig+failure
- thisone 13y agoSaying the program was scrapped is a bit off. It makes it sound like nothing has come out of the project at all. Choose and Book is widely in use, the Spine exists, as well as the N3 network. Data format and transfer standards are coming into play now. So instead of one system to rule them all, it's working towards interop. Which probably should have been the overall goal to begin with. People like to think of the NHS as a monolith, but it's really hundreds of tiny fiefdoms.
- hahainternet 13y agoTiny fiefdoms all on 172.16 They could easily have picked ipv6 and done it the smart way. Oh how things could have been.
- jamespo 13y agoWhen I was at BBC Online ~10 years back they wasted a couple of million on a content management system through a "leading" consultancy that never made it to production.
- Finster 13y agoI've personally worked at 3 large organizations that have tried to formulate and develop an all-encompassing content management system. One of those organizations is a broadcaster that was also trying to build a digital content management system. Every single one of those projects has been a total boondoggle. Always over budget. Always changing scope. Always missing every single milestone and deadline. Now, I know that this is not a statistically significant sample size, but this sort of thing seems to be more common that I initially thought. It's like the worst-case scenario of design-by-committee.
- podperson 13y agoThe only reason failed content management system isn't an oxymoron is that content management system is three words.
- caycep 13y agoSounds like a typical hospital EMR/EPIC/Cerner/Meditech deployment!
- handelaar 13y agoWas wondering if anybody here would notice that the "CMS" being described here was a project to handle every single second of radio and TV footage from ingestion through recoding through editing to broadcast, and then beyond that to transcoding everything for whatever devices want to watch or hear it once it's finished. People here hear "CMS" and think we're talking about text. Incidentally the parts of this boondoggle which actually exist are being used as components in the BBC. The headline last week was about the decision to terminate that particular project because it had got far, far out of its depth. Also to put the highly-charming person responsible for it on gardening leave indefinitely pending the investigation. It's a massive cockup, but it has had some useable outcomes, and it is a product class that doesn't exist outside the world of the Beeb, or other national/international broadcasters. There's nothing off the shelf to buy here. In the 90s this sort of thing wouldn't have got cancelled, so I'm seeing this as a sort of progress.
- ttrreeww 13y agoNow they didn't. They "transfered" $152 million of public money into private hands. It's a success. Plus, they get to do it again.
- clientbiller 13y agoThey were trying to build netflix?
- bifrost 13y agoI'd say "well this is what happens when government runs something technical" but I also watched similar stuff happen at MSN when I was there. Not to say that the Beeb is like MSN, but I do know that there were a lot of dim bulbs to go around...
- pero 13y agoRelated further reading for trainwreck geeks: EDS (now HP) infamously botched a 1-year ~50million-turned-4 year-multi-hundred million dollar CRM project for BskyB a decade ago, and eventually lost a years-long lawsuit. http://www.linklaters.com/Publications/Publication1403Newsletter/20100317/Pages/BSkyBvEDSTimetoReassess.aspx http://www.linklaters.com/Publications/Publication1403Newsle...