9 ms·
Europe’s biggest city council faces £100M bill in Oracle ERP project disaster
- MayeulC 3y agoIs that an example of the sunk cost fallacy, or corruption/lobbying?
- rightbyte 3y agoMy take is that it boils down to insane requirements from the buyer and yes-men sellers. The technical problem is wanting one end-all goto system instead of like 20 different utilities I guess.
- magicalhippo 3y agoI haven't been on projects of this size, but with many of our customers a big issue is that the decisions are taken by people sufficiently high up in the organization that they don't know what's really going on, and without including key resources from lower levels to provide that insight. In my experience it only takes one or two levels to get to that point. Regarding the insane requirements, it's also my experience that a lot of companies think they're snowflakes, thinking they do something really special, when they're really just doing what everyone else is. For historical reasons they ended up doing it this way instead of that way and now they'd rather have the world change around them than change their ways.
- mschuster91 3y ago> Regarding the insane requirements, it's also my experience that a lot of companies think they're snowflakes At least with anything government, there is another problem: changing processes often requires a lot of work, which means explaining to (completely clueless) politicians why something is necessary, get cost approval for each change, convince majorities... it's easier to have an omnibus "migrate to newest version of XYZ" package and to spend the money on making the system operate according to your whims than to fix the underlying bullshit.
- cmh89 3y ago> haven't been on projects of this size, but with many of our customers a big issue is that the decisions are taken by people sufficiently high up in the organization that they don't know what's really going on, and without including key resources from lower levels to provide that insight. This is my experience working in government IT. The people whose job it is to go to meetings and talk end up being the ones representing the businesses interest and the vendor ends up running circles around them because they aren't in any way qualified to talk about something technical.
- mistrial9 3y agoseveral valid viewpoints in this thread, describing portions of a difficult-to-describe situation. The word "qualified" jumps out for me here.. It could mean things like "sufficient technical background" and "of sound mind and problem solving skills" .. but in the govt context, "qualified" means exactly "you have legal permission" .. this is one of the several places that these projects break down, exactly. Those in business and those delivering permits, over time, learn that permits can be used to make money, and control future money, not work. The permit process for qualification also works at the bidding stage, where new entries are required to have their high school gradepoint checked, and hundred page report(s) to show "context" for the bid, yet existing payment contracts continue on in zombie mode for huge values, on a regular basis. This is like the force of gravity for these deals.. both sides, govt and big business, want lots of cash flow and no end in sight.
- notdang 3y agoThey changed from SAP ERP to Oracle ERP. There were no 20 different utilities to begin with.
- red-iron-pine 3y ago> The technical problem is wanting one end-all goto system instead of like 20 different utilities I guess. At some point these systems have to share info, dump info somewhere (data lake, backups, whatever), and output info. Inevitably there will be one or two systems that connect them. Plus the overhead of dealing with 20+ different vendors, systems, updates, vulnerabilities, interconnections, etc. is a nightmare. There is a point where a "single source of truth" is too much effort and too expensive, but a clusterufck of different systems is not a great solution.
- daenney 3y ago¿Por qué no los dos?
- peapicker 3y agoImplementation partner was EY (Ernst & Young). Taking a heavily customized SAP into anything else is going to be painful, and the wrong partner for implementation can really cost.
- namdnay 3y agoIn my experience, with these types of project, 20% of the error is on useless partners, 80% on useless clients, who have no knowledge of how their current system works, are not willing to compromise on any existing features (because it's always easier to say "no" than to try and convince others to say "yes"), and don't really care about the migration one way or the other, because they weren't really asked in the first place If Apple had been in the B2B enterprise space, the iphone would be an exact clone of the blackberry
- rqtwteye 3y ago“ 80% on useless clients, who have no knowledge of how their current system works,” I tend to think that’s true. I have been involved in some process improvements at my company and it was almost always that: * Nobody really knows how things are working now * A lot of people don’t want to adapt to new processes * Execs give you a lot of verbal commitment but don’t do anything to really help It’s really hard to implement a good software system within these constraints.
- sonotathrowaway 3y agoI wasn’t aware of how true this is until I got to see larger companies up close. People are incredibly siloed and very incentivized to make the issue someone else’s problem - if you have a vendor you can shift the problem onto, even better.
- phkahler 3y agoDoes ERP really need customisation? That seems to be a big cost driver as well as causing lock-in.
- namdnay 3y agoit's less "customisation" than "implementation". they're not changing core code in the system, they're just writing all the config and the workflow apps on top. this is the case for any large organisation, whatever the ERP. And even more the case for a local government that is trying to use a tool designed for industry
- jrochkind1 3y agoI think that's the right question that gets at the heart of the ERP mess, but it doens't have a great answer. In theory, you should be able to have your software support the workflow that works for your specific organization, business, and context. Rather than having to change your workflow to what the machine demands as universal. The latter seems really inhumane. In practice, it never seems to work out very well. Does _anyone_ like their ERP and it's implementation? These products are designed around being customizable -- if they gave that up, they'd be much simpler products, which means they'd probably work a lot better (for what they are designed for, which may not be what you need, which is the problem), and be a lot cheaper (which is not necessarily in the interests of the companies that sell them -- the complexity is fine with them, as long as they can keep charging for it, including with their consultants). It almost makes me wish yo ucould do things without software at all -- but that's a disaster too, this isn't 1970 and pretending it is doesn't work. There's really no way out. We've built a society where you can't do this without software but the software that would be _good_ takes more skill and resources than society can possibly afford.
- kodah 3y agoThe alternative is to build purpose built software, but that would require this municipality to pay SWEs (among other folks) directly. An interesting case study would be if it were actually cheaper in the end to just employ some software engineers and write something catered to themselves.
- orwin 3y agoLeaving Sap was a good idea, but i'm pretty sure replacing Sap by oracle ERP wasn't. Especially, it seems it's one to one in functionalities? The better idea is to take a more modular ERP imho, and slowly transition.
- stakhanov 3y ago> The better idea is to take a more modular ERP imho, and slowly transition. ...or just imagine what £100M in bounties to open source developers could accomplish. It would probably get their requirements met and, as a side-effect, contribute billions to the world economy by liberating many others from lock-in, besides just Birmingham City Council. As it stands, all they seem to have accomplished with this ludicrous amount of money is getting rid of one evil by inviting another.
- mschild 3y agoEven just normal competent salaried employees would've been an improvement. With that amount you could've reasonably employed a 50 person department with roughly 100k cost of employment for 20 years.
- justrealist 3y agoOpen-source only helps organizations that take ownership of their own design and decision-making. Contracting disasters are always because the customer is trying to outsource thinking. Usually it ends poorly. Either way, you don't solve the problem with a configurable OSS tool.
- RobotToaster 3y agoIt does sound like a very "out of the frying pan" decision.
- generic92034 3y ago> Leaving Sap was a good idea From the article: "Standard SAP is the same, but BCC customized SAP to get it working really well and apart from some minor annoyances, SAP was a good product that should never have been ditched," the insider said.
- daenney 3y ago"The UK's biggest city council faces £100M bill in Oracle ERP project disaster". There's really no need to rope in the whole continent for a specific thing in a specific country. The fact that they're the biggest local authority by population in Europe is really not relevant to anything that's happening here.
- KptMarchewa 3y agoI don't understand how they are "biggest local authority by population in Europe". Surely Istanbul is larger.
- pjc50 3y agoIs Istanbul a single unitary local authority or done by boroughs/arrondisements?
- terminalcommand 3y agoIstanbul has an elected mayor and a local parlament. The current mayor is Ekrem İmamoğlu. Big cities in Turkey have a single metropolitan municipality each. Below the metropolitan municipality each district also has its own local mayor and local parlament. They have a duty sharing system. For example water and public transportation is responsibilty of the metropolitan municipality. Collecting property taxes is responsibility of the smaller local municipalities etc.
- justrealist 3y agoEurope is a state of mind.
- fredoralive 3y agoIt's a bit of a weird semi-pedantic stats claim, they're basically counting "largest lowest level of government". Birmingham is a large unitary authority, there are no district or borough councils below it. Looking at Wikipedia Istanbul seems to be split into districts which have their own local governments (similar to London with its boroughs). So they're counting the 514,900 population of Başakşehir or 259,146 of Hackney vs the 1.1 million of Birmingham, rather than the total populations of Istanbul or Greater London.
- throwuwu 3y agoWhat does ERP mean in this context? Asking because it means something else that is making all this discussion of ERP projects and consultants sound quite funny.
- candiodari 3y agoAt one point I contributed to "Odoo" (OpenERP). It essentially means having lots of Applications work against a single database so that everything about the whole company uses this single database. ERP vendors generally have a whole bunch of standard applications ready. Things like inventory, sending out stuff, payroll (for X number of countries), ...
- throwuwu 3y agoThanks! Still does nothing to break the association with the other definition for me though so I’ll add ERP vendors to the list of things to giggle at
- candiodari 3y agoA great deal of complex software "paradigms" aren't nearly as complex as they appear to be at first glance. Most of these are built on a gimmick, the same as in the Open Source world. Every database has "their thing". Mysql "is fast and simple". PostgreSQL supports everything you could possibly want. Oracle has their "fantastic" SQL compatibility and their fast clustering solutions. SAP has their "live indexes". Of course, both have implemented each other's features, and so has PostgreSQL at this point. So that apparently makes them great developers of timesheet software, right? Apparently to MBAs this makes sense. These companies are essentially consultancy software development companies that have an in-house "core" software (Oracle, SAP, ...) that they can work with really well (in theory), and have standard solutions for many problems on that core. But 3 ABAP developers aren't that different from 3 PHP developers: they all do things slightly differently. And, yes, I do think SAP ABAP is "kind of" PHP. You can even make it look pretty similar, though it uses their internal stuff, not HTML. These companies are selling "quick" and "standard" business software development. There are major differences too, though. SAP stores all it's code in the database itself, which prevents all development tools, from vim over git to CI/CD from working. Of course this would mean Wordpress is ERP software ... and it kind of is, with enough plugins.
- pjc50 3y agoThis happens a lot. Another way of looking at it is that this is a successful outcome for Oracle, because they've hugely increased their billable hours. Councils are I think too small to access GDS? If they could I'm sure the superior GDS project management would mitigate this kind of disaster. I remember when the University of Cambridge had a similar disaster and ultimately published an inquiry: https://www.admin.cam.ac.uk/reporter/2001-02/weekly/5861/1.html https://www.admin.cam.ac.uk/reporter/2001-02/weekly/5861/1.h...
- toyg 3y ago> Councils are I think too small to access GDS? Councils probably don't want to access GDS. Why lose all those sweet vendor kickbacks? Free ticket to this, free travel to that... There is a lot of corruption in Britain at the lower level of government. It's all in plain sight, it's all legal, and because it benefits the same networks (someone would say "classes") that are structurally in power in Westminster, nothing gets done about it.
- revelio 3y agoBirmingham council is run by Labour but Westminster is run by the Conservatives. Unless by classes you mean all politicians, it's not the case that it's all linked back to central government. Besides, corruption isn't needed to explain this. ERP projects fail all the time, with SAP as well.
- laputan_machine 3y agoThey're all the politician class. Labour don't represent the lower classes any better than the tories imo.
- revelio 3y agoDoesn't really make sense to say that politicians are structurally in power in Westminster though. I mean, yeah, kind of by definition in a democracy the political classes are structurally in power.
- AngeloAnolin 3y agoNot surprised. Enterprise software is always *enter-pricey*. Lots of $$$ poured into consultants and very little to the folks who actually do the work and implementation. And of course, a project (or program in corporate speak) of this magnitude will have a lot of meetings involving a lot of people that usually ends up in very meaningless decisions.
- revelio 3y agoDoes anyone know why they decided to leave SAP?
- pera 3y agoAnother one six months ago: University of Edinburgh orders investigation into Oracle finance disaster https://news.ycombinator.com/item?id=33739657 https://news.ycombinator.com/item?id=33739657
- AlbertCory 3y agoThe term for ginormous software packages like SAP and Oracle ERP is HostageWare: Some people inside these customer organizations are fanatically committed to it, since what choice do they have? A disaster like this?
- drumhead 3y agoIf it's not a fail with Oracle, it's SAP or dynamics or some other "ERP". Do these implementations ever work, and why do these big organisations keep going for them?surely they've seen the string of failures preceding them? I suspect it's ego, and the need to pad CVs that does it. And the unwillingness of the people who should know better to say no.
- rossdavidh 3y agoThey "work" just fine for Oracle, SAP, etc.
- nradov 3y agoMost implementations do "work" to the extent that they are better than nothing. I mean what's the alternative? Running business processes on paper is no longer a practical option. And most enterprises lack the scale and competence that would be needed to build their own custom ERP in a cost effective way.
- user568439 3y agoYes, they do work and sometimes pretty good. It mostly depends on the implementation team. SAP which is the one I know, has plenty of defects but also a lot of virtues. A good implementation and customizing team can make it run properly.
- red-iron-pine 3y agoSAP S4/HANA was insanely expensive but generally worked, and on time. Finding consultants who could help was like pulling teeth, though. The offshored Indian ones were terrible, and the European ones often didn't speak English well enough. Mexico City was had good offshoring options but couldn't supply enough bodies. Never had a strictly Oracle project go well though. Exadata is a dirty word in these parts. Only thing that was more of a PITA was Azure...
- taylodl 3y agoOverpromise, underdeliver, and overcharge. It's the Oracle Way. In my experience, Oracle only ever built one product that was good, and you can replace it with PostgreSQL for significantly less money and gain greater flexibility to boot. I think the source of Oracle's execution woes lies in part with their Fusion Middleware. It's garbage. Even though JEE has become somewhat outdated, it's still viable. That isn't the issue. The problem is the horrible memory management, continually stuck threads, and their poor messaging implementation. It makes solutions built on the platform prone to constant failure and requiring lots and lots of hardware - which of course increases licensing costs back to Oracle. I've given up trying to figure out why people keep buying their stuff.
- peapicker 3y agoEY was the implementation partner, Oracle the vendor. The article doesn't make that clear, but a linked one does. PWC and KPMG were brought in to try and fix after EY failed at additional cost. All three are known for putting brain or three and a bunch of new hires at top rates on projects. Oracle is a red herring here.
- taylodl 3y agoOracle is not a red herring. These are just Oracle's "implementation partners." They're all in on the take.
- red-iron-pine 3y ago> Oracle ERP project disaster Redundant headline. Oracle & ERP almost always implies disaster. Seriously, never again. Ditto for IBM, and a lesser extent, SAP. I wish Red Hat had a real competitor, cuz Canonical ain't there yet.
- ripperdoc 3y agoIt seems to me that most city councils will operate similar services and therefore have similar needs from their software. Therefore it doesn't make sense for each council to buy and customize their own system. Even better, they could co-fund an open source tool that can be used by all. I realize it's a simplistic view but it seems like there is a lot of money to be saved.
- richliss 3y agoBirmingham Council has always been effective at wasting money.