6 ms·
About 15 years ago, I became a full time employee of the Canadian Government. I was part of a team that was rebuilding a pretty huge and critical system. It was
by latch 7y ago
About 15 years ago, I became a full time employee of the Canadian Government. I was part of a team that was rebuilding a pretty huge and critical system. It was a disaster. I left after a year. The gun registry before it and the Phoenix pay system since, and many other failures in between..it seems pretty hopeless.
I initially put all of the blame on the expensive consultants, that were doing most of the work, for being technically awful (writing software like Fowler's Analysis Pattern book was a holy grail).
While I still think they bear the brunt of the blame, in retrospect everything was awful. The project was both a business and technical transformation at once. There were too many managers, too many business experts and overall poor leadership. Rather than breaking it down into manageable chunks, it just kept growing and growing. There was no concern or accountability for waste.
Sometimes replacing a legacy system with a new system is difficult. But most of the time, it can be done as a slow and steady transition...so long as egos and ambitions stay out of it.
- diego_moita 7y agoNot all government departments are like that, but too many are, probably the majority. What I've noticed on the ones that work is that they have a core group of technical managers and architects that hold in their heads the whole system architecture and organization. These are people that for more than 10 years decide what will be done and how it will be done. This seems to happen on the more financially crucial and essential areas of the government (e.g: revenue from resources). It is in the more accessory/non-essential departments where there is rotation of technical staff that the chaos thrives. Profit centers are usually organized and efficient. Cost centers aren't.
- willhslade 7y agoAre you familiar with our tax website? Google CCRA.
- jbay808 7y agoIt's very slow, but I'm always impressed by how functional it is and how easy it is to find what I'm looking for. I'd rate it 7/10, which is higher than I'd rate a lot of Google's stuff.
- derefr 7y agoIt’s pretty clear that it’s a new frontend talking to a very old (probably green-screen style mainframe application) backend through some kind of gateway, though. Maybe with some modern DBMSes serving as ancillary data storage, but with the legacy system definitely being the system-of-record that everything has to interface with in the end. If a new-ish web app for a bank or government service has a “maintenance period” where it becomes inaccessible between 5PM and 9AM each day—without the actual web layer going out, just seemingly changing its access ACLs on the data layer—that’s because they’ve got the web app fronting a mainframe that was designed for explicit OLTP ingest / OLAP batch ETL phase separation. The new record event stream gets collated at night to build queriable/indexed tables. Often on hardware so slow that it needs the entire maintenance period to finish building those tables+indices. (This is why better performance in these systems is hard: if you want higher OLTP ingest throughput on the same hardware, you can have it, but then they’ll need longer maintenance windows, so long that they’ll now break their SLAs by being unavailable during business hours!)
- jbay808 7y agoThanks, I found this post fascinating and informative.
- dredmorbius 7y agoLate reply to a now-closed thread: https://news.ycombinator.com/item?id=22094219 https://news.ycombinator.com/item?id=22094219 I've quasi-blogged at https://old.reddit.com/r/dredmorbius https://old.reddit.com/r/dredmorbius When I do re-start, that will at least get updates. There's a new blog, I promise, eventually coming to GitLab, likely: https://gitlab.com/dredmorbius/lair-of-the-id https://gitlab.com/dredmorbius/lair-of-the-id Or https://gitlab.com/dredmorbius/synopticon https://gitlab.com/dredmorbius/synopticon And yes, technology as predicated on energy flux is a principle I've largely accepted. Vaclav Smil's work is singular in this, though I find Daniel Yergin's to be so as well, though that seems largely unintentional / accidental. It's oil, not money, that changes everything. (Or at least changed it.)
- albertabound 7y agoI did some contract work for a federal agency in Ottawa about 15 years ago. One day in the coffee room I heard a bunch of consultants reminiscing about what a great project the gun registry had been. Coming from Alberta, where that project was a notorious and spectacular failure ($1B for a CRUD app that was years late and didn't work), at first I was shocked. But it was fascinating to get a window into how the other half lives! They were nostalgic for the gravy train. I'm glad I didn't say anything, but it was a learning experience in how one person's boondoggle is someone else's windfall.
- alex_anglin 7y agoYears ago I interviewed a PM who listed the Gun Registry and they had the same feelings, as though the project was a complete success. Unfortunately, said person did get the job... and no surprise things didn't go well thereafter.
- mattkrause 7y agoYou interviewed Jean Chrétien?
- parliament32 7y agoProject Manager
- vorpalhex 7y agoI've had consultants submit projects and go on these long talks where they genuinely felt they had produced something awesome and awe inspiring - when in reality the output was poor. While some of this is bluster, one person's lifetime achievement is another's hungover mad dash attempt to finish a project.
- flukus 7y agoConsultants can be great for filling in very specialized gaps, but beyond that I don't know how anyone can think it's a good idea. Aside from a history of failure and consulting companies acting as parasites the incentives simply don't align in the first place. Why do a good job that will last years when a bad job will make you more money and keep the gravy train rolling? Of course building up a good internal team inside a government department also seems to be frought with peril, so maybe it's the only option.
- altitudinous 7y agoYou're describing every internal government system migration in the world :( Managers, subject matter experts and consultants are paid by the hour unfortunately, so the incentive is not to deliver the project, but to instead make it take as long as possible.
- refurb 7y agoCanadian here, but left 20 years ago, but I read CBC news daily. That Phoenix pay system was a total cluster that lasted for years. I remember reading headlines and thinking “they haven’t fix it yet?”. Made the ACA website in the US look like a resounding success. People were literally not getting paid for months (years?) and the gov’ts was like “working on it!”.
- alex_anglin 7y agoNarrator: "they haven't fixed it yet"
- Harvey-Specter 7y agoThey still haven't fixed it. My partner works for the federal government. She was bumped up to a new pay scale and suddenly stopped being paid for 2 months. When she did start getting paid it was at the old level and she hasn't received the missing 2 months salary yet. She had to open three separate cases to 1) receive the missing pay, 2) have her pay scale updated in the system, and 3) receive the difference between her old salary and new salary for however long they continue to pay her at the old rate. One of her coworkers hasn't been paid in almost 6 months.
- refurb 7y agoNuts! You’d think after a few weeks they’d hire a ton of folks and do it by paper until it could be fixed. How many people can go months without a paycheck.
- dleslie 7y agoI have family in Government; they still aren't getting paid properly.
- dsfyu404ed 7y agoIt's not the egos and ambitions, it's the oversight and accountability. You even said it yourself, there was no accountability for waste. When the top 1/3 of the org chart is gonna get fired by the next administration regardless there's no reason (for the current administration) to hold them accountable for not doing good work (not to mention that most people aren't going to be harsh to their own political appointees) and that sort of culture rolls down hill really well.
- thrower123 7y agoYup. Years ago, an employer sold some software to a Canadian government department. Every couple years, it becomes somebody new's bailiwick, and I get a couple emails asking what it is and how it works. I am not sure that it has ever been actually used, but they keep paying the renewal invoices.
- latch 7y agoI don't think this is accurate. Aside from the minister, they're all civil servants. For example, the current Deputy Minister of Finance was appointment by a different leader (from a different party) over 5 years ago.
- dsfyu404ed 7y agoJust because they are not politicians does not mean they do not get changed out. In my experience in a particular US state the top three layers (director and two levels of reports) pretty much all get changed out with the administration. Positions below that are (mostly) union positions and therefore protected by the union.
- derefr 7y ago> writing software like Fowler's Analysis Pattern book was a holy grail I feel like there’s something to be said for constructing an architecture entirely out of “formalized” architectural patterns, when you know that 1. you’re not going to be around to explain the architecture to whoever has to maintain the project 20 years down the line, and 2. Anyone who does come onto the project to maintain it is going to have very little time to get up to speed, since they’re very likely on a short contract to implement a single fix (“Y2K compliance” being a good example of when this happened.) At least, if all the architectural arrangements are copied out of a book, you can just put a note in a README at the project root to read said book. Might skip a lot of “software forensic anthropology.” I mean, that, or heavily document every piece of the architecture, to the point of writing a book on your architectural choices (e.g. the reams of documentation on Erlang’s OTP-framework architecture, originally produced as a side-effect of Ericsson engineers documenting their architectural choices while writing code for their network switches.) But, well, did the customer pay for that? No? Too bad. (I find this is one of the major differences between the way a regular IT consultancy handles these kinds of contracts, and the way a bigcorp like IBM handles a contract. The bigcorps factor things like “reams of documentation for later maintenance (not necessarily by us)” into their negotiated price, because they’ve been on both sides of that, and also have had customers who have been burned by a lack of it. But this makes their bids higher, so they’re often underbid by novice consultancies who don’t have share this mindset...)
- 908B64B197 7y ago> But this makes their bids higher, so they’re often underbid by novice consultancies who don’t have share this mindset...) Or the bid has no clause about documentation at all!
- dleslie 7y ago> There were too many managers, too many business experts and overall poor leadership. This is Canadian Government Bureaucracy in a nutshell. Decades of (often minor) scandals have caused successive Governments to restructure in such a way that it is nigh-impossible to assign responsibility; and have done so by making it extraordinary difficult to form clear direction and make decisions. What we need is a massive downsizing of the public service at the managerial level, and begin placing the burden of responsibility upon departments. They need both the power to make decisions and the associated risk of failure; whereas now the decision making power is nigh-ephemeral and the risk is distributed broadly and thinly.