9 ms·
I work for a Danish municipality with roughly 10,000 employees. I’m not sure if you know, but our public sector has been competing with Estonia at being the mos
by moksly 7y ago
I work for a Danish municipality with roughly 10,000 employees. I’m not sure if you know, but our public sector has been competing with Estonia at being the most digitised in the world for a decade. We operate an estimated 300-500 different IT-systems, some of them enterprise sized SAP solutions deployed on old IBM mainframes with multiple layers of APIs to make their fronts somewhat web-based (don’t ask). Others are minor time registration systems or automated vacation-payouts. I said estimated because a day-care institution is free (though advices not to) buy it-systems without talking with any part of the centralised organisation.
Microsoft has been one of our better partners in all of this. They aren’t cheap, but they listen to us when we need something. We have a direct line to Seattle, and parts of what we ring up at tickets have made it into the global 365 infrastructure. Stuff like making it easier to hide teams-emails from the company-wide outlook address-book.
More than that though, our tech-team is trained and certified in Microsoft technologies. The combination of in-house staff and a 30+ year long good business relationship makes Azure such an obvious choice for cloud. Some of the co-municipal systems we buy through a joint-owned organisation called KOMBIT operate in AWS (support and operations is handled by private sector companies), and it’s not like we’re religious about not using AWS or something other, but we’d need to build a relationship and retrain staff to do so.
- polskibus 7y agoAt this scale, why not invest in your own datacenter, since you already have a lot of servers to maintain? I understand you have to use Office 365, but what is the value prop of Azure to you beyond that ?
- ptd 7y agoOne point(though I will admit this is the cynical part of my brain speaking) immediately jumps to mind. If the server goes down, you can blame Microsoft. Even the least technical person can’t blame you for that. If you create your own data center however, if it goes down, you may potentially be on the chopping block if service is interrupted. Self preservation is a strong motivator, perhaps the strongest in a business environment.
- adev_ 7y ago> Self preservation is a strong motivator, perhaps the strongest in a business environment. And it's the most bullshit one. Outsourcing the risk does not mean avoiding it, it means however putting it out of control. This is a typical manager bullshit attitude "nobody got fired to buy IBM" that generally lead to adopt unadapted, bloated, overpriced solutions to trivial problems. Just because they do not have the balls to do things properly. Running out of its responsibilities should be a criteria to get fired when things go badly wrong. The outsource of the 737 MAX MCAS code to India today is a perfect example of that
- scarface74 7y agoYes, surprisingly enough, people look out for their own self interests. What makes outsourcing hardware different from the dozens of other software as a service vendors that most companies depend on? Why spend the time developing competency in managing servers if that doesn’t give you a competitive advantage. But, going with the biggest most stable vendor is usually good. If you bought IBM hardware in the 70s you can still buy new hardware that supports your software. If you went with their competitors - not so much.
- adev_ 7y ago> But, going with the biggest most stable vendor is usually good. If you bought IBM hardware in the 70s you can still buy new hardware that supports your software. If you went with their competitors - not so much. For 20 times the cost of commodities x86 that you should have bought by doing the right thing. And this 20 times the cost will very likely also please your own competitors because you will yourself be less competitive. That's how you finish with entire airline industry or bank still running on COBOL with no possibility of migration. IBM itself is very happy about it however.
- scarface74 7y agoHow much will it cost to rewrite everything? What were they suppose to choose in the 70s? Do you think that all of the people still using IBM and running legacy code are dumb or just maybe they did a cost benefit analysis and decided they didn’t need to rewrite everything in Node and React?
- jeswin 7y agoActive Directory, Domain Services, Exchange Server and SharePoint Servers (which can use integrated Windows auth). Also many internal Enterprise apps are integrated with AD for auth and permissions.
- luckydata 7y agoIt would be a tremendous hassle for little to no payoff as they probably don’t have the skill set to actually run a data center, and probably their size is actually not that large, they just run a lot of software I suppose (that’s usually the case in the public administration). It would be rampant malpractice to open your own data center for something like that.
- deleted 7y ago[deleted]
- fulafel 7y agoAn potential problem is that now half of your IT recruits go to reinventing this stuff instead of working on domain problems. There aren't that many it staff per 10k municipal employees. (Of course this is assuming Azure specific hassles take much less staff time than running your own infra, not a given...)
- nprateem 7y agoHaha. Here's another one: "At this scale, why not write your own OS, since you already have a lot of servers to maintain?"
- corporateslave5 7y agoHaha!
- sgt101 7y agoAt this scale why not contact all your neighbouring or peer municipalities and go for a joint cloud migration?
- JohannesH 7y agoAs someone who worked in the very same municipality... one word; bureaucracy. More specific reasons were. Legal boundaries between ownership and taxpayer money separation. Of cause GDPR is also a big obstacle in joining forces on this scale. I worked on a project which tried to make a joined venture between several municipalities. I'm not sure if that partnership has ended yet, but overall you could only progress if there were people on both sides collaborating with a "better ask forgiveness, than permission" mindset. As soon as the municipality lawyers or their security officers got involved, all progress stopped immediately.
- scarface74 7y agoOr not dealing with infrastructure gatekeepers.
- moksly 7y agoOur setup is mixed, all our own servers are virtual on rented space at a local server rental shop. I’m not a hardware tech, so I’m afraid that’s as technical I can get. So our SQL cluster, and most of our web applications run on our own iron. Anything external as well as a lot of managed services run in Azure where the added security and operations dev-tools are invaluable compared to what we had when we self-hosted. We do make a conscious choice about everything that moves to the cloud. We’ve spent a lot of time figuring out if buying new rental space for our SQL cluster would be cheaper than moving it to Azure. With the current political climate, we’re also a little more hesitant about moving things to clouds operated by American companies, because there is risk attached, in that we may have to move it back rather suddenly. Not something we expect will ever happen, but we don’t like risk in the public sector. So my post wasn’t so much a “we run everything in Azure” as it was “well Azure is the obvious choice for the things we do operate in the cloud”. The relationship Microsoft has with enterprise, and the expertise staff has with their products means they become the best choice for a lot of enterprise. Unless Microsoft solutions are radically more expensive than their competitors, they simply have an advantage by already being a big, and typically well liked, player in most large organisation.
- chairmanwow1 7y agoI say this as someone who used to firmly agree with this position, but in the last year my mind has been changed without reservation. The time where this was a good idea has genuinely passed. There is no earthly way that any reasonable organization will be able to provide even the security that is present by relying on a large-scale cloud provider. There is no way they will come out on top money-wise either. The big cloud players have a _absurd_ number of servers that _teams_ of some of the best CA talent the globe has to offer stressing to improve KPIs for. 10k employees in a municipality is small potatoes to what azure / AWS / GCP have dedicated to their products.
- dx034 7y agoMaybe not own dc, but colocation could still make sense? You don't need to be more efficient than Azure & Co. They have pretty solid profit margins, even being 30% less efficient should still be cheaper for you. And beyond a few thousand servers, I'm not even sure if scale matters that much (for server virtualization only).
- vidarh 7y agoI have literally never seen a setup where cloud came out less than ~twice the cost of dedicated hosting. I have seen instances where dedicated hosting comes out cheaper than putting your own equipment in a colo, but even that depends on being in a location where energy and property prices (and so colo rental costs) are high. And this does factor in devops - when doing consulting I earned consistently more (because of more hours) from clients that went for cloud setups; often they'd end up spending more times solving problems that generally didn't exist in the first place in a dedicated setup. I do see lots of people that keep assuming that the cloud players must be cheap because they're so big, but I just have never seen that bear out in practice - in part because of that attitude, the margins they can charge are far higher. Cloud providers are great for ease and for the number of services they provide, but they are generally an expensive step up.
- dcolebatch 7y agoI suspect you’re doing it wrong, or there’s some niche type of computing you specialize in. In general purpose IT, looking across an enterprise portfolio of applications, we consistently see customers of Tidal Migrations replatform their applications to cloud and save 95+% in OpEx vs dedicated on-premise hosting. IMO, The first step to realizing those cost benefits is recognizing that the cloud is not your datacenter and you need to architect differently. Yes, cloud spend can grow as you open up access to more developers, but that’s why we have a plethora of tools and governance people to help make that manageable. I believe the business benefit of the agility gains that come from instant and decentralized resource provisioning will always trump any cloud bill... especially if you’re in a competitive industry & don’t want to get left behind.
- christophilus 7y agoMy experience with home-rolled solutions suggests there are plenty of hidden costs. Got a new project, and want to spin up some experimental servers? Gotta wade through a bunch of IT guys. Documentation and training around Azure, etc is also going to be better than homegrown alternatives, most likely. Things like that. It’s hard to put a monetary figure on the frictions costs, but they are there.
- sergiosgc 7y agoYou're comparing one extreme (public cloud) to another extreme (sysadmin-managed infrastructure). The alternative, today, is either a private or a hybrid cloud, which have all the advantages of short-circuiting sysadmins that public clouds have.
- username90 7y agoPublic institutions usually have a lot of systems but not a lot of traffic (if we compare to tech companies), so it makes sense for them to put things in the cloud instead of building everything themselves. Also public institutions rarely get top talent and instead do everything via contractors, do you really think it is cheaper to get government contractors to build and run infrastructure instead of letting Microsoft do it?
- deleted 7y ago[deleted]
- sgt101 7y agoErrm. "Estimated" 300-500 systems? I worked at a corporate with (at one point) 1891 systems; we had several ways of counting them... Some ways said 44 ( business platforms) But count them we could! If I was your CIO counting would be my number one mission!
- moksly 7y agoThe m not sure why this was downvoted. I think it’s correct and I think we could save a lot of money by collecting this knowledge. I say this, because I’ve witnessed departments where the managers of two teams in the same building had bought two different info-screen display systems. You could literally walk twenty meters and see two different info-displays. Both with hefty expenses, and both completely unnecessary because we have a centralised into system they can use for free, that they just didn’t know about. Gathering this information and centralising the decision making to make sure things like duplicates don’t happen isn’t on our political agenda however. In fact the political agenda is quite the opposite, favouring even more decentralisation. At least right now. I mean, you say CIO like that’s something we should have, we don’t. We’re 10 techies to run those 300-500 systems, develop our own, navigate the national strategies, implement and manage our national it architecture Rammearkitekturen and support those 10,000 employees. ;)
- unethical_ban 7y agoPerhaps they are part of one team in a larger org, perhaps the org has terrible asset management, perhaps "systems" means "applications" that can be counted different ways and this post wasn't a report to the CIO.
- ailideex 7y ago> They aren’t cheap, but they listen to us when we need something Yet amazingly their interfaces and quality is incredibly horrible. Azure like most other things Microsoft makes feels like a half baked and over fitted platform with serious quality issues.
- GordonS 7y agoI completely disagree - I find the Azure interface, delightful to work with. And best of all, it's consistent, which can't be said for AWS' UI, which mostly looks very dated and differs from service to service. Years ago the Azure UI was a bit slow (but still beautiful to look at), but those issues were solved long ago.
- arethuza 7y agoThe change over from the "old" Azure portal to the new one was a bit painful at the time - but that was a while ago now and the current one is, I think, pretty good .
- eropple 7y agoHaving functionality splattered all through the god-awful, inexplicably horizontally-scrolling interface is not my idea of “delightful”. Parts of it are URL-stateful, other parts aren’t, and there’s no rhyme or reason for the difference. AWS might look “dated”, but I can’t imagine the mindset that cares about that. If a console is how you’re evaluating your cloud provider you are several steps behind where you need to be. And, great news: in addition to having a really dire console, Azure’s APIs are just no good, particularly if you have to step into the shitshow that is Azure AD (and paying even more for Microsoft Graph, which should just be included) or IAM, and they don’t step to either AWS or GCP to a degree that I just won’t work for an Azure company again. Trying to get anything done in those environments makes me feel stupid and personally bad for having the temerity to try, between inscrutable errors where the Microsoft support is barely English-literate even when you’re paying for it and the aforementioned API barf, and that’s a bad way to live one’s life.
- 7y ago
- dustinmoris 7y ago> We have a direct line to Seattle, and parts of what we ring up at tickets have made it into the global 365 infrastructure. This is great for one business, but bad for the whole of businesses IMHO. Microsoft shoehorns every little thing into their products, never asking themselves if they should. As long as they think they can they will. This leads to stupid things like Durable (aka stateful) Azure Functions. Durable functions is a product of a business either not knowing how to use FaaS properly, or they were misusing FaaS for something where they should have chosen a different tech. But Microsoft being Microsoft will try to accommodate any stupidity they can as long as it will please some big customer. In the beginning they get away with it, but over the years that's how they always end up with half baked, slow and buggy products which are inconsistent, incoherent and just awful to use. Azure is certainly on that trajectory from everything I've seen so far and I use Azure every day at a client at the moment so I know what I'm speaking of.
- rjzzleep 7y agoMicrosoft is here to stay mainly because once you got your foot in the door at the big companies you will stay forever. The reasoning is indeed since we already have Microsoft guys Azure will fit us well but in practice none of these Microsoft guys will be able to help you on any Azure issue. So when Azure is involved in a big corp, accenture is usually not far behind. The experience with the solutions this doom duo comes up with are absolute hell to deal with. In upfront cost Azure looks better but in general that's rarely the case. All the azure API's seem half baked. Once you're doing anything more advanced you will run into issues, just look at the terraform azure provider issue tracker for a bunch of issues that people run into because it's not clear until you actually try out the apis. Here's another example if you want to use shared storage on kubernetes with any reasonable iops, the azurefile premium storage increases IOPS per Gigabyte allocated. So if you want any kind of reasonable experience/price you have to easier spin up your own nfs server, use azure netapp or allocate 10TB shared premium filesystem per share, which is something like 70k a year.
- oblio 7y agoI like AWS, but AWS EFS has the same problem. They've improved it a bit through some recent changes, but it's not much better. The way it would work: they gave you absolutely pitiful base IOPS credits for EFS and everything else was related to disk space used. So more disk space used (and paid), more IOPS. After that they'd completely detroy your IOPS if you used up all the credits. By destroy I mean IOPS at the level of a HDD from 1995. I set up a Jenkins using EFS and initially it went well. It barely had any activity and after about 2 weeks it used up all the credits. After that even the login page would take 20 seconds to load.
- kakoni 7y agoWhat sort of tech-team do you have btw? Inhouse? (Or should I say inmunicipality;)
- Pigo 7y agoI'm not managing anything near what you, and many people on here, do in your job. But from the perspective of a developer who has worked with several cloud platforms, Azure and Devops make the barrier for entry very easy, and the functionality available keeps growing. For personal stuff I like using Firebase, but that's just because it's free. I just feel like you get what you pay for with Azure, and even for a small team like mine, someone at MS has always gotten back to us quickly when we need them.
- Pigo 7y agoI'm curious why this got down-voted. Was my opinion invalid or something?
- CodeSheikh 7y ago"Stuff like making it easier to hide teams-emails from the company-wide outlook address-book" is not this an implementation thing instead of infrastructure? Say you use LDAP to manage user and permissions etc and it is up to you how you want to set it up. I am curious to find out if you are using some out of box solutions form AZURE for this.