38 ms·

Microsoft’s Azure DevOps: An Unsatisfying Adventure

260 points by jedikv 8y ago

deleted 8y ago
[deleted]
markcubed 8y ago
While I certainly have had a few annoying issues myself, full credit to the team behind DevOps (nee VSTS) for doing an amazing job over the last >12 months to overhaul and improve the product. I have been impressed at the pace things are improving. I have not had the same issues with build machines disappearing into the ether (we use Windows hosts however) but definitely agree that the (lack of) caching is maddening. Our apps are not huge but the NuGet & npm restore times now account for 25-30% of our build time.
- taspeotis 8y ago
  I exchanged an email with the Product Manager of some Azure DevOps component that comprises the build pipelines. They’re well aware of the need for caching. Hopefully it comes soon!
  - GordonS 8y ago
    Unfortunately, they've been aware of this problem for years - I'm an Azure DevOps fan, but I really just don't understand why they didn't fix this an age ago :/
- ethomson 8y ago
  Thanks for the kind words; I agree that caching is a problem. We're working on it now.
  - cbHXBY1D 8y ago
    Any idea on when it will land? I'm on a team at building a C++ component and just installing boost can take over 20 minutes for a build.
- avinium 8y ago
  Yeah, this has been my experience. I haven't had a build agent disappear, but lack of build caching is very frustrating. The hosted build agents are slow to begin with, let alone needing 5 minutes for every build to restore packages. AFAIK there's no way to "only run changed tests" too. EDIT: Forgot to add that, despite the above, I'm pretty happy with it. The whole VSTS experience (Wiki/code/PRs/etc) is pretty great, it's nice to have everything in a single place.
  - taspeotis 8y ago
    > AFAIK there's no way to "only run changed tests" too. https://docs.microsoft.com/en-us/azure/devops/pipelines/test/test-impact-analysis?view=vsts https://docs.microsoft.com/en-us/azure/devops/pipelines/test...
    - avinium 8y ago
      That looks great - must have overlooked that last time I checked.
Havoc 8y ago
wow that sounds pretty seriously broken. Has anybody else here actually used it & can comment?
- GordonS 8y ago
  I've been using Azure DevOps for around 2 years, and TFS for several. Most of what this article says, I simply don't see in (almost) every day usage. The single issue I agree with is the lack of caching - installing NPM packages can take up to 15 minutes in the worst case. MS have known about this for years, so I honestly don't understand why they haven't done something about it. I also think the hosted build machines are underpowered, despite their claims of Xeon CPUs - build on even an Azure B1ms VM (a low-powered, burstable SKU) are up to 3 times faster. In fact this is how we solve the NPM caching and performance issues - by running the VSTS Build Agent (it's OSS!) on relatively cheap Azure VMs. Aside from these niggles though, I actually think the Azure DevOps product is fantastic.
  - toxicbakery 8y ago
    I'm interested to hear you have not had the same CI issues I have. One theory I have with no substantiating evidence is that Linux agents are least commonly used. The Linux agents on hosted builds run in a hypervisor but I've seemingly had the same issues running self hosted agents (as I mentioned AWS). I can't find an actual issue and again as I mentioned in the post, replacing my AWS instance with a new AWS instance also eventually hit the same issue. Maybe it's some perfect storm of problems but that is just more speculation.
  - WorldMaker 8y ago
    Regarding caching, with NPM you actually have a couple good options (best together): 1.) Make sure you are using `npm ci` instead of `npm install`. The `ci` install command is often much faster and skips a lot of stuff that `install` does. It's also better at sticking purely to your package-lock.json versions. I suggested on UserVoice at one point that Azure Pipelines should switch to `npm ci` as the npm task default. `npm ci` is a relatively recent npm command and I don't everyone has caught on to it yet. 2.) You can use Azure Artifacts for NPM packages. It acts as an alternate NPM feed and caches packages to your Azure DevOps account. Artifacts only supports NPM and NuGet, so it isn't a general caching solution, but in this specific case it helps. I've noticed the builds I have using Artifacts are a bit faster than going straight to NPM. (I started using Artifacts for private NPM packages, so I'm not using it in every build yet.) The current biggest caveat to using Artifacts as your main NPM feed for a project is that it doesn't support NPM audits currently. (I also made sure to report that on UserVoice.)
nikolay 8y ago
We are actually pretty satisfied and everything's been running smoothly!
codereflection 8y ago
Tangential: Reading this _really_ makes me appreciate the extremely stable system my team has build internally with GitLab and TeamCity. Azure DevOps sounds extremely frustrating (besides the point that I cannot stand the name they gave this product, and GitLab made that same mistake with the Auto DevOps feature name).
- sytse 8y ago
  Thanks for using GitLab! We're open to a better name for Auto DevOps https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ but we haven't been able to find one. Suggestions are very welcome. BTW We're planning to add per per minute for the CI runners and add shared MacOS and Windows runners so people that don't want to manage their own no longer need to.
  - codereflection 8y ago
    That's awesome. We have so much customization around TeamCity that moving away from it doesn't make sense, but we're starting to look at the CI runners for other things, along with the k8s integration, all great stuff in GitLab. Yeah, the DevOps name though, it's unfortunate. DevOps is a culture, not a thing, product, tool, or process. Seems more like calling it Auto CI/CD is more appropriate, maybe? Anyways, thanks for the response!
    - sytse 8y ago
      Thanks for the Auto CI/CD suggestion. I struggle with it since it doesn't cover things like license management and monitoring. I agree DevOps is a culture foremost. But at the same time it is starting to turn into a product category (DevOps tools). Hope you are able to adopt GitLab with ease.
Rauchg 8y ago
We moved to Azure DevOps for Windows testing for multiple large projects (Next.js, Now Desktop, Hyper) and it's been an absolute pleasure. Much better than our previous Windows CI solutions.
FlorianRappl 8y ago
We use Azure DevOps for a fairly large project and so far its working good (i.e., not great, but better than okay-ish). The point with the lost communication is something we also experienced more often than I would wish for, however, on a standard day I do not see the issue appearing at all. Yes, the mobile page could be optimized (but this is not a deal breaker) and yes some pages do not subscribe for notifications, but usually all these things are not in the hot path. TL;DR: Lots of improvements in the recent year and many things to come - it may not be perfect but it certainly is usable and provides value to our team.
- toxicbakery 8y ago
  The majority of the product indeed does work pretty well. We make heavy use of their Boards and Test Plan features and outside of specific but minor UI/UX complaints, it is a competent tool.
CyanLite2 8y ago
TL;DR: The author works for a Microsoft competitor and doesn't like MS products.
- fourthark 8y ago
  Strong words. Can you back that up? I could find no evidence, looks like the guy is an android developer of some sort.
  - devandanger 8y ago
    Same conclusion.
  - toxicbakery 8y ago
    You are correct, I'm just an Android dev but the company I work for is a long time user of Microsoft products like Visual Studio and obviously Windows. Over time, if anything, we have become more invested in their products and I don't see that changing in the foreseeable future as they generally work well for our use cases.
- toxicbakery 8y ago
  You got me, I use Linux :)
ethomson 8y ago
PM for Azure DevOps here. We've been investing heavily in our user experience and our CI/CD experience, so I'm sad to see that we've disappointed here. Some of these complaints I would agree with - in particularly, we're not (yet) caching build resources - though we're working on this now. But most of these complaints I was quite surprised to hear; not an experience I would want someone to have or what I see from the majority of our customers. So this is certainly a place where I'll follow up with the author to learn more.
- top-flight 8y ago
  Funny I got an email from an Azure PM after I stopped using the free tier asking if I ran into problems. I explained my issue and what I gave up on, and.... never heard a response. I thought that was kind of weird, why even ask?
  - krob 8y ago
    They ask the question, but any response is prohibited unless they cycle it through a pr department. It's about public narrative, and how it changes sentiment. So unless that response pays them, it's better to avoid saying anything than saying the wrong thing and pissing someone off, especially the wrong influencer.
    - bluepanda 8y ago
      (Former Azure DevOps PM here) That's not quite true - did you hear that from someone? When I was there, I had pretty much full freedom to talk with customers, as long as they hadn't opted out of contact. Never had to talk to PR or have emails looked at. Sorry that you didn't hear back though :(
      - iofiiiiiiiii 8y ago
        What does "opted out of contact" mean? I wonder if I am missing some communications because I unchecked some "don't send me spam" checkbox now!
dragonshed 8y ago
I think title should be updated to "Microsoft's Azure DevOps CI: An Unsatisfying Adventure", as all the complaints here are about builds but this service supports lots of other features. My experience with using DevOps nee VSTS was mostly satisfying. For various reasons unrelated to the post, my team ended up installing the build agent on our own bare metal hardware, and other than needing to restart the agent, I didn't have any mission critical issues. Totally agree with all the UI complaints.
rblatz 8y ago
I've been using this day in and day out for several years now on a ton of projects. We mostly use hosted git, and pipelines. The lack of NPM cache is a giant pain, and that's why we run our own build agents. Other than that, I can't say enough about it. Compared to some of the monsters I've seen built with Jenkins, I love the simplicity of the builds and releases we have created in Azure DevOps.
smadurange 8y ago
This article is nonsense. Azure DevOps is a delight to work with. PR system is probably the best I've worked with so far. Build pipeline can improve but it's a far cry from a "horrifically broken". IMHO, Microsoft has been getting a lot of things right lately. Keep up the good work.
- toxicbakery 8y ago
  Horrifically broken was an exaggeration, you are correct. Much of this was written after a stressful day compounded by dealing with our CI failing to do the thing it is supposed to do. I also agree with you that Microsoft has been making great strides in the direction of "getting a lot of things right lately" as you said.
  - chrisrpatterson 8y ago
    We really hate to hear you are having such a bad experience. The engineering manager has reached out on Twitter and included me. I would love to see what I can do to help
    - stevenacreman 8y ago
      I personally don't really understand this approach. In fact, this approach annoys me. In this situation I feel like the product has already wasted my time and now people are contacting me to waste more of my time. The blog is really detailed and includes screenshots. You could fix the obvious issues and then follow up with an update once it's released.
      - toxicbakery 8y ago
        In the opening paragraphs I stated it was my hope that this gives me a gateway to communicate with someone that isn't support and isn't the void that is their robot response issue tracker. That said, they generously reached out to me and appear to genuinely be concerned by my experience. I plan to keep posting as I run into issues and also as they correct my mistakes and also solve my problems.
achievingApathy 8y ago
What I never really understood is why does Microsoft Test Manager only come with Visual Studio Enterprise? Do only enterprises test their software?
- mandeepj 8y ago
  > Do only enterprises test their software? Microsoft also has operating expenses. They need to bring home some income to keep the lights on.
  - Twirrim 8y ago
    Isn't that what Azure is for? Use the software as a loss leader to get them using your services, which is where you make the real ongoing money.
  - achievingApathy 8y ago
    It just strikes me as an odd thing to put on a luxury tier like that. Reiterates that quality and testing is often an afterthought rather than something everyone should include. If MS wanted the revenue they would put something the most senior higher paid developers wanted rather than the testing team.
    - darkcha0s 8y ago
      You have plenty of alternatives...
      - achievingApathy 8y ago
        True, but I thought the goal of Azure DevOps was to be a complete package solution. And so while all the pipelines and code and work items are stored there, it would be a pain to keep test cases in something else. And why no integration with Excel?
- harishkragarwal 8y ago
  (Azure DevOps PM here) Test Manager comes with VS Enterprise, VS Test Professional, MSDN Subscriptions and is also available for direct purchase through https://marketplace.visualstudio.com/items?itemName=ms.vss-testmanager-web https://marketplace.visualstudio.com/items?itemName=ms.vss-t.... You are welcome to take your pick from what suits your needs.
dmarlow 8y ago
I've yet to look, let alone try, Azure DevOps. There would have to be some compelling reasons to choose that over TeamCity, which I have had nothing but great and positive experiences with. Any big reasons DevOps is something to consider, or is it mostly ideal for new projects?
Someone1234 8y ago
We use paid Azure DevOps for Source Control (although we're watching GitHub closely). We tried to also use Azure Boards (Work Items, Boards, Backlogs, etc). Ouch. It is a complete UI mess of disjointed ideas. Instead of implementing one thing well, they implemented two dozen things terribly. We actually migrated from Azure Boards to Office 365's Planner (KanBan) just because the UI wasn't so horrible. It is simple and gets out of your way. In fact O365's Planner, Teams, and OneNote Online products might be superior to anything Azure DevOps has produced. The only nice thing I have to say about Azure DevOps Boards is that it integrates into Visual Studio, but frankly even that is barely passable. I'm predicting they scrap DevOps completely and just use GitHub as a clean slate.
- tracker1 8y ago
  I don't even use VS if I can avoid it... web dev is so much smoother in VS Code, and the laginess of VS is unbearable to me. I do have a couple .Net Core projects I mostly use in VS Code and one MS SQL DB project that I have to use VS for. May want to checkout GitLab as an option as well.
devandanger 8y ago
We are using the on-premise solution with our own build agents to build iOS. Looking to move actually into Azure Devops to reduce the effort of maintenance on our own build agents. I've been using AppCenter for hobbyist projects and I feel its been a good replacement for BuddyBuild. Seems to be pretty reliable additionally we've thrown a lot of builds at Azure Devops to validate how well those hosted-agents can handle a large project which also worked out well. So all in all, we are excited about the hosted-Mac agents as well as the flexibility to BYO-agent (helps in the transition). I have seen a lot of these same problems noted in this link as well, just didn't see to prevent from getting our work done as much. So YMMV depending on your product/devops use case. Personally have been using their APIs with some custom CLI scripts to get around some of these issues.
deleted 8y ago
[deleted]
zeeshanejaz 8y ago
We migrated from TeamCity and OctopusDeploy a couple of months ago and we are already regretting it. The reliability of this system is just horrendous. The documentation is pretty much outdated, and quite a few plugins do not work. I have to write a bunch of powershell script to get it to work the way I want. And today, all hell broke loose. Since this morning, we are stuck with an error message "TF400893: Unable to contact the server. This is most likely caused by a network error. Please check your connection and try again.". This has cause a couple of release pipelines to be get stuck, with cancel option not working either. The paid "parallel jobs" are being consume in this stuck pipeline and no other project from my organisation can be built or deployed because of it. Pretty much regretting having spent so much time configuring the pipelines. We might switch it doesn't get resolved soon or if it happens even one more time.
ianceicys 8y ago
At my company we use a mixture of Azure DevOps, Jenkins, CircleCI, Github, Jira, Artifactory, and a host of AWS and Azure, and my view is that Azure DevOps (formerly VSTS, formerly VSO, formerly TFService, formerly TFS, formerly VSS---i kid i kid) is a semi-strong product that just can't get out of its own way. The New UI is a clear step backward...and the Github integration leaves a lot to be desired. If you have Github Enterprise...why use Azure Repos...why are MSFT teams moving from Azure Repos to Github...and others moving from Github to Azure Repos. The SCM platform and strategy obviously hasn't settled yet...so Gitlab is just as viable an offering that's growing more compelling by the day. The Azure DevOps hosted build system hasn't been viable in my opinion since it launched, so on-prem agents are a much better option. Take a look at CloudBees offerings, they set the poll position. The key thing that continues to be a pain point is the reporting and project organization data structure limitations with Azure DevOps: is it 1 TPC per business unit, 1 TPC for the whole company, 1 team project for the whole company. What happens when you have 2...how do you consolidate...what's the path forward if you have to move things around. The answer for the last 8 years has been silence. The request for a clean solution has been asked for since 2011. Yes 2011, look at the link below. 8 long years...and basically silence...and "it's a hard problem"...and multiple updates saying "We will provide an update once we start planning for the second half of 20XX." https://visualstudio.uservoice.com/forums/330519-azure-devops-formerly-visual-studio-team-services/suggestions/2037613-make-it-possible-to-move-a-team-project-between-te https://visualstudio.uservoice.com/forums/330519-azure-devop... Eventually, the promises just ring hollow, the proof is in shipped code, and in the meantime other platforms and competitors stacks keep building out better enterprise stories for migrating from Azure DevOps (here's to look at Github, Gitlab, Atlassian's platforms, Jenkins/CircleCI, TeamCity, Artifactory, Xebia Labs etc). Honestly, the world is a MUCH more competitive place with lots of best in class point solutions that aren't that hard to stitch together and get better performance/features. Hope springs eternal...that with the new vertical offerings the DevOps team can deliver rapid value...but 8 years is a long time to only hear crickets. If you're starting out and you need a one-size fits all solution and you're a SMB company with two doze devs...go hog wild with Azure DevOps...that's not to say it doesn't work for large companies like Shell and Microsoft itself...but talk with teams about the limitations they run into and make your own decision on your DevOps transformation adventure...oh and never create more than 1 team project in Azure Devops...seriously...if you do, game over.
thejosh 8y ago
Their build pipeline has been down all morning as well, for both hosted and self-hosted. Just sits pending. No deploys this morning, woot!
AdeptusAquinas 8y ago
I don't know... having been on openshift, Jenkins and bitbucket for the last year I'd give a large sum of money to go back to azure devops.
markbnj 8y ago
> For a few hours, builds ran smoothly. As expected, eventually the agent died during a build. Builds sat queued, my frustration grew, and AWS notified my instance needed a doctor. After an hour I gave up hope for the unresponsive server and commanded it to reboot. Sounds a lot like some process on the build server is leaking memory. We have a couple of older services that have this problem, and if we don't catch it early enough the server will become unresponsive and unable to even handle ssh connections.
mattbillenstein 8y ago
How do all of you people get on this broken stuff on Azure with GCP and AWS (I think) being so good? Is it like free credits?
- throw_away2 8y ago
  I can only assume windows people are used to pain
  - kkarakk 8y ago
    pretty much it, also azure is slightly better at pricing in asia
    - throw_away2 8y ago
      This mimics the mac vs windows argument I had with my brother-in-law where it boiled down to the fact that you could run windows on something cheaper than a macbook. Even though in order to remap caps lock to something reasonable you have to edit the registry. In 2019. The best answer I think involves windows server 2003 resource kit and its UI, which is so old it could get a driver's license. You save money but what kind of life is that?
      - skriticos2 8y ago
        It's not even about the price. I have no problem tweaking my Linux desktop to use an international Dvorak layout (useful for those European languages with ä's and é's) and switching the caps and ctrl keys. Say what you want about Gnome 3, but this works just fine. On Windows.. I get a US only bare bone Dvorak layout and few ways to remap the caps/ctrl keys. Messing with the registry is at best annoying on my gaming (only) PC but impossible on a corporate locked down desktop. Then I'm lucky if the corporate package repo comes with AutoHotkey that can remap keys in a very hacky way. A lot of Windows 98 still shines through in the basic usability suite even on the newest versions of Windows. Even the on-screen keyboard on a (Android) phone is more fun than the HID mess in Windows.
vezycash 8y ago
It seems no one wants to learn the lesson Facebook was forced to learn - No wholesale UI changes.
lqqkout4elfy 8y ago
The silver lining here is that Azure DevOps's code review tool is lightyears ahead of all other repos. I would like to see this support for github repos
manigandham 8y ago
VM Agent based builds are an obsolete model. A docker/container-based pipeline is the current state of the art and allows for very fast builds with caching built in and isolated stages with narrow responsibilities.
- GordonS 8y ago
  There is an official Linux image for a VSTS Build Agent container - but, frustratingly, not for a Windows one! People have been begging for such an image for a long time now, but Microsoft do nothing... I just don't understand why they treat Windows as a 2nd class citizen!
- vtbassmatt 8y ago
  We offer container jobs [1] and recently added the ability to use service containers [2] as well. Disclosure: I'm a PM for Azure Pipelines. [1] https://docs.microsoft.com/azure/devops/pipelines/process/container-phases https://docs.microsoft.com/azure/devops/pipelines/process/co... [2] https://docs.microsoft.com/azure/devops/pipelines/process/service-containers https://docs.microsoft.com/azure/devops/pipelines/process/se...
  - toxicbakery 8y ago
    Funny enough I had not had the time to write about this yet. This resolves one of the other issues on my list. Thank you.
BuckRogers 8y ago
I use Azure DevOps everyday and like it quite a bit. It'll keep improving but I've had no complaints. Microsoft is in a very enviable position, not just having one of the best IDEs but having control over potentially an entire CI/CD pipeline from Github to Azure, while having one of the major programming platforms in .Net to enable best-in-class integration. No one else can really say they have all of that, not Atlassian, Oracle or Amazon. The CI component, Azure Pipelines, is the only hosted CI tool that supports all major platforms, at a minimum Windows/macOS/Linux. The value of essentially fully automating out the role of DevOps engineers is a long time coming, and I used to do that sort of work but saw the writing on the wall with things like Azure DevOps. I think a lot of existing DevOps roles will transition over time to 'Cloud Engineer', specializing in setting up and maintaining hosted CI/CD platforms. There will also be a lot less of them as a result. It's becoming a commodity, MS is just one player there. I love what they're doing and eagerly embrace it, Microsoft has been knocking it out of the park. It works well for us.
stonewhite 8y ago
I remember high CPU alarms waking me up, just to see that Azure OMS Monitoring agents were consuming %99 CPU on every node of our client's database cluster. I'm not sure why that happened, yet the immediate reaction was to uninstall it and never use it again. OP's story about build agents turning unresponsive makes me think of this. MS is the best cloud vendor to deal as a consulting partner, yet the overall suffering caused by Azure almost makes me feel like it is not worth it.
Annatar 8y ago
I'm genuinely curious: what exactly is so hard about installing and system administering one's own servers inhouse that so many people would put up with these complications?
- sl1ck731 8y ago
  For us its: availability outside of VPN and avoiding the run around of making a public site with security, integration with active directory, has repos + a wiki + boards all in one place. No configuring 3 different OSS products to integrate with AD. We use the other stuff more than the CI, but will probably begin using it for some more smaller projects soon. It'll be in the same place which is great and no need to tinker with servers which is awesome.
  - chupasaurus 8y ago
    If you have problem with configuring service users in AD, I guess you have problems with security or group policies on a regular basis.
    - sl1ck731 8y ago
      Or AD isn't our responsibility and we can just use the tool (Azure devops) that is already done before we even launch it and provides all the features we need? I guess you don't have more important things to be doing than screwing with ancillary tools.
- patrickg_zill 8y ago
  There isn't any true difficulty; people are running multiple sites that could all fit into 1 dedicated 64gb RAM server from Hetzner on a different infrastructure that allows them to develop their resume...
  - Annatar 8y ago
    Are you saying that it's simply a case of resume inflation at employer's expense?
equasar 8y ago
At my company, we are pretty happy with Boards, Repos and Pipelines. One thing that needs to be improved so badly is their Test Plans section, not so fan for their way to setup agents for Load Testing.
fastbeef 8y ago
Look, developer tooling is always going to be hugely dividing. Some will love it, some will hate it, some will hate it because it's not like the Thing They Loved At Their Previous Job. Now, can we talk about the real WTF, which is the name. "Azure Devops", really? Two extremely potent keywords each on their own? You didn't like people being able to google answers to their questions about your product?
- tracker1 8y ago
  TFS, VSTS, "Azure DevOps" and often the results are outdated... "ADO" won't work either. They probably should have stuck with VSTS and just kept the abbreviation or made it "Azure VSTS" where vsts would at least continue to be searchable. Another nicety would be to pay StackExchange to add an Azure DevOps subsite, assuming there isn't already one.
timvisee 8y ago
I've had similar issues with Microsofts CI implementation. I'll describe some of the problems I've had here, note that the last time I've used it is about half a year ago, maybe things have changed though I doubt it. This was with VisualStudio TeamServices on an instance hosted by Microsoft itself. I deployed self-hosted build agents as they were orders of magnitude faster. Probably the biggest problem I've had was that build agents are stateful. The agent checksout a repository when the build starts, and removes the directory when it's done. It doesn't however clean up other changes made on the system, such as cache directories for tooling and other data you might move out of your build directory. I believe that builds must be reproducible, and therefore must start with a clean and consistent environment. You don't want builds to fail down the line because a precious job broke the agents environment, or even worse, having jobs succeed only on the agent because some dependent files are left on the agent. Other CI solutions use Docker containers for this to build in, which is great, as they are immutable (or stateless, rather) by default. This is in my opinion the single worst 'feature', making these CI builds distrustful. Secondly, automating these agents is hard. If you want to automatically scale these self-hosted build agents horizontally based on demand, good luck! When starting an agent for the first time, you've to go through some cumbersome configuration wizard to set it up. There is no simple static configuration file available to use instead, or at least, I wasn't able to find it at the time. To mitigate the previous problem with stateful agents I created a Docker container having an agent in it, which was in fact quite difficult to properly achieve. I automatically restarted these agent containers multiple times a day in order to clean-up any garbage that was unintentionally left by a job. This is a horrible hack to slightly improve their (in my opinion) broken system, but it had the bonus of making these agents more easily scalable. Another thing is the build configuration. I prefer having a configuration in a simple YAML file in the same repository, to have it as _close_ to the code as possible. Although Microsoft does provide such functionality it doesn't seem to be common to do it this way, and if I remember correctly it's much more difficult to properly set-up and use than for example, a Travis CI configuration. They do provide a graphical interface to configure these build steps though, which is useful when getting started with the system so you can get a feel of what is possible. It provides some graphical blocks/steps to achieve various thighs, such as doing an FTP upload or running a CLI command, with all kinds of weird properties. Maybe that's because Microsofts things are generally less scriptable. But yet again, I prefer a simple script in an in-repository configuration file, because it feels somewhat constrained and limited. Sadly, it doesn't stop here: - sometimes jobs just didn't want to stop, and no, bashing the abort button a million times didn't fix things. In some situations it took an hour before jobs were properly killed. - the interface (VisualStudo TeamServices) is horrible to navigate, have had to wild-click many times to find what I need. I'm sure this will improve once you use it for a longer period though. - and there are many other inconsistencies and little problems, such as build logs loading half of the time. Maybe I'm grumpy. Maybe I'm ranting about nothing. I don't know. Maybe things have changed, or will change, at least I hope it will. I didn't have a great time with their CI system, my experience was horrible as you can probably tell by now. It feels like an unfinished product, that was released to quickly just to satisfy customers, so they didn't have to switch to some other platform not owned by Microsoft for their CI needs. What amazes me is that there are quite a few free alternatives available which are much simpler and more elegant, such as GitLab, Travis and Circle CI. They are easier to navigate, are usually quite quick, don't have the issues I've (and the article have) described. And greatest of all, you can get it up and running in minutes without having to worry about it afterwards.
raxxorrax 8y ago
Interesting post. I feel for him using this tool. Had my own experience with team foundation server on premise. Although a different tool, many of the problems seem quite similar. A little rant: To the cloud infrastructure of MS: I don't know if anyone uses it for anything serious and doesn't complain about severe performance issues not only related to builds, but everything. From administrative sites to hosted environments. I read a lot of excuses mainly saying I should adjust my expectations regarding speed for cloud solutions... Problem is that there are other providers that don't have these issues. Seriously, you need only to use office 365 if you want to get an impression of bad performance. Because everything is slow. This is from a location in Europe, but I really doubt there are significantly more performant locations.
- stevenacreman 8y ago
  This is true of all of the services I've tried across Azure, AWS and Google cloud. Microsoft are always last place in terms of performance and reliability. For Kubernetes I actually keep track of exactly how bad they are with an automated testing tool. Results of the last test are here: https://kubedex.com/is-azure-kubernetes-aks-any-less-terrible/ https://kubedex.com/is-azure-kubernetes-aks-any-less-terribl... I've complained about UX before in the context of the Azure portal being, in my opinion, terrible. But that's subjective and I respect that others may like it, although not sure how they do.
  - raxxorrax 8y ago
    This is very interesting data, thank you for that. Maybe MS has improved a bit although I haven't felt any improvements yet. But the changes might not be rolled out completely. Performance wise it can only get better in my opinion.
  - dtrailin 8y ago
    You should do an updated test comparing AWS as well. I feel like there should be more continuous benchmarking efforts of cloud vendors. This is the last one I remember in this genre (https://www.azurefromthetrenches.com/azure-functions-significant-improvements-in-http-trigger-scaling/ https://www.azurefromthetrenches.com/azure-functions-signifi...). I wonder if someone could make a business doing that. Also, I've found in terms of UX GCP > Azure > AWS personally.
jopx 8y ago
I don't have much complains about DevOps UX, the only problems that sometimes impact me are the caching/random broken builds. But Azure Portal UX is another story, the Blade UI is one of the worst things I've ever used, I hate scrolling horizontally, I can't open things in a new tab, it's really frustrating.
dustinmoris 8y ago
Makes me really appreciate my setup of: - GitHub for source control - AppVeyor for CI - BlazeMeter for Load Testing - Google Cloud for Cloud Hosting - Slack for Team Chat - MyGet for private NuGet feeds - Trello for task management - ELK cloud for log management Yes it is many different services, but as a small team I like that we can pick the best tool for the job instead of trying to find a hammer for all nails which doesn't exist. It also makes us less worried about changes, because we are not invested in any vendor so much that if they do something which makes our lives too difficult that we couldn't easily move on elsewhere. Also it forces us to design our software in such a way that we don't lock ourselves into a specific vendor. For example we would have all our builds scripted instead of configured via some GUIs with the configuration stored on some vendor's cloud. The build script runs from anywhere. We can run our builds on Azure DevOps, TravisCI, CircleCi or anywhere. We just prefer to use AppVeyor because it's the smoothest CI server IMHO, but the transition elsewhere is just a matter of invoking a single build script. I genuinely also prefer to work with a product which is doing one thing really good instead of having one product which does everything mediocre. From an operations POV it is just as simple as having everything in one place. We have one browser tab open for each service and it makes us actually a lot more productive. I am always only 1-2 clicks max away from what I need. Teams/Companies who choose an all-in-one solution (e.g. everything in Azure + Azure DevOps) don't care about the productivity of their teams IMHO. They care more about the one person who has to do the accounting for their subscriptions or something like that :/
- Kuinox 8y ago
  Sadly MyGet is no longer free :(.
- seanjregan 8y ago
  I was just building a presentation on this topic and the power of an open approach.
whatupmd 8y ago
Sounds mostly like networking problems with the Azure Cloud.
- toxicbakery 8y ago
  I initially thought the same thing but I'm less convinced of that after trying an aws based build agent. I think the agents themselves may have a bug that combined with our pipeline needs, triggers some perfect storm issue. At the same time, we have seen builds fail to start at all and and sometimes fail to even checkout from GitHub.
kapilvt 8y ago
As a maintainer on an opensource project thats recently switched out to azure pipelines, I'm actually overall happy with azure pipelines, its considerably faster (hosted agent) for us in matrix builds then travis both on latency to start a build and actual compute power, and cheaper then self hosted drone. Hosted builds have been reliable for us, we notice a hiccup less than once a month on a single build (avg 20+ builds a day), minus a service incident yesterday. That said I agree with a lot of the criticisms noted here (no retrigger on web hooks to update status, complicated web ui, log ux, lack build cache). It is possible to bring your own cache implementation, I just stuffed/restored things into object storage, also filed an issue for build cache https://github.com/Microsoft/azure-pipelines-tasks/issues/9190 https://github.com/Microsoft/azure-pipelines-tasks/issues/91... noting several other closed issues for the same. It does feel like the platform is moving forward. All that said there's a lot of complexity and feature set in azure DevOps and pipelines that I haven't explored and try to ignore in the ui. At the moment its the only cloud provider build solution that actually even attempts to work well for opensource projects afaics.
- edglas 8y ago
  Thanks @kapilvt, I'm on the Azure DevOps engineering team. Are there specific improvements you'd like to see in the logs UX?
nbevans 8y ago
My mind was blown when we invested a week in setting up DevOps, then realised it has a critical flaw in that the automated build agent will just go into hibernation if there was no build activity for a few hours. So then you have to write a script that invokes a manual build via the API just to keep DevOps "alive". It's like they forgot to properly setup the IIS Worker Pool timeout values or something.
- chrisrpatterson 8y ago
  @nbevans could you provide a few more details here on the agent and where it is hosted? The agent itself does not rely on the IIS worker pool and we are not aware of any bugs in this area. If we have some we would love to get them addressed.
- edglas 8y ago
  @nbevans I'm an engineer on Azure DevOps. This is the first we've heard of this issue, can you provide more details? What error do you see? By hibernation do you mean Windows hibernation? Feel free to reach out to me at my HN name at microsoft.com.
ryanackley 8y ago
Came here to say that I've been impressed with Azure DevOps so far. We don't have the same requirements as the Author. For example, we don't deploy artifacts to a dependency management system and we don't build on Linux. I have small complaints here and there but I couldn't find another build service that has Windows and OS X agents for anywhere near what Microsoft charges.
alkonaut 8y ago
There were some enormous UX issues with the old look & feel of TFS/VSTS as well. Navigation was nearly impossible to understand, for example. While I appreciate the work they put into DevOps, it still feels like the same person that designed the Azure Portal UI has had a say in the UI redesign of TFS into DevOps and that must either mean that a) Microsoft never realized what a horrible UX the Azure portal is, or b) they do realize but they are too invested in this new "design language" to do anything, so instead DevOps adopted the same style as Azure. I'm in the process of migrating TO DevOps (on prem) so obviously I have a keen interest in the direction its taking. Build pipelines being declarative, navigation being clean up etc. are fantastic improvements. But it still feels clunky to use, especially around the backlogs/boards.
mcrommert 8y ago
Thats not 1080p
- toxicbakery 8y ago
  I'm not sure what you mean by this but a few other people have asked me about the described screen size problem so I decided to add some clarification. https://toxicbakery.github.io/vsts-devops/dimensions/ https://toxicbakery.github.io/vsts-devops/dimensions/
jhacker123 8y ago
I'm using Azure DevOps (Previously VSTS) more than two years, and meanwhile I create couple of Pipeline to perform CI/CD. > .NET MVC Web Application Build and Deploy Using FTP > .NET MVC Web Application Deploy in Cloud (Both Azure and AWS) > .NET MVC Web Application Build on Linux Visual Studio Agent Create Docker Image and Published I didn't face any errors from devops, my experience overall good with DevOps. but i agree that it's not user friendly in term of UI and other workflow.

260 points by jedikv 8y ago

deleted 8y ago

[deleted]

While I certainly have had a few annoying issues myself, full credit to the team behind DevOps (nee VSTS) for doing an amazing job over the last >12 months to overhaul and improve the product. I have been impressed at the pace things are improving. I have not had the same issues with build machines disappearing into the ether (we use Windows hosts however) but definitely agree that the (lack of) caching is maddening. Our apps are not huge but the NuGet & npm restore times now account for 25-30% of our build time.

taspeotis 8y ago
I exchanged an email with the Product Manager of some Azure DevOps component that comprises the build pipelines. They’re well aware of the need for caching. Hopefully it comes soon!
- GordonS 8y ago
  Unfortunately, they've been aware of this problem for years - I'm an Azure DevOps fan, but I really just don't understand why they didn't fix this an age ago :/
ethomson 8y ago
Thanks for the kind words; I agree that caching is a problem. We're working on it now.
- cbHXBY1D 8y ago
  Any idea on when it will land? I'm on a team at building a C++ component and just installing boost can take over 20 minutes for a build.
avinium 8y ago
Yeah, this has been my experience. I haven't had a build agent disappear, but lack of build caching is very frustrating. The hosted build agents are slow to begin with, let alone needing 5 minutes for every build to restore packages. AFAIK there's no way to "only run changed tests" too. EDIT: Forgot to add that, despite the above, I'm pretty happy with it. The whole VSTS experience (Wiki/code/PRs/etc) is pretty great, it's nice to have everything in a single place.
- taspeotis 8y ago
  > AFAIK there's no way to "only run changed tests" too. https://docs.microsoft.com/en-us/azure/devops/pipelines/test/test-impact-analysis?view=vsts https://docs.microsoft.com/en-us/azure/devops/pipelines/test...
  - avinium 8y ago
    That looks great - must have overlooked that last time I checked.

Havoc 8y ago

wow that sounds pretty seriously broken. Has anybody else here actually used it & can comment?

GordonS 8y ago
I've been using Azure DevOps for around 2 years, and TFS for several. Most of what this article says, I simply don't see in (almost) every day usage. The single issue I agree with is the lack of caching - installing NPM packages can take up to 15 minutes in the worst case. MS have known about this for years, so I honestly don't understand why they haven't done something about it. I also think the hosted build machines are underpowered, despite their claims of Xeon CPUs - build on even an Azure B1ms VM (a low-powered, burstable SKU) are up to 3 times faster. In fact this is how we solve the NPM caching and performance issues - by running the VSTS Build Agent (it's OSS!) on relatively cheap Azure VMs. Aside from these niggles though, I actually think the Azure DevOps product is fantastic.
- toxicbakery 8y ago
  I'm interested to hear you have not had the same CI issues I have. One theory I have with no substantiating evidence is that Linux agents are least commonly used. The Linux agents on hosted builds run in a hypervisor but I've seemingly had the same issues running self hosted agents (as I mentioned AWS). I can't find an actual issue and again as I mentioned in the post, replacing my AWS instance with a new AWS instance also eventually hit the same issue. Maybe it's some perfect storm of problems but that is just more speculation.
- WorldMaker 8y ago
  Regarding caching, with NPM you actually have a couple good options (best together): 1.) Make sure you are using `npm ci` instead of `npm install`. The `ci` install command is often much faster and skips a lot of stuff that `install` does. It's also better at sticking purely to your package-lock.json versions. I suggested on UserVoice at one point that Azure Pipelines should switch to `npm ci` as the npm task default. `npm ci` is a relatively recent npm command and I don't everyone has caught on to it yet. 2.) You can use Azure Artifacts for NPM packages. It acts as an alternate NPM feed and caches packages to your Azure DevOps account. Artifacts only supports NPM and NuGet, so it isn't a general caching solution, but in this specific case it helps. I've noticed the builds I have using Artifacts are a bit faster than going straight to NPM. (I started using Artifacts for private NPM packages, so I'm not using it in every build yet.) The current biggest caveat to using Artifacts as your main NPM feed for a project is that it doesn't support NPM audits currently. (I also made sure to report that on UserVoice.)

nikolay 8y ago

We are actually pretty satisfied and everything's been running smoothly!

codereflection 8y ago

Tangential: Reading this _really_ makes me appreciate the extremely stable system my team has build internally with GitLab and TeamCity. Azure DevOps sounds extremely frustrating (besides the point that I cannot stand the name they gave this product, and GitLab made that same mistake with the Auto DevOps feature name).

sytse 8y ago
Thanks for using GitLab! We're open to a better name for Auto DevOps https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ but we haven't been able to find one. Suggestions are very welcome. BTW We're planning to add per per minute for the CI runners and add shared MacOS and Windows runners so people that don't want to manage their own no longer need to.
- codereflection 8y ago
  That's awesome. We have so much customization around TeamCity that moving away from it doesn't make sense, but we're starting to look at the CI runners for other things, along with the k8s integration, all great stuff in GitLab. Yeah, the DevOps name though, it's unfortunate. DevOps is a culture, not a thing, product, tool, or process. Seems more like calling it Auto CI/CD is more appropriate, maybe? Anyways, thanks for the response!
  - sytse 8y ago
    Thanks for the Auto CI/CD suggestion. I struggle with it since it doesn't cover things like license management and monitoring. I agree DevOps is a culture foremost. But at the same time it is starting to turn into a product category (DevOps tools). Hope you are able to adopt GitLab with ease.

Rauchg 8y ago

We moved to Azure DevOps for Windows testing for multiple large projects (Next.js, Now Desktop, Hyper) and it's been an absolute pleasure. Much better than our previous Windows CI solutions.

FlorianRappl 8y ago

We use Azure DevOps for a fairly large project and so far its working good (i.e., not great, but better than okay-ish). The point with the lost communication is something we also experienced more often than I would wish for, however, on a standard day I do not see the issue appearing at all. Yes, the mobile page could be optimized (but this is not a deal breaker) and yes some pages do not subscribe for notifications, but usually all these things are not in the hot path. TL;DR: Lots of improvements in the recent year and many things to come - it may not be perfect but it certainly is usable and provides value to our team.

toxicbakery 8y ago
The majority of the product indeed does work pretty well. We make heavy use of their Boards and Test Plan features and outside of specific but minor UI/UX complaints, it is a competent tool.

CyanLite2 8y ago

TL;DR: The author works for a Microsoft competitor and doesn't like MS products.

fourthark 8y ago
Strong words. Can you back that up? I could find no evidence, looks like the guy is an android developer of some sort.
- devandanger 8y ago
  Same conclusion.
- toxicbakery 8y ago
  You are correct, I'm just an Android dev but the company I work for is a long time user of Microsoft products like Visual Studio and obviously Windows. Over time, if anything, we have become more invested in their products and I don't see that changing in the foreseeable future as they generally work well for our use cases.
toxicbakery 8y ago
You got me, I use Linux :)

ethomson 8y ago

PM for Azure DevOps here. We've been investing heavily in our user experience and our CI/CD experience, so I'm sad to see that we've disappointed here. Some of these complaints I would agree with - in particularly, we're not (yet) caching build resources - though we're working on this now. But most of these complaints I was quite surprised to hear; not an experience I would want someone to have or what I see from the majority of our customers. So this is certainly a place where I'll follow up with the author to learn more.

top-flight 8y ago
Funny I got an email from an Azure PM after I stopped using the free tier asking if I ran into problems. I explained my issue and what I gave up on, and.... never heard a response. I thought that was kind of weird, why even ask?
- krob 8y ago
  They ask the question, but any response is prohibited unless they cycle it through a pr department. It's about public narrative, and how it changes sentiment. So unless that response pays them, it's better to avoid saying anything than saying the wrong thing and pissing someone off, especially the wrong influencer.
  - bluepanda 8y ago
    (Former Azure DevOps PM here) That's not quite true - did you hear that from someone? When I was there, I had pretty much full freedom to talk with customers, as long as they hadn't opted out of contact. Never had to talk to PR or have emails looked at. Sorry that you didn't hear back though :(
    - iofiiiiiiiii 8y ago
      What does "opted out of contact" mean? I wonder if I am missing some communications because I unchecked some "don't send me spam" checkbox now!

dragonshed 8y ago

I think title should be updated to "Microsoft's Azure DevOps CI: An Unsatisfying Adventure", as all the complaints here are about builds but this service supports lots of other features. My experience with using DevOps nee VSTS was mostly satisfying. For various reasons unrelated to the post, my team ended up installing the build agent on our own bare metal hardware, and other than needing to restart the agent, I didn't have any mission critical issues. Totally agree with all the UI complaints.

rblatz 8y ago

I've been using this day in and day out for several years now on a ton of projects. We mostly use hosted git, and pipelines. The lack of NPM cache is a giant pain, and that's why we run our own build agents. Other than that, I can't say enough about it. Compared to some of the monsters I've seen built with Jenkins, I love the simplicity of the builds and releases we have created in Azure DevOps.

smadurange 8y ago

This article is nonsense. Azure DevOps is a delight to work with. PR system is probably the best I've worked with so far. Build pipeline can improve but it's a far cry from a "horrifically broken". IMHO, Microsoft has been getting a lot of things right lately. Keep up the good work.

toxicbakery 8y ago
Horrifically broken was an exaggeration, you are correct. Much of this was written after a stressful day compounded by dealing with our CI failing to do the thing it is supposed to do. I also agree with you that Microsoft has been making great strides in the direction of "getting a lot of things right lately" as you said.
- chrisrpatterson 8y ago
  We really hate to hear you are having such a bad experience. The engineering manager has reached out on Twitter and included me. I would love to see what I can do to help
  - stevenacreman 8y ago
    I personally don't really understand this approach. In fact, this approach annoys me. In this situation I feel like the product has already wasted my time and now people are contacting me to waste more of my time. The blog is really detailed and includes screenshots. You could fix the obvious issues and then follow up with an update once it's released.
    - toxicbakery 8y ago
      In the opening paragraphs I stated it was my hope that this gives me a gateway to communicate with someone that isn't support and isn't the void that is their robot response issue tracker. That said, they generously reached out to me and appear to genuinely be concerned by my experience. I plan to keep posting as I run into issues and also as they correct my mistakes and also solve my problems.

achievingApathy 8y ago

What I never really understood is why does Microsoft Test Manager only come with Visual Studio Enterprise? Do only enterprises test their software?

mandeepj 8y ago
> Do only enterprises test their software? Microsoft also has operating expenses. They need to bring home some income to keep the lights on.
- Twirrim 8y ago
  Isn't that what Azure is for? Use the software as a loss leader to get them using your services, which is where you make the real ongoing money.
- achievingApathy 8y ago
  It just strikes me as an odd thing to put on a luxury tier like that. Reiterates that quality and testing is often an afterthought rather than something everyone should include. If MS wanted the revenue they would put something the most senior higher paid developers wanted rather than the testing team.
  - darkcha0s 8y ago
    You have plenty of alternatives...
    - achievingApathy 8y ago
      True, but I thought the goal of Azure DevOps was to be a complete package solution. And so while all the pipelines and code and work items are stored there, it would be a pain to keep test cases in something else. And why no integration with Excel?
harishkragarwal 8y ago
(Azure DevOps PM here) Test Manager comes with VS Enterprise, VS Test Professional, MSDN Subscriptions and is also available for direct purchase through https://marketplace.visualstudio.com/items?itemName=ms.vss-testmanager-web https://marketplace.visualstudio.com/items?itemName=ms.vss-t.... You are welcome to take your pick from what suits your needs.

dmarlow 8y ago

I've yet to look, let alone try, Azure DevOps. There would have to be some compelling reasons to choose that over TeamCity, which I have had nothing but great and positive experiences with. Any big reasons DevOps is something to consider, or is it mostly ideal for new projects?

Someone1234 8y ago

We use paid Azure DevOps for Source Control (although we're watching GitHub closely). We tried to also use Azure Boards (Work Items, Boards, Backlogs, etc). Ouch. It is a complete UI mess of disjointed ideas. Instead of implementing one thing well, they implemented two dozen things terribly. We actually migrated from Azure Boards to Office 365's Planner (KanBan) just because the UI wasn't so horrible. It is simple and gets out of your way. In fact O365's Planner, Teams, and OneNote Online products might be superior to anything Azure DevOps has produced. The only nice thing I have to say about Azure DevOps Boards is that it integrates into Visual Studio, but frankly even that is barely passable. I'm predicting they scrap DevOps completely and just use GitHub as a clean slate.

tracker1 8y ago
I don't even use VS if I can avoid it... web dev is so much smoother in VS Code, and the laginess of VS is unbearable to me. I do have a couple .Net Core projects I mostly use in VS Code and one MS SQL DB project that I have to use VS for. May want to checkout GitLab as an option as well.

devandanger 8y ago

We are using the on-premise solution with our own build agents to build iOS. Looking to move actually into Azure Devops to reduce the effort of maintenance on our own build agents. I've been using AppCenter for hobbyist projects and I feel its been a good replacement for BuddyBuild. Seems to be pretty reliable additionally we've thrown a lot of builds at Azure Devops to validate how well those hosted-agents can handle a large project which also worked out well. So all in all, we are excited about the hosted-Mac agents as well as the flexibility to BYO-agent (helps in the transition). I have seen a lot of these same problems noted in this link as well, just didn't see to prevent from getting our work done as much. So YMMV depending on your product/devops use case. Personally have been using their APIs with some custom CLI scripts to get around some of these issues.

deleted 8y ago

[deleted]

zeeshanejaz 8y ago

We migrated from TeamCity and OctopusDeploy a couple of months ago and we are already regretting it. The reliability of this system is just horrendous. The documentation is pretty much outdated, and quite a few plugins do not work. I have to write a bunch of powershell script to get it to work the way I want. And today, all hell broke loose. Since this morning, we are stuck with an error message "TF400893: Unable to contact the server. This is most likely caused by a network error. Please check your connection and try again.". This has cause a couple of release pipelines to be get stuck, with cancel option not working either. The paid "parallel jobs" are being consume in this stuck pipeline and no other project from my organisation can be built or deployed because of it. Pretty much regretting having spent so much time configuring the pipelines. We might switch it doesn't get resolved soon or if it happens even one more time.

ianceicys 8y ago

At my company we use a mixture of Azure DevOps, Jenkins, CircleCI, Github, Jira, Artifactory, and a host of AWS and Azure, and my view is that Azure DevOps (formerly VSTS, formerly VSO, formerly TFService, formerly TFS, formerly VSS---i kid i kid) is a semi-strong product that just can't get out of its own way. The New UI is a clear step backward...and the Github integration leaves a lot to be desired. If you have Github Enterprise...why use Azure Repos...why are MSFT teams moving from Azure Repos to Github...and others moving from Github to Azure Repos. The SCM platform and strategy obviously hasn't settled yet...so Gitlab is just as viable an offering that's growing more compelling by the day. The Azure DevOps hosted build system hasn't been viable in my opinion since it launched, so on-prem agents are a much better option. Take a look at CloudBees offerings, they set the poll position. The key thing that continues to be a pain point is the reporting and project organization data structure limitations with Azure DevOps: is it 1 TPC per business unit, 1 TPC for the whole company, 1 team project for the whole company. What happens when you have 2...how do you consolidate...what's the path forward if you have to move things around. The answer for the last 8 years has been silence. The request for a clean solution has been asked for since 2011. Yes 2011, look at the link below. 8 long years...and basically silence...and "it's a hard problem"...and multiple updates saying "We will provide an update once we start planning for the second half of 20XX." https://visualstudio.uservoice.com/forums/330519-azure-devops-formerly-visual-studio-team-services/suggestions/2037613-make-it-possible-to-move-a-team-project-between-te https://visualstudio.uservoice.com/forums/330519-azure-devop... Eventually, the promises just ring hollow, the proof is in shipped code, and in the meantime other platforms and competitors stacks keep building out better enterprise stories for migrating from Azure DevOps (here's to look at Github, Gitlab, Atlassian's platforms, Jenkins/CircleCI, TeamCity, Artifactory, Xebia Labs etc). Honestly, the world is a MUCH more competitive place with lots of best in class point solutions that aren't that hard to stitch together and get better performance/features. Hope springs eternal...that with the new vertical offerings the DevOps team can deliver rapid value...but 8 years is a long time to only hear crickets. If you're starting out and you need a one-size fits all solution and you're a SMB company with two doze devs...go hog wild with Azure DevOps...that's not to say it doesn't work for large companies like Shell and Microsoft itself...but talk with teams about the limitations they run into and make your own decision on your DevOps transformation adventure...oh and never create more than 1 team project in Azure Devops...seriously...if you do, game over.

thejosh 8y ago

Their build pipeline has been down all morning as well, for both hosted and self-hosted. Just sits pending. No deploys this morning, woot!

AdeptusAquinas 8y ago

I don't know... having been on openshift, Jenkins and bitbucket for the last year I'd give a large sum of money to go back to azure devops.

markbnj 8y ago

> For a few hours, builds ran smoothly. As expected, eventually the agent died during a build. Builds sat queued, my frustration grew, and AWS notified my instance needed a doctor. After an hour I gave up hope for the unresponsive server and commanded it to reboot. Sounds a lot like some process on the build server is leaking memory. We have a couple of older services that have this problem, and if we don't catch it early enough the server will become unresponsive and unable to even handle ssh connections.

mattbillenstein 8y ago

How do all of you people get on this broken stuff on Azure with GCP and AWS (I think) being so good? Is it like free credits?

throw_away2 8y ago
I can only assume windows people are used to pain
- kkarakk 8y ago
  pretty much it, also azure is slightly better at pricing in asia
  - throw_away2 8y ago
    This mimics the mac vs windows argument I had with my brother-in-law where it boiled down to the fact that you could run windows on something cheaper than a macbook. Even though in order to remap caps lock to something reasonable you have to edit the registry. In 2019. The best answer I think involves windows server 2003 resource kit and its UI, which is so old it could get a driver's license. You save money but what kind of life is that?
    - skriticos2 8y ago
      It's not even about the price. I have no problem tweaking my Linux desktop to use an international Dvorak layout (useful for those European languages with ä's and é's) and switching the caps and ctrl keys. Say what you want about Gnome 3, but this works just fine. On Windows.. I get a US only bare bone Dvorak layout and few ways to remap the caps/ctrl keys. Messing with the registry is at best annoying on my gaming (only) PC but impossible on a corporate locked down desktop. Then I'm lucky if the corporate package repo comes with AutoHotkey that can remap keys in a very hacky way. A lot of Windows 98 still shines through in the basic usability suite even on the newest versions of Windows. Even the on-screen keyboard on a (Android) phone is more fun than the HID mess in Windows.

vezycash 8y ago

It seems no one wants to learn the lesson Facebook was forced to learn - No wholesale UI changes.

lqqkout4elfy 8y ago

The silver lining here is that Azure DevOps's code review tool is lightyears ahead of all other repos. I would like to see this support for github repos

manigandham 8y ago

VM Agent based builds are an obsolete model. A docker/container-based pipeline is the current state of the art and allows for very fast builds with caching built in and isolated stages with narrow responsibilities.

GordonS 8y ago
There is an official Linux image for a VSTS Build Agent container - but, frustratingly, not for a Windows one! People have been begging for such an image for a long time now, but Microsoft do nothing... I just don't understand why they treat Windows as a 2nd class citizen!
vtbassmatt 8y ago
We offer container jobs [1] and recently added the ability to use service containers [2] as well. Disclosure: I'm a PM for Azure Pipelines. [1] https://docs.microsoft.com/azure/devops/pipelines/process/container-phases https://docs.microsoft.com/azure/devops/pipelines/process/co... [2] https://docs.microsoft.com/azure/devops/pipelines/process/service-containers https://docs.microsoft.com/azure/devops/pipelines/process/se...
- toxicbakery 8y ago
  Funny enough I had not had the time to write about this yet. This resolves one of the other issues on my list. Thank you.

BuckRogers 8y ago

I use Azure DevOps everyday and like it quite a bit. It'll keep improving but I've had no complaints. Microsoft is in a very enviable position, not just having one of the best IDEs but having control over potentially an entire CI/CD pipeline from Github to Azure, while having one of the major programming platforms in .Net to enable best-in-class integration. No one else can really say they have all of that, not Atlassian, Oracle or Amazon. The CI component, Azure Pipelines, is the only hosted CI tool that supports all major platforms, at a minimum Windows/macOS/Linux. The value of essentially fully automating out the role of DevOps engineers is a long time coming, and I used to do that sort of work but saw the writing on the wall with things like Azure DevOps. I think a lot of existing DevOps roles will transition over time to 'Cloud Engineer', specializing in setting up and maintaining hosted CI/CD platforms. There will also be a lot less of them as a result. It's becoming a commodity, MS is just one player there. I love what they're doing and eagerly embrace it, Microsoft has been knocking it out of the park. It works well for us.

stonewhite 8y ago

I remember high CPU alarms waking me up, just to see that Azure OMS Monitoring agents were consuming %99 CPU on every node of our client's database cluster. I'm not sure why that happened, yet the immediate reaction was to uninstall it and never use it again. OP's story about build agents turning unresponsive makes me think of this. MS is the best cloud vendor to deal as a consulting partner, yet the overall suffering caused by Azure almost makes me feel like it is not worth it.

Annatar 8y ago

I'm genuinely curious: what exactly is so hard about installing and system administering one's own servers inhouse that so many people would put up with these complications?

sl1ck731 8y ago
For us its: availability outside of VPN and avoiding the run around of making a public site with security, integration with active directory, has repos + a wiki + boards all in one place. No configuring 3 different OSS products to integrate with AD. We use the other stuff more than the CI, but will probably begin using it for some more smaller projects soon. It'll be in the same place which is great and no need to tinker with servers which is awesome.
- chupasaurus 8y ago
  If you have problem with configuring service users in AD, I guess you have problems with security or group policies on a regular basis.
  - sl1ck731 8y ago
    Or AD isn't our responsibility and we can just use the tool (Azure devops) that is already done before we even launch it and provides all the features we need? I guess you don't have more important things to be doing than screwing with ancillary tools.
patrickg_zill 8y ago
There isn't any true difficulty; people are running multiple sites that could all fit into 1 dedicated 64gb RAM server from Hetzner on a different infrastructure that allows them to develop their resume...
- Annatar 8y ago
  Are you saying that it's simply a case of resume inflation at employer's expense?

equasar 8y ago

At my company, we are pretty happy with Boards, Repos and Pipelines. One thing that needs to be improved so badly is their Test Plans section, not so fan for their way to setup agents for Load Testing.

fastbeef 8y ago

Look, developer tooling is always going to be hugely dividing. Some will love it, some will hate it, some will hate it because it's not like the Thing They Loved At Their Previous Job. Now, can we talk about the real WTF, which is the name. "Azure Devops", really? Two extremely potent keywords each on their own? You didn't like people being able to google answers to their questions about your product?

tracker1 8y ago
TFS, VSTS, "Azure DevOps" and often the results are outdated... "ADO" won't work either. They probably should have stuck with VSTS and just kept the abbreviation or made it "Azure VSTS" where vsts would at least continue to be searchable. Another nicety would be to pay StackExchange to add an Azure DevOps subsite, assuming there isn't already one.

timvisee 8y ago

I've had similar issues with Microsofts CI implementation. I'll describe some of the problems I've had here, note that the last time I've used it is about half a year ago, maybe things have changed though I doubt it. This was with VisualStudio TeamServices on an instance hosted by Microsoft itself. I deployed self-hosted build agents as they were orders of magnitude faster. Probably the biggest problem I've had was that build agents are stateful. The agent checksout a repository when the build starts, and removes the directory when it's done. It doesn't however clean up other changes made on the system, such as cache directories for tooling and other data you might move out of your build directory. I believe that builds must be reproducible, and therefore must start with a clean and consistent environment. You don't want builds to fail down the line because a precious job broke the agents environment, or even worse, having jobs succeed only on the agent because some dependent files are left on the agent. Other CI solutions use Docker containers for this to build in, which is great, as they are immutable (or stateless, rather) by default. This is in my opinion the single worst 'feature', making these CI builds distrustful. Secondly, automating these agents is hard. If you want to automatically scale these self-hosted build agents horizontally based on demand, good luck! When starting an agent for the first time, you've to go through some cumbersome configuration wizard to set it up. There is no simple static configuration file available to use instead, or at least, I wasn't able to find it at the time. To mitigate the previous problem with stateful agents I created a Docker container having an agent in it, which was in fact quite difficult to properly achieve. I automatically restarted these agent containers multiple times a day in order to clean-up any garbage that was unintentionally left by a job. This is a horrible hack to slightly improve their (in my opinion) broken system, but it had the bonus of making these agents more easily scalable. Another thing is the build configuration. I prefer having a configuration in a simple YAML file in the same repository, to have it as _close_ to the code as possible. Although Microsoft does provide such functionality it doesn't seem to be common to do it this way, and if I remember correctly it's much more difficult to properly set-up and use than for example, a Travis CI configuration. They do provide a graphical interface to configure these build steps though, which is useful when getting started with the system so you can get a feel of what is possible. It provides some graphical blocks/steps to achieve various thighs, such as doing an FTP upload or running a CLI command, with all kinds of weird properties. Maybe that's because Microsofts things are generally less scriptable. But yet again, I prefer a simple script in an in-repository configuration file, because it feels somewhat constrained and limited. Sadly, it doesn't stop here: - sometimes jobs just didn't want to stop, and no, bashing the abort button a million times didn't fix things. In some situations it took an hour before jobs were properly killed. - the interface (VisualStudo TeamServices) is horrible to navigate, have had to wild-click many times to find what I need. I'm sure this will improve once you use it for a longer period though. - and there are many other inconsistencies and little problems, such as build logs loading half of the time. Maybe I'm grumpy. Maybe I'm ranting about nothing. I don't know. Maybe things have changed, or will change, at least I hope it will. I didn't have a great time with their CI system, my experience was horrible as you can probably tell by now. It feels like an unfinished product, that was released to quickly just to satisfy customers, so they didn't have to switch to some other platform not owned by Microsoft for their CI needs. What amazes me is that there are quite a few free alternatives available which are much simpler and more elegant, such as GitLab, Travis and Circle CI. They are easier to navigate, are usually quite quick, don't have the issues I've (and the article have) described. And greatest of all, you can get it up and running in minutes without having to worry about it afterwards.

raxxorrax 8y ago

Interesting post. I feel for him using this tool. Had my own experience with team foundation server on premise. Although a different tool, many of the problems seem quite similar. A little rant: To the cloud infrastructure of MS: I don't know if anyone uses it for anything serious and doesn't complain about severe performance issues not only related to builds, but everything. From administrative sites to hosted environments. I read a lot of excuses mainly saying I should adjust my expectations regarding speed for cloud solutions... Problem is that there are other providers that don't have these issues. Seriously, you need only to use office 365 if you want to get an impression of bad performance. Because everything is slow. This is from a location in Europe, but I really doubt there are significantly more performant locations.

stevenacreman 8y ago
This is true of all of the services I've tried across Azure, AWS and Google cloud. Microsoft are always last place in terms of performance and reliability. For Kubernetes I actually keep track of exactly how bad they are with an automated testing tool. Results of the last test are here: https://kubedex.com/is-azure-kubernetes-aks-any-less-terrible/ https://kubedex.com/is-azure-kubernetes-aks-any-less-terribl... I've complained about UX before in the context of the Azure portal being, in my opinion, terrible. But that's subjective and I respect that others may like it, although not sure how they do.
- raxxorrax 8y ago
  This is very interesting data, thank you for that. Maybe MS has improved a bit although I haven't felt any improvements yet. But the changes might not be rolled out completely. Performance wise it can only get better in my opinion.
- dtrailin 8y ago
  You should do an updated test comparing AWS as well. I feel like there should be more continuous benchmarking efforts of cloud vendors. This is the last one I remember in this genre (https://www.azurefromthetrenches.com/azure-functions-significant-improvements-in-http-trigger-scaling/ https://www.azurefromthetrenches.com/azure-functions-signifi...). I wonder if someone could make a business doing that. Also, I've found in terms of UX GCP > Azure > AWS personally.

jopx 8y ago

I don't have much complains about DevOps UX, the only problems that sometimes impact me are the caching/random broken builds. But Azure Portal UX is another story, the Blade UI is one of the worst things I've ever used, I hate scrolling horizontally, I can't open things in a new tab, it's really frustrating.

dustinmoris 8y ago

Makes me really appreciate my setup of: - GitHub for source control - AppVeyor for CI - BlazeMeter for Load Testing - Google Cloud for Cloud Hosting - Slack for Team Chat - MyGet for private NuGet feeds - Trello for task management - ELK cloud for log management Yes it is many different services, but as a small team I like that we can pick the best tool for the job instead of trying to find a hammer for all nails which doesn't exist. It also makes us less worried about changes, because we are not invested in any vendor so much that if they do something which makes our lives too difficult that we couldn't easily move on elsewhere. Also it forces us to design our software in such a way that we don't lock ourselves into a specific vendor. For example we would have all our builds scripted instead of configured via some GUIs with the configuration stored on some vendor's cloud. The build script runs from anywhere. We can run our builds on Azure DevOps, TravisCI, CircleCi or anywhere. We just prefer to use AppVeyor because it's the smoothest CI server IMHO, but the transition elsewhere is just a matter of invoking a single build script. I genuinely also prefer to work with a product which is doing one thing really good instead of having one product which does everything mediocre. From an operations POV it is just as simple as having everything in one place. We have one browser tab open for each service and it makes us actually a lot more productive. I am always only 1-2 clicks max away from what I need. Teams/Companies who choose an all-in-one solution (e.g. everything in Azure + Azure DevOps) don't care about the productivity of their teams IMHO. They care more about the one person who has to do the accounting for their subscriptions or something like that :/

Kuinox 8y ago
Sadly MyGet is no longer free :(.
seanjregan 8y ago
I was just building a presentation on this topic and the power of an open approach.

whatupmd 8y ago

Sounds mostly like networking problems with the Azure Cloud.

toxicbakery 8y ago
I initially thought the same thing but I'm less convinced of that after trying an aws based build agent. I think the agents themselves may have a bug that combined with our pipeline needs, triggers some perfect storm issue. At the same time, we have seen builds fail to start at all and and sometimes fail to even checkout from GitHub.

kapilvt 8y ago

As a maintainer on an opensource project thats recently switched out to azure pipelines, I'm actually overall happy with azure pipelines, its considerably faster (hosted agent) for us in matrix builds then travis both on latency to start a build and actual compute power, and cheaper then self hosted drone. Hosted builds have been reliable for us, we notice a hiccup less than once a month on a single build (avg 20+ builds a day), minus a service incident yesterday. That said I agree with a lot of the criticisms noted here (no retrigger on web hooks to update status, complicated web ui, log ux, lack build cache). It is possible to bring your own cache implementation, I just stuffed/restored things into object storage, also filed an issue for build cache https://github.com/Microsoft/azure-pipelines-tasks/issues/9190 https://github.com/Microsoft/azure-pipelines-tasks/issues/91... noting several other closed issues for the same. It does feel like the platform is moving forward. All that said there's a lot of complexity and feature set in azure DevOps and pipelines that I haven't explored and try to ignore in the ui. At the moment its the only cloud provider build solution that actually even attempts to work well for opensource projects afaics.

edglas 8y ago
Thanks @kapilvt, I'm on the Azure DevOps engineering team. Are there specific improvements you'd like to see in the logs UX?

nbevans 8y ago

My mind was blown when we invested a week in setting up DevOps, then realised it has a critical flaw in that the automated build agent will just go into hibernation if there was no build activity for a few hours. So then you have to write a script that invokes a manual build via the API just to keep DevOps "alive". It's like they forgot to properly setup the IIS Worker Pool timeout values or something.

chrisrpatterson 8y ago
@nbevans could you provide a few more details here on the agent and where it is hosted? The agent itself does not rely on the IIS worker pool and we are not aware of any bugs in this area. If we have some we would love to get them addressed.
edglas 8y ago
@nbevans I'm an engineer on Azure DevOps. This is the first we've heard of this issue, can you provide more details? What error do you see? By hibernation do you mean Windows hibernation? Feel free to reach out to me at my HN name at microsoft.com.

ryanackley 8y ago

Came here to say that I've been impressed with Azure DevOps so far. We don't have the same requirements as the Author. For example, we don't deploy artifacts to a dependency management system and we don't build on Linux. I have small complaints here and there but I couldn't find another build service that has Windows and OS X agents for anywhere near what Microsoft charges.

alkonaut 8y ago

There were some enormous UX issues with the old look & feel of TFS/VSTS as well. Navigation was nearly impossible to understand, for example. While I appreciate the work they put into DevOps, it still feels like the same person that designed the Azure Portal UI has had a say in the UI redesign of TFS into DevOps and that must either mean that a) Microsoft never realized what a horrible UX the Azure portal is, or b) they do realize but they are too invested in this new "design language" to do anything, so instead DevOps adopted the same style as Azure. I'm in the process of migrating TO DevOps (on prem) so obviously I have a keen interest in the direction its taking. Build pipelines being declarative, navigation being clean up etc. are fantastic improvements. But it still feels clunky to use, especially around the backlogs/boards.

mcrommert 8y ago

Thats not 1080p

toxicbakery 8y ago
I'm not sure what you mean by this but a few other people have asked me about the described screen size problem so I decided to add some clarification. https://toxicbakery.github.io/vsts-devops/dimensions/ https://toxicbakery.github.io/vsts-devops/dimensions/

jhacker123 8y ago

I'm using Azure DevOps (Previously VSTS) more than two years, and meanwhile I create couple of Pipeline to perform CI/CD. > .NET MVC Web Application Build and Deploy Using FTP > .NET MVC Web Application Deploy in Cloud (Both Azure and AWS) > .NET MVC Web Application Build on Linux Visual Studio Agent Create Docker Image and Published I didn't face any errors from devops, my experience overall good with DevOps. but i agree that it's not user friendly in term of UI and other workflow.