9 ms·
System Initiative: Second Wave DevOps
- convolvatron 3y agoAt least a couple words about what that might look like?
- ericand 3y agoThe homepage has a video demo: https://www.systeminit.com/ https://www.systeminit.com/
- ericand 3y agoCheck out Adam's "What if infrastructure as code never existed" talk from a month or two ago. Great framing for SI and entertaining. https://www.youtube.com/watch?v=5lPa2U239C4 https://www.youtube.com/watch?v=5lPa2U239C4
- holoway 3y agoHi! Adam Jacob here. Happy to answer questions!
- skywhopper 3y agoWhat does the performance curve look like with larger numbers of resources? ie, How soon does it get bogged down? 1,000 resources? 10,000? 1,000,000?
- holoway 3y agoIt’s too early to know with real data. Architecturally, it should scale - but you know it’s going to need optimization when we start getting into high numbers.
- 2023throwawayy 3y agoI applaud the effort here, but just looking at how messy the graph is to deploy one docker image doesn't exactly make me want to try this for anything with a level of complexity beyond that.
- holoway 3y agoThis is a reasonable reaction! :) There are a lot of ways to make the visual interface scale - examples from things like Blender, Figma, etc. We believe we can use them to make it scale semantically as the complexity climbs.
- chologrande 3y agoAfter reading this post, I've browsed the site. I'm not sure how this is anything but significantly worse than the current model? I've been around long enough to know that any "no code" style interface or GUI are typically the _problem_ not the solution. Regardless of the code they export, you end up with fat fingers, misclicks, forgotten UI paths to follow... Taking a software eng approach to shipping infra is a stable, known process that the infra team and the software teams can understand, no specialized GUI tool knowledge required. I've been using the same basic terraform modules, jenkins pipelines, and infra architecture for nearly 7 years across multiple companies and numerous cloud deployments. It's not fancy but it justworks.jpg. Every time I re-use that code for a new deployment or account I save TONS of time. Devops doesn't have to be hard. Infrastructure doesn't have to be complex. Deploying every day isn't _that_ difficult. KISS Method is key, especially when you're looking for speed. Using _less_ tools from the CNCF is better, and will let you move faster, not adding a new one.
- totallywrong 3y ago> Devops doesn't have to be hard. Infrastructure doesn't have to be complex That's simply not true for anything larger than a few services and a small dev team. The cloud is very complex to do right when you focus on security, performance, and scalability. And Terraform invariably devolves into a nightmare when you have a ton of resources with dependencies between them.
- chologrande 3y agoI'm definitely not google scale, but we're global, in over 300 cities spanning ~30 countries. On an avg day we process well over 25k rps on multiple services. Simple architecture and IaC like terraform is exactly how we manage the dependencies. It's the solution, not the problem.
- dipperdottydoo 3y agoYou think you have simple architecture when you’ve introduced Terraform to what is, based on your statistics, a two server use case. A PlayStation is capable of 25 kRPS and probably its data iops, too. Buy another one and you’re HA. You’re trapped in the complexity of the method and think you’ve achieved nirvana. This comment reminds me of those demos when Hadoop was the rage, where people would do a $4 million Hadoop ETL on their laptop and shut up a room.
- aftbit 3y ago>Doing “DevOps work” is unquestionably the worst part of building a modern application. It’s full of tiny papercuts, indignities we suffer in our toolchains, our feedback loops, and our software. It’s a city of brutalist buildings filled with sharp-edged couches pretending to be comfortable. Think of all the advances in how we interact with tools in other domains - then take a look at the way you build, deploy, and operate your software, at all the crazy gyrations you use to glue it all together - and ask yourself why you accept it. I dunno, usually I find databases and migrations to be the hard part. At this point, I have enough examples of app deploys that I can have a new app up and running on a pair of VMs with a robust blue/green deploy and backups inside of an hour or two, with deploy by Github Actions responding to pushes to prod branch. Even if you don't have my company's half-decade worth of example devops, you can do something easier, like a single instance on a Digital Ocean machine with deploy by "ssh -A server 'cd yourapp && git pull && sudo systemctl restart yourapp'". Sure, you'll have a few seconds of downtime, and you'll expose your SSH keys to anyone on that box for those few seconds, but if you know some Linux and nginx, you can get this working inside of an hour from scratch.
- hinkley 3y agoFundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups. They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of cards, which is a kind of hypocrisy that makes even mediocre devs bristle. Don’t lecture me on robust software, boyo. Your tools are made of string cheese and staples.
- pmoriarty 3y ago"They historically took on the reliability role, of nobody else did, but they were implementing reliability on top of a house of cards, which is a kind of hypocrisy that makes even mediocre devs bristle. Don’t lecture me on robust software, boyo. Your tools are made of string cheese and staples." I don't know why you'd blame ops for the crappyness of the tools they have at their disposal. Yes, Ansible, Salt, Puppet, and Chef are spaghetti-code inducing congealed messes of design. So are large collections of complex shell scripts. So what's the alternative? What spherical cow of a configuration management tool from Platonic dev heaven shall be foisted on us this time? I'm sure it'll be super clean and elegant this time, unlike the last thousand shitty tools they made. And don't get me started on devs that think they're qualified to do ops when all they know is their language of choice (if even that) and have never thought about the network, security, capacity, redundancy, failover, reliabililty, hardware, backups, the rest of the company or other users.
- holoway 3y agoIf you want to know more about some of the technical details, we wrote something up: https://www.systeminit.com/blog-five-breakthroughs https://www.systeminit.com/blog-five-breakthroughs
- deleted 3y ago[deleted]
- nickstinemates 3y agoThe core value proposition is really valuable and the demo video nailed it. Change propagation as requirements change is ripe for error. Have been following progress, excited for where it goes.
- mailund 3y ago> Things like using source control, shared observability, feature flags, dark launching, continuous integration, and continuous delivery are widely considered best practices. I seriously want to know which places this is! I've been at 5 different companies, and I've never been a place where people don't look at me like I'm speaking French when I suggest dark launching a feature or introducing feature toggles. I've yet to experience a place that actually integrates continuously, as opposed to merely having a ci pipeline without actually doing continuous integration.
- peteridah 3y agoI feel you – I had heard a _lot_ about dark launching and never actually worked at a company that actually did it until I worked at Hashicorp. Now in my mind it seems mainstream. I truly am in a bubble.
- esafak 3y agoIt's mainstream in Silicon Valley.
- mailund 3y agoInteresting! I'm not in SV, but consulting in a European city that portraits itself as having a fairly advanced tech community. I've yet to encounter anyone on a team I've been on that is familiar with it Didn't know the differences could be that huge.
- arpyzo 3y agoI don't understand why the prevailing opinion is that it's acceptable for software development to be complex. It's simply the nature of the beast! Not infrastructure though. Infrastructure should be simple because...? Perhaps the reason your organization is only deploying once a month is the same reason it takes it a month to make simple code changes, which is because you haven't hired sufficiently capable engineers, and not because you're missing some magic. edited: grammatical error
- deadeye 3y agoPerhaps we don't deploy six times a day because we're responsible for something a little more important and delicate than a free photo sharing website.
- negus 3y agoI guess GitHub is enough important and delicate. How often do they deploy?
- esafak 3y agoI think the new wave of devops is called platform engineering and developer experience.
- anotherhue 3y agoWe could begin by accepting that operations work often deals with far more complex problems than feature work. Yes, the tooling is bad, all tooling is bad, but hearing the same old 'Infrastructure should "just work"' trope is getting old. Such developers should stop grandstanding and roll up their sleeves. Learning about TCP isn't beneath you.
- Coryodaniel 3y ago> Learning about TCP isn't beneath you. Its literally beneath you as an application developer in the TCP/IP stack :budumptss:
- Graffur 3y agoSo it is Flickr I have to thank for this devops nonsense
- Mutlut 3y agoI'm very curious about this as normally all tools i know still have a higher entry point than i realize. My current setup is 'get a k8s cluster spup up and configured properly as fast and easy as possible' and than just use argocd. Argocd is by far the best tool i have been using in the last 15 years: It does exactly what it should do (syncing and showing me k8s insight vs my git repo), can manage itself through the same mechanism (IaC) and people of different backgrouns are very fast in using it. This tool either might bridge the gap for people and potentially solve problems but i do have to say: argocd. Even if you think you want to start small and just use kubectl: start with argocd.
- solatic 3y agoThe tools aren't the problem, leadership and culture is. Tools can't fix problems with leadership and culture. "Executive buy-in" is conspicuously missing from the post. When companies are "doing DevOps" by wiring up manual deploys and manual approvals in GitHub Actions, it's not the tool that's at fault. When developers are applauded for deploying once a month, it's not the fault of the YAML engineers who were hired to build the pipeline that's used once per month, it's the fault of the executives putting their hands together and clapping. No tool in the world is going to convince an executive to trust their people, to take risks with uptime and stability, and to break production as a necessary part of organizational learning. No, that requires executives to feel supported by other executives. Tools do not create collaboration and trust; people do.
- oofnik 3y agoI'm happy to see someone really trying to color outside the lines with deployment tooling. I think we've fallen into a number of paradigms for system operations that we know are kind of bad, but we tell ourselves about how much more awful it used to be to numb the pain. That sort of attitude is the real killer of innovation. I say bring it on; more variance and more disruption in this space as people try new approaches might be what we need to get us out of the rut we've been stuck in for too long. No idea if it will work, but good luck to Adam and his team. https://youtube.com/watch?v=5lPa2U239C4 https://youtube.com/watch?v=5lPa2U239C4
- rossmohax 3y agoThis talk is must see to understand what SI tries to achieve.
- trabant00 3y agoAnother attempt by Dev to make Ops a commodity? I was there when Microsoft announced ~20 years ago they're doing away with sysadmins with a GUI. Then Suse tried as well if I remember correctly. The conferences erupted with anger and booing, I was shaking my head and laughing. Then came the great YAML plague and we had to give up our title and general purpose languages in favor of silly names, templates and DSLs. But you still have to understand the OS, the hardware, and have real world experience with availability, redundancy, etc. So the new generation of "DevOps" who was raised directly on terraform and k8s failed miserably in achieving any results. Anybody saw any junior Ops (DevOps, SRE, GitOps, wtfeverops) job openings in the last few years? No? I wonder why. The tools are better, no? It should be easier than ever to deploy. We have all this micro-service orchestration and all those beautiful public clouds. All the conferences and the marketing are saying it's a breeze. You don't have to worry about ha, replication, iops and so on, we'll put that on your bill thank you very much. So here comes Dev again with a solution: click-click-drag Ops. Surely this will fix things, surely you can now hire right from the street and train to deploy. Or will my old admin ass get even pricier as the demand ever rises and the supply is dwindling? Stay tuned to find out.
- holoway 3y agoNothing about SI is intended to make ops a commodity. This is a power tool built by Ops people.
- rossmohax 3y agoI like the appeal of model being bidirectional. Also modelling sequence of actions is really not solved problem in Terraform & Pulumi: canary change, check metrics, rollout to the rest of the region, check metrics, then all regions, if they solved it all while being "declarative" and high level it can be the next tool of choice for me. I am not worried about UI representation of the model like many comments, it is not the main point of this project as I understand. UI just that - a representation, same relationships might as well be coded in HCL or the like of it.
- erilbeth 3y agoIt is never about the tools. it is the people who creates problems. examples in my case: * A manager whose dev team cannot meet any schedule is starting to blame devops. * A dev architect implements a brand new bleeding edge tech stack and ranaway, devops people have to wake up every night to fix poor-designed software. * A boss who doesn't hire a sec team, wants pci-dss passed, guess who's gonna do. * A team lead who abuses you, calls the company who hired you and tells them don't hire it, it is useless. I've been working as a devops engineer for 6 years. I'm done. I quit and never gonna do this job again. Good luck with creating more complexity with tools.
- nathants 3y agowhat makes infra hard is the large delta between what you think exists and what actually exists. the larger the delta, the worse everything gets. the only mitigation to this is less. less tooling, less infra, less abstractions. you want that delta approaching zero as uptime goes to infinity. i’m not sure how replacing walls of code with walls of yaml of walls of gui graphs change anything at all. i suppose it’s possible some paradigm leap in infra understandability is hiding in crazy nextgen ui/ux, but i’m not holding my breath. the last leap i encountered was moving from the python sdk to the go sdk for manipulating aws. this was significant, but still more of a qol improvement than something fundamental to the solution space.
- silok 3y agoI think it looks pretty promising, as long as the deployment setup is not only configurable via the UI, but also can be specified decoratively with code and hierarchical data (eg json). IaC is a really powerful concept, and system initiative does not need to be in conflict with that paradigm, just another layer of abstraction that still allows IaC. The main issue is how to combining UI state + and manual state. The worst thing you can do imo, is to use a common representation, eg the UI would try and edit your manually written declarations. That is just a recipe for disaster. The answer to this type of mixed editing is a layer approach, eg what is being done in the USD format (https://openusd.org/release/index.html https://openusd.org/release/index.html) Each authoring "instance" has full control of its layer, and composition semantics define how the layers compose to the final declarative structure.