5 ms·
Mitogen, an infrastructure code baseline that sucks less
- merlincorey 9y agoFascinating information and ideas presented here. As someone who has many years of experience (and tears) with Ansible, I am looking forward to exploring mitogen and the possibility of it being able to drop in my existing Ansible playbooks is pretty compelling.
- grabcocque 9y agoA thought occurs that by making this backwards compatible with Ansible playbooks, by having an AoT compiler that compiles a playbook down to pure Python and then feeds that into Mitogen, then as a migration path it could look indistinguishable from Ansible from the perspective of a playbook user.
- guhcampos 9y agoI believe this is not meant to replace Ansible, but work like an "engine" that Ansible and other similar tools can run under the hood. This is quite low level code even for an everyday Python developer.
- _wmd 9y agoAnsible internally abstracts all its SSH bits into a 'connection plug-in', the initial goal is implementing another one of those. So with one or two extra lines in ansible.cfg you will get a much faster Ansible, with no other changes necessary. I've no desire to replace Ansible, I'm just sick of wasting hours waiting for it to finish
- simonw 9y agoThis looks amazing. I know the "getting started" doc is missing at the moment, but any chance of a 4 line example of what I'd type on my computer to start trying this out?
- _wmd 9y agoHi Simon, The n-liner is: import subprocess import logging import mitogen.utils def main(router): # http://mitogen.readthedocs.io/en/latest/api.html#context-factories remote_host = router.ssh(hostname='myvps.mycorp.com') root = router.sudo(via=remote_host, username='root', password='god') rc = root.call(subprocess.check_call, 'ifconfig lo', shell=True) print 'Command return code was:', rc logging.basicConfig(level=logging.INFO) mitogen.utils.run_with_router(main) Which gives me: INFO:mitogen.ctx.local.22050:stdout: lo Link encap:Local Loopback INFO:mitogen.ctx.local.22050:stdout: inet addr:127.0.0.1 Mask:255.0.0.0 INFO:mitogen.ctx.local.22050:stdout: inet6 addr: ::1/128 Scope:Host INFO:mitogen.ctx.local.22050:stdout: UP LOOPBACK RUNNING MTU:65536 Metric:1 INFO:mitogen.ctx.local.22050:stdout: RX packets:323421007 errors:0 dropped:0 overruns:0 frame:0 INFO:mitogen.ctx.local.22050:stdout: TX packets:323421007 errors:0 dropped:0 overruns:0 carrier:0 INFO:mitogen.ctx.local.22050:stdout: collisions:0 txqueuelen:1 INFO:mitogen.ctx.local.22050:stdout: RX bytes:109087018382 (101.5 GiB) TX bytes:109087018382 (101.5 GiB) INFO:mitogen.ctx.local.22050:stdout: Command return code was: 0 Seems context name generation has broken :/ But it's running on the remote machine, you get the idea. I'm working on a "mitop" example utility that implements a multi-host top replacement, with hierarchical statistical roll-ups, and documenting the implementation of that will be the basis for the Getting Started guide, but it depends on a bunch of stuff I'm not ready to work on (async connect), while there are still critical issues where hangs may still occur due due to an exception, and basic stuff like module preloading being incomplete (currently day 13 of working on that). It will come eventually, but it will likely encourage many more bug reports than my hands can respond to once it is written, so it is kind of a feature that it is missing right now. :)
- simonw 9y ago"How Mitogen Works" is fantastic reading - http://mitogen.readthedocs.io/en/latest/howitworks.html http://mitogen.readthedocs.io/en/latest/howitworks.html - an unbelievable collection of devious hacks.
- alexnewman 9y agoI think infastructure as code is mostly overrated. I can't tell you the amount of startups wasting time cargo culting best practices without knowing why or how. Most people would be better off just launching shit and keeping it simple
- lowmagnet 9y agoIt's super useful if you embrace disposable environments. The easier it is to build a system or cluster, the more different situations you can assist your internal testing teams in replicating.
- toomuchtodo 9y agoMost environments that persist data are not disposable. Works great for web servers and microsevices though.
- talideon 9y agoIf you're doing something with a couple of machines and it's simple, sure. However, the benefit of 'infrastructure as code' is that you have a _description_ of your infrastructure. Once you get above a small number of machines, keeping things consistent (because humans are humans) when you're managing pets, not cattle, is difficult and time-consuming; rolling out changes is difficult and time-consuming; understanding why something is the way it is is difficult and time-consuming. OTOH, if you have descriptions of your infrastructure in version control, you can read a description of how everything is set up and know how a given machine or VM with a given role ought to look. You can also dig through history, even if the people committing changes were lazy, and get an idea of why something ended up the way it did. Sure, doing it as a cargo cult is a bad idea, but that's a criticism that can be levelled at just about anything a start-up might do with regard to the infrastructure side of things, because they'll tend to skimp too much on people with an ops background. But that doesn't mean that it's overrated, just that it's something that you should understand the reasons for doing before doing it.
- jhundal 9y ago
- e12e 9y agoCool. This is essentially about creating a single image cluster that runs python. I suppose if one can accept the need for python and a python module be installed on all "nodes" - this would start to look a lot like parallel python: http://www.parallelpython.com/ http://www.parallelpython.com/ From an architectural point of view, an si cluster could be viewed as code+communication (+data, but the data could be "wrapped" in code). Ssh and shell provides this - shell basically wraps libc and binary blobs as callable code (built-in + assumed-available code) - and if the cluster isn't too heterogeneous can be augmented with anything from c/go code (compiler assumed available) to machine code (scp /usr/bin/<someprog>...). And it is of course possible to build this on top of python. But the result can be much more than "just" for infrastructure - if it really works it becomes a transparent cluster platform.
- _wmd 9y agoA zero byte Python script can grow up to be anything in the absence of constraints, the infrastructure label makes it simple to say no when thinking about including a feature, and to ease selecting one implementation strategy over another. The nerd within would love to implement subtree detachment ( https://github.com/dw/mitogen/blob/master/docs/images/disconnected-subtree.png https://github.com/dw/mitogen/blob/master/docs/images/discon... ): this is basically the beginnings of an Internet worm, but it has zero practical use for infrastructure, so it's complexity I'd prefer to avoid. Commercially, the killer feature for me in this library will be its Python 2.4 support. I need that for consulting work, but that constraint stands in direct contrast to a general purpose framework for use in application development. Developers want shiny new Python 3 and suchlike, commercially I've never needed that and will be unlikely to need it for a few more Debian/RHEL/Ubuntu releases (I'm guessing, probably 2019) I don't want to rewrite Pyro4 or end up with a design as complex as CORBA (with all the pain transparent object proxies entail).. it should have just enough complexity to make it ideal for the target use case.
- sudosteph 9y agoNice! I have been doing some experimentation with running ansible on AWS Lambda for some time now, and now I'm wondering if this wouldn't be even better for that. I had to pin ansible to an old version and do some weird stuff when building the deployment zip, but it does work. I'm not all that tied to my playbooks, and actually would prefer to write in python rather than yaml (incidentally this is the same reason I still like chef's use of ruby so much, even though ansible being agentless is more useful to me) Seems like this could potentially do the same thing, even faster. Though I admit I have not dived in all that deep yet. Anyone tried it or see anything that would prevent it from running that way? If not I guess I'll give it a go this weekend.
- noir_lord 9y agoAnsible is strange like that, lots of stuff only works if you don't touch it after you get it working, for a period of time the 'apt' module was broken so I had to do similar. It's fantastic software but it can be a bit fickle.
- johnpython 9y agoWith Kubernetes and other container schedulers gaining popularity and the host OS requiring less effort to be production ready, there is much less need for configuration management tooling. I don't see a lot of appeal for another tool in this space.
- _wmd 9y agoThere'll always be a need for better tooling in this space: what configures your Kubernetes cluster?
- shoo 9y ago> given a remote machine and an SSH connection, just magically make Python code run on that machine, with no hacks involving error-prone shell snippets, temporary files, or hugely restrictive single use request-response shell pipelines, and suchlike. I wonder if some of this functionality would be easier with a language that inherently supported distributing functions between nodes, e.g. Joe Armstrong's "favorite Erlang Program", the Universal Server: https://joearms.github.io/published/2013-11-21-My-favorite-erlang-program.html https://joearms.github.io/published/2013-11-21-My-favorite-e... I am missing a few of the points, re: * how to bootstrap this onto a system with very limited assumptions about resources/access * performance * market effects of being able to re-use a bunch of existing libraries/scripts/tools written for configuration management or what-have-you in popular python edit - and there's documentation for erlang over ssh: * http://erlang.org/doc/apps/ssh/using_ssh.html http://erlang.org/doc/apps/ssh/using_ssh.html * http://erlang.org/pipermail/erlang-questions/2007-January/024715.html http://erlang.org/pipermail/erlang-questions/2007-January/02... * http://erlang.org/doc/man/slave.html http://erlang.org/doc/man/slave.html
- marmaduke 9y agoThe appeal of Ansible is that you can declaratively spec infrastructure and it happens (more or less). I don't see how this competes with that.
- _ix 9y agoIf I understand it correctly, this is intended to replace the dependencies or subsystems of applications like Ansible rather than Ansible itself.
- anentropic 9y agoThis looks very clever, but is it a good idea?