8 ms·
Hydra: A framework that simplifies development of complex applications
- 52-6F-62 7y agoThe name of this library almost feels a little too self-aware...
- synt4x1k0 7y agohttps://cli.dev/docs/intro https://cli.dev/docs/intro
- thruflo22 7y agoCode examples: https://github.com/facebookresearch/hydra/tree/master/examples https://github.com/facebookresearch/hydra/tree/master/exampl...
- w_t_payne 7y agoSo ... what does it do, exactly?
- barce 7y agoIt looks like configuration management for devops. If I have an app and want it run in 3 different environments that each have rapidly changing configs, I might use this. But I won't b/c Facebook.
- chrisweekly 7y ago"But I won't b/c Facebook." Do I understand correctly that you also personally boycott React?
- tobr 7y agoAre you insinuating that it’s unimaginable that someone would boycott React because Facebook?
- erichocean 7y agoI did (and do).
- Nicksil 7y ago>"But I won't b/c Facebook." Do I understand correctly that you also personally boycott React? I do. Is that surprising, though? Given the variety of tools/frameworks to build interactive web stuff, there isn't a single thing React offers that cannot be found elsewhere.
- Existenceblinks 7y agoI do not boycott React because of Facebook. It's because its inferiority.
- aphextron 7y agoInferiority to what? I hate Facebook as much as the next self respecting Netizen, but I still use React at work because it's just the best tool for the job. I'd be happy to hear some arguments to the contrary.
- sergioisidoro 7y agoThis is nice, especially the multi run. But it seems to be a boiled down version of click at first glance... https://click.palletsprojects.com/en/7.x/ https://click.palletsprojects.com/en/7.x/
- omry 7y agoClick let's you build really nice command line libraries, but those are rigid and their structure is hard coded. A good example of what you wouldn't do with Click is logging configuration. You have a bunch of appenders, formatters and loggers and this is just not good with something like Click. Hydra actually comes with built in Python logging configuration, it's just a part of the configuration object and Hydra is setting it up for the app: https://cli.dev/docs/tutorial/logging https://cli.dev/docs/tutorial/logging You can override this from the command line (unlikely but could be useful), or completely replace it with your own logging config: https://cli.dev/docs/configure_hydra/logging https://cli.dev/docs/configure_hydra/logging This kind of flexibility is a core part of how Hydra build the config, it's not limited to it's internal config (like logging) but extends naturally to the user config as well.
- choppaface 7y agoIt might be nice if mlflow adopted something like this as a feature.
- omry 7y agoTell them to take a look at Hydra. Hydra has some overlap with what MLFlow is already doing, but I am sure MLFlow can benefit from configuration composition. Another way to look at it is: What kind of Hydra plugins can be backed by MLFlow. for example, remote launching to any cloud can be implemented as a Hydra Launcher plugin. Currently the included launcher plugin is trivial (local execution), and the more interesting launchers to launch remotely are not open sourced as hey rely on internal FB infrastructure. Supporting the major cloud providers in Hydra through plugins is definitely something I would like to see.
- saadshamim 7y agoI initially thought this was post for https://github.com/ory/hydra https://github.com/ory/hydra
- Semiapies 7y agoOr the Emacs library.
- m3kw9 7y agoComplex stuff to do supposedly complex stuff
- viraptor 7y agoThis is pretty similar to openstack's oslo.config, but without the schema for the parameters. If you want validation, and ini rather than yaml, oslo.config may be your thing.
- omry 7y agoThe config is the schema to some extent. see the section about strict mode here: https://cli.dev/docs/tutorial/config_file#strict-mode https://cli.dev/docs/tutorial/config_file#strict-mode At the moment it's not as rich as all of the other command line libraries in term of types, enums etc but it's richer what it you can easily express with it. For example, I challenge you do express logging configuration with oslo.config, or Click, org argparse. You would be writing a whole lot of code to come up with something very rigid. With Hydra composing hierarchical configs is trivial so logging configuration is not harder than anything else. What you would do with those other CLI libraries is just is either either hard code your logging config (eew) or separate it into a different config file and add logic to deal with it. In my book this logic is an example of boilerplate that Hydra can handle cleanly.
- rad_gruchalski 7y agoA framework that simplifies development of complex applications in Python. There, I fixed it.
- omry 7y agoGood point. The concepts of Hydra are not language specific though. At some point there might be something similar for other languages.
- rad_gruchalski 7y agoIndeed, the concepts are not language specific. I was hoping to see something with a grammar and a reference implementation. Looking briefly at this project, not all concepts can be abstracted but a specification with a set of integration tests would make it implementable in other languages. Edit: it surprises me every time that a project like this always goes for a single language implementation. There are plenty of good concepts in there. I know purists will say “it does too much”. But when one can crank out a complex cli app and spend half time on it thanks to things like this, I’m all up for it.
- omry 7y agoThe project grew organically. Evolution was something like: 1. Let's build a framework for model based RL, we can start by implementing the PETS paper (https://arxiv.org/abs/1805.12114 https://arxiv.org/abs/1805.12114) 2. It sure does have a lot of degrees of flexibility, and the PETS reference implementation is full of hard coded configuration. It would be nice to be able to combine configuration files somehow and override them from the command line. 3. Looking around, I didn't find anything that was quiet what I was after so I created OmegaConf (https://github.com/omry/omegaconf https://github.com/omry/omegaconf) 4. Building the framework, I evolved my understanding of what composing configuration actually means, and added additional features as I needed them. 5. The MBRL framework became really nice and useful, but was still primarily an implementation of PETS. I tried to make it a bit more generic to allow others in my team to use it for other things, and realized the better thing to do is to go for something totally generic and make it a standalone package, Thus Hydra was born. It evolved into what you see today, and it was almost exclusively my work - so at no point did I have the cycles to really support development in other languages. One of the challenges is to convince people that it's useful, It can do many things as some people point out and people can get uneasy because of that. For Hydra to succeed, people should be using it, there is no point in spending years building something no one would use. ML, which is where Hydra grew from is dominated by Python today, That's why Python is the first implementation. Building Hydra in another language is possible, especially now that there is a reference implementation, but it's a lot of work. If you are interested in trying it out I would be happy to discuss and advise.
- whalesalad 7y agoThis is one giant antipattern. I hope people don’t really use this. It does too much. Configuration? Running apps remotely? This is a big cluster of miscellaneous helper tools glued into a ball called a library. I don’t understand how it’s useful.
- armitron 7y agoIt can be a good fit for quick throwaway projects, typically found in various research groups. Python itself is a giant antipattern for anything more ambitious I would say. Go provides far more rigor and stable foundations. Rust is also there for systems programming.
- deleted 7y ago[deleted]
- whalesalad 7y agoThat’s a very interesting opinion for sure.
- crimsonalucard 7y agoMake no mistake. In terms of abstractions.. rust is a higher level language then golang. You have more safety features of a higher level language in rust than you have in golang and you get it at zero cost. The exception is concurrency primitives which is golangs only saving grace.
- peteradio 7y agoLooks hella useful to me, having developed plenty of python cli before this organizes in a way I've not really succeeded before.
- omry 7y agoGlad you like it!
- zeronone 7y agoJava Spring!!
- crimsonalucard 7y agoSounds like inheritance.
- crimsonalucard 7y agoI mean how else can you 'compose' data? Either through overriding things (inheritance) or as a Product (ConfigA, ConfigB) (Also inheritance). Technically you can do CoProduct (ConfigA | ConfigB) (not inheritance) but the logic of unraveling such a type is too much for configs so it's very unlikely Hydra does this.
- omry 7y agoIt does, it's called configuration groups. https://cli.dev/docs/tutorial/config_groups https://cli.dev/docs/tutorial/config_groups (Assuming I understand your exact meaning).
- crimsonalucard 7y agoI mean that there's only a couple of things it can be doing and inheritance is it... what else can it possibly be doing. I'm responding to a vote down.
- omry 7y agoI am actually not seeing "Hey, it's like inheritance!" as a vote down, although I am not normally reading HN so I may be naive.
- crimsonalucard 7y agoI’m responding to my own comment because it’s getting voted down, ha.
- tyrion 7y agoA genuine question. How come command line options are passed like so: python my_app.py db.driver=mysql db.user=omry No intention to criticize. Just wondering if there is any reason you did not choose the more standard way: python my_app.py --db-driver=mysql Wouldn't it be nice if all CLI applications could agree on a way to pass parameters? :S
- omry 7y agoSomething like this could be implemented, but Hydra is using argparse for a few basic flags (--help, --run, --cfg etc) and argparse would not like us passing arbitrary undefined flags. Philosophically speaking, I am not seeing a big advantage for Hydra overrides syntax to line up with getopt. What it does is different. For example, unlike with getopt - you MUST specify a value with every override, there is no support for flags like --verbose. it would have to be verbose=true, the reason is that those are editing the configuration object, and in 99% of the cases you want to put in a value and not default to true. Also, you should be thinking of those command line arguments as editing a complex object. db.driver=mysql means: in the db map, change value of the driver key to mysql.