13 ms·
'New' Python modules of 2015
- pougetj 11y agoprogressbar2, can also takes iterables as input, for easy display of progress bars
- maxerickson 11y agoThere's something to be said for the nearly non existent api provided by tqdm.
- scrollaway 11y agoNot really 2015 but Q! https://pypi.python.org/pypi/q https://pypi.python.org/pypi/q Print-debugging on steroids. This really does make things so much easier, especially when dealing with huge apps you don't have time to learn. Not just useful as a dev but also as a sysadmin.
- wesleyy 11y agoWhat's the advantage of Q over pdb?
- scrollaway 11y agoThey don't do the same thing. Q does printing. import q; q(var) -> prints var to /tmp/q, with syntax highlighting, separate files for big output, etc. It can also do lots of other cool things, cf 1-page documentation in the link. :)
- StavrosK 11y agoFirst of all, pudb is fantastic, just use it over pdb all the time. q is for when you want to log data, pudb is for when you want to step through and evaluate lines in-context. It's very possible that you'll want to use both together.
- willvarfar 11y agoObiwan pypi.python.org/pypi/obiwan/ validating JSON also: type checking on function signatures
- vram22 11y agojsonschema: Using JSON Schema with Python to validate JSON data: http://jugad2.blogspot.in/2015/12/using-json-schema-with-python-to.html http://jugad2.blogspot.in/2015/12/using-json-schema-with-pyt... https://pypi.python.org/pypi/jsonschema https://pypi.python.org/pypi/jsonschema http://json-schema.org/ http://json-schema.org/
- lunchladydoris 11y agoInteresting list. I love Anaconda! A few years ago I tried to set up a Mac with a scientific computing stack and it took me days to hack my way through all the various dependencies and incompatible versions. Anaconda now lets me do that in a minutes.
- chestervonwinch 11y agoNo kidding! Thankfully brew makes this more pleasant.
- danso 11y agoMe too...I taught a python class by making everyone download Anaconda's distribution of 3.x...and everyone could do the assignments no matter what kind of computer they used. Anaconda does a little too much for me to have it be my own default install but it does quite well in on boarding beginners. I use pyenv to install maintain Anaconda on my own machine when I need to replicate student work
- bmer 11y agoWhat does it do that ends up being a little too much for your use?
- danso 11y agoIt takes precedence in the path over everything...and in the last version I used (before I upgraded to OS X El Capitain and wiped out everything), things like `curl` were provided [1] ...which I completely understand for Anaconda's use case, but it caused a lot of confusing grief to me when I hadn't expected that and OpenSSL was having its rough times. I don't know if that's the case (curl being part of the package) now, with Anaconda 3 2.4.0+? It certainly isn't so when installed via pyenv, so I'm happy with that. But there were other issues in the past build...BeautifulSoup was inexplicably broken. I mean that it simply did not correctly parse non-trivial HTML pages and yet threw no errors. The results could be replicated for all of my students but I never could isolate the issue... I installed Python 3 and the same version of BS4 from scratch and had no problems, but I can't imagine where the Anaconda build would have gotten wrong. It ended up being OK since I just switched to lxml which I now happily use over BS4 on any day, but it was frustrating to not be able to diagnose the problem (I didn't get a response in the support forums either). I'm assuming this problem has gone away in subsequent versions of Anaconda though I haven't tried since lxml is perfectly fine to me. And finally...well, I have to admit it, but I use Python like a goddamned moron in that I still don't know how to use virutalenv/venv to do proper dev isolation. And from the brief research I did, I see that Anaconda has its own conventions, or work flow...something with the conda utility. Again, I can see why it's necessary for Anaconda's use case (people who want to do data science and not hand-tweak their environment every time they upgrade a package over pip), but it added too many layers for me at the time. [1] https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/XCSUmZbNqn4 https://groups.google.com/a/continuum.io/forum/#!topic/anaco...
- baldfat 11y ago> I went looking for a pure-Python NoSQL database and came across TinyDB…which had a simple interface, and has handled everything I’ve thrown at it so far! Why would anyone need a simple NoSQL? Why would you go the NoSQL route if it isn't a HUGE complex database?
- mavidser 11y agoIt's really good for prototyping things, actually.
- GFK_of_xmaspast 11y agoYou've already got a perfectly fine KVS built into the language.
- nashequilibrium 11y agoCould you name it?
- vram22 11y agoThis is probably what he meant: $ py Python 2.7.8 |Anaconda 2.1.0 (64-bit) ... on win32 >>> import bsddb >>> print bsddb.__doc__ Support for Berkeley DB 4.3 through 5.3 with a simple interface. For the full featured object oriented interface use the bsddb.db module instead. It mirrors the Oracle Berkeley DB C API.
- nashequilibrium 11y agoAhhh... Thanks, never knew that!
- alelos 11y agobsddb is deprecated as of python 2.6 and removed in python 3
- 11y ago
- bbayer 11y agoI am using Scrapy a lot. http://scrapy.org/ http://scrapy.org/ It is very well designed web crawling library.
- vaibhavkul 11y agoNo Python 3 support yet?
- elyase 11y agounfortunately not yet, just tried it yesterday. There has been some progress but it looks like it might still take a while [1]. [1] https://github.com/scrapy/scrapy/issues/263 https://github.com/scrapy/scrapy/issues/263
- IanCal 11y agoAnother vote for scrapy, massively useful for web crawling. Particularly love the shell for debugging things.
- scrollaway 11y agoI found out, through reddit a couple of days ago, about Pomp: https://bitbucket.org/estin/pomp https://bitbucket.org/estin/pomp It looks like a much cleaner Scrapy-inspired spider framework, without the twisted dependency. And it's python 2+3 compatible. I'm very excited to try it out.
- bbayer 11y agoIt looks very promising. I will give it a try.
- daturkel 11y agoJust seeing this for the first time. Got a proof-of-concept demo working in no time (after fussing with install requirements...). Looks like a great tool for non-devs like me who still need to scrape things occasionally for data collection and analysis.
- 11y ago
- rffn 11y agoThe Python module I learned to love this year is Click. Gets me better command line interfaces fast. URL is http://click.pocoo.org http://click.pocoo.org. It does progress bars too...
- danso 11y agoWow, thanks for this...I was literally in the middle of writing a CLI tool to fetch a URL and parse out metadata and was getting a little tired of the argparse route. I browsed the documentation and honestly can't say that I immediately grok the advantages but given who its author is, I'm more than happy to switch libraries in mid-coding :)
- w_t_payne 11y agoI'm loving Click too.
- agumonkey 11y agodocopt was inspiring in its own way, but click was 'simpler' do toy with.
- gh02t 11y agoI find click to be much more natural too. Docopt was always touchy about more complicated CLIs when I used it (probably because a lot of "magic" happens behind the scenes), whereas click lets you drill down and arrange things just so. It also feels very well designed, hats of to Ronacher as usual for being good at designing Python libraries. I also find myself using click even when I don't want a CLI. The pretty-printer (`secho`) and progress bars are extremely handy, plus some of the other stuff in utilities. It's quite nice that they handle detecting when output is an interactive terminal versus piping to a file.
- ibejoeb 11y agoI'm looking for a new CLI library. I like Click because it does very simple things very well. It gets hairy when you want to build more complex CLIs. For example, value options and validators don't play nicely because Click doesn't distinguish between the absence of a value and an invalid value, so you wind up dropping Click features and rewriting your own plumbing for that kind of stuff. The alternative is writing a more verbose CLI grammar, which leads to a really clunky UI.
- emehrkay 11y agoI really like lists like this. I get updates daily on which of my github friends (is that what they're called?) have starred and there is no real reason why they're following a project. I can look at the README and guess. I did see someone start following this project the other day https://github.com/elastic/elasticsearch-dsl-py https://github.com/elastic/elasticsearch-dsl-py, which seems pretty interesting. Has anyone used it?
- jefurii 11y agoIt's something like Django models but with Elasticsearch. You can create object classes and then save them to Elasticsearch, query them, etc. It's built on the lower-level https://github.com/elastic/elasticsearch-py https://github.com/elastic/elasticsearch-py. Very handy.
- wclax04 11y agoWe've transitioned our local/dev/prod instances to use conda on Heroku, and couldn't be happier. It was a tiny bit of work to get it set up, but now everything is consistent, and we can set up new local environments in seconds.
- sandGorgon 11y agoSo I have been considering this. does conda track pypi or does it lag it? I have been concerned about moving over my requirements.text for a webapp with lots of dependencies
- wclax04 11y agoIt slightly lags, but you can include pip requirements in an environment.yml file, and they install normally. I really only use conda for the non-python bits of our stack: numpy/scipy/pandas etc - packages that are a pain to install on Heroku.
- gh02t 11y agoIt's also pretty straightforward to set up your own Conda package tree. Nice for packaging your app for deployment or making sure you have very precise dependencies. http://conda.pydata.org/docs/custom-channels.html http://conda.pydata.org/docs/custom-channels.html
- sandGorgon 11y agoI think deployment is a solved problem with docker. Its libraries like blas,etc that are a huge pain. I'm not sure why static linked bumpy is not possible - even anaconda could not achieve it.
- gh02t 11y agoIf you've ever tried to dive into the NumPy build process you'd see why. It's unbelievably complicated... not that they really could do it better given that they are compiling about a billion scientific libraries and support alternatives and optimizations (like MKL).
- teekert 11y agoI'm feeling a lot of love for Pandas. Any (biology related) project I work on starts with multi-headered dataframes and ends in beautiful Seaborn graphs. In combination with Jupyter notebook I breeze through large data sets while leaving a perfect trail of what goes on in the data pipeline. Python is great.
- gh02t 11y agoSeaborn is great for visualization; it basically packages up some of the more specialized R plots for MPL. The other thing I really love about it are plotting contexts, which make it really easy to properly size and format the same plot for e.g. a poster and a paper. Interestingly enough, my biggest use of Pandas is to serialize to and from HDF5. I work with a lot of large datasets and Pandas simplifies using HDF5 quite a lot.
- chestervonwinch 11y agoFor those interested, you can also get nice styles with just matplotlib (including seaborn style) using stylesheets: http://matplotlib.org/users/style_sheets.html http://matplotlib.org/users/style_sheets.html
- mrswag 11y agoI had no idea it existed, very useful, thanks!
- SalmoShalazar 11y agoI've been loving Pandas used alongside Seaborn as well. It's really just so easy to manipulate/visualize my data (and make it look gorgeous) with the combo.
- IndianAstronaut 11y agoPandas are great. Looking forward to Dask.
- bru 11y agoLooks down (503), here is a cached version: http://webcache.googleusercontent.com/search?q=cache:Bf9Iv63AAD0J:blog.rtwilson.com/my-top-5-new-python-modules-of-2015/ http://webcache.googleusercontent.com/search?q=cache:Bf9Iv63...
- PLenz 11y agoFolium + Geopandas is my new goto GIS toolkit this year
- atmosx 11y agoAre there any ruby alternatives to tqdm?
- Hortinstein 11y agoor node?
- bpicolo 11y agoIt's only around 100 lines of code. Should be easy to replicate in language of choice: https://github.com/noamraph/tqdm/blob/master/tqdm.py https://github.com/noamraph/tqdm/blob/master/tqdm.py
- chippy 11y agoAlso interested in the answer, because if there are none, it's something I could look into doing over a weekend!
- StavrosK 11y agoHands down, my favorite new library is schema: https://pypi.python.org/pypi/schema https://pypi.python.org/pypi/schema Here's a schema I use in production, see how readable it makes the parameters of the API and how quick all the validation and normalization is: https://www.pastery.net/mhwwnv/ https://www.pastery.net/mhwwnv/ At the end, you get an object called data, and you can do data.title, data.language, etc, and be sure that everything is as you expect.
- deleted 11y ago[deleted]
- lsh 11y agoAny relation to the Clojure Schema? https://github.com/Prismatic/schema https://github.com/Prismatic/schema
- StavrosK 11y agoPossibly? It certainly looks similar.
- bpicolo 11y agoOptional being an attribute of the key rather than the value is pretty bizarre to me, I've seen that patten a couple times. Feels like the wrong way to go about it
- StavrosK 11y agoI see what you mean, but it's the key that's optional, not the value.
- bpicolo 11y agoAhh. That makes sense then. The value of the key being optional is indeed a different concept. Fwiw, this is one huge downside of Swagger APIs. Swagger supports the concept of optional keys but not optional values >.> I have yet to find a validation library that supports all the sorts of things I expect it to. One place they tend to fail is the ways they can fulfill default values. Say a field has an error and I want to just give it a default value when it's broken? That particular feature doesn't exist in any library i've found so far (for python) =/ Granted, this sort of blends into the usual "It's a validation library not a serialization library". But they all make a half-assed attempt at the other side in my experience.
- krylon 11y agoI haven't used Python in a while, but I shall once more look into it. Python was always fun. Guess I'll do that over christmas, see where it takes me. I might even get around to learning Python 3 after only ... what? seven-or-so years?
- StavrosK 11y agoJust write the next thing in Python 3, there's not really that much to learn right away as much as there are minor surprises that you can very quickly get up to speed on as you encounter them.
- fit2rule 11y agoI have been a bit 'off Python' for a while, but this list article prompted me to take a renewed look at it because of Jupyter Notebook, and I have to say I'm quite impressed .. this is a really nice way of working on code, wow .. especially using folium this way is very cool.
- amyjess 11y agoI've been dragging my heels on this for a long time, but I'm finally starting to take the plunge into 3. I think for any greenfield project, there's very little reason to use CPython 2 anymore. If you want performance, use PyPy. If you want features, use CPython 3. From now on, that's the philosophy I'm following whenever I write something new.
- bpicolo 11y agoWhile we're on the subject of python modules: sqlalchemy is, and will probably continue to be, my favorite library for any language.
- bubalus 11y agoAnaconda has been a lifesaver, because it can be installed and managed quite easily without root privileges (it even installs pip). Some of the sysadmins where I work are slower than molasses when it comes to installing python packages (as in, it takes months of repeated emails from multiple people to get anything done), and what is installed is often years out of date.
- xyzzy4 11y agoThis is a tangent, but the most annoying change in the latest Python versions is you can no longer write print "foo". Now it has to be print("foo"). Damn kids ruining my language.
- dragonwriter 11y ago> This is a tangent, but the most annoying change in the latest Python versions is you can no longer write print "foo". Now it has to be print("foo"). The statement-to-function migration for print is, IMO, generally an improvement, but in any case its not a change in the latest versions of python, except with an unusually broad interpretation of latest; its a change in Python 3.0, which was released a little over 7 years ago.
- zo1 11y agoI switched to Python three a few months ago. And it still gets me and I end up typing "print variable", only to have Python complain. It's probably stuck with me because of it's simplicity, though I do understand why they've removed it. It's an aberration in the syntax, for lack of a better way of putting it.
- njharman 11y agoI was always way more annoyed having to remove/add parans when I swapped a logging, Exception, or write to a print. It was a glaring inconsistency.
- JustSomeNobody 11y agoYou just wake up Rip Van Winkle?
- robohamburger 11y agotqdm looks super promising. progressbar and progressbar2 end up being complicated and weird enough to use that my company ended up making wrappers. Why maintain that when you can just use a library that works out of the box. It would be great if it had ipython notebook support. I often end up doing long operations that scrape services for data but have no idea what their progress is. For me 2015 has been the year of tox. It is a great tool and worth using for just about any python project.
- hiciu 11y agotqdm does have ipython notebook support :). Progress is displayed in stderr field under cell. But no progress bar AFAIR, just textual information.
- toyg 11y agotqdm is failing for me on Windows at the moment. To be fair, it might not be its fault (I'm mixing it with Blinker signals), but still I'm slightly disappointed. A lot of these "magic" tools fall apart when you're trying to do something slightly more structured than "throwaway bunch o' functions".
- antman 11y agoEasy platform independent gui as webpage. Used it on my pc a robot and raspberry pi. https://github.com/dddomodossola/remi/blob/master/README.md https://github.com/dddomodossola/remi/blob/master/README.md
- Drdrdrq 11y agoPyrasite. When you have a running python app that is behaving oddly and you can't replicate the bug elsewhere, you can run python code inside the running process - without any preparation beforehand - to display stack trace, output vars,...
- craig552uk 11y agoI've got a lot of love for py.test right now. http://pytest.org/latest/ http://pytest.org/latest/ I feel I'm able to write much more concise test scripts than I could with unittest.
- rotten 11y agoWow, there are some great new tools to explore. Thanks! Some of the new libraries I'm using this year that I've found really handy include: Odo - http://odo.readthedocs.org/en/latest/ http://odo.readthedocs.org/en/latest/ It is ridiculously handy for converting data from one format to another - especially for transforming a table from a database or csv into a DataFrame and back. Arrow - Makes for quick datetime processing. - http://crsmithdev.com/arrow/ http://crsmithdev.com/arrow/ Xlsxwriter - http://xlsxwriter.readthedocs.org http://xlsxwriter.readthedocs.org - I'm building beautiful reports, with charts, using this tool. As someone who moves data around a lot, but has to work with less technical business and analyst folks, this is becoming my goto for handing them some data to play with. Blessings - https://pypi.python.org/pypi/blessings https://pypi.python.org/pypi/blessings - as I get older staring at simple black and white text on the screen seems to be getting harder. Putting a little color and flare in my command line interfaces cheers me up even if it doesn't do much in the way of actually getting the job done. Lastly, switching from curl to httpie was a huge help in working with API's of all sorts. It solved a problem I didn't even know I had. https://pypi.python.org/pypi/httpie https://pypi.python.org/pypi/httpie
- tenfingers 11y ago"python-bond" also came out in 2015: https://pypi.python.org/pypi/python-bond https://pypi.python.org/pypi/python-bond allows a simple interface between python runtimes and php/perl/nodejs.
- collyw 11y agoThere is a lot of overlap between these languages, and I wonder what any of them can do that can't be done in pure Python.
- tenfingers 11y agoIt's not much about language, it's mostly about code/ecosystem re-use (that is: if you have a library available in system X and you're writing for Y you can still take advantage of it).