10 ms·
Welcome to the Python Engineering blog
- sdegutis 11y ago> Python is obviously a big part of data science these days Off topic, but why is that? I mean, Python doesn't seem any more suitable to "data science" than Ruby or Lua or Clojure or any number of other similar languages. So why do I keep hearing that Python is super popular with scientists?
- nnutter 11y agoBecause of the community and libraries; not the language.
- softinio 11y agoAgreed. Plus Python being taught at more universities now probably helping.
- melling 11y agoAre there specific data science libraries or are we talking about the large number of libraries in general?
- anton_gogolev 11y agoNumPy and SciPy are the ones that I know of (and I don't even write Python, so there is surely more).
- masklinn 11y agonumpy, scipy, scikit-learn, matplotlib, pandas, biopython, sage, ipython and ipython notebook — now jupyter — (with both matplotlib and clusters management integrations), non-C accelerators (numba, cython), anaconda (a science and engineering semi-proprietary bundling & package manager). Many of these have their roots in the late 90s or early 00s. I guess Python got its fangs (haha) into scientific computing at the turn of the century, from the late 90s you find articles mentioning the introduction of scripting languages as glue in scientific pipelines variously mentioning Perl, Python and Tcl, and by and large it seems to have slowly accreted around Python.
- MengerSponge 11y agoSome of the Numpy and Scipy routines go back earlier than the 90's. AFAIK, they trace their roots to numeric routines on Fortran punch cards.
- masklinn 11y ago> AFAIK, they trace their roots to numeric routines on Fortran punch cards. Oh yes they can bind to much older libraries (e.g. BLAS), I was talking about the packages themselves. For instance Numpy is the unification of Numeric and Numarray, the former having gotten started circa 1995[0] on matrix-sig[1] and the latter originally indented to replace the former. So numpy itself has its roots in 1995. [0] https://mail.python.org/pipermail/matrix-sig/1995-August/000002.html https://mail.python.org/pipermail/matrix-sig/1995-August/000... [1] https://mail.python.org/pipermail/matrix-sig/ https://mail.python.org/pipermail/matrix-sig/ There are gems in these early messages too, for instance one of the early threads suggested allowing parens-less tuples and slice literals (a:b) in tuples within indexing: https://mail.python.org/pipermail/matrix-sig/1995-September/000044.html https://mail.python.org/pipermail/matrix-sig/1995-September/... Slice literals, let alone slice literals in tuples, remain indexing-only features to this day. So the language has a long history of adapting itself to numerical computing concerns.
- deleted 11y ago[deleted]
- giancarlostoro 11y agoWhy can't it be both? If the language is easy to adapt to then that's a plus to anyone.
- masklinn 11y agoThe language probably had something to do with it, at least in that scientists got involved very early on[0] and language features were added as a result. [0] https://mail.python.org/pipermail/matrix-sig/ https://mail.python.org/pipermail/matrix-sig/ note the direct involvement of GvR
- eximius 11y agoNetwork effect and a few really big names in it?
- mixmastamyk 11y agoAnd it is easy to use for people that want to get work done. i.e. clear syntax, and a goal of one-way to do things.
- dxhdr 11y agoIt's the libraries like NumPy, Pandas, SciPy, Matplotlib, SymPy, and so on that make the difference. There's been a ton of work put into developing those Python frameworks for "data science" which don't exist to such an extent in languages like Lua.
- xiaq 11y agoLua has Torch.
- dxhdr 11y agoOk thanks for pointing that out, I stand corrected. Regardless I think Python has reached a sort of critical mass of data science packages.
- maxerickson 11y agoNumpy existing (more than) a decade ago helped with getting mind share.
- deleted 11y ago[deleted]
- GFK_of_xmaspast 11y agoOK, now compare that to "literally everything else" in python.
- toyg 11y agoA lot of those actually started as wrappers to existing C/C++/other libs, something that is very easy to achieve in Python and not so much with other runtimes.
- dalke 11y agoTo be sure, NumPy is the third implementation (Numeric, numarray, NumPy) of the basic array data type, so getting it right isn't as easy to achieve even in Python.
- gjkood 11y agoFrom personal experience, having programmed in a small number of languages C/C++, Java, COBOL, Pascal, Python, Ruby, various shells, Assembly (Intel) etc, there are a few things that keep attracting me back to Python: a) Somehow the fact that indentation is meaningful forces my code to be neat automatically without having to run any beautifier through it. I consider myself a bit OCD on how the code looks (can't explain it) and I like how Python code finally looks. I like code to be maintainable and self-explanatory without leaving puzzles for my successors to solve. b) Batteries included. Granted there are libraries for most things in other languages, but nothing with the ease of access as Python. c) Portability without having to know the compiler intricacies of each Port/Platform/OS d) Being able to deliver solutions on time and within budget without putzing around with irrelevant stuff that the customer never cares about. Not being a Data Scientist, I don't know if any of this is relevant. But being educated in CS and responsible for delivering on promises I care about some of this a lot.
- dr_zoidberg 11y agoa) Having "beautiful code" helps keep focus where it matters: the "difficult" sutff (be data data science, machine learning, math problems, physics simulations, etc). b) True, there are beautiful libraries all around, and easy to use. In particular to data science, I think NumPy+SciPy+matplotlib are comparable to MATLAB. With beautiful code, portability and no need to pay an expensive license. c) Totally agree. Though there are small nitpicks in different platforms, most of the times they can be overcome. d) Totally agree. Also: the Anaconda distribution solves most of the problems that anyone could have with (b), (c) and (d).
- Taren 11y agoI heard from a couple people that they don't exactly want to learn Fortran. Python stays out of your way and there is a major ecosystem around it by this point.
- Adam_O 11y agoI saw they ran a small "python for data science" intro course this year on edx, so I figure they're definitely behind using the language in some capacity. https://www.edx.org/course/introduction-python-data-science-microsoft-dat208x https://www.edx.org/course/introduction-python-data-science-...
- vram22 11y agoI checked out that course a little; it looked very basic, at least the part I checked. Then left it. Saw the course via an ad, FWIW.
- danso 11y agoAFAIK, Ruby doesn't have anything like Python's numpy. There has been some effort to fix that, but not one driven by necessity or an organized movement: https://github.com/SciRuby/sciruby/wiki/The-SciRuby-Manifesto https://github.com/SciRuby/sciruby/wiki/The-SciRuby-Manifest... Fun bit of trivia: Hadley Wickham (of R's ggplot2 and dplyr fame, among many other things) said that if R hadn't been around when he first got into statistical programming, Ruby would have been his next choice, because of Ruby on Rails. He says now, his choice would be Python or Julia ("or maybe JavaScript"): https://www.reddit.com/r/dataisbeautiful/comments/3mp9r7/im_hadley_wickham_chief_scientist_at_rstudio_and/cvgybce https://www.reddit.com/r/dataisbeautiful/comments/3mp9r7/im_... Fun to imagine ggplot2 in R...or as a competitor to D3 in data visualization.
- ElliotH 11y agoAs well as the packages that people have mentioned, there are nice bundles like Anaconda or Canopy which makes it easier to get started. Certainly working with people who know data but don't necessarily know much Python (even if they know it they might not know tools like pip, or want to spend time understanding the versions of their tools) it's way easier to say "just install anaconda" than "install python and all these packages we use". I don't know if other languages have these bundles but Python's are very good. Additionally, the jupyter/ipython notebook environment is very good, but I'm not sure that's converting people, I still see many researchers working in plain text editors.
- imh 11y agoIt's in a sweet spot. It's good enough for unstructured exploratory analysis, for production code, it's super easy to get started in, and there are some good libraries. There's really low overhead to poking around data in it, and you can put something into production with it.
- mickeyp 11y agoThey are all fine languages. There is another (key) reason why Python is keeping its momentum: the availability of skilled, professional python developers with a lot commercial experience. I've been called in a number of times to help out small businesses or teams that've started using Python because they were: researchers, scientists, or even just regular developers, and have found themselves with a growing business and customer requirements. Having a large, professional body of capable of (especially contract, here in London) developers makes a world of difference. You can find expert Julia, Haskell, Clojure, etc. developers -- but they are fewer and far between.
- BuckRogers 11y agoI'm not a scientist, but I think the answer would be orthogonal to why I (and probably many others) pick Python. Is it tied to a single source? No, many implementations exist and with the 2/3 split, the BDFL is weaker than ever (a good thing, overthrow all BDFLs). Is it portable across platforms? Yes. It runs anywhere C89 will. Does it have a broad range of uses? Yes. Already in largescale use for systems, web, administration scripting, game scripting, 3d applications, data modeling, game creation, crossplatform GUI applications including iOS/Android. Is it easily maintainable? Yes, as much as anything else. Nothing is a cure for the complexity issues at 500KLOC scale, not static typing, nothing. Will it be around for a long time? Since 1991, so likely in some form yes. The syntax has proven too popular with folks already. I'm considering creating own Pythonic language. Is it popular, to get help easily and borrow code? Yes. #3 on Github (practical comparison) and #5 on Tiobe (abstract comparison).
- fencepost 11y agoSince the data science folks don't also have a goal of becoming programmers, a language that had some early design goals of being easy to teach (see also CP4E [0]) and has appropriate "good enough" libraries lets them build custom analysis software without having to spend nearly as much time learning programming instead of more relevant things. Also, inertia: * Python, 1991 * Lua, 1993 * Ruby, 1995 (Japan) 1999 (English) * Clojure, 2009 [0] https://www.python.org/doc/essays/cp4e/ https://www.python.org/doc/essays/cp4e/
- afarrell 11y agoThe early design goal of being easy to teach also caused python to have a culture that places a high value on doing the work to make the developer experience better for beginners. You can see this in things like the Django tutorial, but it is more powerfully felt as an undercurrent in conversations at python conferences (based on PyTennessee, PyConUK, DjangoCon, and PyCaribbean)
- antod 11y agoSide note about the author: Steve Dower from MS has also been helping out core Python development. He's done great work on improving how Python gets built, maintained, packaged etc on Windows. Although I'm not a Windows user, it has been great to see the progress made in that area recently.
- afarrell 11y agoHe's also a nice fellow and pretty good at pool.
- levemi 11y agoI recently started doing more development on Windows and coming from os x and linux it's a bit weird to use `py` instead of `python` but other than that it's pretty straight forward. I also want to mention that I'm loving all the recent attention for python on HN. It's a great language, 2 or 3, and has many great use cases and has a great ecosystem.
- ianamartin 11y agoI used to use a Windows box at work, and still use one at home for games. Python is getting much better for Windows. But some of the package management still completely sucks. Cmndr helps out a lot, but some of the math heavy packages (I'm looking at you, cryptography) have to be compiled with the version of the Windows C compiler that was originally used to write the package, which is not always easy to find. Or you have to find a precompiled version, which is tricky to install into a virtual environment. Of course, for general computing, things are good and getting better, and I really appreciate the work Microsoft has put into Python Tools for Visual Studio. I love my sublime text, but I also have a mad crush on visual studio from my C# days. If VS would support Pyramid as well as it does Flask and Django, I would seriously consider switching to a Windows machine full time.
- dr_zoidberg 11y agoThe compiler issue has been solved by Microsoft with MSVC for Python[0]. For anything that still fails with that (may happen) or if you need NumPy with Intel's MKL, check Christoph Gohlke's Python Module repository[1]. [0] https://www.microsoft.com/en-us/download/details.aspx?id=44266 https://www.microsoft.com/en-us/download/details.aspx?id=442... [1] www.lfd.uci.edu/~gohlke/pythonlibs/
- Negative1 11y agoMy biggest gripe with Python on Windows is all the libraries that sport a GCC toolchain for *nix environments, so completely fail to work. Yes, that is the library contributors fault, but I hate that Python even allows that. Had to chuckle at the python on HN comment; Python tends to get universal praise around these parts (supposedly some investors won't even waste their time with you unless you're a Python shop).
- smortaz 11y agoHi, team lead here. Very Happy to see this on HN. If there are any requests for the team pls feel free to voice them. Smiles, frowns, ... Anything from making pkg installation easier to cross platform dev tools. Cheers.
- ableal 11y ago> making pkg installation easier http://www.pyinstaller.org http://www.pyinstaller.org Best thing since bread. OK, sliced bread, but it's close ...
- smortaz 11y agoThanks ableal - improving pkg installation is under heavy discussion currently. We'll take a look. Cheers.
- tanlermin 11y agoActually, I think support for nuitka would be amazing. Its already 2x faster than cpython on pystone, but its even more promising to compile everything into a binary blob for easy distribution. http://nuitka.net/ http://nuitka.net/
- smortaz 11y agoThanks tanlermin. We are aware of nuitka and in general very excited about the various Turbo options that are showing up for Python. Our own effort is pyjion and we have a talk at PyCon 2016: https://github.com/Microsoft/Pyjion https://github.com/Microsoft/Pyjion The high order bit is to restructure CPython to enable easy plugging in of various JIT optimizer/codegenerators into the interpreter. Our own attempt will be based on the cross-platform .Net JIT (generating machine code).
- tanlermin 11y agoThis is really cool..hope to see it succeed :)
- 11y ago
- dethos 11y agoNot a windows user but for what is written in the post, it looks like it will cover more transversal topics. Will follow it for a while, to see what it brings to the table.
- ksec 11y agoIf there was ever a race between Python and Ruby, then Python has won. Everywhere from Google, Microsoft, Facebook are now using Python. This isn't to say Ruby is dead, but it will remains in a niche.