3 ms·

Have you tried the iPython notebook? Don't know about the GIS software but I do most of what you mention inside the my notebooks.

by elyase 13y ago

dalek2point3 13y ago
right, I have not. I'm very much wedded to emacs, and I like the iPython style of working. Maybe one of these days I will setup emacs + iPython and see how far it gets me. The main problem I've found so far is that python is quite bad with advanced statistical analysis, like producing regressions with the right standard errors -- charts and table are better, but still not there. But maybe things have changed ...
- jofer 13y ago
  With regards to regressions, have you tried statsmodels?
- denzil_correa 13y ago
  > The main problem I've found so far is that python is quite bad with advanced statistical analysis, like producing regressions with the right standard errors Can you tell me an example? Scipy, NumPy and statsmodels etc. pretty much handle these.
  - dalek2point3 13y ago
    Looking at statsmodel Google Summer of Code 2013, "Google Summer of Code 2013: We have had two students accepted to work on statsmodels as part of the Google Summer of Code 2013. The first project will focus on improving the discrete choice models, adding, for example, Conditional Logit, Nested Logit, and Mixed Logit models. The second project will focus on time series analysis, including regime-switching models such as SETAR, STAR, and Markov Switching models." These are basics that I would expect any decent statistical language to have. I can see potential, but not for many years do I see the kind of support that I'm looking for in python.
  - Myrmornis 13y ago
    Hardcore python user here. It sounds like you haven't used R. The OP is a social scientist and researcher. These things are vastly easier and more comprehensively implemented and documented in R. (upvoted you accidentally in the android app...) statsmodels is WIP and has an emphasis on econometrics.
  - Fomite 13y ago
    Especially with more cutting edge and Not Machine Learning based statistics, R has packages where Python does not. I've found SciPy/NumPy and statsmodels to be an adequate "What if I need a GLM in my Python code?" solution, but less of a great solution to heavy stats work.
- urschrei 13y ago
  Pandas + Statsmodels for linear regression / GLM / Discrete Choice, ARMA / ARIMA, and various non-parametric ops. For GIS: Fiona (Python OGR operations); Shapely for geometric object manipulation, analysis, and set-theoretic operations; RasterIO (self-explanatory); Descartes and/or Basemap for drawing; PySAL for spatial analysis. GeoPandas (an unholy, massively useful combination of most of the above) is also under heavy development.
- Myrmornis 13y ago
  I think I have some good news for you at least (but sadly not about the fact that statistics is easiest in R than anything else though!). For context, I'm a longtime, daily emacs and python user; I used to use R and ESS more than I do now. 1. You say you use sweave and emacs . So I assume you are using Emacs Speaks Statistics[1]. Running a python REPL in emacs now works very well thanks to Fabian Gallina's python.el which is in recent releases of emacs 24. I use it as my main python REPL. Use ipython instead of python as the interpreter running inside emacs. For config, see the comments at the top of the python.el file[2]. 2. ipython notebook is amazing! [3] But, we don't want to write code in a browser. Good news. You can have ipython notebook in the browser share the same python kernel as a python shell in the terminal or in emacs. The steps are: (1) `ipython notebook` in shell and watch for the localhost URL it prints out, (2) Open the URL in a browser and start a new notebook, (3) Back in the shell, it has just printed out the name of the kernel, like `Kernel started: 5df5d119-85b5-4f33-8ba1-1c6b2eaa950f`; copy that. (4) Now start a shell ipython sharing the same kernel: `ipython console --existing 5df5d119-85b5-4f33-8ba1-1c6b2eaa950f`. To do (4) in emacs, one way to add the arguments is to pass a prefix argument to run-python. So `C-u M-x run-python` and then edit the interpreter arguments in the minibuffer. So now you can define variables, functions etc in your shell/emacs ipython, and they are available in the web browser notebook. 3. No need to write code in the browser notebook! Implement functions in .py files in emacs, and in the web browser just import modules and call functions from within them. 4. Now, I don't have experience trying to replicate the sweave workflow in ipython notebook, but it is very flexible. See ipython author Min RK's reply here: http://stackoverflow.com/a/13222501/583763 http://stackoverflow.com/a/13222501/583763. That suggests you can indeed replicate much of sweave functionality. (Typesetting mathematical notation in latex and having MathJax display it in the browser I know works beautifully). 5. You can call R from ipython notebook, see e.g. http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/notebooks/R%20Magics.ipynb http://nbviewer.ipython.org/github/ipython/ipython/blob/mast.... Rather than getting fancy, you may just want to have the R graphics device write figures to file explicitly and then work in python with those graphics files on disk. 6. There is also RPy2, I don't have much experience with that. [1] For those not familiar, ESS is an implementation of an R REPL in emacs with extremely good usability and "feel". I would always use it over the vanilla R shell. [2] `M-x find-library python` or https://github.com/fgallina/python.el/blob/master/python.el#L76 https://github.com/fgallina/python.el/blob/master/python.el#... [3] https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks https://github.com/ipython/ipython/wiki/A-gallery-of-interes...

Have you tried the iPython notebook? Don't know about the GIS software but I do most of what you mention inside the my notebooks.

by elyase 13y ago

Have you tried the iPython notebook? Don't know about the GIS software but I do most of what you mention inside the my notebooks.

dalek2point3 13y ago

right, I have not. I'm very much wedded to emacs, and I like the iPython style of working. Maybe one of these days I will setup emacs + iPython and see how far it gets me. The main problem I've found so far is that python is quite bad with advanced statistical analysis, like producing regressions with the right standard errors -- charts and table are better, but still not there. But maybe things have changed ...

jofer 13y ago
With regards to regressions, have you tried statsmodels?
denzil_correa 13y ago
> The main problem I've found so far is that python is quite bad with advanced statistical analysis, like producing regressions with the right standard errors Can you tell me an example? Scipy, NumPy and statsmodels etc. pretty much handle these.
- dalek2point3 13y ago
  Looking at statsmodel Google Summer of Code 2013, "Google Summer of Code 2013: We have had two students accepted to work on statsmodels as part of the Google Summer of Code 2013. The first project will focus on improving the discrete choice models, adding, for example, Conditional Logit, Nested Logit, and Mixed Logit models. The second project will focus on time series analysis, including regime-switching models such as SETAR, STAR, and Markov Switching models." These are basics that I would expect any decent statistical language to have. I can see potential, but not for many years do I see the kind of support that I'm looking for in python.
- Myrmornis 13y ago
  Hardcore python user here. It sounds like you haven't used R. The OP is a social scientist and researcher. These things are vastly easier and more comprehensively implemented and documented in R. (upvoted you accidentally in the android app...) statsmodels is WIP and has an emphasis on econometrics.
- Fomite 13y ago
  Especially with more cutting edge and Not Machine Learning based statistics, R has packages where Python does not. I've found SciPy/NumPy and statsmodels to be an adequate "What if I need a GLM in my Python code?" solution, but less of a great solution to heavy stats work.
urschrei 13y ago
Pandas + Statsmodels for linear regression / GLM / Discrete Choice, ARMA / ARIMA, and various non-parametric ops. For GIS: Fiona (Python OGR operations); Shapely for geometric object manipulation, analysis, and set-theoretic operations; RasterIO (self-explanatory); Descartes and/or Basemap for drawing; PySAL for spatial analysis. GeoPandas (an unholy, massively useful combination of most of the above) is also under heavy development.
Myrmornis 13y ago
I think I have some good news for you at least (but sadly not about the fact that statistics is easiest in R than anything else though!). For context, I'm a longtime, daily emacs and python user; I used to use R and ESS more than I do now. 1. You say you use sweave and emacs . So I assume you are using Emacs Speaks Statistics[1]. Running a python REPL in emacs now works very well thanks to Fabian Gallina's python.el which is in recent releases of emacs 24. I use it as my main python REPL. Use ipython instead of python as the interpreter running inside emacs. For config, see the comments at the top of the python.el file[2]. 2. ipython notebook is amazing! [3] But, we don't want to write code in a browser. Good news. You can have ipython notebook in the browser share the same python kernel as a python shell in the terminal or in emacs. The steps are: (1) `ipython notebook` in shell and watch for the localhost URL it prints out, (2) Open the URL in a browser and start a new notebook, (3) Back in the shell, it has just printed out the name of the kernel, like `Kernel started: 5df5d119-85b5-4f33-8ba1-1c6b2eaa950f`; copy that. (4) Now start a shell ipython sharing the same kernel: `ipython console --existing 5df5d119-85b5-4f33-8ba1-1c6b2eaa950f`. To do (4) in emacs, one way to add the arguments is to pass a prefix argument to run-python. So `C-u M-x run-python` and then edit the interpreter arguments in the minibuffer. So now you can define variables, functions etc in your shell/emacs ipython, and they are available in the web browser notebook. 3. No need to write code in the browser notebook! Implement functions in .py files in emacs, and in the web browser just import modules and call functions from within them. 4. Now, I don't have experience trying to replicate the sweave workflow in ipython notebook, but it is very flexible. See ipython author Min RK's reply here: http://stackoverflow.com/a/13222501/583763 http://stackoverflow.com/a/13222501/583763. That suggests you can indeed replicate much of sweave functionality. (Typesetting mathematical notation in latex and having MathJax display it in the browser I know works beautifully). 5. You can call R from ipython notebook, see e.g. http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/notebooks/R%20Magics.ipynb http://nbviewer.ipython.org/github/ipython/ipython/blob/mast.... Rather than getting fancy, you may just want to have the R graphics device write figures to file explicitly and then work in python with those graphics files on disk. 6. There is also RPy2, I don't have much experience with that. [1] For those not familiar, ESS is an implementation of an R REPL in emacs with extremely good usability and "feel". I would always use it over the vanilla R shell. [2] `M-x find-library python` or https://github.com/fgallina/python.el/blob/master/python.el#L76 https://github.com/fgallina/python.el/blob/master/python.el#... [3] https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks https://github.com/ipython/ipython/wiki/A-gallery-of-interes...