3 ms·

IPython/Jupyter, Pandas/Numpy and Python will get you everywhere you need to go. Currently, until maybe Go gets decent DataFrame support, in terms of the total

IPython/Jupyter, Pandas/Numpy and Python will get you everywhere you need to go. Currently, until maybe Go gets decent DataFrame support, in terms of the total time to get to your solution, I'd be amazed if any other setup got you there quicker.

threeseed 9y ago
> get you everywhere you need to go No it won't. That combination can't handle large datasets that are typical for most data science teams i.e. maybe include PySpark. And then it's very limited so far as ML/DL technologies.
- fredley 9y ago
  > i.e. maybe include PySpark Pandas and Spark are both DataFrame libraries, and seem to offer very similar functionality to me. Why do you prefer Spark over Pandas? > very limited so far as ML/DL technologies I mean, getting Tensorflow up and running with GPU support isn't trivial, but it's not exactly hard, and Keras[1] provides excellent support for a wide variety of other backends. What, in your experience, is less limited? [1]: https://keras.io/ https://keras.io/
  - jwilbs 9y ago
    Spark sits on top of YARN/Mesos, and is used for data processing scalability that pandas can't handle. Personally, I think two areas often lacking are software development skills and general statistics knowledge. The former is necessary for writing production-quality code, assisting with an sort of data engineering pipeline, writing reliable, reusable code, and creating custom solutions. Unfortunately, the latter is often skimped on (if not skipped entirely) in favor of more 'hot' fields like ml/dl, with the result being a fuzzy understanding across the board. (You'd be amazed at the quantity of candidates lacking fundamental knowledge about glm's, basic nonparametric stats, popular distributions, etc).
- ppod 9y ago
  >typical for most data science teams I would bet that the mean size of dataset people are dealing with is a lot bigger than the median size.

threeseed 9y ago

> get you everywhere you need to go No it won't. That combination can't handle large datasets that are typical for most data science teams i.e. maybe include PySpark. And then it's very limited so far as ML/DL technologies.

fredley 9y ago
> i.e. maybe include PySpark Pandas and Spark are both DataFrame libraries, and seem to offer very similar functionality to me. Why do you prefer Spark over Pandas? > very limited so far as ML/DL technologies I mean, getting Tensorflow up and running with GPU support isn't trivial, but it's not exactly hard, and Keras[1] provides excellent support for a wide variety of other backends. What, in your experience, is less limited? [1]: https://keras.io/ https://keras.io/
- jwilbs 9y ago
  Spark sits on top of YARN/Mesos, and is used for data processing scalability that pandas can't handle. Personally, I think two areas often lacking are software development skills and general statistics knowledge. The former is necessary for writing production-quality code, assisting with an sort of data engineering pipeline, writing reliable, reusable code, and creating custom solutions. Unfortunately, the latter is often skimped on (if not skipped entirely) in favor of more 'hot' fields like ml/dl, with the result being a fuzzy understanding across the board. (You'd be amazed at the quantity of candidates lacking fundamental knowledge about glm's, basic nonparametric stats, popular distributions, etc).
ppod 9y ago
>typical for most data science teams I would bet that the mean size of dataset people are dealing with is a lot bigger than the median size.