wesm

10 ms·

61.
by wesm 10y ago
No, it's all closely inter-related. pandas as a project is responsible for data manipulation and in-memory analytics. These other projects are all complementary technologies.
62.
GitHub's one-dimensional view of open source contributions(wesmckinney.com)
3 points by wesm 10y ago|0 comments
63.
by wesm 10y ago
On plain text accounting, you should check out Beancount (I am unaffiliated, but know the author): http://furius.ca/beancount/
64.
by wesm 10y ago
Spurious argument. See http://www.apache.org/legal/resolved.html#category-a
65.
by wesm 11y ago
Concurrent writes would not be possible but parallel reads definitely are. Haven't put any multithreading work into the library yet but it shouldn't be a large task.
66.
by wesm 11y ago
Dask is a compute framework. So you could use dask to create lots of Feather files, then perform computations.
67.
by wesm 11y ago
This is precisely the goal of the Apache Arrow project http://arrow.apache.org/ -- and I've been working very hard to bring together diverse groups of data system developers to work on this problem together. Exciting r
68.
by wesm 11y ago
Feather doesn't support a number of pandas features (but these are features that R doesn't really support, either): https://github.com/wesm/feather/blob/master/python/README.md (not compre
69.
by wesm 11y ago
CSVs, in addition to being slow to parse (in practice, 40-100MB/s is the typical window, see some benchmarks here http://wesmckinney.com/blog/pandas-and-apache-arrow/ ), drop types (which have to be inferred),
70.
by wesm 11y ago
I'd like to see SPARK-13534 completed so that we can bring Spark to Python/R data access performance reach a level we can deem "acceptable". I dug into this issue a bit here: http://wesmckinney.com/blog&#
71.
by wesm 11y ago
Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many differe
72.
by wesm 11y ago
Drawing a parallel!
73.
by wesm 11y ago
I never stated what language LAPACK was written in.
74.
by wesm 11y ago
By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive? I believe what the article says is that "Python has tools that enable a savv
75.
by wesm 11y ago
Claiming that using wrapped libraries written in another programming language (LAPACK, anyone?) is an inauthentic usage of the language (here, Python) is pedantic and unhelpful. C is just a wrapper for assembly, then, right?
76.
by wesm 11y ago
Completely bizarre attitude (creator of pandas here).
77.
by wesm 11y ago
I hope you all know that the people who have invested most in actually building this software care the least about this discussion.
78.
by wesm 11y ago
Having both designed pandas and suffered from some of the rough edges of pandas when used in a database-like way, I tried to be thoughtful in designing the Ibis API to be semantically complete with SQL (i.e. so you shouldn't have to
79.
by wesm 11y ago
Makes most sense to compare Impala and Spark architecturally. Ibis will eventually integrate with Spark. We've been focusing on Impala integration for reasons cited here: http://blog.cloudera.com/blog/2015/07&
80.
by wesm 12y ago
Integration between Box and Office was announced at BoxWorks in September, so this shouldn't be a shocker: http://techcrunch.com/2014/09/03/box-pushes-its-office-365-i...
81.
by wesm 13y ago
It's used by Metamarkets in production ( http://metamarkets.com/); I've seen talks about it at Strata and they're getting really impressive query times on terascale problems. Shameless plug: if you're int
82.
by wesm 14y ago
This is really a mischaracterization. The point of the book was to address data tooling topics (and the bare essentials: IPython and NumPy and matplotlib); for most data tasks (especially Chapters 7, 9, and 10) I would challenge you to repl
83.
by wesm 14y ago
Comparisons with data.table on performance are much more favorable than with vanilla R or plyr; a lot of progress has been made last couple years, too. I personally find the data.table syntax to be a bit obtuse at times but it's a great lib
84.
by wesm 14y ago
Careful, you might hurt some people's feelings. Has anyone done a systematic comparison of LuaJIT's performance with PyPy's?
85.
by wesm 14y ago
Whoa there cowboy. I'm pretty sure that tkf (if he's the same one who's submitted pull requests to my projects) is from the land of the rising sun. And his second statement was a joke. And funny IMHO
86.
by wesm 14y ago
I'd be interested to see some of your R use cases where you perceive that things could be improved in pandas; a year ago there were lots of things you couldn't do, but a lot has changed :) Nowadays, the tables have turned and there are lots
87.
Pandas (Python data analysis tookit) version 0.8.0 released(pandas.pydata.org)
97 points by wesm 14y ago|16 comments
88.
by wesm 14y ago
Could you explain how Christoph Gohlke's .exe installers solve the programmatic deployment problem (clicking through an .exe installer is not "programmatic deployment")?
89.
by wesm 14y ago
Programmatic deployment on Windows is still a massive problem (especially for extension modules-- e.g. the entire scientific Python ecosystem).
90.
by wesm 14y ago
I was also very disapointed with DAwOST; it's a book about data analysis techniques with a bit of code mixed in. This is a book about _programming tools for working with data_, pretty much the exact opposite.

More ›