8 ms·
Feather: A Fast On-Disk Format for Data Frames for R and Python
- eximius 11y agoIgnorant question: What does this solve that a csv doesn't? Type information?
- rspeer 11y agoCSV parsing is relatively slow.
- phillc73 11y agoNot necessarily. Importing CSV data from disk in R can be pretty quick with 'fread', from the data.table package.[1] Having said that I'm pretty excited about feather and where development will lead. I use RDS files quite heavily, mostly because of the compression which allows much smaller file sizes for distribution.[2] However, there's a trade off with parsing spread and also interoperability between languages. Looks like feather already has the interoperability part sorted, just waiting for compression now. Reading slices direct from disk is pretty exciting too. [1] http://www.starkingdom.co.uk/faster-csv-import-with-r/ http://www.starkingdom.co.uk/faster-csv-import-with-r/ [2] http://www.starkingdom.co.uk/faster-import-with-r-redux/ http://www.starkingdom.co.uk/faster-import-with-r-redux/
- roel_v 11y agoAnything that needs to parse text into numbers will be slow. Your links just compare various slow methods and conclude that one isn't quite as slow as the others. Of course it all depends on what you call 'slow'. Reading a few hundreds of megabytes of megabytes of csv's isn't going to be 'really' slow on modern hardware even if it was fgetc'd character by character. Either way: anything that represents data as text will become a bottleneck when the size of the dataset grows.
- phillc73 11y agoThe original comment was, "CSV parsing is relatively slow." Relative to what? The links I provided show that, using fread from data.table in R, parsing CSV data can be quicker than returning the same data from, for example, an SQLite database. I believe the links do show CSV parsing relative to a couple of other common data parsing methods in R. Whether they're all "slow methods" is an entirely different question. If you'd like to compare parsing CSV data relative to a "fast method", I'd very much like to read the analysis.
- roel_v 11y agoI guess in re-reading my comment, I phrased it in a tone that was more combative than I meant, so don't take the GP as an attack. My point was, all conversions from text into numbers (I'm assuming here that the target use case is reading large amounts of numeric data) are slow and the slow part isn't the IO, but the conversion from text to numbers. In that light, it isn't a surprise that sqlite isn't much faster than csv, because sqlite doesn't have 'strong typing' itself. Any storage format that is concerned with speed will store data in binary format. But of course text formats are a lot easier to work with, and to interface between programs. I sometimes (when I have a lot of data that I know need multiple passes of reading) build quick and dirty 'caches' where I read file.csv and do what is basically a memory dump of the parsed data into file.csv.bin. My read functions can then check if that file exists and skip the parsing step. In my experience, this can be easily an order of magnitude (10x) faster. It's not portable or even elegant, of course. Apart from that - when speed is of importance, one wouldn't use R in the first place, of course (I say that as someone who likes R for what it is and for the things it does well). I don't do write ups of speed analyses, nor do I know of any, so I have to cop out on that one.
- rspeer 11y agoThe fact that storing data in binary format gives you more speed is exactly why you'd want to use Feather. It means you don't have to translate things into an unspecified .bin format. This is the answer to the original question about "why not use CSV". I don't know about R, but in Python, operations on objects such as NumPy arrays and Pandas DataFrames are all implemented using fast C code, and so is Feather. You can be concerned about speed.
- oddthink 11y agoType information is one. CSV is slow, since it has to parse everything on load. CSV has no random access, since rows can be arbitrary length. CSV takes up a lot of disk space, since a 8-byte double gets expanded into a 15+ digit string.
- roywiggins 11y agoAlso there's no single, official CSV standard.
- jabl 11y agoThere's RFC 4180. Problem is, there is no universal acceptance in actually following it.. :)
- wesm 11y agoCSVs, in addition to being slow to parse (in practice, 40-100MB/s is the typical window, see some benchmarks here http://wesmckinney.com/blog/pandas-and-apache-arrow/ http://wesmckinney.com/blog/pandas-and-apache-arrow/), drop types (which have to be inferred), sometimes in a way that cannot be recovered (factor / category levels). By comparison, Feather performs very close to disk performance. So speeds exceeding 500 MB/s (versus < 100MB/s for CSV) are common.
- th0ma5 11y agoAfter reading through it, the spec seems to promise superior data interchange in pretty much every way while sacrificing about the only thing good about CSV... That most things can be coerced to produce or consume CSV with a simple read or print.
- baldeagle 11y agoAny luck in getting it to work with sas... Maybe in a 10.12 future release? My daily grind is SAS to csv to R, because not everyone else has seen the light.
- hadley 11y agoIf SAS wanted to support it, it would be super easy for them to implement it.
- sandGorgon 11y agoQuick request - could you hook into save.image in R and give the ability to save the entire workspace in R? That would be awesome. Incidentally I had filed a bug request for a functionality to save the entire workspace in Pandas...but was rejected as being unpythonic. Oh and the devs claimed Apache Arrow was vaporware ! https://github.com/pydata/pandas/issues/12381 https://github.com/pydata/pandas/issues/12381
- hadley 11y agoI think saving your entire workspace is a bad idea too, sorry!
- sandGorgon 11y agoCould you talk about why? Other than convenience factor (and R already does it), could you talk about why. Is it stemming from a fundamental aspect of the data format - for example can you save two data frames to the same file? Because if you can save two - why not save two hundred.
- peatmoss 11y agoIt thwarts reproducibility. By saving your workspace, it drags a lot of state from session to session that isn't accounted for. If you share code with someone else, their workspace space won't be the same, and thus the code may not function the same.
- sandGorgon 11y agoPoint taken. But we are again delving dangerously close to thou-shall-not . from my perspective, it is a quick and convenient way to save all the data frames in my code. It's a boon for productivity. If not this, then I pray for Feather to be able to save multiple data frames innone file.
- peatmoss 11y ago
- wallstquant 11y agoNow if only there was a Julia package for this too
- th0ma5 11y agoAny plans to support a pipe-aware POSIX command, or get this into PostgreSQL?
- bjt 11y agoA Postgres foreign data wrapper to Arrow or Feather sounds pretty reasonable, maybe even inevitable. https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers
- hadley 11y agoNot by us, but we expect that many other projects will add feather support now that it's used by both R and Python.
- vannevar 11y agoIs there any functional distinction between a data frame and a relational table? Could you implement a persistent frame just by wrapping sqlite?
- deleted 11y ago[deleted]
- hadley 11y agoBoth Wes and I (project authors) will be tracking this thread in case you have questions!
- _Wintermute 11y agoAs one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?
- peatmoss 11y agoI mentioned it elsewhere in the thread, but libdataframe.c seems like a pretty natural continuation to me. EDIT: What I DON'T know is how much libdataframe would look than libsqlite.
- hadley 11y agoYes! Anything that is mostly C/C++ for performance could be shared between R and Python now that we have an easy interchange format.
- maxerickson 11y agoIn discussing Apache Arrow in the context of Python and R, we wanted to see if we could use the insights from feather to design a very fast file format for storing data frames that could be used by both languages. Thus, the Feather format was born. Is the feather in insights from feather the right word? It reads awkwardly to me, which could just be me lacking context.
- hadley 11y agoShould be insights from _arrow_
- data_scientist 11y agoGreat idea! Some questions: - Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather? - Feather is "not for long term data storage". Will it be standardize in a distant future? - Do you plan to integrate it into Pandas?
- stephenl 11y agowhat are some practical uses for this if living in a pure R world? Is this like BigMemory but for data frames? https://cran.r-project.org/web/packages/bigmemory/index.html https://cran.r-project.org/web/packages/bigmemory/index.html Thanks.
- hadley 11y agoIt's often much faster than rds. And in the long long term there will be tools for computing on feather files that don't require loading it into memory. (In the short term I'll add ways to pull in slices of the full dataset)
- alsocasey 11y agoFaster because it isn't (currently) using compression (which rds uses by default) or faster period? Either way, the idea of mixed Python/R pipelines with feather file intermediates input/outputs is pretty sweet. Learn in scikit, save to feather, plot in ggplot2... using Make to tie the pieces together?
- hadley 11y agoIt's usually faster than either compressed or uncompressed RDS - but if you have heavily duplicated data, compressed RDS can be faster than feather (depending on some tradeoff between compression speed and disk speed). Feather will probably gain compression support eventually.
- sshillo 11y agoAre there any plans to support larger than RAM datasets? Like hdf5 or bcolz does.
- hadley 11y agoThe format already supports larger than RAM data, but we don't yet have an API for creating those files or just extracting slices. That will come in the future.
- dbyte 11y agoThanks for working on this. Really a great effort. Hope your guys can get good inspiration from projects like h5df, bcolz, pytables having the option of incrementally adding features and maintaining an open spec.
- mziel 11y agoI can see that Wes is a reporter for Spark issue (https://issues.apache.org/jira/browse/SPARK-13534 https://issues.apache.org/jira/browse/SPARK-13534), what are the plans (if any) for tighter Spark/Python/R DataFrames integration?
- wesm 11y agoI'd like to see SPARK-13534 completed so that we can bring Spark to Python/R data access performance reach a level we can deem "acceptable". I dug into this issue a bit here: http://wesmckinney.com/blog/pandas-and-apache-arrow/ http://wesmckinney.com/blog/pandas-and-apache-arrow/
- peatmoss 11y agoMy heart is palpitating. I love where this is going. Jake Vanderplas talked about the desire for a common data frame lib to unite the warring tribes in his PyCon keynote a year or so ago, and I couldn't have agreed more. This appears to be "only" a serialization format ("oh, my unicorn only lays golden eggs"). I really hope this is the start of some common library infrastructure that can be used for all aspects of in- and out-of-memory data frames. Great work, and I hope it is a harbinger of good things to come. Also, I'll treat this as tangible evidence that the "language wars" are stupid.
- wesm 11y agoThis is precisely the goal of the Apache Arrow project http://arrow.apache.org/ http://arrow.apache.org/ -- and I've been working very hard to bring together diverse groups of data system developers to work on this problem together. Exciting road ahead!
- peatmoss 11y agoAh jeez, I read the feather announce, but not the arrow docs. How did I not know about this!?
- jph00 11y agoIf anyone wants to get this running on Windows, I've made a start. In cpp\thirdparty, you can run the .sh scripts with cygwin, but in build_thirdparty you'll need to add to the appropriate section: elif [[ "$OSTYPE" == "cygwin"* ]]; then PARALLEL=$NUMBER_OF_PROCESSORS And you'll need to use msbuild rather than make, e.g.: if [[ "$OSTYPE" == "cygwin"* ]]; then msbuild gtest.sln /p:configuration=release That got the 3rd party stuff working. But then I hit a snag, because building python 2.7 modules on Windows requires an old MSVC version that doesn't support stdint.h, which is used by feather in ext.cpp . Maybe a simple conditional compilation for the appropriate header will be enough to fix that, but I haven't got time to check today. So hopefully someone else can fix that...
- jzwinck 11y agoHDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if the Feather API is preferable?
- andreasvc 11y agoI found it quite frustrating to use HDF5. It does not handle variable-length strings well (very common). In Pandas, categoricals and MultiIndex are not supported. I found that settling for CSV and pickle is more reliable & robust. Also, HDF5 basically implements a hierarchical file system, which is overengineering IMO.
- nhatbui 11y agoAre you talking about a different type of MultiIndex? http://pandas.pydata.org/pandas-docs/stable/advanced.html http://pandas.pydata.org/pandas-docs/stable/advanced.html
- andreasvc 11y agoThat's the one. Don't recall the specifics, but combinations of MultiIndex as index and columns and the other features were not supported with HDF5.
- chubot 11y agoHDF5 apparently only has one implementation, which is ridiculously bloated: https://news.ycombinator.com/item?id=10858189 https://news.ycombinator.com/item?id=10858189
- deadmutex 11y agoHDF5 is not thread-efficient. https://www.hdfgroup.org/hdf5-quest.html#tsafe https://www.hdfgroup.org/hdf5-quest.html#tsafe So, python packages like h5py do not even try to release the GIL. This makes working with HDF5 very annoying in python (when using multiple threads).
- zzleeper 11y agoThis looks amazing! Hadley, Wes, what are your thoughts on how to implement compression? I recall some open source columnar datastores (e.g. infobright) that achieved very VERY fast compression rates with just a few tricks: https://news.ycombinator.com/item?id=8354416 https://news.ycombinator.com/item?id=8354416 In particular, compression is extremely fast for columnar datastores (its the same type one after the other). Since a lot of times the data is sorted by some ID (date, individual, etc.), you should see large improvements in both speed and disk space.
- hadley 11y agoCompression is on the long term to do list - Wes knows a lot more about it than me.
- joelthelion 11y agoIt seems like C++ (C?) is also supported. Why don't you advertise it? In my (limited) experience, exchanging data with C/C++ is even more painful than between Python and R.