3 ms·

In-memory only... How is this better than the SFrame implementation from Dato (2015) that was posted here a couple of days ago? https://news.ycombinator.com/it

by rch 11y ago

In-memory only... How is this better than the SFrame implementation from Dato (2015) that was posted here a couple of days ago? https://news.ycombinator.com/item?id=11106501 https://news.ycombinator.com/item?id=11106501

sandGorgon 11y ago
even i had that question - especially when people are talking about using SFrame as the underlying structure for Julia. but then I saw this: "Arrow's cross platform and cross system strengths will enable Python and R to become first-class languages across the entire Big Data stack," said Wes McKinney, creator of Pandas. Code committers to Apache Arrow include developers from Apache Big Data projects Calcite, Cassandra, Drill, Hadoop, HBase, Impala, Kudu (incubating), Parquet, Phoenix, Spark, and Storm as well as established and emerging Open Source projects such as Pandas and Ibis. this is pretty much nuke-from-orbit.
- rch 11y ago
  > this is pretty much nuke-from-orbit That analogy might imply overkill, thus highlighting the tactical advantages of the SFrame approach in processing a month's worth of 1-10GB daily-generated SQLite files, for instance.
  - tanlermin 11y ago
    Python's Dask out of core dataframe can also do that.
    - infinite8s 11y ago
      Dasks' out of core dataframes are just a thin wrapper around pandas dataframes (aided by the recent improvement in pandas to release the GIL on a bunch of operations)
      - tanlermin 11y ago
        Uh, no they are not. They lazy- scale pandas to on disk and distributed files. http://dask.pydata.org/en/latest/dataframe.html http://dask.pydata.org/en/latest/dataframe.html "Dask dataframes look and feel like pandas dataframes, but operate on datasets larger than memory using multiple threads." http://blaze.pydata.org/blog/2015/09/08/reddit-comments/ http://blaze.pydata.org/blog/2015/09/08/reddit-comments/
        infinite8s 11y ago
        You haven't refuted anything I said. Internally the dask dataframe operations sit on top of pandas dataframes. All dask does is automatically handle the chunking into in-memory pandas dataframes and interpret dask workflows as a series of pandas operations.
        sandGorgon 11y ago
        Why doesn't Pandas have anything to save the entire workspace to disk (like .RData). There are all these cool file formats like Castra, HDF5, even the vanilla pickle - but I don't see anything with a one shot save of the workspace (something like Dill) Is this an antipattern for Pandas?
  - sandGorgon 11y ago
    Well no. It means the same situation as git vs mercurial. One may be better than the other (as you mentioned), but it really doesn't matter. If Pandas and other high profile products are endorsing it (and may adopt it), it's going to be very hard for 99% of people to choose something else.

by rch 11y ago

sandGorgon 11y ago

even i had that question - especially when people are talking about using SFrame as the underlying structure for Julia. but then I saw this: "Arrow's cross platform and cross system strengths will enable Python and R to become first-class languages across the entire Big Data stack," said Wes McKinney, creator of Pandas. Code committers to Apache Arrow include developers from Apache Big Data projects Calcite, Cassandra, Drill, Hadoop, HBase, Impala, Kudu (incubating), Parquet, Phoenix, Spark, and Storm as well as established and emerging Open Source projects such as Pandas and Ibis. this is pretty much nuke-from-orbit.

rch 11y ago
> this is pretty much nuke-from-orbit That analogy might imply overkill, thus highlighting the tactical advantages of the SFrame approach in processing a month's worth of 1-10GB daily-generated SQLite files, for instance.
- tanlermin 11y ago
  Python's Dask out of core dataframe can also do that.
  - infinite8s 11y ago
    Dasks' out of core dataframes are just a thin wrapper around pandas dataframes (aided by the recent improvement in pandas to release the GIL on a bunch of operations)
    - tanlermin 11y ago
      Uh, no they are not. They lazy- scale pandas to on disk and distributed files. http://dask.pydata.org/en/latest/dataframe.html http://dask.pydata.org/en/latest/dataframe.html "Dask dataframes look and feel like pandas dataframes, but operate on datasets larger than memory using multiple threads." http://blaze.pydata.org/blog/2015/09/08/reddit-comments/ http://blaze.pydata.org/blog/2015/09/08/reddit-comments/
      - infinite8s 11y ago
        You haven't refuted anything I said. Internally the dask dataframe operations sit on top of pandas dataframes. All dask does is automatically handle the chunking into in-memory pandas dataframes and interpret dask workflows as a series of pandas operations.
      - sandGorgon 11y ago
        Why doesn't Pandas have anything to save the entire workspace to disk (like .RData). There are all these cool file formats like Castra, HDF5, even the vanilla pickle - but I don't see anything with a one shot save of the workspace (something like Dill) Is this an antipattern for Pandas?
- sandGorgon 11y ago
  Well no. It means the same situation as git vs mercurial. One may be better than the other (as you mentioned), but it really doesn't matter. If Pandas and other high profile products are endorsing it (and may adopt it), it's going to be very hard for 99% of people to choose something else.