9 ms·

JDF.jl – Julia DataFrames serialization format

55 points by xgdgsc 3y ago

jstx1 3y ago
I don't see the value of a language-specific format when we have things like parquet.
- thumbuddy 3y ago
  I think Julia is one of those researcher languages like Haskell so it's probably just someone having fun, writing a paper, or whatever.
  - Buttons840 3y ago
    I don't agree with this. Julia does not have new ideas the same way Haskell does, and Julia has done a lot of work to be practical, like having a good way of calling Python and R code from early on in its life. If Julia has done anything new, it's probably related to being fast, but that's not exactly a novel idea.
    - thumbuddy 3y ago
      How come you never see anything practical made in Julia in production somewhere that isn't a pile of research mathematics? I personally can't think of a single tool or company. It's not like it's a young language either? I have seen dozens of niche papers written using it though. Being fair, I have seen dozens of not hundreds of products and mass deployed projects written in similar languages. Even Haskell made its way to a lot of desktops in the Linux world, and Haskell is super niche research territory imo.
      - ChrisRackauckas 3y ago
        The clinical analysis for the COVID-19 vaccines was pretty practical and that was Pumas used by Moderna. There's a JuliaCon talk about that from the Director Head of Pharmacometrics at Moderna. There's Formula 1 usage, there's a some satellites running Julia onboard, etc. Some are documented here https://juliahub.com/case-studies/ https://juliahub.com/case-studies/ though some people post what they are doing on the Discourse and Slack and so those end up being the most comprehensive source. Of course you can always argue whether 30 public examples is enough, or 50, or etc. but it's far from 0. And of course, most examples are never for public release.
        thumbuddy 3y ago
        Fifteen years ish and there's maybe ten-thirty production use cases that kinda can't be verified because they all sound like research projects doesn't sound super great to me. No offense, but I do wonder if all of those cases could have been accomplished more easily/sustainably with another tool and if they exist simply because someone was selling the language to a company as a contractor as a form of vendor lock in or something. I guess I'll be curious once I install some software or an OS and see it brings in Julia in as a dependency or something. Otherwise I worry it's Matlab 2.0 with less of a mindshare...
    - gugagore 3y ago
      Julia is unique in using multiple dispatch pervasively. It did not invent multiple dispatch, but it certainly popularized it.
    - mrfox321 3y ago
      Sure it does. Have you seen the type system used to generically dispatch matrices to GPUs or cpus. Or the dispatching to give you auto-differentiation? Maybe it's not your definition of "new ideas", but they are really useful and original.
  - VHRanger 3y ago
    Julia is still niche, but it's definitely used in production? The NY Fed have their main model of the economy in Julia (ported from Matlab)
    - thumbuddy 3y ago
      Sounds like research to me? I dunno...
    - agumonkey 3y ago
      this https://juliahub.com/case-studies/ny-fed/ https://juliahub.com/case-studies/ny-fed/ ?
      - VHRanger 3y ago
        This: https://frbny-dsge.github.io/DSGE.jl/latest/ https://frbny-dsge.github.io/DSGE.jl/latest/
- ChrisRackauckas 3y ago
  The repo is pretty clear on its intentions to not be a language-specific format: > Development Plans > I fully intend to develop JDF.jl into a language neutral format by version v0.4. However, I have other OSS commitments including R's {disk.frame} and hence new features might be slow to come onboard. But I am fully committed to making JDF files created using JDF.jl v0.2 or higher loadable in all future JDF.jl versions. It's beyond v0.4 and at v0.5.1 right now so it seems delayed, but the repo describes intentions to be a language-neutral format after the dust settles. I think we're just taking a snapshot too early in the timeframe to really know what this project is truly all about. And given that the author is an OSS dev for multiple languages, this intention makes a lot of sense.
zX41ZdbW 3y ago
Interesting to see a comparison with ClickHouse Native format.
SilverBirch 3y ago
The difficulty with serialization formats is that you're serializing for a reason. What if I want to read these dataframes into C++ or Python or some other language? It's pretty rare for pure performance to be the only consideration in choosing a serialization format - and unfortunately all the other reasons favour well-supported, long-established projects that can afford to hand out libraries in every language you could want.
ElectronBadger 3y ago
Thanks, added to the Big Book of Julia (https://adamwysokinski.codeberg.page/bbj https://adamwysokinski.codeberg.page/bbj).
moomin 3y ago
Isn’t the Python data frame format becoming an established format? I mean, there’s already commercial products that talk it natively.
- jzwinck 3y ago
  There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files no other library understands in an intuitive way. They mostly recommend "to_csv" with all the usual problems of CSV. NumPy does have an official documented format which is straightforward to read and write from other languages if you write some code yourself, but still doesn't have very wide support outside of Python.
  - dijksterhuis 3y ago
    Parquet is a binary file format and is very widely supported. I think there’s even an Excel plug-in for parquet these days.
    - radarsat1 3y ago
      > I think there’s even an Excel plug-in for parquet these days. While CSV comes with plenty of known problems, its one advantage, which apparently remains a significant one, is that it doesn't require any plugins or special software support at all, just a text editor. It seems that's a big enough feature that no attempt to fully replace it has really worked out. Amazing when you think about it, the power of plain text encoding. Of course I agree that it's often not the best choice but somehow it remains incredibly useful and can even be hard to make arguments against it on a team project because it just kind or works well enough in a lot of cases and is simply a lowest common denominator when people can't agree on what format to use.
  - benglish11 3y ago
    Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.
amj7e 3y ago
This is a pretty cool package and very useful. Is the project stale? What are the plans going forward?
- markkitti 3y ago
  There was a release 25 days ago. The HN submitter might even be the author.
  - amj7e 3y ago
    Yeah, but it was just a project.toml update. There haven't been serious updates to the src for a long time that's why Im asking. Hopefully, it resumes.
zarkenfrood 3y ago
Looks interesting and the syntax appears to be very similar to R data.table. It will be interesting to keep an eye on once it gets to v1.0.
tjrgergw 3y ago
What exactly is the advantage of this vs Arrow.jl?
- ChrisRackauckas 3y ago
  If I'm not mistaken, Arrow is an in-memory format and not necessarily a storage format. Though it is common to use Arrow with the Parquet serialization format, and the benchmarks here show a much faster read performance than Parquet while achieving around the same file size. From my read, that seems to be the impetus for the project (with comparisons made on Linkedin by the author as well) and so in theory Arrow and this could be complementary like Arrow and Parquet if someone setup the links.
  - tjrgergw 3y ago
    The benchmark comparison is vs Feather.jl.
  - zokier 3y ago
    > the benchmarks here show a much faster read performance than Parquet That felt bit surprising.. is parquet slow to read in general or is the julia implementation just slow?
    - tjrgergw 3y ago
      Arrow.jl, when compression is off, is just MMAP. I don't know how it can be faster than that.
braza 3y ago
It’s sad that Julia lost the so much preference in the Data Ecosystem. I think the syntax it’s clean, it’s fast, a lovely community and a clear vision on what needs to be done.
- goerz 3y ago
  Has it “lost” preference? I was more under the impression that it just hasn’t yet gained as much popularity as it deserves based on its technical merits, due to being quite a young language and the entrenchment of a large existing Python ecosystem.
bobbylarrybobby 3y ago
I think the initialism JDF may already be taken https://en.m.wikipedia.org/wiki/Jewish_Defense_League https://en.m.wikipedia.org/wiki/Jewish_Defense_League
- derstander 3y ago
  > I think the initialism JDF may already be taken Acronyms are lossy compression. I’d bet there’s a JDF that even predates the one you reference in one of your descendant comments. I’d be interested in knowing (though admittedly not enough to look) if there’s even a three letter acronym using the English alphabet that’s not already in use.
- eindiran 3y ago
  That's "JDL", which isn't the same 3 letters.
  - bobbylarrybobby 3y ago
    Whoops, I meant https://www.jewishdefensefund.org/ https://www.jewishdefensefund.org/
akdor1154 3y ago
If the parquet julia lib is slow, just go and fix that? Sorry I don't mean to discourage people trying cool new things, but seriously parquet is great, and I'd much rather see a diversity of great parquet implementations than a diversity of language specific parquet competitors.
- markkitti 3y ago
  The comparison in the README is to the parquet package for R.

55 points by xgdgsc 3y ago

jstx1 3y ago

I don't see the value of a language-specific format when we have things like parquet.

thumbuddy 3y ago
I think Julia is one of those researcher languages like Haskell so it's probably just someone having fun, writing a paper, or whatever.
- Buttons840 3y ago
  I don't agree with this. Julia does not have new ideas the same way Haskell does, and Julia has done a lot of work to be practical, like having a good way of calling Python and R code from early on in its life. If Julia has done anything new, it's probably related to being fast, but that's not exactly a novel idea.
  - thumbuddy 3y ago
    How come you never see anything practical made in Julia in production somewhere that isn't a pile of research mathematics? I personally can't think of a single tool or company. It's not like it's a young language either? I have seen dozens of niche papers written using it though. Being fair, I have seen dozens of not hundreds of products and mass deployed projects written in similar languages. Even Haskell made its way to a lot of desktops in the Linux world, and Haskell is super niche research territory imo.
    - ChrisRackauckas 3y ago
      The clinical analysis for the COVID-19 vaccines was pretty practical and that was Pumas used by Moderna. There's a JuliaCon talk about that from the Director Head of Pharmacometrics at Moderna. There's Formula 1 usage, there's a some satellites running Julia onboard, etc. Some are documented here https://juliahub.com/case-studies/ https://juliahub.com/case-studies/ though some people post what they are doing on the Discourse and Slack and so those end up being the most comprehensive source. Of course you can always argue whether 30 public examples is enough, or 50, or etc. but it's far from 0. And of course, most examples are never for public release.
      - thumbuddy 3y ago
        Fifteen years ish and there's maybe ten-thirty production use cases that kinda can't be verified because they all sound like research projects doesn't sound super great to me. No offense, but I do wonder if all of those cases could have been accomplished more easily/sustainably with another tool and if they exist simply because someone was selling the language to a company as a contractor as a form of vendor lock in or something. I guess I'll be curious once I install some software or an OS and see it brings in Julia in as a dependency or something. Otherwise I worry it's Matlab 2.0 with less of a mindshare...
  - gugagore 3y ago
    Julia is unique in using multiple dispatch pervasively. It did not invent multiple dispatch, but it certainly popularized it.
  - mrfox321 3y ago
    Sure it does. Have you seen the type system used to generically dispatch matrices to GPUs or cpus. Or the dispatching to give you auto-differentiation? Maybe it's not your definition of "new ideas", but they are really useful and original.
- VHRanger 3y ago
  Julia is still niche, but it's definitely used in production? The NY Fed have their main model of the economy in Julia (ported from Matlab)
  - thumbuddy 3y ago
    Sounds like research to me? I dunno...
  - agumonkey 3y ago
    this https://juliahub.com/case-studies/ny-fed/ https://juliahub.com/case-studies/ny-fed/ ?
    - VHRanger 3y ago
      This: https://frbny-dsge.github.io/DSGE.jl/latest/ https://frbny-dsge.github.io/DSGE.jl/latest/
ChrisRackauckas 3y ago
The repo is pretty clear on its intentions to not be a language-specific format: > Development Plans > I fully intend to develop JDF.jl into a language neutral format by version v0.4. However, I have other OSS commitments including R's {disk.frame} and hence new features might be slow to come onboard. But I am fully committed to making JDF files created using JDF.jl v0.2 or higher loadable in all future JDF.jl versions. It's beyond v0.4 and at v0.5.1 right now so it seems delayed, but the repo describes intentions to be a language-neutral format after the dust settles. I think we're just taking a snapshot too early in the timeframe to really know what this project is truly all about. And given that the author is an OSS dev for multiple languages, this intention makes a lot of sense.

zX41ZdbW 3y ago

Interesting to see a comparison with ClickHouse Native format.

SilverBirch 3y ago

The difficulty with serialization formats is that you're serializing for a reason. What if I want to read these dataframes into C++ or Python or some other language? It's pretty rare for pure performance to be the only consideration in choosing a serialization format - and unfortunately all the other reasons favour well-supported, long-established projects that can afford to hand out libraries in every language you could want.

ElectronBadger 3y ago

Thanks, added to the Big Book of Julia (https://adamwysokinski.codeberg.page/bbj https://adamwysokinski.codeberg.page/bbj).

moomin 3y ago

Isn’t the Python data frame format becoming an established format? I mean, there’s already commercial products that talk it natively.

jzwinck 3y ago
There is no practical Pandas DataFrame binary format on disk. They have "tofile" which is not portable across machines. They have "to_hdf" which produces files no other library understands in an intuitive way. They mostly recommend "to_csv" with all the usual problems of CSV. NumPy does have an official documented format which is straightforward to read and write from other languages if you write some code yourself, but still doesn't have very wide support outside of Python.
- dijksterhuis 3y ago
  Parquet is a binary file format and is very widely supported. I think there’s even an Excel plug-in for parquet these days.
  - radarsat1 3y ago
    > I think there’s even an Excel plug-in for parquet these days. While CSV comes with plenty of known problems, its one advantage, which apparently remains a significant one, is that it doesn't require any plugins or special software support at all, just a text editor. It seems that's a big enough feature that no attempt to fully replace it has really worked out. Amazing when you think about it, the power of plain text encoding. Of course I agree that it's often not the best choice but somehow it remains incredibly useful and can even be hard to make arguments against it on a team project because it just kind or works well enough in a lot of cases and is simply a lowest common denominator when people can't agree on what format to use.
- benglish11 3y ago
  Arrow is the more universal data format. It is already used to serialize non-python data structures in libraries like spark , snowflake-snowpark and polars. Pandas can read/write in arrow format as well. In pandas 2.0 the numpy backend can even be replaced with a pyarrow backend. I think this directly contradicts your statement about “no practical pandas binary format exists”. That may have been true a few years ago but certainly is not an issue any more.

amj7e 3y ago

This is a pretty cool package and very useful. Is the project stale? What are the plans going forward?

markkitti 3y ago
There was a release 25 days ago. The HN submitter might even be the author.
- amj7e 3y ago
  Yeah, but it was just a project.toml update. There haven't been serious updates to the src for a long time that's why Im asking. Hopefully, it resumes.

zarkenfrood 3y ago

Looks interesting and the syntax appears to be very similar to R data.table. It will be interesting to keep an eye on once it gets to v1.0.

tjrgergw 3y ago

What exactly is the advantage of this vs Arrow.jl?

ChrisRackauckas 3y ago
If I'm not mistaken, Arrow is an in-memory format and not necessarily a storage format. Though it is common to use Arrow with the Parquet serialization format, and the benchmarks here show a much faster read performance than Parquet while achieving around the same file size. From my read, that seems to be the impetus for the project (with comparisons made on Linkedin by the author as well) and so in theory Arrow and this could be complementary like Arrow and Parquet if someone setup the links.
- tjrgergw 3y ago
  The benchmark comparison is vs Feather.jl.
- zokier 3y ago
  > the benchmarks here show a much faster read performance than Parquet That felt bit surprising.. is parquet slow to read in general or is the julia implementation just slow?
  - tjrgergw 3y ago
    Arrow.jl, when compression is off, is just MMAP. I don't know how it can be faster than that.

braza 3y ago

It’s sad that Julia lost the so much preference in the Data Ecosystem. I think the syntax it’s clean, it’s fast, a lovely community and a clear vision on what needs to be done.

goerz 3y ago
Has it “lost” preference? I was more under the impression that it just hasn’t yet gained as much popularity as it deserves based on its technical merits, due to being quite a young language and the entrenchment of a large existing Python ecosystem.

bobbylarrybobby 3y ago

I think the initialism JDF may already be taken https://en.m.wikipedia.org/wiki/Jewish_Defense_League https://en.m.wikipedia.org/wiki/Jewish_Defense_League

derstander 3y ago
> I think the initialism JDF may already be taken Acronyms are lossy compression. I’d bet there’s a JDF that even predates the one you reference in one of your descendant comments. I’d be interested in knowing (though admittedly not enough to look) if there’s even a three letter acronym using the English alphabet that’s not already in use.
eindiran 3y ago
That's "JDL", which isn't the same 3 letters.
- bobbylarrybobby 3y ago
  Whoops, I meant https://www.jewishdefensefund.org/ https://www.jewishdefensefund.org/

akdor1154 3y ago

If the parquet julia lib is slow, just go and fix that? Sorry I don't mean to discourage people trying cool new things, but seriously parquet is great, and I'd much rather see a diversity of great parquet implementations than a diversity of language specific parquet competitors.

markkitti 3y ago
The comparison in the README is to the parquet package for R.