5 ms·

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers an

by whakim 3y ago

To me this criticism feels excessive. It feels like the author is describing their frustrations with internal usage of CSVs - there's no mention of customers and non-technical stakeholders at all. I think it goes without saying that Parquet files and other non-human-readable formats are a nonstarter when working with external stakeholders and the last paragraph makes that clear - if the end-user wants CSV, give them CSV. I also think we shouldn't blindly dismiss the performance drawbacks of CSV when working with data pipelines. At even modest scales it becomes hard to work with very large CSVs because the data often doesn't fit into memory, a problem easily solved by Parquet and other formats assuming you only need a subset.

chaps 3y ago
Why is this getting downvoted? They're right that the criticism is pretty excessive: "Maybe his next article will be about how airlines should speak Esperanto because English is such a flawed language. That’s a clever and unique observation." Hm.
- prepend 3y ago
  I got a little snarky but I think the analogy holds. Esperanto is a superior language to English. And English has many flaws. Theoretically it would be better to have all pilots and airports learn an efficient language. But it would be stupid and immature to seriously write a blog post about that, especially without talking about all the flaws in that plan.
  - lmm 3y ago
    > Esperanto is a superior language to English. Not really. You can say it's more regular but that's because it sees barely any actual use; if it ever gained popularity it wouldn't stay regular. (And given that pilots speak in set phrases anyway, irregularity isn't really an issue). It's not a great language by any stretch, it's an awkward mismash of four european languages; sure it sounds kind of nice in an Italianate way, but if that's what you want then why not just speak Italian?
    - nrdvana 3y ago
      It would stay regular if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". English really is a disaster of a language. There was a(nother) great XKCD about it just a few days ago. https://xkcd.com/2907/ https://xkcd.com/2907/
      - Stranger43 3y ago
        Correction it's a disaster of at least 4 languages and this is probably why English is so hard to dethrone as it have no strict ownership so everyone is kind of equal in speaking it incorrectly. Sometimes lack of rigidity is actually an feature that allows for things to sort of work that would be politically impossible if thinks had to be specified formally before being used.
        nrdvana 3y ago
        I'm pretty sure the reason English is hard to dethrone is because Britain ~helped~ forced the various colonies to join world commerce using English, so they started teaching it to entire generations as the national second-language, and then because the USA dominated world commerce after that in a sort of "we'll let you in on the game if you speak our language and use our money" sort of way.
      - prmoustache 3y ago
        > if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". There is no way it can work. People don't care about governing bodies when they speak a language.
        maxcoder4 3y ago
        They kind of do. In my language there exists a central "governing" body that decides what is correct, and some "incorrect" regionalisms are disappearing because of it.
  - vouaobrasil 3y ago
    > Esperanto is a superior language to English. And English has many flaws. At least one one perspective, English is superior. That perspective is that you can actually use it in almost any modern situation because it has been tried and tested globally.
    - prepend 3y ago
      This is the point of my analogy. English:Esperanto::csv:parquet (Although I think parquet is much more useful than Esperanto and may eventually end up dethroning csv)
jgord 3y ago
btw, xsv has solved most of my problems dealing with 'large' 40GB csv files
- throwaway2037 3y ago
  xsv? I never heard of it. This one? https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv If yes, looks very cool. Plus, bonus HN/Internet points for being written in Rust!
  - jgord 3y ago
    yep .. his utils are most excellent.
- vmchale 3y ago
  its parser is buggy! https://github.com/BurntSushi/xsv/issues/337 https://github.com/BurntSushi/xsv/issues/337 (I ran into this issue myself)
  - burntsushi 3y ago
    I just responded to that. It isn't the parser that's a buggy. The parser handles the quotes just fine. If it didn't, that would be a serious bug in the `csv` crate that oodles of users would run into all the time. There would be forks over it if it had persisted for that long. The problem is that `xsv table` doesn't print the parsed contents. It just prints CSV data, but with tabs, and then those tabs are expanded to spaces for alignment. Arguably it ought to print the parsed contents, i.e., with quotes unescaped. It almost looks like it's doing that because the quotes are removed in one case, but that's only because the CSV writer knows when it doesn't need to write quotes.
    - mardifoufs 3y ago
      Ok this might sound stupid, and a bit unrelated, but you make so many great tools that I can't help but ask. How do you start planning and creating for a tool that needs to "follow standards"(in this case I know CSV is under specified but still!), is it by iteration or do you try to set and build a baseline for all the features a certain tool needs? Or do you just try to go for modularity from the get go even if the problem space is "smaller" for stuff like csv for example.
prepend 3y ago
I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records. Csv is not very performant, but it doesn’t matter for these use cases. I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my internal processes. Just for when someone sends me data or I have to send it back to them. That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. But that’s usually just for dev or debugging and anything that sticks around will be working with data in some proper store (usually parquet distributed across nodes).
- Certhas 3y ago
  If I run a simulation workload it's pretty easy to generate gigabytes of data per second. CSV encoding adds a huge overhead space and time wise, so saving trajectories to disc for later analysis can easily become the bottleneck. I have had many other situations where CSV was the bottleneck. I still would default to CSV first in many situations because it's robust and easily inspected by hand.
- smcin 3y ago
  > That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. How much RAM do you have? What's the ratio of [smallest CSV file which bottlenecks]/[your RAM]?
  - prepend 3y ago
    My dev workstation has 96gb. I don’t work with massive data files so I’ve never really hit my limit. I think the biggest raw data file I’ve opened was 10-20gb.
- whakim 3y ago
  That may be your experience, but certainly not a universal experience (and apparently not the author's, either). In my experience, it's pretty easy to have CSVs (or Parquet files, or whatever) that are tens or hundreds of GBs in size. The space savings from a more modern file format are significant, as is the convenience of being able to specify and download/open only a subset of rows or columns over the network. Most of us don't have workstations with 50GB of RAM, because it's far more cost-effective to use a Cloud VM if you only occasionally need that much memory. That being said, the real point here is that folks blindly use CSVs for internal-facing processes even though there's no particular reason to, and they have plenty of drawbacks. If you're just building some kind of ETL pipeline why wouldn't you use Parquet? It isn't as if you're opening stuff in Excel.
strken 3y ago
I very much agree with this. For an integration where you have control over both ends of the pipeline, CSV is not optimal unless there's existing work to build on, and even then it's a legacy choice. Parquet and Avro are widely supported in backend languages and also in data analysis. I don't think the article is talking about exported-like-a-jpeg, but instead exported-like-a-daily-report-run: the data scientist doing the exporting is probably using R or Pandas instead of Excel, and can reasonably be expected to read https://arrow.apache.org/docs/r/reference/read_parquet.html https://arrow.apache.org/docs/r/reference/read_parquet.html.

by whakim 3y ago

chaps 3y ago

Why is this getting downvoted? They're right that the criticism is pretty excessive: "Maybe his next article will be about how airlines should speak Esperanto because English is such a flawed language. That’s a clever and unique observation." Hm.

prepend 3y ago
I got a little snarky but I think the analogy holds. Esperanto is a superior language to English. And English has many flaws. Theoretically it would be better to have all pilots and airports learn an efficient language. But it would be stupid and immature to seriously write a blog post about that, especially without talking about all the flaws in that plan.
- lmm 3y ago
  > Esperanto is a superior language to English. Not really. You can say it's more regular but that's because it sees barely any actual use; if it ever gained popularity it wouldn't stay regular. (And given that pilots speak in set phrases anyway, irregularity isn't really an issue). It's not a great language by any stretch, it's an awkward mismash of four european languages; sure it sounds kind of nice in an Italianate way, but if that's what you want then why not just speak Italian?
  - nrdvana 3y ago
    It would stay regular if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". English really is a disaster of a language. There was a(nother) great XKCD about it just a few days ago. https://xkcd.com/2907/ https://xkcd.com/2907/
    - Stranger43 3y ago
      Correction it's a disaster of at least 4 languages and this is probably why English is so hard to dethrone as it have no strict ownership so everyone is kind of equal in speaking it incorrectly. Sometimes lack of rigidity is actually an feature that allows for things to sort of work that would be politically impossible if thinks had to be specified formally before being used.
      - nrdvana 3y ago
        I'm pretty sure the reason English is hard to dethrone is because Britain ~helped~ forced the various colonies to join world commerce using English, so they started teaching it to entire generations as the national second-language, and then because the USA dominated world commerce after that in a sort of "we'll let you in on the game if you speak our language and use our money" sort of way.
    - prmoustache 3y ago
      > if there was a strict governing body for it that wasn't a Webster-style "whatever people are speaking is the new definition of correct". There is no way it can work. People don't care about governing bodies when they speak a language.
      - maxcoder4 3y ago
        They kind of do. In my language there exists a central "governing" body that decides what is correct, and some "incorrect" regionalisms are disappearing because of it.
- vouaobrasil 3y ago
  > Esperanto is a superior language to English. And English has many flaws. At least one one perspective, English is superior. That perspective is that you can actually use it in almost any modern situation because it has been tried and tested globally.
  - prepend 3y ago
    This is the point of my analogy. English:Esperanto::csv:parquet (Although I think parquet is much more useful than Esperanto and may eventually end up dethroning csv)

jgord 3y ago

btw, xsv has solved most of my problems dealing with 'large' 40GB csv files

throwaway2037 3y ago
xsv? I never heard of it. This one? https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv If yes, looks very cool. Plus, bonus HN/Internet points for being written in Rust!
- jgord 3y ago
  yep .. his utils are most excellent.
vmchale 3y ago
its parser is buggy! https://github.com/BurntSushi/xsv/issues/337 https://github.com/BurntSushi/xsv/issues/337 (I ran into this issue myself)
- burntsushi 3y ago
  I just responded to that. It isn't the parser that's a buggy. The parser handles the quotes just fine. If it didn't, that would be a serious bug in the `csv` crate that oodles of users would run into all the time. There would be forks over it if it had persisted for that long. The problem is that `xsv table` doesn't print the parsed contents. It just prints CSV data, but with tabs, and then those tabs are expanded to spaces for alignment. Arguably it ought to print the parsed contents, i.e., with quotes unescaped. It almost looks like it's doing that because the quotes are removed in one case, but that's only because the CSV writer knows when it doesn't need to write quotes.
  - mardifoufs 3y ago
    Ok this might sound stupid, and a bit unrelated, but you make so many great tools that I can't help but ask. How do you start planning and creating for a tool that needs to "follow standards"(in this case I know CSV is under specified but still!), is it by iteration or do you try to set and build a baseline for all the features a certain tool needs? Or do you just try to go for modularity from the get go even if the problem space is "smaller" for stuff like csv for example.

prepend 3y ago

I deal with gig size csvs all the time and don’t have any performance issues. These aren’t huge files, but decent sized. And most are just a few megs and only thousands to millions of records. Csv is not very performant, but it doesn’t matter for these use cases. I’ll also add that I’m not working with the csvs, they are just I/o. So any memory issues are handled by the load process. I certainly don’t use csvs for my internal processes. Just for when someone sends me data or I have to send it back to them. That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. But that’s usually just for dev or debugging and anything that sticks around will be working with data in some proper store (usually parquet distributed across nodes).

Certhas 3y ago
If I run a simulation workload it's pretty easy to generate gigabytes of data per second. CSV encoding adds a huge overhead space and time wise, so saving trajectories to disc for later analysis can easily become the bottleneck. I have had many other situations where CSV was the bottleneck. I still would default to CSV first in many situations because it's robust and easily inspected by hand.
smcin 3y ago
> That being said my workstation is pretty big and can handle 10s of gigs of csv before I care. How much RAM do you have? What's the ratio of [smallest CSV file which bottlenecks]/[your RAM]?
- prepend 3y ago
  My dev workstation has 96gb. I don’t work with massive data files so I’ve never really hit my limit. I think the biggest raw data file I’ve opened was 10-20gb.
whakim 3y ago
That may be your experience, but certainly not a universal experience (and apparently not the author's, either). In my experience, it's pretty easy to have CSVs (or Parquet files, or whatever) that are tens or hundreds of GBs in size. The space savings from a more modern file format are significant, as is the convenience of being able to specify and download/open only a subset of rows or columns over the network. Most of us don't have workstations with 50GB of RAM, because it's far more cost-effective to use a Cloud VM if you only occasionally need that much memory. That being said, the real point here is that folks blindly use CSVs for internal-facing processes even though there's no particular reason to, and they have plenty of drawbacks. If you're just building some kind of ETL pipeline why wouldn't you use Parquet? It isn't as if you're opening stuff in Excel.

strken 3y ago

I very much agree with this. For an integration where you have control over both ends of the pipeline, CSV is not optimal unless there's existing work to build on, and even then it's a legacy choice. Parquet and Avro are widely supported in backend languages and also in data analysis. I don't think the article is talking about exported-like-a-jpeg, but instead exported-like-a-daily-report-run: the data scientist doing the exporting is probably using R or Pandas instead of Excel, and can reasonably be expected to read https://arrow.apache.org/docs/r/reference/read_parquet.html https://arrow.apache.org/docs/r/reference/read_parquet.html.