4 ms·

Data isn't "locked up" in CSV, rather it is available in CSV. CSV is one of the most open things possible. As a programmer I do a lot of one-time makeshift da

Data isn't "locked up" in CSV, rather it is available in CSV. CSV is one of the most open things possible. As a programmer I do a lot of one-time makeshift data reports for other people, and I always use CSV (or precisely, tab separated) because that's what every program happily emits and consumes. If it does not, it's trivial to transform thanks to UNIX sort, awk and uniq.

thejteam 13y ago
If I were to build a MVP for any type of data charting or visualization or analysis the first data format that I would think of supporting would be CSV.
- sehrope 13y ago
  You'd think so but when the use case involves dynamically retrieving fresh data it becomes one of the last things to work on. Lots of people have CSV data and it makes sense to support it but since it's generally a manual process it doesn't deliver a good "flow" for the customer. If they have to repeatedly export/import their data manually to have the chart data update it'll seem like a chore. Compare this to connecting to a customer's database directly. You can refresh your (cached) chart data on your own schedule and the customer doesn't need to be involved at all. Oh and if anyone is thinking, "Yeah but I can schedule a cron job to extract the CSV file and publish it every X minutes", yes you could do that but again that's work for the customer. If the pitch is "sign up, plug in to your database, and boom charts!" there really should be a "schedule automated cron job" component. All that aside, it is useful to be able to manually add data like this. Particularly for static data where the manual work is infrequent.
zbanks 13y ago
Once your CSV (or TSV) files start having quoted fields, they become very tricky to parse using standard multi-purpose tools like sort, awk, & uniq. It's hard enough when you have delimiters in quoted fields, but dealing with quoted newlines starts to become unreasonable, especially for line-based tools. CSV files, as you say, are absolutely wonderful to create. Problems come up when you try to parse files other people write. Not everyone follows RFC 4180.
- guard-of-terra 13y ago
  With tab separated values this is not a big problem in practice. On the other hand, you can sort tsv, but you can't sort quoted csv.
  - sopooneo 13y ago
    Why not? I ask in earnest.
    - guard-of-terra 13y ago
      sort -n gets confused by quotes.
- jwilliams 13y ago
  Problems come up when you try to parse files other people write. Not everyone follows RFC 4180. Plus you've got encodings. If you're accepting CSVs from users, they'll generally come from Excel, which will produce different encoding in different circumstances.
voltagex_ 13y ago
OK, here's one to do with "just" sort, awk and uniq ;) http://naa.gov.au/naaresources/govhack-2013/PassengersArrivalIndex-Final.csv http://naa.gov.au/naaresources/govhack-2013/PassengersArriva...

thejteam 13y ago

If I were to build a MVP for any type of data charting or visualization or analysis the first data format that I would think of supporting would be CSV.

sehrope 13y ago
You'd think so but when the use case involves dynamically retrieving fresh data it becomes one of the last things to work on. Lots of people have CSV data and it makes sense to support it but since it's generally a manual process it doesn't deliver a good "flow" for the customer. If they have to repeatedly export/import their data manually to have the chart data update it'll seem like a chore. Compare this to connecting to a customer's database directly. You can refresh your (cached) chart data on your own schedule and the customer doesn't need to be involved at all. Oh and if anyone is thinking, "Yeah but I can schedule a cron job to extract the CSV file and publish it every X minutes", yes you could do that but again that's work for the customer. If the pitch is "sign up, plug in to your database, and boom charts!" there really should be a "schedule automated cron job" component. All that aside, it is useful to be able to manually add data like this. Particularly for static data where the manual work is infrequent.

zbanks 13y ago

Once your CSV (or TSV) files start having quoted fields, they become very tricky to parse using standard multi-purpose tools like sort, awk, & uniq. It's hard enough when you have delimiters in quoted fields, but dealing with quoted newlines starts to become unreasonable, especially for line-based tools. CSV files, as you say, are absolutely wonderful to create. Problems come up when you try to parse files other people write. Not everyone follows RFC 4180.

guard-of-terra 13y ago
With tab separated values this is not a big problem in practice. On the other hand, you can sort tsv, but you can't sort quoted csv.
- sopooneo 13y ago
  Why not? I ask in earnest.
  - guard-of-terra 13y ago
    sort -n gets confused by quotes.
jwilliams 13y ago
Problems come up when you try to parse files other people write. Not everyone follows RFC 4180. Plus you've got encodings. If you're accepting CSVs from users, they'll generally come from Excel, which will produce different encoding in different circumstances.

voltagex_ 13y ago

OK, here's one to do with "just" sort, awk and uniq ;) http://naa.gov.au/naaresources/govhack-2013/PassengersArrivalIndex-Final.csv http://naa.gov.au/naaresources/govhack-2013/PassengersArriva...