5 ms·

Total clickbait title, obviously being able to make an HTTP request to an endpoint, and get a CSV back, is an API. Respecting the “accept” header to return diff

by yashap 2y ago

Total clickbait title, obviously being able to make an HTTP request to an endpoint, and get a CSV back, is an API. Respecting the “accept” header to return different data formats, based on what the client requests, is totally fine/reasonable, but it doesn’t magically make an interface that’s designed for applications to program against not an API. Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass. I did think it was interesting to learn about fsspec, didn’t know about that! And this style of client library does seem like a good/convenient one for this specific Python data analysis use case. Enjoyed that part of the article, but had to wade through a fair bit of clickbait style writing to get there.

cogman10 2y ago
CSV is an awful non-standard standard that nobody should use. To understand how awful it is, you just have to ask two questions "How do I represent empty values?" "What do I do when my separator is contained in the data?" There are answers to these questions, but there's not a standard backing up those answers. And that's what makes CSV is PITA to deal with. It's such a loose "standard" that anything goes. xml and json are FAR better options even when you just want a table of data.
- taeric 2y ago
  JSON is not without its own problems. Not the least for getting very close to what folks want, but just falling short when things get complicated. https://en.wikipedia.org/wiki/CDATA https://en.wikipedia.org/wiki/CDATA is something far too few people pay attention to when considering the complications of object notation and why markup languages have some escape hatches.
- nradov 2y ago
  The CSV standard is RFC 4180, although it's merely informational. https://www.ietf.org/rfc/rfc4180.txt https://www.ietf.org/rfc/rfc4180.txt
- maxcoder4 2y ago
  I don't understand the question about empty values. A row with there empty values is just: ,, And yeah, dealing with separators is annoying, but pretty much every reader supports quoted values and escapes (not every writer cares up add them, though). In practice I use csv to process large amount of tabular data (like logs, events, etc) where I care about greppability and performance more than about potential for 0.001% of corrupted data. YMMV of course, if getting it 100% right is important use something else (JSON is not without sins too, consider that JSON numbers are often parsed as floats).
  - lelandbatey 2y ago
    I would guess that their point is that CSV does have any official null-vs-empty-string differentiation mechanism.
  - cogman10 2y ago
    Is it ''? Or is it ,,,,,,, Or is it tabtabtabtab Or perhaps it's "NULL,NULL,NULL" My part of my work is data ingestion and I've seen all these (and more) as answers to the "empty values" question. I'm not saying that other formats aren't without their problems, they certainly are. However, CSV doesn't just have those problems, it has multiple other problems on top of them. It's a basic idea with really obvious edge cases addressed in multiple ways depending on who is producing these documents.
    - jorams 2y ago
      CSV is extremely underspecified, but that doesn't mean it deserves the blame for software that fails to implement even the one thing it inherently specifies. A sequence of tabs is one value in a CSV. A sequence of semicolons is one value in a CSV. Any software that thinks otherwise is buggy, and unfortunately that includes some extremely popular software that is supposed to be good at tables (Excel).
- magicalhippo 2y ago
  > To understand how awful it is, you just have to ask two questions Those are rookie questions. What about when the data contains newline/crlf? What if the data contains the quote character and newline? And why is the file mostly Windows-1252 encoded except some fields that are sometimes, at random, UTF-8 encoded?
  - kevindamm 2y ago
    Still basic questions. The real kicker is when your fellow users are opening the CSV in a spreadsheet with a locale that prefers commas for fractional currency amount and then saving the file back.
- user_of_the_wek 2y ago
  It always pains me to say, but when processing tabular data from sources I don't personally control, I prefer xlsx format. It's usually well supported by most export processes (in contrast to e.g. the LibreOffice equivalent) and I never had problems finding a library that can parse it. And you never have to solve the "anything goes" CSV problems you're talking about.
- RockRobotRock 2y ago
  json lines/ndjson is great.
- paulddraper 2y ago
  > How do I represent empty values Empty string. For example, here are three in a row: ,, > What do I do when my separator is contained in my values? Use quotes. For example, here is a single comma: ","
calpaterson 2y ago
> Total clickbait title, obviously being able to make an HTTP request to an endpoint, and get a CSV back, is an API. Respecting the “accept” header to return different data formats, based on what the client requests, is totally fine/reasonable, but it doesn’t magically make an interface that’s designed for applications to program against not an API. I'm sorry about the title. As I said below; that was really meant in the spirit of fun. On the subject of csvbase's content negotiation - yes that is an API. That was covered here some time ago when I wrote about it before: https://news.ycombinator.com/item?id=37526047 https://news.ycombinator.com/item?id=37526047 The "no API" bit I'm talking about in this article is basically the "trick" (or whatever word you want to use) of avoiding having any user-facing interface and just hooking into stuff that is already there. There is no "API surface" here for the user to learn beyond a url scheme. I think that's nice. And it's mainly what I'm talking about. > Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass. Without wanting to sound too much like a sales pitch: csvbase does offer JSON. Try https://csvbase.com/calpaterson/opcodes-6502.jsonl https://csvbase.com/calpaterson/opcodes-6502.jsonl for JSON lines (or https://csvbase.com/calpaterson/opcodes-6502.json https://csvbase.com/calpaterson/opcodes-6502.json (no 'l') for a paged plain-JSON interface). I personally think there is no ideal format for this at the moment. JSON is very very large and slow to parse. CSV has well known problems though has massive compatibility and often works well in practice. Parquet is probably closest to the ideal and excellent in many respects but is quite complicated to parse (moreso than CSV? perhaps) and anyway is effectively unstreamable - actually quite annoying for something like csvbase where you really _don't_ want to materialise the dataset while serving it. > I did think it was interesting to learn about fsspec, didn’t know about that! Yes it is cool isn't it. Millions of downloads, terabytes of bandwidth of PyPI and no one has heard of it.
- lolinder 2y ago
  > I'm sorry about the title. As I said below; that was really meant in the spirit of fun. There's no need to apologize! I and many others understood exactly what you meant by "no API". Some other people didn't understand the distinction you were drawing and chose to interpret their lack of understanding as you misleading them somehow, but that's on them, not you. It was a great article that I thoroughly enjoyed! Thanks for sharing!
jrochkind1 2y ago
The site has an API. The python library has no python API, serving instead just as a plugin for fsspec. You then use pandas or anything else the same as you did before, the python library adds no (python) API. i guess technically "use the custom `csvbase://` scheme in the URIs you supply as input, instead of `http`" could be called an "API" if you really want to play gotcha. I think the point is legit -- the python programmer has to reference nothing specific to the client library here other than the custom URI scheme, to then use remote data from the site with any one of several existing python data libraries, via their own python APIs.

by yashap 2y ago

cogman10 2y ago

CSV is an awful non-standard standard that nobody should use. To understand how awful it is, you just have to ask two questions "How do I represent empty values?" "What do I do when my separator is contained in the data?" There are answers to these questions, but there's not a standard backing up those answers. And that's what makes CSV is PITA to deal with. It's such a loose "standard" that anything goes. xml and json are FAR better options even when you just want a table of data.

taeric 2y ago
JSON is not without its own problems. Not the least for getting very close to what folks want, but just falling short when things get complicated. https://en.wikipedia.org/wiki/CDATA https://en.wikipedia.org/wiki/CDATA is something far too few people pay attention to when considering the complications of object notation and why markup languages have some escape hatches.
nradov 2y ago
The CSV standard is RFC 4180, although it's merely informational. https://www.ietf.org/rfc/rfc4180.txt https://www.ietf.org/rfc/rfc4180.txt
maxcoder4 2y ago
I don't understand the question about empty values. A row with there empty values is just: ,, And yeah, dealing with separators is annoying, but pretty much every reader supports quoted values and escapes (not every writer cares up add them, though). In practice I use csv to process large amount of tabular data (like logs, events, etc) where I care about greppability and performance more than about potential for 0.001% of corrupted data. YMMV of course, if getting it 100% right is important use something else (JSON is not without sins too, consider that JSON numbers are often parsed as floats).
- lelandbatey 2y ago
  I would guess that their point is that CSV does have any official null-vs-empty-string differentiation mechanism.
- cogman10 2y ago
  Is it ''? Or is it ,,,,,,, Or is it tabtabtabtab Or perhaps it's "NULL,NULL,NULL" My part of my work is data ingestion and I've seen all these (and more) as answers to the "empty values" question. I'm not saying that other formats aren't without their problems, they certainly are. However, CSV doesn't just have those problems, it has multiple other problems on top of them. It's a basic idea with really obvious edge cases addressed in multiple ways depending on who is producing these documents.
  - jorams 2y ago
    CSV is extremely underspecified, but that doesn't mean it deserves the blame for software that fails to implement even the one thing it inherently specifies. A sequence of tabs is one value in a CSV. A sequence of semicolons is one value in a CSV. Any software that thinks otherwise is buggy, and unfortunately that includes some extremely popular software that is supposed to be good at tables (Excel).
magicalhippo 2y ago
> To understand how awful it is, you just have to ask two questions Those are rookie questions. What about when the data contains newline/crlf? What if the data contains the quote character and newline? And why is the file mostly Windows-1252 encoded except some fields that are sometimes, at random, UTF-8 encoded?
- kevindamm 2y ago
  Still basic questions. The real kicker is when your fellow users are opening the CSV in a spreadsheet with a locale that prefers commas for fractional currency amount and then saving the file back.
user_of_the_wek 2y ago
It always pains me to say, but when processing tabular data from sources I don't personally control, I prefer xlsx format. It's usually well supported by most export processes (in contrast to e.g. the LibreOffice equivalent) and I never had problems finding a library that can parse it. And you never have to solve the "anything goes" CSV problems you're talking about.
RockRobotRock 2y ago
json lines/ndjson is great.
paulddraper 2y ago
> How do I represent empty values Empty string. For example, here are three in a row: ,, > What do I do when my separator is contained in my values? Use quotes. For example, here is a single comma: ","

calpaterson 2y ago

> Total clickbait title, obviously being able to make an HTTP request to an endpoint, and get a CSV back, is an API. Respecting the “accept” header to return different data formats, based on what the client requests, is totally fine/reasonable, but it doesn’t magically make an interface that’s designed for applications to program against not an API. I'm sorry about the title. As I said below; that was really meant in the spirit of fun. On the subject of csvbase's content negotiation - yes that is an API. That was covered here some time ago when I wrote about it before: https://news.ycombinator.com/item?id=37526047 https://news.ycombinator.com/item?id=37526047 The "no API" bit I'm talking about in this article is basically the "trick" (or whatever word you want to use) of avoiding having any user-facing interface and just hooking into stuff that is already there. There is no "API surface" here for the user to learn beyond a url scheme. I think that's nice. And it's mainly what I'm talking about. > Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass. Without wanting to sound too much like a sales pitch: csvbase does offer JSON. Try https://csvbase.com/calpaterson/opcodes-6502.jsonl https://csvbase.com/calpaterson/opcodes-6502.jsonl for JSON lines (or https://csvbase.com/calpaterson/opcodes-6502.json https://csvbase.com/calpaterson/opcodes-6502.json (no 'l') for a paged plain-JSON interface). I personally think there is no ideal format for this at the moment. JSON is very very large and slow to parse. CSV has well known problems though has massive compatibility and often works well in practice. Parquet is probably closest to the ideal and excellent in many respects but is quite complicated to parse (moreso than CSV? perhaps) and anyway is effectively unstreamable - actually quite annoying for something like csvbase where you really _don't_ want to materialise the dataset while serving it. > I did think it was interesting to learn about fsspec, didn’t know about that! Yes it is cool isn't it. Millions of downloads, terabytes of bandwidth of PyPI and no one has heard of it.

lolinder 2y ago
> I'm sorry about the title. As I said below; that was really meant in the spirit of fun. There's no need to apologize! I and many others understood exactly what you meant by "no API". Some other people didn't understand the distinction you were drawing and chose to interpret their lack of understanding as you misleading them somehow, but that's on them, not you. It was a great article that I thoroughly enjoyed! Thanks for sharing!

jrochkind1 2y ago

The site has an API. The python library has no python API, serving instead just as a plugin for fsspec. You then use pandas or anything else the same as you did before, the python library adds no (python) API. i guess technically "use the custom `csvbase://` scheme in the URIs you supply as input, instead of `http`" could be called an "API" if you really want to play gotcha. I think the point is legit -- the python programmer has to reference nothing specific to the client library here other than the custom URI scheme, to then use remote data from the site with any one of several existing python data libraries, via their own python APIs.