4 ms·
No, it's a terrible API if retrieving the result '2000-10-26' requires downloading a 565 KB file. I don't want to seem overly negative -- zipped CSV files are
by crazygringo 3y ago
No, it's a terrible API if retrieving the result '2000-10-26' requires downloading a 565 KB file.
I don't want to seem overly negative -- zipped CSV files are fantastic when you want to import lots of data that you then re-serve to users. I would vastly prefer it over e.g. protobufs that are currently used for mass transit system's live train times, but have terrible support in many languages.
But it's incredibly wasteful to treat it like an API to retrieve a single value. I hope nobody would ever write something like that into an app...
(So the article is cool, it's just the headline that's too much of a "hot take" for me.)
- smallpipe 3y agoHow much data do I need to download before I can do a protobuf request ?
- eddythompson80 3y agoAre you counting the client size? TLS handshake? Because otherwise the answer is none.
- crazygringo 3y agoMass transit files are often a single protobuf file that is like a megabyte or something. I'm talking about zipped CSV files as easier to use than protobuf files. Neither is a request in this comparison.
- theLiminator 3y agoThey should ship parquet, supports predicate pushdown and is dramatically more compact, while you can't get row level data, it's great for analytical queries.
- IanCal 3y agoIMO this is the best simple option right now. For sorted data you only need the relevant row groups which can be tunable to sensible sizes for your data and access pattern.
- RobinL 3y agoTotally agree! So much that I wrote a whole article about it a while back! "Why parquet files are my preferred API for bulk open data" https://www.robinlinacre.com/parquet_api/ https://www.robinlinacre.com/parquet_api/
- swader999 3y agoParquet is a level better than csv but difficult to get customers to adopt and transmit in that format.
- calpaterson 3y agocsvbase does ship parquet! Just add ".parquet" - eg https://csvbase.com/meripaterson/stock-exchanges.parquet https://csvbase.com/meripaterson/stock-exchanges.parquet
- theLiminator 3y agoNice!
- pixl97 3y ago> I hope nobody would ever write something like that into an app... I have got some bad news for you... Not directly API related, but I remember supporting some land management application, and a new version of their software came out. Before that point it was working fine on our slow satellite offices that may have been on something like ISDN at the time. New version didn't work at all. The company said to run it on an RDP server. I thought their answer was bullshit and investigated what they were doing. One particular call, for no particular reason was doing a 'SELECT * FROM sometable' for no particular reason. There were many other calls that were using proper SQL select clauses in the same execution. I brought this up with the vendor and at first they were confused as hell how we could even figure this out. Eventually they pushed out a new version with a fixed call that was usable over slow speed lines, but for hells sake, how could they not figure that out in their own testing and instead pushed customers to expensive solutions?
- gray_-_wolf 3y ago> how could they not figure that out in their own testing This one is easy. Testing with little data on fast network (likely localhost).
- codetrotter 3y agoAlso, if there are any ORMs involved it could be that it’s not immediately obvious from their code itself that this would happen. I’ve seen code that was using an ORM, where they needed to find some data that matched certain criteria. With plain SQL it would have resulted in just a few rows of data. Put instead with their use of the ORM they ended up selecting all rows in the table from the database, and then looping over the resulting rows in code. The result of that was that the code was really slow to run, for something that would’ve been super fast if it wasn’t for the way they used the ORM in that code.
- mrighele 3y agoI just see recently an example of that, a REST call returning a few KB of data that would fetch a few million rows from the database and use 10+ GB of memory (unfortunately some people think that you should always use join fetches with JPA...).
- mhh__ 3y agoBut what if I want the average over time? The query depends on every value, should everything be computed on the server?
- netsharc 3y agoI'm an old fart, seeing the curl in every example is like nails on chalkboard. 565 KB, that's about 3 minutes download on a 28.8kbps modem I started getting online with...
- zamadatix 3y agoWhen you got that modem were you concerned with how long it would have taken to type/write the information sent over it or were you just happy it didn't matter anymore? I often wonder what I'll think of technology in say another 20 years but I can never tell if it's all just some general shift in perspective or, as you look farther back, if certain people were always just about a certain perspective (e.g. doing the most given the constraints) and technology changes enough that different perspectives (e.g. getting the most done easily for the same unit of human time) become the most common users for the newer stuff and that these people will also have a different perspective than the ones another 20 years down the line from them and so on. For example, maybe you think it's crazy to ask for the exact piece of data you need, I think it's crazy to do all the work to not just grab the whole half MB and just extract what you need quickly on the client side as often as you want, and someone equidistant from me will think it's nuts to not just feed all the data into their AI tool and ask it "what is the data for ${thing}" instead of caring about how the data gets delivered at all. Or maybe that's just something I hope for because I don't want to end up a guy who says everything is the same just done slower on faster computers since that seems... depressing in comparison :).
- maxbond 3y agoHere's an API built on top of this data that allows for more fine tuned queries. https://exchangeratesapi.io/ https://exchangeratesapi.io/ https://github.com/exchangeratesapi/exchangeratesapi https://github.com/exchangeratesapi/exchangeratesapi
- Hamuko 3y agoHow much bandwidth does the average API documentation page use?
- eddythompson80 3y agoDoes your client query the API documentation every time it's querying an API?
- ajcp 3y agoWell of course it does; how else would it know how to query the API? /s
- piaste 3y agoIt's historical data. There is zero reason to request it more than once a day, and the users of such data will be interested in wildly different filters or aggregates over that data. If this were being used to get the current rates, then yes, it would be a terrible design choice. But there are other services for that. This one fits its typical use case.
- jrm4 3y agoKinda feels like 20 years ago called and wants its argument back, like have you seen any javascript ever? 565 KB + the logic to get the big one is miniscule today by any reasonable factor.
- scubbo 3y agoTrue, though ironic that also OP is implicitly making the argument for GraphQL-like interfaces ("just specify the data that you want returned" rather than "get all the data, then process it"), which are themselves the New Hotness in some areas.
- thekashifmalik 3y agoAt risk of sounding pedantic, REST also allows for "just specify the data that you want returned" style APIs. Something like: curl https://www.ecb.europa.eu/stats/eurofxref/api?fields=Date&order=USD&limit=1
- scubbo 3y agoA very fair point well-made!
- Telemakhos 3y agoI think a lot of people in this thread are glossing over a difference in definitions. Some people see "API" as "how to get data, even if that means all data with no filtering." Personally, I regard downloading an entire table as downloading a data model without any logic operating on the model, and an API as logic that returns only part of the model filtered in some way that interests me.
- berkes 3y agoWhy do you assume that 565KB even matters? I've been building loads of financial software, both back and front ends. In frontend world it, sadly is quite common to send that amount of "data" over wires before even getting to actual data And in backends it's really a design decision. There's nothing faster than a Cron job parsing echange rates nightly and writing them to a purpose designed todays-rates.json served as static file to your mobile, web or microservices apps. Nothing implies your mobile app has to consume this zip-csv_over_http
- dang 3y ago> (So the article is cool, it's just the headline that's too much of a "hot take" for me.) It's one of those headlines that we'd never allow if it were an editorialization by the submitter (even if the submitter were the author, in this case), but since it's the article's own subtitle, it's ok. A bit baity but more in a whimsical than a hardened-shameless way. (I'm sure you probably noticed this but I thought the gloss might be interesting)