11 ms·
Client libraries are better when they have no API?
- vsnf 3y agoI don’t know if the post lives up to its title but it’s a very nice introduction to a little known piece of Python machinery
- btown 3y agoExaggerative title aside, it's really cool to learn about how fsspec lets you essentially make a custom filesystem in a few lines of code, as well as the (important) notion that the very existence of a Python package can cause a plugin to be installed at a deep level, to be used automatically by other packages. https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend https://filesystem-spec.readthedocs.io/en/latest/developer.h... https://setuptools.pypa.io/en/latest/userguide/entry_point.html#advertising-behavior https://setuptools.pypa.io/en/latest/userguide/entry_point.h... (As a side note here, you might think that "pip install" can only add functionality, and that you can add extra packages locally for your convenience, or keep them around when you switch branches, without having your dev environment diverge from production behavior. But if they register plugins with other packages via the entry point functionality, you might end up coding things that depend on behavior that's not present in other environments! This is especially common in the pytest ecosystem. CI is vital here!)
- calpaterson 3y ago> Exaggerative title aside I apologise for the title. Not intended to mislead, just to be a bit of fun :) > t's really cool to learn about how fsspec [...] I'm glad you took something from it. At the last place I worked I spent a lot of time building a library to persist dataframes (intended audience: data analysts) and in retrospect I wish we'd thought of the approach of just having a `csvbase://` url scheme via fsspec. Currently it doesn't support Parquet which they would have needed though. Still some work to do for csvbase to accept Parquet uploads.
- toolz 3y agoI'm curious what the author's loose definition of "API" is, if all of those API's that were presented were somehow not API's to the author (strictly speaking, they were definitely all API's).
- thomascgalvin 3y agoMy initial reaction was also "if I can talk to your software, that's an API." Reading through the article, though, the author isn't advocating for "no APIs," they're advocating for minimalist APIs that use agent detection to "do the right thing." Chrome tells the API that it can accept HTML, so the server sends data formatted inside of a web page, with an HTML table. Curl doesn't send that header, so the server sends unformatted CSV. But you could send an Accept header to get the HTML if you wanted. The benefit of this is that for most use cases, this will "just work." The downside is, if you want to view the CSV in a browser, or the web page in curl, you need to know (or guess) how the server is deciding what to send you, and take the correct action. An API documented with OpenAPI, while more complicated, explicitly tells you what you can do.
- lenerdenator 3y ago> My initial reaction was also "if I can talk to your software, that's an API." I've long held that software ages far better if you just eliminate the ability of people to provide it input.
- actionfromafar 3y agoNot much use then either, it's like encased in resin. Limit the ability to provide input, maybe?
- magicalhippo 3y agoAnd users. We wouldn't have nearly as many issues if it weren't for those darned users...
- 3y ago
- kennydude 3y ago> That barely qualifies as an API, it's just HTTP. So what possible use could a client library be? Wikipedia: API stands for Application Programming Interface. In the context of APIs, the word Application refers to any software with a distinct function. Interface can be thought of as a contract of service between two applications. This is an interface between two applications. I do like the simplicity of the APIs provided though, but they _are_ APIs
- dheera 3y agoPerhaps the author is referring to "annoying APIs"? - Where there is an "API key" and a "secret key" when in reality the idiot maintainers could have just concatenated the two and called it a "key". The customer doesn't need to know the abstraction on the server side - Where one needs to create an "account" and then a "project" before one can even create a "key". WTF is a project - Where one needs to do a dumb SMS 2FA to even get an API key - Where one needs to do multiple steps and try/except blocks to get a result in a plain format
- byteknight 3y agoI smell an OAuth2.0 hater =]
- dheera 3y agoYeah I hate Oath. I really wish APIs were as just having a free tier that you can just use, and if you want more, just POST $10 worth of Solana to some endpoint and the API gives your IP 10000 more requests. No accounts, no keys, just simple. I'm not a cryptocurrency fanatic but making APIs easier to access is one really good use case for it. It makes paying for an API as easy as paying for a lemonade at a street stand with cash. No accounts, no billing addresses, no subscriptions, just POST over a nickel along with your API request and get a response, and API owner gets paid.
- pjc50 3y agoThe accounts are usually an anti-abuse feature; few people want to provide services to faceless bad actors on the internet.
- adql 3y agoReturning CSV is an api thought. A terrible one, as "CSV" means a dozen ways of how to encode the special cases/escaping in the column
- robertclaus 3y agoThis is a great example of digging into existing functionality, understanding how it works, and building on it. I've seen way too many examples of re-inventing the wheel just because someone wasn't willing to put in the effort to research what's already out there. This is the type of example I would share with my junior developers on how to do good dev, especially in an older enterprise codebase.
- vladms 3y agoI think the thing that prevents efficient research is presentation of what is already there. There are lots of great things presented very poorly/undocumented. Anybody developing should worry much more about explaining clearly/presenting what is out there rather than build amazing features that nobody will use that they are not aware of. But this must be a management vision and effort, if you are always pressed just to deliver something working, you will end up with lots of people reinventing the wheel internally.
- hosh 3y agoThose examples in the article are what I would consider APIs. It is just that it is written in a way that is _composable_, and the primitives are scoped in a way that composing them versatile. That there was a library that facilitates that composition allows for an ecosystem. The Rack protocol did something similar for http servers in the Ruby world, allowing for a number of middleware. Whole frameworks (Rails, Sinatra, as examples) could be mounted on specific routes.
- kstrauser 3y agoConfession: I hate that. Hate it. It does fiddly magic at package installation time and then there’s zero indication that anything’s happened. It’s magic. When I’m looking at the end result, there’s no hint whatsoever at how those new URLs are being handled. Did Python somehow learn csvbase://? Pandas? Where’s that coming from? How do I fix it if it breaks? Please don’t do that. Edit: Explicit patching would be just fine, like: from my_project import csvbase_patch csvbase_patch() p = pandas.pd(“csvbase://…”) Then there’s a giant indicator inside the same module that there’s magic happening. If I cmd-f “csvbase”, the magic string in the URL, I’ll stumble across that patch function. Then I can jump to its definition to see what’s happening. This is Python. Hidden monkeypatching isn’t how we do things.
- corytheboyd 3y agoPlease, for the love of god, listen to this commenter. I’m coming from the Ruby side of things, and hidden monkeypatching will be the death of us all.
- actionfromafar 3y agoI was going to say "this reminds me of Ruby". :-D
- kstrauser 3y agoWhen I wrote “This is Python”, I resisted the urge to finish with “not Ruby”.
- sloowm 3y agoThanks for the edit. I agree it should be explicit and then I actually think it's quite a nice solution.
- brigadier132 3y agoIt's absolutely terrible. My first job out of college I was basically only fixing bugs in a python codebase. 99% of the bugs were from monkey patching and null pointer exceptions where people were trying to access properties on objects in places they were not instantiated yet. People that write code like this should be forced into bug jail like i was for a year.
- jesprenj 3y agoWouldn't it be better to just register http as a protocol with fsspec and use http://csvbase.example/username/dataset/table instead of creating a whole protocol handler just for an online service?
- Macha 3y agoWhat happens if N projects attempt this?
- calpaterson 3y agohttp is already registered as a protocol and you can indeed do that already. It's printed on every table page. eg on https://csvbase.com/calpaterson/opcodes-6502 https://csvbase.com/calpaterson/opcodes-6502: import pandas as pd; opcodes_6502 = pd.read_csv('https://csvbase.com/calpaterson/opcodes-6502', index_col=0) But writing back is harder. It's not easy to make pandas do an HTTP PUT and then insert the HTTP basic auth and so on. Plus (not discussed in the article) there is a cache to avoid redownloading the data when it hasn't changed, which is my personal bete noire in "data science" such as it is.
- jesprenj 3y agoAh, now I understand the motive! I wasn't thinking about writing back to csvbase at all.
- byteknight 3y agoFrom a low to low-mid developer, I feel like this is a great introduction (or at least good reason for) inheritence (from a 10k foot perspective) and why its awesome
- zzzeek 3y ago> On csvbase, you can just curl any table url and get a csv file. that's still an API
- calpaterson 3y agoYes, though as I try to explain in https://news.ycombinator.com/item?id=39993176 https://news.ycombinator.com/item?id=39993176 the bit I was trying to get at is that for `pd.read_csv('csvbase://calpaterson/opcodes-6502')` there is no API surface at all. And you can do writes with `pd.DataFrame.to_csv`. I personally think that is fun. PS: Thanks for all of your work over the years (decades?) on sqlalchemy. I'm sure you're not too surprised to learn that the lions share of code in csvbase is calls to SQLA core.
- zzzeek 3y ago> I'm sure you're not too surprised to learn that the lions share of code in csvbase is calls to SQLA core. Really! a bit surprised sure, wasn't really sure what the tool was actually doing to persist data :)
- layer8 3y agoAfter reading the article and the comments, I’m still not sure if TFA means “API” in the HTTP/REST sense or in the programming-language library sense.
- dbrower 3y agoHow is that not an API? It’s offloaded the work to an adapter.
- therealwardo 3y agoserverless - someone else's servers adaptors - someone else's APIs
- yashap 3y agoTotal clickbait title, obviously being able to make an HTTP request to an endpoint, and get a CSV back, is an API. Respecting the “accept” header to return different data formats, based on what the client requests, is totally fine/reasonable, but it doesn’t magically make an interface that’s designed for applications to program against not an API. Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass. I did think it was interesting to learn about fsspec, didn’t know about that! And this style of client library does seem like a good/convenient one for this specific Python data analysis use case. Enjoyed that part of the article, but had to wade through a fair bit of clickbait style writing to get there.
- cogman10 3y agoCSV is an awful non-standard standard that nobody should use. To understand how awful it is, you just have to ask two questions "How do I represent empty values?" "What do I do when my separator is contained in the data?" There are answers to these questions, but there's not a standard backing up those answers. And that's what makes CSV is PITA to deal with. It's such a loose "standard" that anything goes. xml and json are FAR better options even when you just want a table of data.
- taeric 3y agoJSON is not without its own problems. Not the least for getting very close to what folks want, but just falling short when things get complicated. https://en.wikipedia.org/wiki/CDATA https://en.wikipedia.org/wiki/CDATA is something far too few people pay attention to when considering the complications of object notation and why markup languages have some escape hatches.
- nradov 3y agoThe CSV standard is RFC 4180, although it's merely informational. https://www.ietf.org/rfc/rfc4180.txt https://www.ietf.org/rfc/rfc4180.txt
- haroldl 3y agoPerson 1 adds code that uses this with Pandas. Person 2 sees the csvbase:// URIs and copy/pastes them to use with some other library Foozle that uses URIs and it works because Foozle also happens to use fsspec. Foozle migrates off of fsspec to some other filesystem wrapper. 3 years later, after persons 1 and 2 have left the company, person 3 bumps the version of Foozle the project uses and suddenly tests are failing because they can't resolve csvbase:// URIs, but the Foozle release notes say nothing about removing support for csvbase:// URIs.
- koliber 3y agoAction at a distance, and across time. I just read this article and have very conflicting feelings about it. It is clever, and the nice kind of clever that does not require one to be a mega-brain. On the other hand, it creates invisible and uncontrollable dependencies, such as the one you describe. Another drawback: something is broken and I want to set a debug point in the code that fetches the CSV data. Unless you know about fsspec it will be hard to follow the breadcrumbs to know how this library injects itself into your code.
- lolc 3y agoIn this hypothetical case, there would be some floundering to discover the cause, and a two-line fix. Rather than having the two extra lines from the start. Seems acceptable risk to me.
- pantsforbirds 3y agoHow is this different from the risk that a dataframe library will stop supporting S3:// URIs? Why wouldn't Foozle's release notes mention a major protocol change? How is this scenario different from any other library making a major change without documenting it?
- reubenmorais 3y agoBecause fsspec handles the extensions directly, Foozle does not have to opt in to get the csvbase:// support, all that's required is for the csvbase Python client to be installed. So users of Foozle might start depending on these extensions without Foozle ever finding out about it. But I guess that's the reality of making libraries, even bugs will be relied on, we really don't have any methods for evolving software ecosystems reliably and compatibly.
- jvmancuso 3y ago> For whatever reason, fsspec is not that well known. It has less than 800 stars on github. Sorry, I was the 800th, your post is now outdated!
- 1vuio0pswjnm7 3y agoWhen I started using the www the term "API" referred to functions in programming languages not to httpd configurations. It has been bizarre to see the appearance of this terminology to refer to websites. In every instance I have seen, the free "web API" involves an extra HTTP header(s) that, in lieu of a common one such as "User-Agent", can be used to track, rate-limit and/or selectively block a www user. The upside of the "web API" idea IMO is the serving of public information in formats other than HTML or PDF. It's great. But why not just do this without using the extra HTTP header(s), tracking and limitations.
- tjtang2019 3y ago[dead]
- wodenokoto 3y agoWhy not use the http link to the csv data?
- Bendy 3y agoThe just swear jar’s overflowing for this article. It sounds like an API and it looks like an API, but don’t let that fool you…