18 ms·

API design is stuck in the past

146 points by kentonv 6y ago

hcarvalhoalves 6y ago
> In other words, the benefit of maintaining type signatures now well outweighs the cost. That's a rather bold claim to begin your argument, but there are no references. There's a good review of related studies here: https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/
- karmakaze 6y ago
  I think a better way of phrasing this would be "the cost of maintaining type signatures has been reduced well below the benefits" since I believe buf makes tools to do just that.
  - kentonv 6y ago
    I think it's both that the costs are going down, and that the benefits are going up, in the form of useful tooling that couldn't exist without schemas to drive it. Quantifying these costs and benefits is, of course, notoriously difficult.
- spinningslate 6y ago
  Agreed. The unconditional nature of the statement detracts from its message. Had it said: "we see benefits in static types for APIs, and it's working for us - here's why" it would have had some credibility. And provided some insight on why it might work for others too. Contrast with DHH's articulation on the corecursive podcast [0]. Paraphrasing: * Dynamic types work for him * People don't all think the same. So static typing is fine too. Jeremy Howard made essentially the same point on TWIML AI [1]. The point is there's far more to success than static vs dynamic typing. Personal preferences, team skills, culture, task at hand, nature of domain, commercial imperatives, code base size, and many other things besides all influence choice. To suggest there's a single, global answer to static vs dynamic - whether for languages or APIs - is a misguided and unhelpful simplification. [0]: https://corecursive.com/045-david-heinemeier-hansson-software-contrarian/ https://corecursive.com/045-david-heinemeier-hansson-softwar... [1]: https://twimlai.com/whats-next-for-fast-ai-w-jeremy-howard/ https://twimlai.com/whats-next-for-fast-ai-w-jeremy-howard/
- mistersys 6y ago
  It's fairly obvious that for the any public facing API, type signatures are going to be a net positive, because you have to write input validation for your API anyway to have any hope of it being secure. You can either write as a series of buggy, hard to understand if statements or use a well-defined schema tool of some sort. Furthermore, once you have the schema you can generate client APIs in e.g Typescript for your APIs instead of wasting engineering effort on writing them manually. In terms of static types generally, outside of APIs, the equation can be different. However, if you are creating a library that many developers will use, given the state of IDEs, your just hurting your users (developers) by not providing static types. I use Typescript in every single project not because of supposed benefit of "type grantees", but because I don't need to spend all day flipping between some developers poorly written, incomplete docs to understand how to use an API. My code editor will tell me the whole format and I can explore the APIs with goto definition, demonstrably reducing development time.
hashamali 6y ago
Surprised there is no mention of OpenAPI. A spec driven API development process solves most of the problems described without having to switch to schematized data serialization formats.
- kentonv 6y ago
  I admit I haven't written an OpenAPI schema myself. But, looking at the docs, I have a hard time imagining that I'd want to do API-driven development with it. Let me explain what I mean by that... When I start a new project, or a major new feature in an existing project, usually the first thing I do is write out the Protobuf or Cap'n Proto schemas describing the APIs and data structures. Even for internal-facing APIs. APIs are how components talk to each other, and defining the APIs defines which components are responsible for what in a very clear way. Often I find writing out the APIs is actually the easiest, most precise way to communicate a design with my teammates; much better than prose in a design doc. For this purpose, it's actually pretty important that the schema language be something that's not cumbersome to write. Protobuf and Cap'n Proto feature dedicated schema languages that feel like a programming language. It looks like OpenAPI has you writing schemas themselves in JSON or YAML. Ick. And once I've written those schemas, I run the code generator and poof, I now have nice, type-safe classes in my programming language of choice implementing my schema. This lets me move a lot faster implementing, being able to use my IDE's auto-complete and whatnot. Does OpenAPI generate code for you? I have worked on projects that use it and didn't have any generated code, so I am assuming not... I think this is what the article is trying to get at. (Disclosure: I'm the author of Cap'n Proto; long ago I was the author of Protobuf v2; and I am a (small) investor in Buf because I like what they're doing, but I don't work with them.)
  - hashamali 6y ago
    OpenAPI is just the format for describing APIs for various use cases such as documentation, runtime validation, and yes, even code generation. Depending in the language you use, the support for code generation may vary though. I’m certainly not against the flow you’re describing. Aside from having to wrangle JSON/YAML instead of a more concise DSL, it’s not too different from what you’d do with OpenAPI.
TeeWEE 6y ago
We are using GRPC and for Rest OpenAPI and write our own codegenerators for OpenAPI... How is this in the past? I guess most developers don't use OpenAPI and just return JSON... Thats indeed an old way of development.
recursivedoubts 6y ago
I don't have strong feelings about this article except for the fact that it once again uses REST to mean "JSON over HTTP" when, in fact, REST requires HATEOAS and the vast majority of APIs that claim to be RESTful aren't, and almost certainly shouldn't be, since they aren't using a hypermedia/hypertext. https://intercoolerjs.org/2016/01/18/rescuing-rest.html https://intercoolerjs.org/2016/01/18/rescuing-rest.html https://intercoolerjs.org/2016/02/17/api-churn-vs-security.html https://intercoolerjs.org/2016/02/17/api-churn-vs-security.h... The API community made a mistake in appropriating the terms and concepts from REST (a description of the original web architecture) and we've been unwinding that mistake for almost two decades now, but the momentum of language is amazing...
- nicholasjarnold 6y ago
  Yes, true pure technical REST probably does require HATEOAS. However, there's still a lot of value in a "RESTful" API that that achieves at least 'Level 2' in the so-called Richardson Maturity Model. https://www.martinfowler.com/articles/richardsonMaturityModel.html https://www.martinfowler.com/articles/richardsonMaturityMode... On my teams I advocate for building Level 2 maturity "RESTfulness" into the design as it seems to strike a good balance between following the principal of least surprise for those who need to consume it and the heavier implementation effort required to get to that Level 3, which as you point out is probably not even appropriate in most cases.
- gipp 6y ago
  My sense is that the article would agree strongly with you here but didn't want to get into that long digression.
- solipsism 6y ago
  Seriously, give it up. It's a losing battle. The term means something different from what you want to to mean. How many HN comments about it are we going to have to suffer through?
  - recursivedoubts 6y ago
    At least one more!
rohan_ 6y ago
I think this only solves the problem for the API owner, not the consumer, right? You can't use cURL with protobufs like you can with REST to quickly play/interact with the data
- cfors 6y ago
  There is tooling built to easily do grpc requests from the command line[0] or even a GUI if that's your thing[1]. The tooling isn't quite as ubiquitous as cURL, but if anything I would think that its even easier because you have introspection into the contract with a .proto file. [0] https://github.com/fullstorydev/grpcurl https://github.com/fullstorydev/grpcurl [1] https://github.com/uw-labs/bloomrpc https://github.com/uw-labs/bloomrpc
  - kentonv 6y ago
    Exactly. JSON is benefitting from the fact that it can ride on top of existing non-dedicated tooling in an ad hoc way. Protobuf/gRPC needs dedicated tooling designed to replace what JSON can do with cURL. But that tooling has the potential to be much better than what can be accomplished with JSON, because it can actually type-check your input (e.g. catching typo'd field names) and maybe even let you explore the schema interactively. E.g. imagine having bash tab completion for these APIs -- easy if you have the schema.
- barumi 6y ago
  >>I think this only solves the problem for the API owner, not the consumer, right? The only problem this solves is their need to sell their product. This so-called "schema-driven development" approach was already tried in the past and failed miserably. Call it SOAP or OData, the world already gave that a try, saw all the mess and operational and productivity problems it creates, and in spite of the gold mine it represented for tooling vendors and consulting firms, it was a collosal mess. It's very weird how their selling pitch is based on veiled insults on hoe "the industry is still twenty years behind", but they failed to do their homework to learn the absolute mess that their so-called "schema-driven development" approach left behind. It's as if they are totally oblivious to why the whole world shifted to "free-form" APIs, which worked far better than the SOAP mess, and their are hell bent on betting on a rehash of the bad old days.
lxe 6y ago
"Breaking changes" happen regardless whether you use protobufs or JSON. Where the schema contract is enforced, run-time or build-time is irrelevant. - If I serve a JSON API and make a change, your client may stop working correctly if I broke the API contract. - If I serve a protobuf API and make a change, your client may also stop working correctly for the same reasons.
- kentonv 6y ago
  With schema-driven protocols, it's possible for linters to automatically detect many kinds of breaking changes and flag them, to stop developers from creating a breaking change by accident. Of course, it's still possible to make breaking changes without changing the schema. But being able to detect and prevent some kinds of bugs is better than not.
  - mixmastamyk 6y ago
    Yes, the contract pushes the break forward, to happen at build time rather than waiting for run time.
- EdwardDiego 6y ago
  Yeah, more tooling is required. We use Schema Registry for our Kafka consumers and producers and it's quite handy, we tend to use it in FORWARD compatibility mode. We've also in the past used build time compatibility checks for the same purpose. https://docs.confluent.io/current/build-applications.html https://docs.confluent.io/current/build-applications.html
J-Kuhn 6y ago
No comment that says something like "Use SOAP instead"?
eikenberry 6y ago
IMO talking about API design as the focus is stuck in the past. If the last 20 years has taught us nothing else it has taught us that protocols won the whole http://wiki.c2.com/?ApiVsProtocol http://wiki.c2.com/?ApiVsProtocol debate. No one can argue the complete dominance of HTTP as the dominant RPC mechanism in use today. We should be talking about how we can structure new, higher level protocols on it or create new protocols to improve on it. Not how to replicate all the previous mistakes on HTTP instead of TCP.
Snitch-Thursday 6y ago
WSDL, JSON-WSP[1], now protobuf, and many more formats abound on the web. In the spirit of 'choose the boring technology', which of these[2] is common enough / flexible enough to be worth learning to use and use well for non-fad web development / homestyle programming? I'd default to WSDL since I happened to grow up learning that in school, but I'm sure that's my baby duck syndrome since CORBA etc came before that. [1] https://en.wikipedia.org/wiki/JSON-WSP https://en.wikipedia.org/wiki/JSON-WSP [2] https://en.wikipedia.org/wiki/Interface_description_language https://en.wikipedia.org/wiki/Interface_description_language
- ako 6y ago
  Maybe have a look at OData, strongly typed data model, standardized query language, selection of data required to optimize performance (similar to graphql), changelist support to enable caching, grouping to enable aggregating large amount of timeseries data, capabilities model so services can indicate what they do and do not support, and support for operations/actions, all conforming to REST.
- techbio 6y ago
  Thanks for "baby duck syndrome." https://en.m.wikipedia.org/wiki/Imprinting_(psychology) https://en.m.wikipedia.org/wiki/Imprinting_(psychology)
- qppo 6y ago
  One day we're going to just return to c structs and live with it.
- therealmarv 6y ago
  wait... I also learned WSDL... is it still a thing? I thought this is SOAP related technology only...
  - foolfoolz 6y ago
    when you use these open api generators or protobufs the transport is abstracted away. yes wsdl is soap and POST for everything, but when it’s abstracted away, does it really matter? the value is from not having to think about the transport layer at all
bww 6y ago
This article seems to view “APIs” as a monolithic concept wherein the trade-offs required are the same under all circumstances. Particularly for public-facing APIs, unless you want to maintain a client library in every language your users could conceivably want — or make them shoulder the burden of dealing with the up-front costs of a far more complex integration that requires a bunch of tooling — plain old JSON over HTTP is going to be very difficult to beat for its familiarity and simplicity. And that isn’t even to consider the various technical hurdles involved in using something like gRPC in a browser on the other side of a load balancer, for example.
- mumblemumble 6y ago
  Note that protocol buffers != gRPC. The reason you can't (directly) consume gRPC from a browser isn't because of protocol buffers, it's because there are no browser APIs to permit client code to work with HTTP2 streaming. Clients consuming protocol buffers does require some tooling, but it's not that much more complicated than the tooling that is needed to consume JSON, which also requires a separate library for almost any language. Once you get past that, though, protocol buffers are relatively more language-friendly. The designers put a lot of thought into trying to ensure that it's difficult to design a protbuf-based format that can't be mapped onto the semantics of virtually any language. That is not the case with JSON, which, by being schema-on-read, makes it relatively easy to accidentally create an API that is fundamentally difficult to consume from many popular programming languages. That isn't to say that it's perfect, but one thing I do like about the protobuf approach is that problems have a tendency to stay solved once you solve them. JSON-over-HTTP seems to always lead to this chronic low-grade messiness that seems minor at first, but the cost adds up over time.
plutonorm 6y ago
Show me studies where static typing is shown to be superior to any of the other options. I'm not saying it's bad, but I'm irritated by the tone which assumes it knows things which it does not.
- kentonv 6y ago
  TypeScript is a thing which exists for the sole purpose of adding type checking to JavaScript, and nothing more. In particular, unlike most historic statically-typed languages, TypeScript gains no performance advantage from types; it literally compiles down to JavaScript before running. The only possible advantage is developer productivity stemming from type checking and other tooling it enables. TypeScript is extremely popular. Obviously there is some subjectivity here. Different people can have different opinions. Different kinds of projects may call for different approaches. But it's hard to argue that all those developers using TypeScript chose to do so without getting any benefit.
  - superfamicom 6y ago
    I've personally found JSDoc gets me what I want out of TypeScript without any compiling or features that don't actually exists in the language (I'm glaring at you enums). Ensuring the documentation is correct ensures the types are correct. You loose some nice-to-haves but to me not having to compile my code or deal with that variety of tooling is a huge plus. I can facilitate folks who like TypeScript by exporting types from the same JSDocs.
    - SkyPuncher 6y ago
      Have you ever done a wide refactor of a type or public method? It's bliss in TypeScript. Nearly impossible in JS. ---- To me the value prop of Typescript isn't the initial writing, it's the maintainability. Having good types makes it significantly easier to maintain and refactor.
      - superfamicom 6y ago
        I have, I didn’t have a bad time, but YMMV.
losvedir 6y ago
As a fan of static typing, the ideas in the blog post do resonate with me. However, one concern I have with this approach over simple JSON, is client size. A JSON API can be consumed by the web client natively, basically. Whereas using, e.g., protobuf will require some tooling to digest a .proto file to generate some client side code for you to use. In practice, if you're embedding that in your web app, how large will that be?
- rpdillon 6y ago
  I've hit this in scripting. I wanted to submit performance metrics from a build script to an analytics service another team was running, but the service only accepted gRPC. This meant the bash script had to somehow pull in the entire protobuf/gRPC stack so it could say "The build on this host for this branch took 12 minutes". I think accessibility from a broad range of environments is a disadvantage of the "protobufs everywhere" approach. I'm one of the few that still believes in dynamic typing though, so I'm probably biased.
awinter-py 6y ago
oof -- I assumed 'schema-driven' meant database schema driven until I read further down tools to automatically generate crud RPC from DB models + roles, like hasura and postgraphile, are smart and as they standardize will be money savers protobuf is okay I guess; last time I touched this 12 months ago the mobile platforms didn't have sufficiently good build + typing tools to take advantage
rubyn00bie 6y ago
They reference Protocol Buffers as a source/definition of a "schema" at the bottom and IMHO they're pretty rotten as schema definitions. There are no meaningful validations or constraints, properties/members can be missing/omitted, cross-language support is pretty rough when you need more advanced features like annotations to fill-the-gap caused by crappy types. Moving to protocol buffers is just pushing the problem out a while until we end up back here. Which is to say; a place where we just need to stop bike shedding how we define our fucking data. To quote the ineffable R. Sanchez: "Sometimes I can't even tell you apart, because I don't go by height or age, I go by amount of pain in my ass." I feel the same about all of these alternatives to HTTP APIs. I can only differentiate them by the amount of pain in my ass they cause; and on that front they're all the same.
- com2kid 6y ago
  > They reference Protocol Buffers as a source/definition of a "schema" at the bottom and IMHO they're pretty rotten as schema definitions. A past team of mine went from "raw C structures over the wire consumed by 4 different implementations on 4 platforms" to protobufs and it was a huge improvement. One schema definition shared by all the teams was an end to the nightmare of some developer putting a field in the wrong order and burning debugging time trying to figure what was going on. Or even better, that bug we hit in .NET that, even though we explicitly defined the struct, size, and ordering, the compile reordered one of our fields and we couldn't find a way around the bug so we had to make changes to the structure on all platforms so the bug didn't exhibit itself. Or the great lengths we went through to avoid making any sort of breaking changes. We didn't originally pad our structures out more than an occasional handful of bytes (embedded, constrained storage and slow transfer speed over BTLE) so we quickly ran into all sorts of horrid issues. Life was much better with protobufs. Our main complaint was the lack of unsigned types. That sucked. We didn't use any of the RPC stuff, we just used it for the schema and as a binary data format. Worked great, would recommend any day over raw C structs.
  - SV_BubbleTime 6y ago
    Funny, I hated Protobufs in C. The generated code felt bloated and I was never at all happy with the memory management. We used NanoPB. This was on embedded so that might be a different perspective than yours.
vmception 6y ago
Step right up, place your bets "Will schema defined apis become a thing in 2021 or continue to stay exclusively in the realm of 'thought leadership' aka blogs"
hinkley 6y ago
I think the biggest way API design is stuck in the past is predicated on the thinking that because the code is available, the degree of documentation we need is diminished. This isn't a rant about documentation. It's about the cognitive dissonance caused by making the internals of our code for ourselves but then expecting all of the users to read - or more likely, step through - that code to figure out why it's behaving oddly with empty strings. What I'd like to see us do is to make the shape and contents of the stack frames one of the criteria by which we judge the quality of our code. This also affects the quality of the error stack traces, but it's more of a 'pro' than a reason. Most of the time I'm only reading your (general you, not just API designers) because I am having an issue. It's a basic psychological reaction that the people near a pleasant or unpleasant experience are partly blamed for that feeling. If I'm trying to be sympathetic to that, or my user, then I have to think about my code as something people mostly look at (and almost entirely remember) when they're already having a bad day. Any work I do to honor that relationship has often paid dividends the next time I'm having a hard day, and especially when we're having a hard day together (eg, a production outage or a missed milestone)
rendall 6y ago
There is definitely need for better tools and discoverability for server-clent APIs. Protobufs seems like an odd take but I'm looking forward to their subsequent articles The best in class right now is Open API 3, which offers a way to describe endpoints, verbs and expected responses with a JSON schema. And even so, working with it feels primitive compared to, for instance, front end focused tools with bundling, type checking, package management and so forth. With Open API 3 / Swagger, you hand write a YAML or JSON file or fill out a form and have it done for you, but there is no code completion nevermind AST analysis. And now you have a schema, but there is no automatic link between that schema and the backend code, nor error typing, nor validation. The code generation options that are available are clunky and not customizable. If you change the code, you just have to go over the schema by hand again and make adjustments. Unit tests help of course, but you build it yourself As for server responses, there aren't really coalescing best practices. There are many possibilities for incoming and outgoing headers, but no language or library to ensure getting them right, the way that TypeScript works for instance I really feel like this is greenfield territory, but that's strange, since it's pretty critical
- swsieber 6y ago
  It definitely is greenfield territory At work we have a java backend and a typescript frontend. I wrote some java code that traverses the api endpoints and generates typescript definitions for them. It's completely bespoke of course. I also had to write a small java compiler plugin to record the type parameters for method return values on our api endpoints.
  - nullsense 6y ago
    In .NET Core there is a nuget package you can add that generates the open api docs based on your controllers. It's a good start, though to make it fully accurate you have to add the right attributes to each endpoint. It works pretty well. What I find really irksome about the .NET openapi tooling is the client generators use RestSharp under the hood which doesn't support HttpClient. There's no way you shouldn't be using HttpClient in 2020. So companies leveraging openapi to generate their SDKs are generating sub-par code.
ijidak 6y ago
I take issue with this statement in the article: > the industry has learned over time that statically typed languages actually enable a whole host of new tooling possibilities, and ultimately, this tooling can drastically improve developer productivity and codebase maintainability I would say, a new generation of developers have learned this. But I don't think the industry as a whole ever thought dynamic everything was something that would scale long-term. Dynamic languages are great for prototypes and tiny teams of 1 to 5 disciplined developers. As soon as the LOC starts to climb, or you have one loosely disciplined developer (which describes 80% of developers on the planet), you're toast. I never felt dynamic would become the central paradigm for most business software development. Just something used by startup founders less than 5 years out of college, hobbyists, and for selective portions of large projects (like maybe plugins and extensions of a larger piece of software).
jugg1es 6y ago
The whole time I read this article, I was thinking "well, there's SOAP...". Was very surprised when I realized he was talking about something like protocol buffer. SOAP is a great way to solve the problem described in this article, but good luck creating a SOAP API in node. "What is old is new again"?
- titanomachy 6y ago
  SOAP has a high serialization overhead and doesn't support schema evolution.
  - jugg1es 6y ago
    Oh, it definitely has a lot of overhead but it solves the primary problem as described in this article.
    - titanomachy 6y ago
      Sure, it's a serialization standard that uses schemas. In that sense we are definitely coming around in a circle.
hn_throwaway_99 6y ago
This is exactly why I use GraphQL for my APIs: 1. All inputs and outputs are strongly typed, and the framework validates the types before my code touches it 2. Can easily generate Typescript types from my GraphQL typedefs. 3. Types can evolve easily to maintain backwards compatibility with existing clients withOUT requiring explicit versions (if so desired).
- ecf 6y ago
  Not only is there benefit for the developer, but for the consumer as well. Having typed APIs allows IDE-like autocomplete when working with API tools like Insomnia. An entire website dedicated to documentation for REST endpoints and their query parameters can be replaced with a simple readme instructing those to point their tool to the GQL client endpoint.
finder83 6y ago
The issue I have with API design goes much deeper...if you have a structured database, a statically typed language, and want validation, you're writing 3 separate schemas that all do slightly different but not really different things. Add protobufs onto that, you're now doing four. Maintaining four different schemas does not make refactoring easy, in fact I'd even say it's harder than just using a nosql database and a dynamic language where all you're maintaining is the validation schema. (Yes, I realize there are other benefits to static typing.) Sure, it might generate the code initially depending on what you use, but you still have to maintain it. And any attempt to combine them has been way too complex/verbose. I want to write a lot less code, not more. What I really want is a static language with a validation logic extensive enough that it can be used to generate CRUD APIs, validation, and still offer static typing. I don't know how this would work in practice, and I might hate it after I try it, but I'm tired of maintaining schemas. Just to add to this, things like authorization, login, registration, etc, seem tiresome to write every time for a new API. I've been trying to find something more along the lines of a Backend as a Service, which Parse is very interesting for...but I wish there were more options and easier to use libraries for crafting APIs. It actually feels like we're stuck in the past no matter what we do. I welcome suggestions for easier to write APIs.
- kentonv 6y ago
  This is exactly why a lot of people choose to use the protobuf-generated data types as their data model internally within their app in addition to for serialization, and many people even derive their database schemas and validation from their protobuf schemas. (When I left Google in 2012, Spanner table schemas were literally defined using protobuf IDL. I haven't followed it since so I have no idea if that's still the case.) Admittedly, protobuf wasn't explicitly designed for these purposes, it just of grew into them because they were convenient, and as a result there are certainly warts. But it's close enough that it works pretty well.
  - finder83 6y ago
    Fascinating, I didn't know it could be used for all of that. Last time I tried it it felt more like static typing than true validation (e.g. validating email addresses, date formats, dependent fields, etc) I'll take a look at it again, thanks.
nostrademons 6y ago
The static/dynamic debate reoccurs every ~20 years and has more to do with the economics of the software industry than any intrinsic merits of each approach. In times of rapid change, technological revolution, and landgrabs for market share, dynamic typing wins out. Why? Because you don't know what information you want to encode in your schema. You get this by putting products in front of users, changing them, and see what resonates and increases your growth rate. This was the case in the early 60s (mainframes & scientific computing), the early 80s (AI & PCs), and the early 00s (Web 2.0). In times of steady-state, where the dominant players have already been established and we're just filling in niches, static interfaces dominate. This is a way of building supplier networks, increasing efficiency, and locking down best practices. At this point the users are largely known, we're just trying to serve them better. This was the case in the 70s (mainframes), 90s (Windows), and 10s (Internet/mobile). The challenge for a new startup selling statically-typed APIs is that at this point, the ecosystem is already established, the big players are in place, and remaining startups are just filling in niches. Somebody is likely to win with a statically-typed API, but it'll likely be Google (gRPC), Facebook (Thrift), or Microsoft (TypeScript?) rather than some startup.
- kentonv 6y ago
  > Because you don't know what information you want to encode in your schema. I'd argue that Protobuf actually embraces this -- its killer feature vs. other binary protocols of the era was that it's always easy to add new fields.
  - Guthur 6y ago
    Conflating binary and schema. Also there are many over schema encodings that have better type systems than protobuf and allow dynamic schema, Avro being one for example.
    - ayewo 6y ago
      Your first sentence is peak HN :) On a serious note, kentonv properly prefaced his opinion with a qualifier “I'd argue that ...” but you somehow interpreted it as a fact ... Anyway, if you think the person you replied to does not understand the difference between binary and schema, after helping author protocol buffers at Google, then no one can really claim to know anything here on HN. Healthy skepticism is fine as it helps keep the discussion grounded, but its blind use will more often than not stifle insightful contributions from being posted by people close to the action. Edited for clarity.
taeric 6y ago
Feels so strange to see JSON and REST starting to get more criticism for not being XML and WSDL. Sure, they don't say that directly, but it is hard for me to see it as anything else. Now, I fully grant that WSDL really faltered over what feels like a proxy battle between Sun and Microsoft now. As I recall, you could make a WSDL that was easy to use from .NET, or from Java; but not both. I don't recall what the other languages were like. Which is all to say I don't really disagree. But it is eye opening to think that this is all getting trodden again.
- deathanatos 6y ago
  Nothing about REST prevents strong validation of inputs. If anything, the notion that the ends should pass data using well-defined media types should push one towards something that can be validated. I don't think "everything chucks application/json around and we have no idea what the JSON represents is in the spirit of REST at all. The set of externally standardized media types is somewhat small. When I worked in real estate, there's not really a standardized set of stuff for that problem domain, and we're pretty much defining our own. And, worse, IMO, we did so badly, because we didn't take the time to lay out a good, well documented specification. Some of the schema languages out there can help with that. (Though I don't think they're a panacea.) Likewise, while I agree with some of the article, protobufs are just not terribly great. The lack of sum types really hampers how you can represent stuff in them. (We had other, more practical issues when we tried to adopt them over JSON in my last org.) But, also, if you want to define application/vnd.foo.bar.baz as "a protocol buf with this definition that contains data representing this" … that can still be as RESTful as JSON. (And, for the XML comparison… JSON is too verbose. XML is just even worse. JSON was a step in the right direction.)
  - taeric 6y ago
    I don't really disagree. But I have yet to see a schema system for JSON that isn't just a crappy version of what XML can do. Similarly, most ways of representing services turn into bike shedding on how to map to HTTP. I will also add that as soon as you think you need complicated models at the API level, you are almost certainly setting a path to problems. Yes, it is possible to come up with a place where sum types will make sense. In most cases, though, you are better representing a flat set of data, and let a validation layer move things into distinct types, if you really need it. I view this as everyone wanting a programming language where you control AST constructs. Sounds great, but working with raw text/data at a layer above will almost certainly be easier. And you can use a compiler/evaluator to get to things you can manipulate at that layer. (Similarly, this idea that files are the bane of computer. In stark contrast to my always just wanting a file that represents whatever I am wanting to move from one place to another. However I want to make that happen.)
tflinton 6y ago
F$$$$ it, everyone just directly connect to everyones database and use SQL.
marta_morena_28 6y ago
> Two decades ago, it was widely argued that dynamic programming languages were more productive because you didn't have to spend time dealing with type signatures. The only reason, then, to use a statically typed language, was for better performance. This boggles the mind. I am using typed languages since I can think. I never once recall an instant where I was saying "Uh uh, this type signature is driving me craaazy". Like seriously, I don't even know what articles like this are talking about. Types are not your enemy, they surface issues early. Yeah you can whip something up in Python and JS, but ultimately you DO have to deal with types, except now you don't have a compiler doing this job anymore, you have to do it yourself... Somehow. The only thing typed languages need is something like `dynamic` from C#, which automatically boilerplates untyped access with reflection, without cluttering your code. I.e. duck typing is one of the things some languages like Java need to get better at. But the situations in which I yearn for this are far and few between. You don't think about types, unless you are new to typed languages... It's really that simple. I never have to think about types. Perhaps its subconcious, but its definitely not slowing me down, its making things faster through robust refactoring, auto-complete and welll doh: TYPE SAFETY!
- mbesto 6y ago
  > The only reason, then, to use a statically typed language, was for better performance. Uhh, someone please correct me if I'm wrong, but aren't statically typed languages about reliability (e.g. testing, mutability)? EDIT: in addition to performance, not solely.
  - zdragnar 6y ago
    Two decades ago, dynamic languages were more typically scripted / interpreted, and statically typed languages were more typically compiled. Scripting does allow for quicker iteration (especially if compared to a language with a slow compiler) and compilation usually does produce faster code (at least pre-dating sophisticated JIT compilers).
  - marta_morena_28 6y ago
    Statically typed languages only eliminate SOME things you'd otherwise have to test. In the end, you can eliminate almost anything besides logical errors, however, those are unfortunately a pretty big portion of bugs :D. So while I would always use statically typed languages for anything that needs to be reliable, I do not see how this is in any way a necessity. You CAN write reliable programs without type safety, you just have to test things you normally wouldn't have to test (i.e. the lack of type safety introduces a whole bunch of null-pointer style scenarios where you get something your code totally didn't expect but has to deal with). As for performance. Statically typed languages are usually faster, mostly because we do not have the technology yet to make dynamically typed ones as fast (in the general case). Not because there is something inherently different about them. However, I imagine the technology to make them on par with statically typed languages will take another few decades. Mainly because untyped languages need sophisticated transformations to be fast. That is the job the human normally does for the compiler in typed languages. Things just fit together and play nicely. With dynamic languages, your get one big spaghetti soup with no structure and now the compiler has to figure out the "intended" types and compile for different estimated function signatures, etc. all while honoring the JIT-style performance considerations (fast startup, low overhead during runtime). This is a gargantuan task that probably will require advanced machine learning before it really takes off.
Razengan 6y ago
I think most Swift APIs, especially SwiftUI and Combine, are pretty impressive, clean and modern.
seanhandley 6y ago
I agree there are many positive aspects to schema-based API systems. What I disagree with is the assertion that "X is the future, Y is the past". Novelty does not automatically equal superiority and there are tangible pros and cons for each. I might be able to take the discussion slightly more seriously if the article wasn't part of a marketing strategy for a company selling protobuf-based products.
Thetawaves 6y ago
My experience with SOAP and wizzles indicates that one of the primary problems with statically typed protocols is the proliferation of types. Corresponding types need to be created in each language, and these object representations reflect the original implementation language (server), rather than whatever idiosyncrasies fit the current language (client). The core benefit the world gained by migrating to REST/JSON was the total simplification of types, and the ease of interacting with them in any language.

146 points by kentonv 6y ago

> In other words, the benefit of maintaining type signatures now well outweighs the cost. That's a rather bold claim to begin your argument, but there are no references. There's a good review of related studies here: https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/

karmakaze 6y ago
I think a better way of phrasing this would be "the cost of maintaining type signatures has been reduced well below the benefits" since I believe buf makes tools to do just that.
- kentonv 6y ago
  I think it's both that the costs are going down, and that the benefits are going up, in the form of useful tooling that couldn't exist without schemas to drive it. Quantifying these costs and benefits is, of course, notoriously difficult.
spinningslate 6y ago
Agreed. The unconditional nature of the statement detracts from its message. Had it said: "we see benefits in static types for APIs, and it's working for us - here's why" it would have had some credibility. And provided some insight on why it might work for others too. Contrast with DHH's articulation on the corecursive podcast [0]. Paraphrasing: * Dynamic types work for him * People don't all think the same. So static typing is fine too. Jeremy Howard made essentially the same point on TWIML AI [1]. The point is there's far more to success than static vs dynamic typing. Personal preferences, team skills, culture, task at hand, nature of domain, commercial imperatives, code base size, and many other things besides all influence choice. To suggest there's a single, global answer to static vs dynamic - whether for languages or APIs - is a misguided and unhelpful simplification. [0]: https://corecursive.com/045-david-heinemeier-hansson-software-contrarian/ https://corecursive.com/045-david-heinemeier-hansson-softwar... [1]: https://twimlai.com/whats-next-for-fast-ai-w-jeremy-howard/ https://twimlai.com/whats-next-for-fast-ai-w-jeremy-howard/
mistersys 6y ago
It's fairly obvious that for the any public facing API, type signatures are going to be a net positive, because you have to write input validation for your API anyway to have any hope of it being secure. You can either write as a series of buggy, hard to understand if statements or use a well-defined schema tool of some sort. Furthermore, once you have the schema you can generate client APIs in e.g Typescript for your APIs instead of wasting engineering effort on writing them manually. In terms of static types generally, outside of APIs, the equation can be different. However, if you are creating a library that many developers will use, given the state of IDEs, your just hurting your users (developers) by not providing static types. I use Typescript in every single project not because of supposed benefit of "type grantees", but because I don't need to spend all day flipping between some developers poorly written, incomplete docs to understand how to use an API. My code editor will tell me the whole format and I can explore the APIs with goto definition, demonstrably reducing development time.

hashamali 6y ago

Surprised there is no mention of OpenAPI. A spec driven API development process solves most of the problems described without having to switch to schematized data serialization formats.

kentonv 6y ago
I admit I haven't written an OpenAPI schema myself. But, looking at the docs, I have a hard time imagining that I'd want to do API-driven development with it. Let me explain what I mean by that... When I start a new project, or a major new feature in an existing project, usually the first thing I do is write out the Protobuf or Cap'n Proto schemas describing the APIs and data structures. Even for internal-facing APIs. APIs are how components talk to each other, and defining the APIs defines which components are responsible for what in a very clear way. Often I find writing out the APIs is actually the easiest, most precise way to communicate a design with my teammates; much better than prose in a design doc. For this purpose, it's actually pretty important that the schema language be something that's not cumbersome to write. Protobuf and Cap'n Proto feature dedicated schema languages that feel like a programming language. It looks like OpenAPI has you writing schemas themselves in JSON or YAML. Ick. And once I've written those schemas, I run the code generator and poof, I now have nice, type-safe classes in my programming language of choice implementing my schema. This lets me move a lot faster implementing, being able to use my IDE's auto-complete and whatnot. Does OpenAPI generate code for you? I have worked on projects that use it and didn't have any generated code, so I am assuming not... I think this is what the article is trying to get at. (Disclosure: I'm the author of Cap'n Proto; long ago I was the author of Protobuf v2; and I am a (small) investor in Buf because I like what they're doing, but I don't work with them.)
- hashamali 6y ago
  OpenAPI is just the format for describing APIs for various use cases such as documentation, runtime validation, and yes, even code generation. Depending in the language you use, the support for code generation may vary though. I’m certainly not against the flow you’re describing. Aside from having to wrangle JSON/YAML instead of a more concise DSL, it’s not too different from what you’d do with OpenAPI.

TeeWEE 6y ago

We are using GRPC and for Rest OpenAPI and write our own codegenerators for OpenAPI... How is this in the past? I guess most developers don't use OpenAPI and just return JSON... Thats indeed an old way of development.

recursivedoubts 6y ago

I don't have strong feelings about this article except for the fact that it once again uses REST to mean "JSON over HTTP" when, in fact, REST requires HATEOAS and the vast majority of APIs that claim to be RESTful aren't, and almost certainly shouldn't be, since they aren't using a hypermedia/hypertext. https://intercoolerjs.org/2016/01/18/rescuing-rest.html https://intercoolerjs.org/2016/01/18/rescuing-rest.html https://intercoolerjs.org/2016/02/17/api-churn-vs-security.html https://intercoolerjs.org/2016/02/17/api-churn-vs-security.h... The API community made a mistake in appropriating the terms and concepts from REST (a description of the original web architecture) and we've been unwinding that mistake for almost two decades now, but the momentum of language is amazing...

nicholasjarnold 6y ago
Yes, true pure technical REST probably does require HATEOAS. However, there's still a lot of value in a "RESTful" API that that achieves at least 'Level 2' in the so-called Richardson Maturity Model. https://www.martinfowler.com/articles/richardsonMaturityModel.html https://www.martinfowler.com/articles/richardsonMaturityMode... On my teams I advocate for building Level 2 maturity "RESTfulness" into the design as it seems to strike a good balance between following the principal of least surprise for those who need to consume it and the heavier implementation effort required to get to that Level 3, which as you point out is probably not even appropriate in most cases.
gipp 6y ago
My sense is that the article would agree strongly with you here but didn't want to get into that long digression.
solipsism 6y ago
Seriously, give it up. It's a losing battle. The term means something different from what you want to to mean. How many HN comments about it are we going to have to suffer through?
- recursivedoubts 6y ago
  At least one more!

rohan_ 6y ago

I think this only solves the problem for the API owner, not the consumer, right? You can't use cURL with protobufs like you can with REST to quickly play/interact with the data

cfors 6y ago
There is tooling built to easily do grpc requests from the command line[0] or even a GUI if that's your thing[1]. The tooling isn't quite as ubiquitous as cURL, but if anything I would think that its even easier because you have introspection into the contract with a .proto file. [0] https://github.com/fullstorydev/grpcurl https://github.com/fullstorydev/grpcurl [1] https://github.com/uw-labs/bloomrpc https://github.com/uw-labs/bloomrpc
- kentonv 6y ago
  Exactly. JSON is benefitting from the fact that it can ride on top of existing non-dedicated tooling in an ad hoc way. Protobuf/gRPC needs dedicated tooling designed to replace what JSON can do with cURL. But that tooling has the potential to be much better than what can be accomplished with JSON, because it can actually type-check your input (e.g. catching typo'd field names) and maybe even let you explore the schema interactively. E.g. imagine having bash tab completion for these APIs -- easy if you have the schema.
barumi 6y ago
>>I think this only solves the problem for the API owner, not the consumer, right? The only problem this solves is their need to sell their product. This so-called "schema-driven development" approach was already tried in the past and failed miserably. Call it SOAP or OData, the world already gave that a try, saw all the mess and operational and productivity problems it creates, and in spite of the gold mine it represented for tooling vendors and consulting firms, it was a collosal mess. It's very weird how their selling pitch is based on veiled insults on hoe "the industry is still twenty years behind", but they failed to do their homework to learn the absolute mess that their so-called "schema-driven development" approach left behind. It's as if they are totally oblivious to why the whole world shifted to "free-form" APIs, which worked far better than the SOAP mess, and their are hell bent on betting on a rehash of the bad old days.

lxe 6y ago

"Breaking changes" happen regardless whether you use protobufs or JSON. Where the schema contract is enforced, run-time or build-time is irrelevant. - If I serve a JSON API and make a change, your client may stop working correctly if I broke the API contract. - If I serve a protobuf API and make a change, your client may also stop working correctly for the same reasons.

kentonv 6y ago
With schema-driven protocols, it's possible for linters to automatically detect many kinds of breaking changes and flag them, to stop developers from creating a breaking change by accident. Of course, it's still possible to make breaking changes without changing the schema. But being able to detect and prevent some kinds of bugs is better than not.
- mixmastamyk 6y ago
  Yes, the contract pushes the break forward, to happen at build time rather than waiting for run time.
EdwardDiego 6y ago
Yeah, more tooling is required. We use Schema Registry for our Kafka consumers and producers and it's quite handy, we tend to use it in FORWARD compatibility mode. We've also in the past used build time compatibility checks for the same purpose. https://docs.confluent.io/current/build-applications.html https://docs.confluent.io/current/build-applications.html

J-Kuhn 6y ago

No comment that says something like "Use SOAP instead"?

eikenberry 6y ago

IMO talking about API design as the focus is stuck in the past. If the last 20 years has taught us nothing else it has taught us that protocols won the whole http://wiki.c2.com/?ApiVsProtocol http://wiki.c2.com/?ApiVsProtocol debate. No one can argue the complete dominance of HTTP as the dominant RPC mechanism in use today. We should be talking about how we can structure new, higher level protocols on it or create new protocols to improve on it. Not how to replicate all the previous mistakes on HTTP instead of TCP.

Snitch-Thursday 6y ago

WSDL, JSON-WSP[1], now protobuf, and many more formats abound on the web. In the spirit of 'choose the boring technology', which of these[2] is common enough / flexible enough to be worth learning to use and use well for non-fad web development / homestyle programming? I'd default to WSDL since I happened to grow up learning that in school, but I'm sure that's my baby duck syndrome since CORBA etc came before that. [1] https://en.wikipedia.org/wiki/JSON-WSP https://en.wikipedia.org/wiki/JSON-WSP [2] https://en.wikipedia.org/wiki/Interface_description_language https://en.wikipedia.org/wiki/Interface_description_language

ako 6y ago
Maybe have a look at OData, strongly typed data model, standardized query language, selection of data required to optimize performance (similar to graphql), changelist support to enable caching, grouping to enable aggregating large amount of timeseries data, capabilities model so services can indicate what they do and do not support, and support for operations/actions, all conforming to REST.
techbio 6y ago
Thanks for "baby duck syndrome." https://en.m.wikipedia.org/wiki/Imprinting_(psychology) https://en.m.wikipedia.org/wiki/Imprinting_(psychology)
qppo 6y ago
One day we're going to just return to c structs and live with it.
therealmarv 6y ago
wait... I also learned WSDL... is it still a thing? I thought this is SOAP related technology only...
- foolfoolz 6y ago
  when you use these open api generators or protobufs the transport is abstracted away. yes wsdl is soap and POST for everything, but when it’s abstracted away, does it really matter? the value is from not having to think about the transport layer at all

bww 6y ago

This article seems to view “APIs” as a monolithic concept wherein the trade-offs required are the same under all circumstances. Particularly for public-facing APIs, unless you want to maintain a client library in every language your users could conceivably want — or make them shoulder the burden of dealing with the up-front costs of a far more complex integration that requires a bunch of tooling — plain old JSON over HTTP is going to be very difficult to beat for its familiarity and simplicity. And that isn’t even to consider the various technical hurdles involved in using something like gRPC in a browser on the other side of a load balancer, for example.

mumblemumble 6y ago
Note that protocol buffers != gRPC. The reason you can't (directly) consume gRPC from a browser isn't because of protocol buffers, it's because there are no browser APIs to permit client code to work with HTTP2 streaming. Clients consuming protocol buffers does require some tooling, but it's not that much more complicated than the tooling that is needed to consume JSON, which also requires a separate library for almost any language. Once you get past that, though, protocol buffers are relatively more language-friendly. The designers put a lot of thought into trying to ensure that it's difficult to design a protbuf-based format that can't be mapped onto the semantics of virtually any language. That is not the case with JSON, which, by being schema-on-read, makes it relatively easy to accidentally create an API that is fundamentally difficult to consume from many popular programming languages. That isn't to say that it's perfect, but one thing I do like about the protobuf approach is that problems have a tendency to stay solved once you solve them. JSON-over-HTTP seems to always lead to this chronic low-grade messiness that seems minor at first, but the cost adds up over time.

plutonorm 6y ago

Show me studies where static typing is shown to be superior to any of the other options. I'm not saying it's bad, but I'm irritated by the tone which assumes it knows things which it does not.

kentonv 6y ago
TypeScript is a thing which exists for the sole purpose of adding type checking to JavaScript, and nothing more. In particular, unlike most historic statically-typed languages, TypeScript gains no performance advantage from types; it literally compiles down to JavaScript before running. The only possible advantage is developer productivity stemming from type checking and other tooling it enables. TypeScript is extremely popular. Obviously there is some subjectivity here. Different people can have different opinions. Different kinds of projects may call for different approaches. But it's hard to argue that all those developers using TypeScript chose to do so without getting any benefit.
- superfamicom 6y ago
  I've personally found JSDoc gets me what I want out of TypeScript without any compiling or features that don't actually exists in the language (I'm glaring at you enums). Ensuring the documentation is correct ensures the types are correct. You loose some nice-to-haves but to me not having to compile my code or deal with that variety of tooling is a huge plus. I can facilitate folks who like TypeScript by exporting types from the same JSDocs.
  - SkyPuncher 6y ago
    Have you ever done a wide refactor of a type or public method? It's bliss in TypeScript. Nearly impossible in JS. ---- To me the value prop of Typescript isn't the initial writing, it's the maintainability. Having good types makes it significantly easier to maintain and refactor.
    - superfamicom 6y ago
      I have, I didn’t have a bad time, but YMMV.

losvedir 6y ago

As a fan of static typing, the ideas in the blog post do resonate with me. However, one concern I have with this approach over simple JSON, is client size. A JSON API can be consumed by the web client natively, basically. Whereas using, e.g., protobuf will require some tooling to digest a .proto file to generate some client side code for you to use. In practice, if you're embedding that in your web app, how large will that be?

rpdillon 6y ago
I've hit this in scripting. I wanted to submit performance metrics from a build script to an analytics service another team was running, but the service only accepted gRPC. This meant the bash script had to somehow pull in the entire protobuf/gRPC stack so it could say "The build on this host for this branch took 12 minutes". I think accessibility from a broad range of environments is a disadvantage of the "protobufs everywhere" approach. I'm one of the few that still believes in dynamic typing though, so I'm probably biased.

awinter-py 6y ago

oof -- I assumed 'schema-driven' meant database schema driven until I read further down tools to automatically generate crud RPC from DB models + roles, like hasura and postgraphile, are smart and as they standardize will be money savers protobuf is okay I guess; last time I touched this 12 months ago the mobile platforms didn't have sufficiently good build + typing tools to take advantage

rubyn00bie 6y ago

They reference Protocol Buffers as a source/definition of a "schema" at the bottom and IMHO they're pretty rotten as schema definitions. There are no meaningful validations or constraints, properties/members can be missing/omitted, cross-language support is pretty rough when you need more advanced features like annotations to fill-the-gap caused by crappy types. Moving to protocol buffers is just pushing the problem out a while until we end up back here. Which is to say; a place where we just need to stop bike shedding how we define our fucking data. To quote the ineffable R. Sanchez: "Sometimes I can't even tell you apart, because I don't go by height or age, I go by amount of pain in my ass." I feel the same about all of these alternatives to HTTP APIs. I can only differentiate them by the amount of pain in my ass they cause; and on that front they're all the same.

com2kid 6y ago
> They reference Protocol Buffers as a source/definition of a "schema" at the bottom and IMHO they're pretty rotten as schema definitions. A past team of mine went from "raw C structures over the wire consumed by 4 different implementations on 4 platforms" to protobufs and it was a huge improvement. One schema definition shared by all the teams was an end to the nightmare of some developer putting a field in the wrong order and burning debugging time trying to figure what was going on. Or even better, that bug we hit in .NET that, even though we explicitly defined the struct, size, and ordering, the compile reordered one of our fields and we couldn't find a way around the bug so we had to make changes to the structure on all platforms so the bug didn't exhibit itself. Or the great lengths we went through to avoid making any sort of breaking changes. We didn't originally pad our structures out more than an occasional handful of bytes (embedded, constrained storage and slow transfer speed over BTLE) so we quickly ran into all sorts of horrid issues. Life was much better with protobufs. Our main complaint was the lack of unsigned types. That sucked. We didn't use any of the RPC stuff, we just used it for the schema and as a binary data format. Worked great, would recommend any day over raw C structs.
- SV_BubbleTime 6y ago
  Funny, I hated Protobufs in C. The generated code felt bloated and I was never at all happy with the memory management. We used NanoPB. This was on embedded so that might be a different perspective than yours.

vmception 6y ago

Step right up, place your bets "Will schema defined apis become a thing in 2021 or continue to stay exclusively in the realm of 'thought leadership' aka blogs"

hinkley 6y ago

I think the biggest way API design is stuck in the past is predicated on the thinking that because the code is available, the degree of documentation we need is diminished. This isn't a rant about documentation. It's about the cognitive dissonance caused by making the internals of our code for ourselves but then expecting all of the users to read - or more likely, step through - that code to figure out why it's behaving oddly with empty strings. What I'd like to see us do is to make the shape and contents of the stack frames one of the criteria by which we judge the quality of our code. This also affects the quality of the error stack traces, but it's more of a 'pro' than a reason. Most of the time I'm only reading your (general you, not just API designers) because I am having an issue. It's a basic psychological reaction that the people near a pleasant or unpleasant experience are partly blamed for that feeling. If I'm trying to be sympathetic to that, or my user, then I have to think about my code as something people mostly look at (and almost entirely remember) when they're already having a bad day. Any work I do to honor that relationship has often paid dividends the next time I'm having a hard day, and especially when we're having a hard day together (eg, a production outage or a missed milestone)

rendall 6y ago

There is definitely need for better tools and discoverability for server-clent APIs. Protobufs seems like an odd take but I'm looking forward to their subsequent articles The best in class right now is Open API 3, which offers a way to describe endpoints, verbs and expected responses with a JSON schema. And even so, working with it feels primitive compared to, for instance, front end focused tools with bundling, type checking, package management and so forth. With Open API 3 / Swagger, you hand write a YAML or JSON file or fill out a form and have it done for you, but there is no code completion nevermind AST analysis. And now you have a schema, but there is no automatic link between that schema and the backend code, nor error typing, nor validation. The code generation options that are available are clunky and not customizable. If you change the code, you just have to go over the schema by hand again and make adjustments. Unit tests help of course, but you build it yourself As for server responses, there aren't really coalescing best practices. There are many possibilities for incoming and outgoing headers, but no language or library to ensure getting them right, the way that TypeScript works for instance I really feel like this is greenfield territory, but that's strange, since it's pretty critical

swsieber 6y ago
It definitely is greenfield territory At work we have a java backend and a typescript frontend. I wrote some java code that traverses the api endpoints and generates typescript definitions for them. It's completely bespoke of course. I also had to write a small java compiler plugin to record the type parameters for method return values on our api endpoints.
- nullsense 6y ago
  In .NET Core there is a nuget package you can add that generates the open api docs based on your controllers. It's a good start, though to make it fully accurate you have to add the right attributes to each endpoint. It works pretty well. What I find really irksome about the .NET openapi tooling is the client generators use RestSharp under the hood which doesn't support HttpClient. There's no way you shouldn't be using HttpClient in 2020. So companies leveraging openapi to generate their SDKs are generating sub-par code.

ijidak 6y ago

I take issue with this statement in the article: > the industry has learned over time that statically typed languages actually enable a whole host of new tooling possibilities, and ultimately, this tooling can drastically improve developer productivity and codebase maintainability I would say, a new generation of developers have learned this. But I don't think the industry as a whole ever thought dynamic everything was something that would scale long-term. Dynamic languages are great for prototypes and tiny teams of 1 to 5 disciplined developers. As soon as the LOC starts to climb, or you have one loosely disciplined developer (which describes 80% of developers on the planet), you're toast. I never felt dynamic would become the central paradigm for most business software development. Just something used by startup founders less than 5 years out of college, hobbyists, and for selective portions of large projects (like maybe plugins and extensions of a larger piece of software).

jugg1es 6y ago

The whole time I read this article, I was thinking "well, there's SOAP...". Was very surprised when I realized he was talking about something like protocol buffer. SOAP is a great way to solve the problem described in this article, but good luck creating a SOAP API in node. "What is old is new again"?

titanomachy 6y ago
SOAP has a high serialization overhead and doesn't support schema evolution.
- jugg1es 6y ago
  Oh, it definitely has a lot of overhead but it solves the primary problem as described in this article.
  - titanomachy 6y ago
    Sure, it's a serialization standard that uses schemas. In that sense we are definitely coming around in a circle.

hn_throwaway_99 6y ago

This is exactly why I use GraphQL for my APIs: 1. All inputs and outputs are strongly typed, and the framework validates the types before my code touches it 2. Can easily generate Typescript types from my GraphQL typedefs. 3. Types can evolve easily to maintain backwards compatibility with existing clients withOUT requiring explicit versions (if so desired).

ecf 6y ago
Not only is there benefit for the developer, but for the consumer as well. Having typed APIs allows IDE-like autocomplete when working with API tools like Insomnia. An entire website dedicated to documentation for REST endpoints and their query parameters can be replaced with a simple readme instructing those to point their tool to the GQL client endpoint.

finder83 6y ago

The issue I have with API design goes much deeper...if you have a structured database, a statically typed language, and want validation, you're writing 3 separate schemas that all do slightly different but not really different things. Add protobufs onto that, you're now doing four. Maintaining four different schemas does not make refactoring easy, in fact I'd even say it's harder than just using a nosql database and a dynamic language where all you're maintaining is the validation schema. (Yes, I realize there are other benefits to static typing.) Sure, it might generate the code initially depending on what you use, but you still have to maintain it. And any attempt to combine them has been way too complex/verbose. I want to write a lot less code, not more. What I really want is a static language with a validation logic extensive enough that it can be used to generate CRUD APIs, validation, and still offer static typing. I don't know how this would work in practice, and I might hate it after I try it, but I'm tired of maintaining schemas. Just to add to this, things like authorization, login, registration, etc, seem tiresome to write every time for a new API. I've been trying to find something more along the lines of a Backend as a Service, which Parse is very interesting for...but I wish there were more options and easier to use libraries for crafting APIs. It actually feels like we're stuck in the past no matter what we do. I welcome suggestions for easier to write APIs.

kentonv 6y ago
This is exactly why a lot of people choose to use the protobuf-generated data types as their data model internally within their app in addition to for serialization, and many people even derive their database schemas and validation from their protobuf schemas. (When I left Google in 2012, Spanner table schemas were literally defined using protobuf IDL. I haven't followed it since so I have no idea if that's still the case.) Admittedly, protobuf wasn't explicitly designed for these purposes, it just of grew into them because they were convenient, and as a result there are certainly warts. But it's close enough that it works pretty well.
- finder83 6y ago
  Fascinating, I didn't know it could be used for all of that. Last time I tried it it felt more like static typing than true validation (e.g. validating email addresses, date formats, dependent fields, etc) I'll take a look at it again, thanks.

nostrademons 6y ago

The static/dynamic debate reoccurs every ~20 years and has more to do with the economics of the software industry than any intrinsic merits of each approach. In times of rapid change, technological revolution, and landgrabs for market share, dynamic typing wins out. Why? Because you don't know what information you want to encode in your schema. You get this by putting products in front of users, changing them, and see what resonates and increases your growth rate. This was the case in the early 60s (mainframes & scientific computing), the early 80s (AI & PCs), and the early 00s (Web 2.0). In times of steady-state, where the dominant players have already been established and we're just filling in niches, static interfaces dominate. This is a way of building supplier networks, increasing efficiency, and locking down best practices. At this point the users are largely known, we're just trying to serve them better. This was the case in the 70s (mainframes), 90s (Windows), and 10s (Internet/mobile). The challenge for a new startup selling statically-typed APIs is that at this point, the ecosystem is already established, the big players are in place, and remaining startups are just filling in niches. Somebody is likely to win with a statically-typed API, but it'll likely be Google (gRPC), Facebook (Thrift), or Microsoft (TypeScript?) rather than some startup.

kentonv 6y ago
> Because you don't know what information you want to encode in your schema. I'd argue that Protobuf actually embraces this -- its killer feature vs. other binary protocols of the era was that it's always easy to add new fields.
- Guthur 6y ago
  Conflating binary and schema. Also there are many over schema encodings that have better type systems than protobuf and allow dynamic schema, Avro being one for example.
  - ayewo 6y ago
    Your first sentence is peak HN :) On a serious note, kentonv properly prefaced his opinion with a qualifier “I'd argue that ...” but you somehow interpreted it as a fact ... Anyway, if you think the person you replied to does not understand the difference between binary and schema, after helping author protocol buffers at Google, then no one can really claim to know anything here on HN. Healthy skepticism is fine as it helps keep the discussion grounded, but its blind use will more often than not stifle insightful contributions from being posted by people close to the action. Edited for clarity.

taeric 6y ago

Feels so strange to see JSON and REST starting to get more criticism for not being XML and WSDL. Sure, they don't say that directly, but it is hard for me to see it as anything else. Now, I fully grant that WSDL really faltered over what feels like a proxy battle between Sun and Microsoft now. As I recall, you could make a WSDL that was easy to use from .NET, or from Java; but not both. I don't recall what the other languages were like. Which is all to say I don't really disagree. But it is eye opening to think that this is all getting trodden again.

deathanatos 6y ago
Nothing about REST prevents strong validation of inputs. If anything, the notion that the ends should pass data using well-defined media types should push one towards something that can be validated. I don't think "everything chucks application/json around and we have no idea what the JSON represents is in the spirit of REST at all. The set of externally standardized media types is somewhat small. When I worked in real estate, there's not really a standardized set of stuff for that problem domain, and we're pretty much defining our own. And, worse, IMO, we did so badly, because we didn't take the time to lay out a good, well documented specification. Some of the schema languages out there can help with that. (Though I don't think they're a panacea.) Likewise, while I agree with some of the article, protobufs are just not terribly great. The lack of sum types really hampers how you can represent stuff in them. (We had other, more practical issues when we tried to adopt them over JSON in my last org.) But, also, if you want to define application/vnd.foo.bar.baz as "a protocol buf with this definition that contains data representing this" … that can still be as RESTful as JSON. (And, for the XML comparison… JSON is too verbose. XML is just even worse. JSON was a step in the right direction.)
- taeric 6y ago
  I don't really disagree. But I have yet to see a schema system for JSON that isn't just a crappy version of what XML can do. Similarly, most ways of representing services turn into bike shedding on how to map to HTTP. I will also add that as soon as you think you need complicated models at the API level, you are almost certainly setting a path to problems. Yes, it is possible to come up with a place where sum types will make sense. In most cases, though, you are better representing a flat set of data, and let a validation layer move things into distinct types, if you really need it. I view this as everyone wanting a programming language where you control AST constructs. Sounds great, but working with raw text/data at a layer above will almost certainly be easier. And you can use a compiler/evaluator to get to things you can manipulate at that layer. (Similarly, this idea that files are the bane of computer. In stark contrast to my always just wanting a file that represents whatever I am wanting to move from one place to another. However I want to make that happen.)

tflinton 6y ago

F$$$$ it, everyone just directly connect to everyones database and use SQL.

marta_morena_28 6y ago

> Two decades ago, it was widely argued that dynamic programming languages were more productive because you didn't have to spend time dealing with type signatures. The only reason, then, to use a statically typed language, was for better performance. This boggles the mind. I am using typed languages since I can think. I never once recall an instant where I was saying "Uh uh, this type signature is driving me craaazy". Like seriously, I don't even know what articles like this are talking about. Types are not your enemy, they surface issues early. Yeah you can whip something up in Python and JS, but ultimately you DO have to deal with types, except now you don't have a compiler doing this job anymore, you have to do it yourself... Somehow. The only thing typed languages need is something like `dynamic` from C#, which automatically boilerplates untyped access with reflection, without cluttering your code. I.e. duck typing is one of the things some languages like Java need to get better at. But the situations in which I yearn for this are far and few between. You don't think about types, unless you are new to typed languages... It's really that simple. I never have to think about types. Perhaps its subconcious, but its definitely not slowing me down, its making things faster through robust refactoring, auto-complete and welll doh: TYPE SAFETY!

mbesto 6y ago
> The only reason, then, to use a statically typed language, was for better performance. Uhh, someone please correct me if I'm wrong, but aren't statically typed languages about reliability (e.g. testing, mutability)? EDIT: in addition to performance, not solely.
- zdragnar 6y ago
  Two decades ago, dynamic languages were more typically scripted / interpreted, and statically typed languages were more typically compiled. Scripting does allow for quicker iteration (especially if compared to a language with a slow compiler) and compilation usually does produce faster code (at least pre-dating sophisticated JIT compilers).
- marta_morena_28 6y ago
  Statically typed languages only eliminate SOME things you'd otherwise have to test. In the end, you can eliminate almost anything besides logical errors, however, those are unfortunately a pretty big portion of bugs :D. So while I would always use statically typed languages for anything that needs to be reliable, I do not see how this is in any way a necessity. You CAN write reliable programs without type safety, you just have to test things you normally wouldn't have to test (i.e. the lack of type safety introduces a whole bunch of null-pointer style scenarios where you get something your code totally didn't expect but has to deal with). As for performance. Statically typed languages are usually faster, mostly because we do not have the technology yet to make dynamically typed ones as fast (in the general case). Not because there is something inherently different about them. However, I imagine the technology to make them on par with statically typed languages will take another few decades. Mainly because untyped languages need sophisticated transformations to be fast. That is the job the human normally does for the compiler in typed languages. Things just fit together and play nicely. With dynamic languages, your get one big spaghetti soup with no structure and now the compiler has to figure out the "intended" types and compile for different estimated function signatures, etc. all while honoring the JIT-style performance considerations (fast startup, low overhead during runtime). This is a gargantuan task that probably will require advanced machine learning before it really takes off.

Razengan 6y ago

I think most Swift APIs, especially SwiftUI and Combine, are pretty impressive, clean and modern.

seanhandley 6y ago

I agree there are many positive aspects to schema-based API systems. What I disagree with is the assertion that "X is the future, Y is the past". Novelty does not automatically equal superiority and there are tangible pros and cons for each. I might be able to take the discussion slightly more seriously if the article wasn't part of a marketing strategy for a company selling protobuf-based products.

Thetawaves 6y ago

My experience with SOAP and wizzles indicates that one of the primary problems with statically typed protocols is the proliferation of types. Corresponding types need to be created in each language, and these object representations reflect the original implementation language (server), rather than whatever idiosyncrasies fit the current language (client). The core benefit the world gained by migrating to REST/JSON was the total simplification of types, and the ease of interacting with them in any language.