4 ms·

https://xkcd.com/927/ https://xkcd.com/927/ Edit: Sorry, that was probably not polite. But I've been here before too many times. IO Docs, RAML, Blueprint... I

https://xkcd.com/927/ https://xkcd.com/927/ Edit: Sorry, that was probably not polite. But I've been here before too many times. IO Docs, RAML, Blueprint... I guess the question is - why should I adopt this if I'm already committed to the Open API ecosystem, and why should I consider it if I am still deciding?

martypitt 4y ago
We started trying to be a better OpenAPI, however the goals have shifted, and today we're trying to build a cross-team semantic language for describing data and APIs - one which describes how data and APIs from independent teams can interact. OpenAPI is a great way for someone to publish their API, so that another person can discover it, generate a Client SDK, and start building integrations. Taxi has the same goal, but for machines - so that software can automatically discover another API, and understand semantically how to interact with it. The two techs can co-exist quite happily, and you get great documentation benefits in embedding Taxi types in your OpenAPI spec [0] IMO, OpenAPI has a poor signal-to-noise ratio - a documentation tool you can't actually read without tooling. That feels misaligned to me. More importantly, OpenAPI doesn't do a good enough job in describing the semantics of a contract. What String should I pass to your API? EmailAddress or CustomerId? In this transaction, where do I place the BuyerCustomerId vs SellerCustomerId? That's an important distinction, which other tools aren't trying to address. With Taxi, we want to document APIs and Data in a way that tooling can understand across the organsiation, without having to fall back to reading docs, and that doesn't require shared models across the org. From there, we can build tooling can understand the semantics of an API, and then automate the integration (That's our other tool, Vyne). [0] https://docs.taxilang.org/generating-taxi-from-source/#examples https://docs.taxilang.org/generating-taxi-from-source/#examp...
- meh2frdf 4y ago
  Unpopular opinion, but I think RDF and shacl solves a lot of this!
  - jcrites 4y ago
    I hadn't heard of RDF. I've tried to review its W3C material right now but the documentation is so abstract that I'm struggling to understand what idea it's trying to convey, or problem to solve. https://www.w3.org/TR/rdf11-concepts/ https://www.w3.org/TR/rdf11-concepts/ I think the concepts this document is trying to convey would be a lot clearer with an example of how one might use the concept to build a concrete system, and what problem the RDF technology helps solve. The overall description is "RDF is a standard model for data interchange on the Web." That seems to fall short from defining services and APIs, and then code generating client and/or server logic, which is the goal of Taxi and Smithy: https://awslabs.github.io/smithy/ https://awslabs.github.io/smithy/ RDF appears to be a data format, and I don't understand what benefit I'd get from using it, except perhaps that automated reasoning tools could potentially interact with my data more easily. But that is rarely a problem I have when building systems: maybe I'd consider this if I was in charge of building a government system that publishes official data to the public (and then also offer a CSV/JSON feed which everyone would probably use in practice), but when working in a private (corporate/organizational) environment, the benefits of RDF seem lackluster. The only immediate benefit I can think of would be that, using unique IRIs to represent business data types, I could perhaps see what systems across a company handle a particular type of data: but this would also be possible when using Smithy or Taxi (if appropriately reusing types and models across many interacting systems). The data format also appears to be reasonably verbose due to the inclusion of the field name and type for every field. But many of the systems I build are RPC and the client and service already have a shared model of the data that they will be exchanging. Including RDF's additional information would seem to result in a wire format that's lower performance than something built on a toolkit like Apache Thrift/gRPC/Capn'Proto, or even hand-coded HTTP APIs (as much as I despise that approach). What benefits would one get using RDF vs. defining an API using Taxi or Smithy, and then binding it to an efficient binary or text HTTP protocol representation? (E.g., BSON, FlatBuffers, etc.) If I wanted to serve self-describing data I'd be more inclined to use a technology like Ion, which is very concrete and simple to understand: https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/ The documents on SHACL are a bit clearer due to the inclusion of concrete examples, but I'm still not sure (1) what engineering problem I would face when building a system that these technologies would help me solve, and (2) is the actually usable (cross-platform) and robust, or is it largely a set conceptual ideas? There didn't seem to be many actively maintained open source projects focused on RDF. https://www.w3.org/TR/2017/REC-shacl-20170720/ https://www.w3.org/TR/2017/REC-shacl-20170720/ Please feel free to let me know if I'm underestimating or misunderstanding RDF. From my skim of its docs it seems like a data format, not a way to model APIs (and get down to business of giving them concrete protocols), which seems like a very crucial distinction from the problem that Taxi and Smithy solve. I suppose since RDF can describe "any" data, one could invent a set of data types that express operations, models, types, etc., but unless there's a standard set of types for expressing operations, models, types, and bindings to protocol interfaces, already, then you're leaving it to the developers of each service to define their own RDF representations of these things, which means that there wouldn't be similarity to modeling of these concepts across services, preventing universal automated reasoning or transformation. (Let me know if there's a spec for using RDF for defining service interfaces ... there are a lot of docs associated with that WG and I can't read them all before writing an HN comment :-) From all of what I've read, it looks like RDF is trying to define a data format that can be understood "universally" through the definition of well known IRIs; that feels like a different problem than defining a service interface, service stubs, and client libraries, that might be used privately within an organization for building client/server systems.
- jcrites 4y ago
  How does Taxi and its goals compare to that of Smithy? https://awslabs.github.io/smithy/ https://awslabs.github.io/smithy/ On the surface it looks like they're attempting to solve very similar problems. Smithy also seems to come with code generators that can produce client libraries in various languages for calling the APIs too -- something that would be an important feature to me. There are implementations for a number of languages apparently, at various stages of maturity: https://awslabs.github.io/smithy/implementations.html https://awslabs.github.io/smithy/implementations.html Among them include TypeScript, Go, Rust, Kotlin, Swift, Scala; server -side generators for TypeScript; and model converters for converting Smithy models to Open API or JSON Schema (presumably where compatible). Cool project. I've definitely hoped that a toolkit would become de facto standard that provides model definition, protocol bindings (abstract operations accepting/returning models aka resources, described as a specific wire protocol like HTTP), and finally code generation for both client and server-sides of the interface for multiple platforms. It looks like Smithy encourages the same approach of defining unique types to represent model fields. A Smithy example pulled from their page (abbreviating some newlines since HN still doesn't support Markdown :-( for code blocks): // "pattern" is a trait. @pattern("^[A-Za-z0-9 ]+$") string CityId resource City { identifiers: { cityId: CityId }, read: GetCity, list: ListCities, resources: [Forecast], } resource Forecast { identifiers: { cityId: CityId }, read: GetForecast, } @readonly operation GetCity { input: GetCityInput, output: GetCityOutput, errors: [NoSuchResource] } @input structure GetCityInput { // "cityId" provides the identifier for the resource and // has to be marked as required. @required cityId: CityId } Where existing RPC toolkits like Apache Thrift, gRPC, and Cap'n Proto define and derive their wire formats from your IDL, and you have no control over them, I like this concept's power because you can: (1) use it to match and model the interface of an existing API, if you need to (2) generate client and server stubs for many platforms, saving the hassle of trying to integrate with raw HTTP APIs (frequently mistermed "REST") (3) ideally the toolkit would also provide bindings or plugins for multiple potential protocols, such as an HTTP resource-oriented representation, or a binary BSON representation, etc., to allow for easily making tradeoffs between APIs that are human-accessible and those that are more highly efficient (although if you can count on HTTP/2 or HTTP/3 at all layers, regular "text" requests should actually be quite efficient). And if you don't need the API to be easily callable from browse JavaScript, all the substantial parameters could be represented by HTTP headers to benefit from those protocols' header compression. (4) Perhaps a plugin could also provide a special-purpose (perhaps configuration-driven, but might require the ability to define state machines; perhaps the protocol translator could be implemented in one language as a sort of VM with a sequence of commands for how the service API, and its operations, models, resources, types are converted into binary wire format. (5) Toolkits like this if designed with foresight could also be useful for modeling (large) data sets -- a completely different use-case than providing APIs. For example, a plugin to convert collections of models/resources into Apache Parque and back, or other similar efficient bulk data formats. You could the toolkit to model your system's log file format (where resource collections convert to newline-delimited JSON text files), or perhaps performance. Anywhere that data is handled, it ought to have a formal model in my opinion. Toolkits like these will simplify translating and manipulating it between formats and systems in a type-safe way given sufficient code generation support. (For example, in languages that support opaque type aliases, CityId, despite being a string, would not be castable to/from string. Instead, a transformation can be defined to convert a CityId to string if necessary -- otherwise, developers are discouraged from conflating many types that might be aliases for string as interchangeable, permitting errors that a strong type system with opaque aliases would catch. (For example, since a CityId must match a regex, creating one would require calling a function that perhaps accepts a string as input, validates the regex, and then returns a CityId as long as the regex matches.) It's interesting to imagine what other semantic validation could be lifted up to the model level to make applications more robust. Perhaps the next stage beyond this one might be relationships between data. For example, perhaps a model could include a set of words, and then a text blob that must consist exclusively of space-delimited words from that list; or a list of banned words. Anyway, it looks like Taxi and Smithy might be close enough in goals to potentially benefit from exploring collaboration. On the other hand some competition might be healthy.

martypitt 4y ago

We started trying to be a better OpenAPI, however the goals have shifted, and today we're trying to build a cross-team semantic language for describing data and APIs - one which describes how data and APIs from independent teams can interact. OpenAPI is a great way for someone to publish their API, so that another person can discover it, generate a Client SDK, and start building integrations. Taxi has the same goal, but for machines - so that software can automatically discover another API, and understand semantically how to interact with it. The two techs can co-exist quite happily, and you get great documentation benefits in embedding Taxi types in your OpenAPI spec [0] IMO, OpenAPI has a poor signal-to-noise ratio - a documentation tool you can't actually read without tooling. That feels misaligned to me. More importantly, OpenAPI doesn't do a good enough job in describing the semantics of a contract. What String should I pass to your API? EmailAddress or CustomerId? In this transaction, where do I place the BuyerCustomerId vs SellerCustomerId? That's an important distinction, which other tools aren't trying to address. With Taxi, we want to document APIs and Data in a way that tooling can understand across the organsiation, without having to fall back to reading docs, and that doesn't require shared models across the org. From there, we can build tooling can understand the semantics of an API, and then automate the integration (That's our other tool, Vyne). [0] https://docs.taxilang.org/generating-taxi-from-source/#examples https://docs.taxilang.org/generating-taxi-from-source/#examp...

meh2frdf 4y ago
Unpopular opinion, but I think RDF and shacl solves a lot of this!
- jcrites 4y ago
  I hadn't heard of RDF. I've tried to review its W3C material right now but the documentation is so abstract that I'm struggling to understand what idea it's trying to convey, or problem to solve. https://www.w3.org/TR/rdf11-concepts/ https://www.w3.org/TR/rdf11-concepts/ I think the concepts this document is trying to convey would be a lot clearer with an example of how one might use the concept to build a concrete system, and what problem the RDF technology helps solve. The overall description is "RDF is a standard model for data interchange on the Web." That seems to fall short from defining services and APIs, and then code generating client and/or server logic, which is the goal of Taxi and Smithy: https://awslabs.github.io/smithy/ https://awslabs.github.io/smithy/ RDF appears to be a data format, and I don't understand what benefit I'd get from using it, except perhaps that automated reasoning tools could potentially interact with my data more easily. But that is rarely a problem I have when building systems: maybe I'd consider this if I was in charge of building a government system that publishes official data to the public (and then also offer a CSV/JSON feed which everyone would probably use in practice), but when working in a private (corporate/organizational) environment, the benefits of RDF seem lackluster. The only immediate benefit I can think of would be that, using unique IRIs to represent business data types, I could perhaps see what systems across a company handle a particular type of data: but this would also be possible when using Smithy or Taxi (if appropriately reusing types and models across many interacting systems). The data format also appears to be reasonably verbose due to the inclusion of the field name and type for every field. But many of the systems I build are RPC and the client and service already have a shared model of the data that they will be exchanging. Including RDF's additional information would seem to result in a wire format that's lower performance than something built on a toolkit like Apache Thrift/gRPC/Capn'Proto, or even hand-coded HTTP APIs (as much as I despise that approach). What benefits would one get using RDF vs. defining an API using Taxi or Smithy, and then binding it to an efficient binary or text HTTP protocol representation? (E.g., BSON, FlatBuffers, etc.) If I wanted to serve self-describing data I'd be more inclined to use a technology like Ion, which is very concrete and simple to understand: https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/ The documents on SHACL are a bit clearer due to the inclusion of concrete examples, but I'm still not sure (1) what engineering problem I would face when building a system that these technologies would help me solve, and (2) is the actually usable (cross-platform) and robust, or is it largely a set conceptual ideas? There didn't seem to be many actively maintained open source projects focused on RDF. https://www.w3.org/TR/2017/REC-shacl-20170720/ https://www.w3.org/TR/2017/REC-shacl-20170720/ Please feel free to let me know if I'm underestimating or misunderstanding RDF. From my skim of its docs it seems like a data format, not a way to model APIs (and get down to business of giving them concrete protocols), which seems like a very crucial distinction from the problem that Taxi and Smithy solve. I suppose since RDF can describe "any" data, one could invent a set of data types that express operations, models, types, etc., but unless there's a standard set of types for expressing operations, models, types, and bindings to protocol interfaces, already, then you're leaving it to the developers of each service to define their own RDF representations of these things, which means that there wouldn't be similarity to modeling of these concepts across services, preventing universal automated reasoning or transformation. (Let me know if there's a spec for using RDF for defining service interfaces ... there are a lot of docs associated with that WG and I can't read them all before writing an HN comment :-) From all of what I've read, it looks like RDF is trying to define a data format that can be understood "universally" through the definition of well known IRIs; that feels like a different problem than defining a service interface, service stubs, and client libraries, that might be used privately within an organization for building client/server systems.
jcrites 4y ago
How does Taxi and its goals compare to that of Smithy? https://awslabs.github.io/smithy/ https://awslabs.github.io/smithy/ On the surface it looks like they're attempting to solve very similar problems. Smithy also seems to come with code generators that can produce client libraries in various languages for calling the APIs too -- something that would be an important feature to me. There are implementations for a number of languages apparently, at various stages of maturity: https://awslabs.github.io/smithy/implementations.html https://awslabs.github.io/smithy/implementations.html Among them include TypeScript, Go, Rust, Kotlin, Swift, Scala; server -side generators for TypeScript; and model converters for converting Smithy models to Open API or JSON Schema (presumably where compatible). Cool project. I've definitely hoped that a toolkit would become de facto standard that provides model definition, protocol bindings (abstract operations accepting/returning models aka resources, described as a specific wire protocol like HTTP), and finally code generation for both client and server-sides of the interface for multiple platforms. It looks like Smithy encourages the same approach of defining unique types to represent model fields. A Smithy example pulled from their page (abbreviating some newlines since HN still doesn't support Markdown :-( for code blocks): // "pattern" is a trait. @pattern("^[A-Za-z0-9 ]+$") string CityId resource City { identifiers: { cityId: CityId }, read: GetCity, list: ListCities, resources: [Forecast], } resource Forecast { identifiers: { cityId: CityId }, read: GetForecast, } @readonly operation GetCity { input: GetCityInput, output: GetCityOutput, errors: [NoSuchResource] } @input structure GetCityInput { // "cityId" provides the identifier for the resource and // has to be marked as required. @required cityId: CityId } Where existing RPC toolkits like Apache Thrift, gRPC, and Cap'n Proto define and derive their wire formats from your IDL, and you have no control over them, I like this concept's power because you can: (1) use it to match and model the interface of an existing API, if you need to (2) generate client and server stubs for many platforms, saving the hassle of trying to integrate with raw HTTP APIs (frequently mistermed "REST") (3) ideally the toolkit would also provide bindings or plugins for multiple potential protocols, such as an HTTP resource-oriented representation, or a binary BSON representation, etc., to allow for easily making tradeoffs between APIs that are human-accessible and those that are more highly efficient (although if you can count on HTTP/2 or HTTP/3 at all layers, regular "text" requests should actually be quite efficient). And if you don't need the API to be easily callable from browse JavaScript, all the substantial parameters could be represented by HTTP headers to benefit from those protocols' header compression. (4) Perhaps a plugin could also provide a special-purpose (perhaps configuration-driven, but might require the ability to define state machines; perhaps the protocol translator could be implemented in one language as a sort of VM with a sequence of commands for how the service API, and its operations, models, resources, types are converted into binary wire format. (5) Toolkits like this if designed with foresight could also be useful for modeling (large) data sets -- a completely different use-case than providing APIs. For example, a plugin to convert collections of models/resources into Apache Parque and back, or other similar efficient bulk data formats. You could the toolkit to model your system's log file format (where resource collections convert to newline-delimited JSON text files), or perhaps performance. Anywhere that data is handled, it ought to have a formal model in my opinion. Toolkits like these will simplify translating and manipulating it between formats and systems in a type-safe way given sufficient code generation support. (For example, in languages that support opaque type aliases, CityId, despite being a string, would not be castable to/from string. Instead, a transformation can be defined to convert a CityId to string if necessary -- otherwise, developers are discouraged from conflating many types that might be aliases for string as interchangeable, permitting errors that a strong type system with opaque aliases would catch. (For example, since a CityId must match a regex, creating one would require calling a function that perhaps accepts a string as input, validates the regex, and then returns a CityId as long as the regex matches.) It's interesting to imagine what other semantic validation could be lifted up to the model level to make applications more robust. Perhaps the next stage beyond this one might be relationships between data. For example, perhaps a model could include a set of words, and then a text blob that must consist exclusively of space-delimited words from that list; or a list of banned words. Anyway, it looks like Taxi and Smithy might be close enough in goals to potentially benefit from exploring collaboration. On the other hand some competition might be healthy.