42 ms·

Why we created Taxi, and why we felt the need for another schema language

59 points by martypitt 3y ago

obi1kenobi 3y ago
I've had my eye on Taxi for a while, and it's neat! I agree that the problems it aims to solve are real and painful in real life. In my experience, I'm not sure people care about schemas or schema languages — they are just implementation details best left under the hood. This is why in my own work, I started on the query end of the spectrum instead. This is why I built Trustfall, a query engine able to query any data source: DBs, APIs, files like PDF, CSV, or JSON, or any combination of those. GitHub: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall Playground showing the query "which GitHub/Twitter users comment on HackerNews stories about OpenAI?": https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVlcnkgKEFsZ29saWEgKyBGaXJlYmFzZSk6CiMgRmluZCBjb21tZW50cyBvbiBzdG9yaWVzIGFib3V0ICJvcGVuYWkuY29tIiB3aGVyZQojIHRoZSBjb21tZW50ZXIncyBiaW8gaGFzIGF0IGxlYXN0IG9uZSBHaXRIdWIgb3IgVHdpdHRlciBsaW5rCnF1ZXJ5IHsKICAjIFRoaXMgaGl0cyB0aGUgQWxnb2xpYSBzZWFyY2ggQVBJIGZvciBIYWNrZXJOZXdzLgogICMgVGhlIHN0b3JpZXMvY29tbWVudHMvdXNlcnMgZGF0YSBpcyBmcm9tIHRoZSBGaXJlYmFzZSBITiBBUEkuCiAgIyBUaGUgdHJhbnNpdGlvbiBpcyBzZWFtbGVzcyAtLSBpdCBpc24ndCB2aXNpYmxlIGZyb20gdGhlIHF1ZXJ5LgogIFNlYXJjaEJ5RGF0ZShxdWVyeTogIm9wZW5haS5jb20iKSB7CiAgICAuLi4gb24gU3RvcnkgewogICAgICAjIEFsbCBkYXRhIGZyb20gaGVyZSBvbndhcmQgaXMgZnJvbSB0aGUgRmlyZWJhc2UgQVBJLgogICAgICBzdG9yeVRpdGxlOiB0aXRsZSBAb3V0cHV0CiAgICAgIHN0b3J5TGluazogdXJsIEBvdXRwdXQKICAgICAgc3Rvcnk6IHN1Ym1pdHRlZFVybCBAb3V0cHV0CiAgICAgICAgICAgICAgICAgICAgICAgICAgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNpdGVQYXR0ZXJuIl0pCgogICAgICBjb21tZW50IHsKICAgICAgICByZXBseSBAcmVjdXJzZShkZXB0aDogNSkgewogICAgICAgICAgY29tbWVudDogdGV4dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICBieVVzZXIgewogICAgICAgICAgICBjb21tZW50ZXI6IGlkIEBvdXRwdXQKICAgICAgICAgICAgY29tbWVudGVyQmlvOiBhYm91dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICAgICMgVGhlIHByb2ZpbGUgbXVzdCBoYXZlIGF0IGxlYXN0IG9uZQogICAgICAgICAgICAjIGxpbmsgdGhhdCBwb2ludHMgdG8gZWl0aGVyIEdpdEh1YiBvciBUd2l0dGVyLgogICAgICAgICAgICBsaW5rCiAgICAgICAgICAgICAgQGZvbGQKICAgICAgICAgICAgICBAdHJhbnNmb3JtKG9wOiAiY291bnQiKQogICAgICAgICAgICAgIEBmaWx0ZXIob3A6ICI%2BPSIsIHZhbHVlOiBbIiRtaW5Qcm9maWxlcyJdKQogICAgICAgICAgICB7CiAgICAgICAgICAgICAgY29tbWVudGVySURzOiB1cmwgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNvY2lhbFBhdHRlcm4iXSkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBAb3V0cHV0CiAgICAgICAgICAgIH0KICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIH0KICAgIH0KICB9Cn0%3D&v=ewogICJzaXRlUGF0dGVybiI6ICJodHRwW3NdOi8vKFteLl0qXFwuKSpvcGVuYWkuY29tLy4qIiwKICAibWluUHJvZmlsZXMiOiAxLAogICJzb2NpYWxQYXR0ZXJuIjogIihnaXRodWJ8dHdpdHRlcilcXC5jb20vIgp9 https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVl...
- martypitt 3y ago
  I agree that querying is where the real value is at, and Trustfall looks like an elegant approach. While Taxi is all about documenting & augmenting specs, Orbital is the query engine (which is a bit similar to Trustfall) that consumes those specs. Orbital's goal is to allow consumers to be able to query for data, without having to be aware of the specifics of the data sets / APIs / DBs, etc they're composing together. IMO GraphQL does a nice job on the query side of keeping consumers away from the wiring, but shifts the obligation to middleware resolvers. How data is wired together is ultimately an implementation detail, and one that changes. If consumers need to know about how stuff is connected, they're subject to breakages when those details change.
  - gavinray 3y ago
    > GraphQL does a nice job on the query side of keeping consumers away from the wiring, but shifts the obligation to middleware resolvers. I'm not sure that I understand how Orbital is different from tools that autogenerate GraphQL API's on top of data sources like databases and API's. From the site, it reads: > "Orbital uses your existing API specs and Database schemas - enriched with Taxi metadata to describe links. Orbital turns this into rich API and data catalog, letting you explore all your data, and how it connects." It sounds like the same concept, except using a new language instead of GraphQL? The schema language looks very similar to something AWS has called Smithy by the way: - https://aws.amazon.com/blogs/developer/introducing-smithy-idl-2-0/ https://aws.amazon.com/blogs/developer/introducing-smithy-id... - https://smithy.io/2.0/index.html https://smithy.io/2.0/index.html
    - martypitt 3y ago
      Autogen tools do a nice job of creating GraphQL schemas from a single source. We're looking more at how to compose / stitch multiple sources into a single queryable API. (ie., Stitch data from a DB, some REST APIs, and a Kafka topic). In GraphQL, you need to write / maintain resolvers that handle the stitching for you - which means as upstream data sources change, resolvers need to be maintained too. Smithy is nice, and we'll probably add first-class support in Orbital for it at some point. Taxi was around before Smithy, so we're the OG :)
sfvisser 3y ago
Couldn’t quickly find it, but does it support enums/sum types?
- martypitt 3y ago
  Enums, yep: https://taxilang.org/language-reference/types-and-models/#enums https://taxilang.org/language-reference/types-and-models/#en... Sum types, no, not yet.
hurril 3y ago
The examples all look like something I can do today in any modern functional language. What if I told you that you do not have to stick with Java or C#?
hhthrowaway1230 3y ago
i love how graphql described its types in its dsl, thats one of things id love to see carried over in other schema formats. it looks like taxi tries to do this to some extend
- martypitt 3y ago
  A descriptive type system is a big part of what makes Taxi work. We took a lot of inspiration from GraphQL (because it's awesome), but I actually think we go further in the type system than GraphQL.
  - deleted 3y ago
    [deleted]
dang 3y ago
I think there was a related thread about this in recent weeks but I can't find it. Anyone?
- martypitt 3y ago
  Taxi was discussed 12 months ago here: https://news.ycombinator.com/item?id=31524950 https://news.ycombinator.com/item?id=31524950
  - dang 3y ago
    That's the one I was looking for. Thanks! Taxi: A language for documenting data models and the contracts of APIs - https://news.ycombinator.com/item?id=31524950 https://news.ycombinator.com/item?id=31524950 - May 2022 (36 comments)
philipodonnell 3y ago
> Taxi shifts the obligation of describing how things stitch together from Consumers to Producers. Traditionally, it falls to consumers to work out how to do this. And, it’s an expensive question to answer… it involves tracking down API specs, reading docs, and building a mental model of how things hang together. I work on enterprise data platforms and this project sounds very much like the "data mesh" concept where responsibility for defining metadata/usability of datasets is supposed to rest with the teams producing the datasets rather than the teams consuming datasets. The same criticism probably applies: * Even though it is expensive/slow for a consumer to figure out how to use data, it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers, rather than focus on their system and have actual consumers figure out how to use the subset of data they need as it changes. * Above a Dunbar-esque-sized organization you can no longer rely on empathy to encourage producers to expend extra effort on consumer's needs, so either monetary or disciplinary encouragement is needed, both of which raise costs even further. Is there a name for this kind of paradox? When an approach will work only in a company below a certain size/complexity/age, but that kind of company probably doesn't feel the pain enough to need the approach?
- martypitt 3y ago
  > it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers I agree, but I don't think Taxi encourages that. (I sure hope it doesn't). All that's really happening is defining a system-agnostic set of terms, embedding those in producer schemas, and then letting consumers using those same terms to ask for data. Those terms can be used to automate integration, and in many (but not all) cases, alleviate the need for low-value plumbing code. Semantic Metadata is a growing trend in the data ecosystem (cube.dev is doing a nice job here too). The Agile folks have had Ubiquitous Language as a first class concept since day dot. > an approach will work only in a company below a certain size/complexity/age I think that GraphQL federation is an example of mid-to-large companies trying to solve the federation problem, suggesting they both feel the pain, and are looking for solutions.
  - mst 3y ago
swalsh 3y ago
This is super interesting, it's like a single spec for your entire enterprise architecture.
AlexITC 3y ago
Not sure if others are aware, there is an Esoteric language called Taxi (https://esolangs.org/wiki/Taxi https://esolangs.org/wiki/Taxi).
davidy123 3y ago
It's a shame they didn't mention Hydra. https://www.hydra-cg.com/spec/latest/core/ https://www.hydra-cg.com/spec/latest/core/ This is designed to let data be exchanged richly between sites, and comes from the earliest ideas of the web, it is the HATEOAS part of REST.
speedbird 3y ago
ASN.1 anybody?
RideAndWave 3y ago
Yeah, it's a common problem with non deterministic api schemas, e.g. openapi and swagger (https://swagger.io/specification/ https://swagger.io/specification/), but Taxi looks similar in some aspects to protoforce (https://www.protoforce.io/ https://www.protoforce.io/) except that the types subset is substantially narrower.
- nimrody 3y ago
  Can you explain what's non-deterministic about swagger/open-api?
malkia 3y ago
How do you extend this later, when you need getReviews(id: FilmId, somethingElse) : FilmReview[], andSomethingElse[] ? service ReviewsApi { @HttpOperation(method = "GET", url = "https://reviews/{id}") operation getReviews(id: FilmId): FilmReview[] }

59 points by martypitt 3y ago

I've had my eye on Taxi for a while, and it's neat! I agree that the problems it aims to solve are real and painful in real life. In my experience, I'm not sure people care about schemas or schema languages — they are just implementation details best left under the hood. This is why in my own work, I started on the query end of the spectrum instead. This is why I built Trustfall, a query engine able to query any data source: DBs, APIs, files like PDF, CSV, or JSON, or any combination of those. GitHub: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall Playground showing the query "which GitHub/Twitter users comment on HackerNews stories about OpenAI?": https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVlcnkgKEFsZ29saWEgKyBGaXJlYmFzZSk6CiMgRmluZCBjb21tZW50cyBvbiBzdG9yaWVzIGFib3V0ICJvcGVuYWkuY29tIiB3aGVyZQojIHRoZSBjb21tZW50ZXIncyBiaW8gaGFzIGF0IGxlYXN0IG9uZSBHaXRIdWIgb3IgVHdpdHRlciBsaW5rCnF1ZXJ5IHsKICAjIFRoaXMgaGl0cyB0aGUgQWxnb2xpYSBzZWFyY2ggQVBJIGZvciBIYWNrZXJOZXdzLgogICMgVGhlIHN0b3JpZXMvY29tbWVudHMvdXNlcnMgZGF0YSBpcyBmcm9tIHRoZSBGaXJlYmFzZSBITiBBUEkuCiAgIyBUaGUgdHJhbnNpdGlvbiBpcyBzZWFtbGVzcyAtLSBpdCBpc24ndCB2aXNpYmxlIGZyb20gdGhlIHF1ZXJ5LgogIFNlYXJjaEJ5RGF0ZShxdWVyeTogIm9wZW5haS5jb20iKSB7CiAgICAuLi4gb24gU3RvcnkgewogICAgICAjIEFsbCBkYXRhIGZyb20gaGVyZSBvbndhcmQgaXMgZnJvbSB0aGUgRmlyZWJhc2UgQVBJLgogICAgICBzdG9yeVRpdGxlOiB0aXRsZSBAb3V0cHV0CiAgICAgIHN0b3J5TGluazogdXJsIEBvdXRwdXQKICAgICAgc3Rvcnk6IHN1Ym1pdHRlZFVybCBAb3V0cHV0CiAgICAgICAgICAgICAgICAgICAgICAgICAgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNpdGVQYXR0ZXJuIl0pCgogICAgICBjb21tZW50IHsKICAgICAgICByZXBseSBAcmVjdXJzZShkZXB0aDogNSkgewogICAgICAgICAgY29tbWVudDogdGV4dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICBieVVzZXIgewogICAgICAgICAgICBjb21tZW50ZXI6IGlkIEBvdXRwdXQKICAgICAgICAgICAgY29tbWVudGVyQmlvOiBhYm91dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICAgICMgVGhlIHByb2ZpbGUgbXVzdCBoYXZlIGF0IGxlYXN0IG9uZQogICAgICAgICAgICAjIGxpbmsgdGhhdCBwb2ludHMgdG8gZWl0aGVyIEdpdEh1YiBvciBUd2l0dGVyLgogICAgICAgICAgICBsaW5rCiAgICAgICAgICAgICAgQGZvbGQKICAgICAgICAgICAgICBAdHJhbnNmb3JtKG9wOiAiY291bnQiKQogICAgICAgICAgICAgIEBmaWx0ZXIob3A6ICI%2BPSIsIHZhbHVlOiBbIiRtaW5Qcm9maWxlcyJdKQogICAgICAgICAgICB7CiAgICAgICAgICAgICAgY29tbWVudGVySURzOiB1cmwgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNvY2lhbFBhdHRlcm4iXSkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBAb3V0cHV0CiAgICAgICAgICAgIH0KICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIH0KICAgIH0KICB9Cn0%3D&v=ewogICJzaXRlUGF0dGVybiI6ICJodHRwW3NdOi8vKFteLl0qXFwuKSpvcGVuYWkuY29tLy4qIiwKICAibWluUHJvZmlsZXMiOiAxLAogICJzb2NpYWxQYXR0ZXJuIjogIihnaXRodWJ8dHdpdHRlcilcXC5jb20vIgp9 https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVl...

martypitt 3y ago
I agree that querying is where the real value is at, and Trustfall looks like an elegant approach. While Taxi is all about documenting & augmenting specs, Orbital is the query engine (which is a bit similar to Trustfall) that consumes those specs. Orbital's goal is to allow consumers to be able to query for data, without having to be aware of the specifics of the data sets / APIs / DBs, etc they're composing together. IMO GraphQL does a nice job on the query side of keeping consumers away from the wiring, but shifts the obligation to middleware resolvers. How data is wired together is ultimately an implementation detail, and one that changes. If consumers need to know about how stuff is connected, they're subject to breakages when those details change.
- gavinray 3y ago
  > GraphQL does a nice job on the query side of keeping consumers away from the wiring, but shifts the obligation to middleware resolvers. I'm not sure that I understand how Orbital is different from tools that autogenerate GraphQL API's on top of data sources like databases and API's. From the site, it reads: > "Orbital uses your existing API specs and Database schemas - enriched with Taxi metadata to describe links. Orbital turns this into rich API and data catalog, letting you explore all your data, and how it connects." It sounds like the same concept, except using a new language instead of GraphQL? The schema language looks very similar to something AWS has called Smithy by the way: - https://aws.amazon.com/blogs/developer/introducing-smithy-idl-2-0/ https://aws.amazon.com/blogs/developer/introducing-smithy-id... - https://smithy.io/2.0/index.html https://smithy.io/2.0/index.html
  - martypitt 3y ago
    Autogen tools do a nice job of creating GraphQL schemas from a single source. We're looking more at how to compose / stitch multiple sources into a single queryable API. (ie., Stitch data from a DB, some REST APIs, and a Kafka topic). In GraphQL, you need to write / maintain resolvers that handle the stitching for you - which means as upstream data sources change, resolvers need to be maintained too. Smithy is nice, and we'll probably add first-class support in Orbital for it at some point. Taxi was around before Smithy, so we're the OG :)

sfvisser 3y ago

Couldn’t quickly find it, but does it support enums/sum types?

martypitt 3y ago
Enums, yep: https://taxilang.org/language-reference/types-and-models/#enums https://taxilang.org/language-reference/types-and-models/#en... Sum types, no, not yet.

hurril 3y ago

The examples all look like something I can do today in any modern functional language. What if I told you that you do not have to stick with Java or C#?

hhthrowaway1230 3y ago

i love how graphql described its types in its dsl, thats one of things id love to see carried over in other schema formats. it looks like taxi tries to do this to some extend

martypitt 3y ago
A descriptive type system is a big part of what makes Taxi work. We took a lot of inspiration from GraphQL (because it's awesome), but I actually think we go further in the type system than GraphQL.
- deleted 3y ago
  [deleted]

dang 3y ago

I think there was a related thread about this in recent weeks but I can't find it. Anyone?

martypitt 3y ago
Taxi was discussed 12 months ago here: https://news.ycombinator.com/item?id=31524950 https://news.ycombinator.com/item?id=31524950
- dang 3y ago
  That's the one I was looking for. Thanks! Taxi: A language for documenting data models and the contracts of APIs - https://news.ycombinator.com/item?id=31524950 https://news.ycombinator.com/item?id=31524950 - May 2022 (36 comments)

philipodonnell 3y ago

> Taxi shifts the obligation of describing how things stitch together from Consumers to Producers. Traditionally, it falls to consumers to work out how to do this. And, it’s an expensive question to answer… it involves tracking down API specs, reading docs, and building a mental model of how things hang together. I work on enterprise data platforms and this project sounds very much like the "data mesh" concept where responsibility for defining metadata/usability of datasets is supposed to rest with the teams producing the datasets rather than the teams consuming datasets. The same criticism probably applies: * Even though it is expensive/slow for a consumer to figure out how to use data, it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers, rather than focus on their system and have actual consumers figure out how to use the subset of data they need as it changes. * Above a Dunbar-esque-sized organization you can no longer rely on empathy to encourage producers to expend extra effort on consumer's needs, so either monetary or disciplinary encouragement is needed, both of which raise costs even further. Is there a name for this kind of paradox? When an approach will work only in a company below a certain size/complexity/age, but that kind of company probably doesn't feel the pain enough to need the approach?

martypitt 3y ago
> it is even more expensive/slow to have producers try to continuously anticipate the needs of all possible current and future consumers I agree, but I don't think Taxi encourages that. (I sure hope it doesn't). All that's really happening is defining a system-agnostic set of terms, embedding those in producer schemas, and then letting consumers using those same terms to ask for data. Those terms can be used to automate integration, and in many (but not all) cases, alleviate the need for low-value plumbing code. Semantic Metadata is a growing trend in the data ecosystem (cube.dev is doing a nice job here too). The Agile folks have had Ubiquitous Language as a first class concept since day dot. > an approach will work only in a company below a certain size/complexity/age I think that GraphQL federation is an example of mid-to-large companies trying to solve the federation problem, suggesting they both feel the pain, and are looking for solutions.
- mst 3y ago

swalsh 3y ago

This is super interesting, it's like a single spec for your entire enterprise architecture.

AlexITC 3y ago

Not sure if others are aware, there is an Esoteric language called Taxi (https://esolangs.org/wiki/Taxi https://esolangs.org/wiki/Taxi).

davidy123 3y ago

It's a shame they didn't mention Hydra. https://www.hydra-cg.com/spec/latest/core/ https://www.hydra-cg.com/spec/latest/core/ This is designed to let data be exchanged richly between sites, and comes from the earliest ideas of the web, it is the HATEOAS part of REST.

speedbird 3y ago

ASN.1 anybody?

RideAndWave 3y ago

Yeah, it's a common problem with non deterministic api schemas, e.g. openapi and swagger (https://swagger.io/specification/ https://swagger.io/specification/), but Taxi looks similar in some aspects to protoforce (https://www.protoforce.io/ https://www.protoforce.io/) except that the types subset is substantially narrower.

nimrody 3y ago
Can you explain what's non-deterministic about swagger/open-api?

malkia 3y ago

How do you extend this later, when you need getReviews(id: FilmId, somethingElse) : FilmReview[], andSomethingElse[] ? service ReviewsApi { @HttpOperation(method = "GET", url = "https://reviews/{id}") operation getReviews(id: FilmId): FilmReview[] }