4 ms·
Trustfall author here, pleasantly surprised to find this posted! The goal of Trustfall is to be the LLVM of data sources. GraphQL, OpenAPI, JSON (with JSON sch
by obi1kenobi 4y ago
Trustfall author here, pleasantly surprised to find this posted!
The goal of Trustfall is to be the LLVM of data sources. GraphQL, OpenAPI, JSON (with JSON schema or not), SQL, RDF/SPARQL -- and none of them can natively talk to each other. Sure, you can stick JSON into Postgres, or compile GraphQL to SQL -- I've done both in production and it's always ultimately a poor fit because you're cramming one system into another when it was never originally designed to support that.
Here's an example: tell me the GitHub or Twitter accounts of HN users that have commented on HN stories about OpenAI. The data is available from the HN APIs on Firebase (for item lookup) and Algolia (for search). I know all of us could write a script to do it -- but would we? Or is it too annoying and difficult, and not worth it? That "activation energy" barrier is something I want to eliminate. Here's that same query in the Trustfall Playground, where it took just a minute or two to put together: https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVlcnkgKEFsZ29saWEgKyBGaXJlYmFzZSk6CiMgRmluZCBjb21tZW50cyBvbiBzdG9yaWVzIGFib3V0ICJvcGVuYWkuY29tIiB3aGVyZQojIHRoZSBjb21tZW50ZXIncyBiaW8gaGFzIGF0IGxlYXN0IG9uZSBHaXRIdWIgb3IgVHdpdHRlciBsaW5rCnF1ZXJ5IHsKICAjIFRoaXMgaGl0cyB0aGUgQWxnb2xpYSBzZWFyY2ggQVBJIGZvciBIYWNrZXJOZXdzLgogICMgVGhlIHN0b3JpZXMvY29tbWVudHMvdXNlcnMgZGF0YSBpcyBmcm9tIHRoZSBGaXJlYmFzZSBITiBBUEkuCiAgIyBUaGUgdHJhbnNpdGlvbiBpcyBzZWFtbGVzcyAtLSBpdCBpc24ndCB2aXNpYmxlIGZyb20gdGhlIHF1ZXJ5LgogIFNlYXJjaEJ5RGF0ZShxdWVyeTogIm9wZW5haS5jb20iKSB7CiAgICAuLi4gb24gU3RvcnkgewogICAgICAjIEFsbCBkYXRhIGZyb20gaGVyZSBvbndhcmQgaXMgZnJvbSB0aGUgRmlyZWJhc2UgQVBJLgogICAgICBzdG9yeVRpdGxlOiB0aXRsZSBAb3V0cHV0CiAgICAgIHN0b3J5TGluazogdXJsIEBvdXRwdXQKICAgICAgc3Rvcnk6IHN1Ym1pdHRlZFVybCBAb3V0cHV0CiAgICAgICAgICAgICAgICAgICAgICAgICAgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNpdGVQYXR0ZXJuIl0pCgogICAgICBjb21tZW50IHsKICAgICAgICByZXBseSBAcmVjdXJzZShkZXB0aDogNSkgewogICAgICAgICAgY29tbWVudDogdGV4dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICBieVVzZXIgewogICAgICAgICAgICBjb21tZW50ZXI6IGlkIEBvdXRwdXQKICAgICAgICAgICAgY29tbWVudGVyQmlvOiBhYm91dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICAgICMgVGhlIHByb2ZpbGUgbXVzdCBoYXZlIGF0IGxlYXN0IG9uZQogICAgICAgICAgICAjIGxpbmsgdGhhdCBwb2ludHMgdG8gZWl0aGVyIEdpdEh1YiBvciBUd2l0dGVyLgogICAgICAgICAgICBsaW5rCiAgICAgICAgICAgICAgQGZvbGQKICAgICAgICAgICAgICBAdHJhbnNmb3JtKG9wOiAiY291bnQiKQogICAgICAgICAgICAgIEBmaWx0ZXIob3A6ICI%2BPSIsIHZhbHVlOiBbIiRtaW5Qcm9maWxlcyJdKQogICAgICAgICAgICB7CiAgICAgICAgICAgICAgY29tbWVudGVySURzOiB1cmwgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNvY2lhbFBhdHRlcm4iXSkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBAb3V0cHV0CiAgICAgICAgICAgIH0KICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIH0KICAgIH0KICB9Cn0%3D&v=ewogICJzaXRlUGF0dGVybiI6ICJodHRwW3NdOi8vKFteLl0qXFwuKSpvcGVuYWkuY29tLy4qIiwKICAibWluUHJvZmlsZXMiOiAxLAogICJzb2NpYWxQYXR0ZXJuIjogIihnaXRodWJ8dHdpdHRlcilcXC5jb20vIgp9 https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVl...
Trustfall is designed for interoperation from day 1. It separates the queries from the data providers, allowing the infrastructure to evolve and change how it serves queries without any of the queries noticing anything except faster execution. In practice, that means you don't have to rewrite your product to make it run faster -- which makes both the product side and the infra side happier :)
Here's a real-world example of that. The `cargo-semver-checks` Rust semantic versioning linter implements its lints as Trustfall queries over JSON files describing the package API, and I recently was able to speed up its execution by over 2000x without changing a single query -- just by changing how those queries execute under the hood. More details in my blog post here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
AMA, I guess :)
- bhawks 4y ago> More details in my blog post here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... Tl;Dr implementations of the adapter api can provide hashmap lookups and the query engine will use this when possible before doing a full scan - this is transparent to the query author. A 3000 word essay takes approximately 10 minutes to read. (Assuming subject familiarity) It is frustrating when the main point is at the end and very little is discussed about the shape of the new api besides the fact that it is experimental. I spent quite awhile on the site trying to understand the features and benefits of the query language. I left unsatisfied by the long commentless query examples and repeated claims of being able to query anything. It'd be nice to have a discussion of the what the language is, how an adapter should be implemented and how the engine will use my adapter. Maybe that is covered in linked video talks but there are no slides to skim to see if thats worth my time. OP I appreciate your passion and enthusiasm but as a potential user I can't figure out where to start with the project.
- obi1kenobi 4y agoThanks for the detailed feedback, I appreciate it! I plan on writing more about all the topics you mentioned, and add docs for writing adapters and queries. I was planning on doing a "Show HN" when more of that was in place, but I'm not the OP here -- someone else submitted it :) Obviously all of the below needs to be more discoverable, and is something I'll work on! Here are the slides from my talk from the HYTRADBOI conference last year, containing some more real-world query use cases: https://docs.google.com/presentation/d/1foUdlEDOQ1WcTadhAxsATJ0yDbZwtr40JclJFArUVeY/edit?usp=sharing https://docs.google.com/presentation/d/1foUdlEDOQ1WcTadhAxsA... This is the current stable adapter API: it's a trait that adapters need to implement https://docs.rs/trustfall_core/latest/trustfall_core/interpreter/basic_adapter/trait.BasicAdapter.html https://docs.rs/trustfall_core/latest/trustfall_core/interpr... It's also available in the Python bindings: https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0.1.6/pytrustfall/trustfall/adapter.py#L22 https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0.... Tl;Dr of the optimizations API is that every call into the adapter gets a reference to a `QueryInfo` struct. It's totally safe to ignore if the adapter isn't interested in optimizing, and otherwise allows the adapter to ask questions like "from this location in the query, will future operations include traversing edge X and then filtering on property Y?" If so, it's possible to use the API to resolve the expected values for that Y property and use them to speed up the lookup, as in the case with the hashtables in cargo-semver-checks. The optimizations API is still pretty raw and I didn't want to dig into it too much in that post (which is mostly about cargo-semver-checks' architecture, not just Trustfall internals) in case it changed dramatically. If you really want to see the current state of the API, here's a link to the draft PR: https://github.com/obi1kenobi/trustfall/pull/131/files#diff-7d824232cb78ba5b9057259aedd7a0bec18d6eeb9c558e7f471fb808c1c7453dR21-R107 https://github.com/obi1kenobi/trustfall/pull/131/files#diff-...