8 ms·

Trustfall: How to Query (Almost) Everything

288 points by sbt567 4y ago

whoopdeepoo 4y ago
Already way too many tabs in the example. I could see this getting completely unreadable very quick.
- paavanb 4y ago
  The tabs are due to mobile limitations, it's hard to fit a full-fledged query editor and documentation browser on such a small screen. Try it on desktop! It's much easier to manage there, more like a typical GraphQL editor. Full disclosure: I wrote it.
sbt567 4y ago
Online playground for querying Hacker News: https://play.predr.ag/hackernews https://play.predr.ag/hackernews
metadat 4y ago
This is crazy cool. Instead of searching Google which returns info from literally any random source (occasional good sites among the ocean of SEO spam, malicious sites, ad-ridden clone sites, annoying trolls, paywalled, etc), you could have your own set of diverse query sources you've deemed to be actually useful and trustworthy. I suspect this is only a basic, naive idea compared to the true potential capabilities Trustfall could unlock.
- xwolfi 4y ago
  It's weird, like, does he pay you to drum up support ? It s a rust code that read a very complex json dsl to do queries, I haven't seen any sort of potential. Your use case even seems to be a UNION and not even a JOIN ?
  - paavanb 4y ago
    It's effectively a subset of GraphQL, not a JSON DSL.
- Terretta 4y ago
  See kagi.com search engine, with custom Lens, for something that queries "occasional good sites among the ocean".
flanked-evergl 4y ago
You may be interested in RDF and SPARQL which supports federated queries. Much simpler than reinventing the wheel. For more context, see https://ontop-vkg.org/guide/ https://ontop-vkg.org/guide/
- hobofan 4y ago
  If one is faced with the complexity and footguns of SPARQL reinventing the wheel is a good option.
debarshri 4y ago
At first glance, the goal of the project feels very similar to SPARQL. Somehow it didn't make it to the mainstream.
- flanked-evergl 4y ago
  Wiki Data runs a sparql endpoint and I think this integrates with Wikipedia. Not the most popular thing in the world but it is not fringe technology.
  - debarshri 4y ago
    True!
  - riku_iki 4y ago
    and do we know how many people use SPARQL for querying wikidata?..
    - obi1kenobi 4y ago
      FWIW, I've genuinely spent time trying to use it, and I've never managed to run any non-trivial query on it without having it time out. SPARQL is very expressive, but its implementations seem impossible to optimize and as a result impossible to use because they have to be locked down so tight.
parhamn 4y ago
When I saw a demo of this I was blow away by how easy it is to add another data source. Great work, looking forward to using this soon.
- tnzk 4y ago
  Curious, I didn't find how to add another data source in the talk at the top of the README[1]. Do you mind sharing the demo you saw? [1]: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin...
  - obi1kenobi 4y ago
    I bumped into him and showed it to him in person one time :) Realistically, the docs are still very sparse -- I wasn't planning on posting this for another few months, but someone else noticed my project and beat me to it! New data providers need a schema and an Adapter implementation. For Rust, this is the Adapter trait that data providers need to implement: https://docs.rs/trustfall_core/latest/trustfall_core/interpreter/basic_adapter/trait.BasicAdapter.html https://docs.rs/trustfall_core/latest/trustfall_core/interpr... This is the equivalent abstract class in Python: https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0.1.6/pytrustfall/trustfall/adapter.py#L22 https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0.... For Python, the package docs have some (underwhelming) additional examples of building a schema and passing an adapter to use to run a query: https://pypi.org/project/trustfall/ https://pypi.org/project/trustfall/ The process in Rust is equivalent, here's the code powering the demo in the talk: https://github.com/obi1kenobi/trustfall/blob/main/demo-hytradboi/src/main.rs#L70 https://github.com/obi1kenobi/trustfall/blob/main/demo-hytra... I realize this is all extremely suboptimal, but like I said, I wasn't ready Please stay tuned! I'm planning to spruce all of this up in the coming weeks!
deleted 4y ago
[deleted]
obi1kenobi 4y ago
Trustfall author here, pleasantly surprised to find this posted! The goal of Trustfall is to be the LLVM of data sources. GraphQL, OpenAPI, JSON (with JSON schema or not), SQL, RDF/SPARQL -- and none of them can natively talk to each other. Sure, you can stick JSON into Postgres, or compile GraphQL to SQL -- I've done both in production and it's always ultimately a poor fit because you're cramming one system into another when it was never originally designed to support that. Here's an example: tell me the GitHub or Twitter accounts of HN users that have commented on HN stories about OpenAI. The data is available from the HN APIs on Firebase (for item lookup) and Algolia (for search). I know all of us could write a script to do it -- but would we? Or is it too annoying and difficult, and not worth it? That "activation energy" barrier is something I want to eliminate. Here's that same query in the Trustfall Playground, where it took just a minute or two to put together: https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVlcnkgKEFsZ29saWEgKyBGaXJlYmFzZSk6CiMgRmluZCBjb21tZW50cyBvbiBzdG9yaWVzIGFib3V0ICJvcGVuYWkuY29tIiB3aGVyZQojIHRoZSBjb21tZW50ZXIncyBiaW8gaGFzIGF0IGxlYXN0IG9uZSBHaXRIdWIgb3IgVHdpdHRlciBsaW5rCnF1ZXJ5IHsKICAjIFRoaXMgaGl0cyB0aGUgQWxnb2xpYSBzZWFyY2ggQVBJIGZvciBIYWNrZXJOZXdzLgogICMgVGhlIHN0b3JpZXMvY29tbWVudHMvdXNlcnMgZGF0YSBpcyBmcm9tIHRoZSBGaXJlYmFzZSBITiBBUEkuCiAgIyBUaGUgdHJhbnNpdGlvbiBpcyBzZWFtbGVzcyAtLSBpdCBpc24ndCB2aXNpYmxlIGZyb20gdGhlIHF1ZXJ5LgogIFNlYXJjaEJ5RGF0ZShxdWVyeTogIm9wZW5haS5jb20iKSB7CiAgICAuLi4gb24gU3RvcnkgewogICAgICAjIEFsbCBkYXRhIGZyb20gaGVyZSBvbndhcmQgaXMgZnJvbSB0aGUgRmlyZWJhc2UgQVBJLgogICAgICBzdG9yeVRpdGxlOiB0aXRsZSBAb3V0cHV0CiAgICAgIHN0b3J5TGluazogdXJsIEBvdXRwdXQKICAgICAgc3Rvcnk6IHN1Ym1pdHRlZFVybCBAb3V0cHV0CiAgICAgICAgICAgICAgICAgICAgICAgICAgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNpdGVQYXR0ZXJuIl0pCgogICAgICBjb21tZW50IHsKICAgICAgICByZXBseSBAcmVjdXJzZShkZXB0aDogNSkgewogICAgICAgICAgY29tbWVudDogdGV4dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICBieVVzZXIgewogICAgICAgICAgICBjb21tZW50ZXI6IGlkIEBvdXRwdXQKICAgICAgICAgICAgY29tbWVudGVyQmlvOiBhYm91dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICAgICMgVGhlIHByb2ZpbGUgbXVzdCBoYXZlIGF0IGxlYXN0IG9uZQogICAgICAgICAgICAjIGxpbmsgdGhhdCBwb2ludHMgdG8gZWl0aGVyIEdpdEh1YiBvciBUd2l0dGVyLgogICAgICAgICAgICBsaW5rCiAgICAgICAgICAgICAgQGZvbGQKICAgICAgICAgICAgICBAdHJhbnNmb3JtKG9wOiAiY291bnQiKQogICAgICAgICAgICAgIEBmaWx0ZXIob3A6ICI%2BPSIsIHZhbHVlOiBbIiRtaW5Qcm9maWxlcyJdKQogICAgICAgICAgICB7CiAgICAgICAgICAgICAgY29tbWVudGVySURzOiB1cmwgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNvY2lhbFBhdHRlcm4iXSkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBAb3V0cHV0CiAgICAgICAgICAgIH0KICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIH0KICAgIH0KICB9Cn0%3D&v=ewogICJzaXRlUGF0dGVybiI6ICJodHRwW3NdOi8vKFteLl0qXFwuKSpvcGVuYWkuY29tLy4qIiwKICAibWluUHJvZmlsZXMiOiAxLAogICJzb2NpYWxQYXR0ZXJuIjogIihnaXRodWJ8dHdpdHRlcilcXC5jb20vIgp9 https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVl... Trustfall is designed for interoperation from day 1. It separates the queries from the data providers, allowing the infrastructure to evolve and change how it serves queries without any of the queries noticing anything except faster execution. In practice, that means you don't have to rewrite your product to make it run faster -- which makes both the product side and the infra side happier :) Here's a real-world example of that. The `cargo-semver-checks` Rust semantic versioning linter implements its lints as Trustfall queries over JSON files describing the package API, and I recently was able to speed up its execution by over 2000x without changing a single query -- just by changing how those queries execute under the hood. More details in my blog post here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... AMA, I guess :)
loveparade 4y ago
Isn't this the same as converting websites into a GraphQL API? Aren't there already dozens of projects and services that do the "convert websites into an API", at scale? What exactly is the innovation?
- obi1kenobi 4y ago
  No, it isn't: it's neither about websites nor GraphQL. The query language is substantially more powerful than GraphQL, much closer to SQL. And the idea is unifying and querying any kind of data, not just websites but also databases, raw files, ML models, anything at all. For example, the cargo-semver-checks semver linter for Rust uses Trustfall to abstract over the different JSON formats in which different Rust versions describe a library's API. The lints are written as Trustfall queries (GraphQL isn't expressive enough for this!) and don't care about how the underlying data is represented. Recently, I was able to tweak how those JSON files are used to serve data, and sped up cargo-semver-checks by over 2000x -- without changing a single query. More info here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... If you want to see more of it in action, there's a 10min talk featuring demos and real-world use cases here: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin...
- leobg 4y ago
  I googled “convert website into a GraphQL API” and literally found no matches. Did you have any specific projects/services in mind that attempt to do that kind of thing?
haolez 4y ago
This looks a lot like what Apache Drill already does.
sdfhbdf 4y ago
Very interesting, sounds kinda like GraphQL counterpart to https://github.com/cube2222/octosql https://github.com/cube2222/octosql
- obi1kenobi 4y ago
  The syntax is reminiscent to GraphQL, but semantically it's a lot more expressive and not compatible. It's much closer to SQL in expressive power, actually: left joins, recursion, arbitrary filters, aggregation, etc. -- none of that stuff works in GraphQL but is supported in Trustfall.
dilawar 4y ago
obligatory, "written in Rust".
- obi1kenobi 4y ago
  It is! But it has Python and JS bindings, and the web playground uses Trustfall compiled to WASM: https://play.predr.ag/ https://play.predr.ag/
vsroy 4y ago
I imagine with these types of things the vast majority of the work is writing integrations. Could you explain how this makes writing integrations easier?
- obi1kenobi 4y ago
  Right now, to be perfectly honest, it doesn't yet. This is by design: the purely integrations-oriented side is a crowded space, and I'm not ready to jump in there yet. Stay tuned :) The focus instead has been in a place where it's easier to stand out: deeper query ability over the data providers than any competing solution, with stronger optimization guarantees. The goal is that any query optimization you could implement by hand, you should be able to implement within Trustfall -- while having the benefit of purely declarative type-checked queries, with integrated docs and autocomplete. If a query is too slow, you don't have to rewrite it -- you can just use Trustfall's APIs to tweak how it executes under the hood and speed it up by using caching, batching, prefetch, or an index. For a real-world demo of that, I wrote a blog post about speeding up cargo-semver-checks (a Rust linter for semantic versioning) by 2272x without changing any query, just by making some indexes available to the execution of the existing queries. This is awesome because it empowers anyone to write queries, knowing that either their query will either already be fast or it can be made fast in the future in a way that is entirely invisible to the query itself. Link: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
  - cormacrelf 4y ago
    I read that yesterday and felt a bit tricked. You said there was a one line diff, which to me suggested you had made a query optimiser and added that to trustfall so that consumer applications could transparently benefit without writing any new code. But really, as it turned out, you just added APIs for for indexing a table and using those indexes, and then used those APIs to do select manual optimisations in a “fast path” in the trustfall-rustdoc-adapter crates. What does it matter that some crates are called trustfall adapters and some are not? You still had to optimise the execution of the query manually. I can see how it’s cool you didn’t need to change the text of the query, but people like SQL because the execution engines are smart enough to optimise for you. They will build a little hash table index on the fly if it saves a lot of full table scanning. The expectations re smart optimisation in the market you’re competing in are very high. If you say it was a one line upgrade to the trustfall executor then people will believe you. The net result is better than what most GraphQL infrastructure gives you. GraphQL doesn’t give you any tools to avoid full table scans, it just tells you “here’s the query someone wants, I will do the recursive queries with as many useless round trips as possible unless you implement all these things from scratch”. At least your API has the concept of an index now. But I think you’re trying to sell it as being as optimisable as SQL while trying to avoid telling users the bad news that it’s going to be them who has to write the optimiser, not you.
unshavedyak 4y ago
Could this be used to manage ingesting data from messier sources? Ala files (pdf/etc), web pages, etc? edit: Admittedly the website/video tends to talk about data sources a bit hand-wavy. I'd have loved some real world examples on how one goes about adding a data source. Also how we handle problems of scale.. ie passing filters to the data source, rather than drinking from a firehose and filtering after the fact. With that said, the idea is becoming interesting to me. At the very least i am liking the idea of a standardized query interface to "things". Just feels like edge cases might drown me.
- obi1kenobi 4y ago
  Yes, in at least two ways: - It could be used as the "ingestion process" itself: write the ingestion as a Trustfall query (or queries) over the data sources, and store the results in a traditional database or other system. - It could be used to make the "ingestion" a mere implementation detail: you could write a query over all the data sources, and that query could be executed against the raw data sources themselves, or some "ingested" format which would presumably be faster and more efficient. The query can't tell the difference, and this frees up the ingestion system to evolve independently of how it's being used. It sounds like you might have a specific use case in mind? I'd love to learn more about it if so! My Twitter DMs are open, or I could give you my email address if you'd prefer that.
- obi1kenobi 4y ago
  To be completely honest, I was planning on doing a "Show HN" in a month or two, but someone posted the link today and it caught me a bit off-guard -- that's why the docs and examples aren't there yet :) There's definitely a bit of a learning curve because the docs are few and far between, but in general, everything is much simpler than it seems. The people that have adopted it already have been able to do so within a couple of hours with minimal guidance from me. For passing filters to the data source, the API there is still very experimental (it was just an idea in my head until last week!) but I've been working on it recently. Here's a blog post showing a small bit of that new API and using it to dramatically optimize a workload: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... This is my full-time project right now, so stay tuned and expect a decently steep progress trajectory :)
wodenokoto 4y ago
Clicking through to the python repo and looking at the examples there, I guess the query language is GraphQL. Reading further, it seems that it is actually a variation of GraphQL. I've never used GraphQL before, maybe one can get used to it, but it doesn't look very nice for defining data queries.
- obi1kenobi 4y ago
  The language is syntactically similar to GraphQL, but the semantics are much richer -- in terms of expressiveness Trustfall is more like SQL than GraphQL. No language syntax makes everyone happy :) But it's definitely is something that isn't hard to get used to -- in my experience, people go from "never seen it before" to "can productively write their own queries" very quickly after trying a few examples. Try out the example queries over HackerNews in the Playground (ideally not on mobile) and see for yourself: https://play.predr.ag/hackernews https://play.predr.ag/hackernews And in the long run, Trustfall is more meant to be like an "LLVM for data sets," with pluggable language frontends. There's already been interest in plugging in other languages and making them generate Trustfall representation internally, to make use of the rest of its functionality without using the language itself. This is fine too!
nathanwallace 4y ago
Readers who like SQL may also enjoy Steampipe [1], an open source tool to live query 99+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc across the services. (Disclaimer - I'm a lead on the project.) 1 - https://github.com/turbot/steampipe https://github.com/turbot/steampipe
- mafro 4y ago
  +1 Steampipe's aggregators to combine multiple datasets into one are great
- sztanko 4y ago
  Very cool! Is it possible to connect to steampipe via jdbc connector?
  - code_biologist 4y ago
    I presume so: "You can also use psql, pgcli, Metabase, Tableau, or any client that can connect to Postgres."
    - nathanwallace 4y ago
      Correct - it's just Postgres, so all your favorite clients / tools work as usual. We have docs for a number of example integrations [1] that work with Steampipe CLI (or cloud). 1 - https://steampipe.io/docs/cloud/integrations/overview https://steampipe.io/docs/cloud/integrations/overview
- xrendan 4y ago
  Is it possible to join those data sources against an existing database like MySQL or SQLServer?
  - nathanwallace 4y ago
    Steampipe runs an embedded Postgres database [1]. One approach would be to connect that Postgres database with other databases using the usual (advanced) techniques like Foreign Data Wrappers [2]? With that connection in place, joins would work as normal. 1 - https://steampipe.io/docs/develop/overview https://steampipe.io/docs/develop/overview 2 - https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers
darkteflon 4y ago
This looks super cool. Going to put this together with Dagster tonight. @obi1kenobi, could you comment a bit on the motivation and background to this project?
- obi1kenobi 4y ago
  Might I direct you to a 10min talk I gave on the subject? https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin... The motivation is that queries that are trivial in a SQL database are painfully impractical over any other data source. Most query languages are tied way too closely to a data format or representation, and fail miserably at the boundaries to other systems or representations. Yes, we can e.g. plug SQL into APIs but there's a severe impedance mismatch there -- we end up forced into all sorts of unintuitive situations and poor UX because SQL was never meant for APIs. Based on experiences I had at work (some of them covered in that talk!), it felt like an "LLVM for data sources" could be useful -- that's what Trustfall is. Yes, it includes a query language. That's because SQL and GraphQL and the like just weren't a good fit to demonstrate true datasource-agnostic queries: we want a modern GraphQL-like user experience, with SQL-like expressiveness but without limitations that make it hard to plug in APIs and ML tool. But the query language is just one layer of the system, and it's entirely possible to replace it with another language parser while keeping the rest of the benefits. A key benefit of Trustfall is that you can write queries over any data sources, and then change the underlying implementation to add optimizations or to migrate to a new system without the queries needing to change. For example, cargo-semver-checks, a Rust semver linter I built on top of Trustfall, supports 4 separate format versions of the underlying data source it uses with the same set of queries, and I was recently able to speed up its execution by over 2000x with a tiny amount of code and effort and without even a single query needing to change -- just by tweaking how the data adapter executed queries to plug in some indexes where they mattered: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... Excited to see what you build, please post a link or email / DM me on Twitter if you'd rather not post publicly? The docs are still sparse (I wasn't planning to do a Show HN for another couple of months, but someone beat me to it today) but I'd be happy to help with anything. There are Python and Rust bindings, and feel free to open issues for anything that's confusing: https://pypi.org/project/trustfall/ https://pypi.org/project/trustfall/ https://crates.io/crates/trustfall_core https://crates.io/crates/trustfall_core
yevpats 4y ago
Also relevant - High Performance Open Source ELT Framework - https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery
srcreigh 4y ago
What join algorithms does this support? How many data sources are integrated so far (what’s the MOM growth?) What’s the largest dataset this integrates with? And how many different people do you estimate have written queries in the language?
- obi1kenobi 4y ago
  As I wrote in reply to your other comment, the join algorithms are the adapter's choice. I wouldn't necessarily recommend using it as a SQL replacement, though. There's no measurable MoM growth yet. This is the first major look the broader community has had at the system outside of a relatively small conference where I gave a talk, and demos I've given to friends and colleagues. I wasn't planning on submitting to HN for another few months -- but someone else beat me to it today :) There are production-grade adapters for HackerNews APIs and for Rust's rustdoc JSON format, used by the Playground (https://play.predr.ag/hackernews https://play.predr.ag/hackernews) and the Rust semver-checking linter cargo-semver-checks, respectively. There are also a few more demo adapters in the Trustfall repo itself: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall I expect the number of adapters to grow significantly in the coming months, so please stay tuned and let me know if you have datasets you'd like to try it with! The "largest dataset" is a bit of a trick question: how big is the dataset of "all HackerNews data available via its Firebase and Algolia APIs"? Because that's what the Playground queries. The Rust semver linter has been used by dozens if not hundreds of crates, and the JSON payloads in question there are in the 100-400MB range. The example in this blog post runs 40 quite complex Trustfall queries (they express semver rules!) over 400MB across two JSON files in 8 seconds: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... You can also see more real-world use cases in this talk I gave last year: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin... Probably around 200-300 people or so have written queries in the language, with experience levels ranging from seasoned engineers to Excel analysts and SQL analysts with no programming experience outside of that domain. The language itself is actually a refinement from an earlier open-source project I developed and open-sourced through my previous job. That open-source project was used to query everything from a multi-TB SQL cluster to APIs to ML models, and with Trustfall I've taken the opportunity to revisit and update design decisions that the previous project got to regret in retrospect.
alexisread 4y ago
Nice! I wrote something similar for my workplace a couple of years ago based around Rx (as Rx has many implementations in different languages - we had a multi-node browser/server requirement, and optimization in a streaming DSL is easier than SQL, as you've done, as you can hint+order lazy materializations) and libs like https://pypi.org/project/lquery/ https://pypi.org/project/lquery/ to do pushdown queries. Are you planning on doing reactive/live/materialized queries? You could use a config file to specify the live queries (in the Trustfall DSL) which can be fed to the engine on startup.
- obi1kenobi 4y ago
  Thanks! Reactive/live/materialized queries are certainly a possibility I've tried to keep open and I'd love to build, but there's so many possible things to build from here that I'm not sure when that might happen. In particular, while that tech is really cool, it's not clear to me how many use cases for it are actually ready to adopt it. I'm hesitant to build something that looks really cool, but is somewhat complex and doesn't get used a lot.
crosen99 4y ago
This is a nifty tool. It’s existence alongside the emerging LLMs reminds me of the two diametrically opposed approaches to harnessing it all: 1. Store the knowledge in a highly structured way and interrogate it with a precise and rigorous query language to extract the exact answer you want based on a well defined set of rules 2. Store the knowledge in whatever ad hoc way it’s produced, and then rely on a higher form of intelligence to take an equally ad hoc query, feed it through the entire universe of knowledge with some attention mechanism, and magically return a (statistically significant) response Both approaches are so satisfying when they work. Of course you also have everything in between and then you have tools like LangChain that start to bring it all together.
- TickleSteve 4y ago
  i.e. Approximate vs precise. But, there is another option. The tool (presumably AI/ML based) that takes the ad-hoc query and turns it into a precise query to the structured-data tool in order to return precise results. I hope that this will be Googles answer to ChatGPT... i.e. a chat interface to their existing search tool. This should then be able to "explain" the query its actually generated to the tool as a side-effect.
  - crosen99 4y ago
    For sure, I just edited my comment to mention LangChain, which starts to get at that capability. Would be nice to see LangChain integrate with Trustfall similar to how it can integrate with Wolfram Alpha.

288 points by sbt567 4y ago

whoopdeepoo 4y ago

Already way too many tabs in the example. I could see this getting completely unreadable very quick.

paavanb 4y ago
The tabs are due to mobile limitations, it's hard to fit a full-fledged query editor and documentation browser on such a small screen. Try it on desktop! It's much easier to manage there, more like a typical GraphQL editor. Full disclosure: I wrote it.

sbt567 4y ago

Online playground for querying Hacker News: https://play.predr.ag/hackernews https://play.predr.ag/hackernews

metadat 4y ago

This is crazy cool. Instead of searching Google which returns info from literally any random source (occasional good sites among the ocean of SEO spam, malicious sites, ad-ridden clone sites, annoying trolls, paywalled, etc), you could have your own set of diverse query sources you've deemed to be actually useful and trustworthy. I suspect this is only a basic, naive idea compared to the true potential capabilities Trustfall could unlock.

xwolfi 4y ago
It's weird, like, does he pay you to drum up support ? It s a rust code that read a very complex json dsl to do queries, I haven't seen any sort of potential. Your use case even seems to be a UNION and not even a JOIN ?
- paavanb 4y ago
  It's effectively a subset of GraphQL, not a JSON DSL.
Terretta 4y ago
See kagi.com search engine, with custom Lens, for something that queries "occasional good sites among the ocean".

flanked-evergl 4y ago

You may be interested in RDF and SPARQL which supports federated queries. Much simpler than reinventing the wheel. For more context, see https://ontop-vkg.org/guide/ https://ontop-vkg.org/guide/

hobofan 4y ago
If one is faced with the complexity and footguns of SPARQL reinventing the wheel is a good option.

debarshri 4y ago

At first glance, the goal of the project feels very similar to SPARQL. Somehow it didn't make it to the mainstream.

flanked-evergl 4y ago
Wiki Data runs a sparql endpoint and I think this integrates with Wikipedia. Not the most popular thing in the world but it is not fringe technology.
- debarshri 4y ago
  True!
- riku_iki 4y ago
  and do we know how many people use SPARQL for querying wikidata?..
  - obi1kenobi 4y ago
    FWIW, I've genuinely spent time trying to use it, and I've never managed to run any non-trivial query on it without having it time out. SPARQL is very expressive, but its implementations seem impossible to optimize and as a result impossible to use because they have to be locked down so tight.

parhamn 4y ago

When I saw a demo of this I was blow away by how easy it is to add another data source. Great work, looking forward to using this soon.

tnzk 4y ago
Curious, I didn't find how to add another data source in the talk at the top of the README[1]. Do you mind sharing the demo you saw? [1]: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin...
- obi1kenobi 4y ago
  I bumped into him and showed it to him in person one time :) Realistically, the docs are still very sparse -- I wasn't planning on posting this for another few months, but someone else noticed my project and beat me to it! New data providers need a schema and an Adapter implementation. For Rust, this is the Adapter trait that data providers need to implement: https://docs.rs/trustfall_core/latest/trustfall_core/interpreter/basic_adapter/trait.BasicAdapter.html https://docs.rs/trustfall_core/latest/trustfall_core/interpr... This is the equivalent abstract class in Python: https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0.1.6/pytrustfall/trustfall/adapter.py#L22 https://github.com/obi1kenobi/trustfall/blob/pytrustfall-v0.... For Python, the package docs have some (underwhelming) additional examples of building a schema and passing an adapter to use to run a query: https://pypi.org/project/trustfall/ https://pypi.org/project/trustfall/ The process in Rust is equivalent, here's the code powering the demo in the talk: https://github.com/obi1kenobi/trustfall/blob/main/demo-hytradboi/src/main.rs#L70 https://github.com/obi1kenobi/trustfall/blob/main/demo-hytra... I realize this is all extremely suboptimal, but like I said, I wasn't ready Please stay tuned! I'm planning to spruce all of this up in the coming weeks!

deleted 4y ago

[deleted]

obi1kenobi 4y ago

Trustfall author here, pleasantly surprised to find this posted! The goal of Trustfall is to be the LLVM of data sources. GraphQL, OpenAPI, JSON (with JSON schema or not), SQL, RDF/SPARQL -- and none of them can natively talk to each other. Sure, you can stick JSON into Postgres, or compile GraphQL to SQL -- I've done both in production and it's always ultimately a poor fit because you're cramming one system into another when it was never originally designed to support that. Here's an example: tell me the GitHub or Twitter accounts of HN users that have commented on HN stories about OpenAI. The data is available from the HN APIs on Firebase (for item lookup) and Algolia (for search). I know all of us could write a script to do it -- but would we? Or is it too annoying and difficult, and not worth it? That "activation energy" barrier is something I want to eliminate. Here's that same query in the Trustfall Playground, where it took just a minute or two to put together: https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVlcnkgKEFsZ29saWEgKyBGaXJlYmFzZSk6CiMgRmluZCBjb21tZW50cyBvbiBzdG9yaWVzIGFib3V0ICJvcGVuYWkuY29tIiB3aGVyZQojIHRoZSBjb21tZW50ZXIncyBiaW8gaGFzIGF0IGxlYXN0IG9uZSBHaXRIdWIgb3IgVHdpdHRlciBsaW5rCnF1ZXJ5IHsKICAjIFRoaXMgaGl0cyB0aGUgQWxnb2xpYSBzZWFyY2ggQVBJIGZvciBIYWNrZXJOZXdzLgogICMgVGhlIHN0b3JpZXMvY29tbWVudHMvdXNlcnMgZGF0YSBpcyBmcm9tIHRoZSBGaXJlYmFzZSBITiBBUEkuCiAgIyBUaGUgdHJhbnNpdGlvbiBpcyBzZWFtbGVzcyAtLSBpdCBpc24ndCB2aXNpYmxlIGZyb20gdGhlIHF1ZXJ5LgogIFNlYXJjaEJ5RGF0ZShxdWVyeTogIm9wZW5haS5jb20iKSB7CiAgICAuLi4gb24gU3RvcnkgewogICAgICAjIEFsbCBkYXRhIGZyb20gaGVyZSBvbndhcmQgaXMgZnJvbSB0aGUgRmlyZWJhc2UgQVBJLgogICAgICBzdG9yeVRpdGxlOiB0aXRsZSBAb3V0cHV0CiAgICAgIHN0b3J5TGluazogdXJsIEBvdXRwdXQKICAgICAgc3Rvcnk6IHN1Ym1pdHRlZFVybCBAb3V0cHV0CiAgICAgICAgICAgICAgICAgICAgICAgICAgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNpdGVQYXR0ZXJuIl0pCgogICAgICBjb21tZW50IHsKICAgICAgICByZXBseSBAcmVjdXJzZShkZXB0aDogNSkgewogICAgICAgICAgY29tbWVudDogdGV4dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICBieVVzZXIgewogICAgICAgICAgICBjb21tZW50ZXI6IGlkIEBvdXRwdXQKICAgICAgICAgICAgY29tbWVudGVyQmlvOiBhYm91dFBsYWluIEBvdXRwdXQKCiAgICAgICAgICAgICMgVGhlIHByb2ZpbGUgbXVzdCBoYXZlIGF0IGxlYXN0IG9uZQogICAgICAgICAgICAjIGxpbmsgdGhhdCBwb2ludHMgdG8gZWl0aGVyIEdpdEh1YiBvciBUd2l0dGVyLgogICAgICAgICAgICBsaW5rCiAgICAgICAgICAgICAgQGZvbGQKICAgICAgICAgICAgICBAdHJhbnNmb3JtKG9wOiAiY291bnQiKQogICAgICAgICAgICAgIEBmaWx0ZXIob3A6ICI%2BPSIsIHZhbHVlOiBbIiRtaW5Qcm9maWxlcyJdKQogICAgICAgICAgICB7CiAgICAgICAgICAgICAgY29tbWVudGVySURzOiB1cmwgQGZpbHRlcihvcDogInJlZ2V4IiwgdmFsdWU6IFsiJHNvY2lhbFBhdHRlcm4iXSkKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBAb3V0cHV0CiAgICAgICAgICAgIH0KICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIH0KICAgIH0KICB9Cn0%3D&v=ewogICJzaXRlUGF0dGVybiI6ICJodHRwW3NdOi8vKFteLl0qXFwuKSpvcGVuYWkuY29tLy4qIiwKICAibWluUHJvZmlsZXMiOiAxLAogICJzb2NpYWxQYXR0ZXJuIjogIihnaXRodWJ8dHdpdHRlcilcXC5jb20vIgp9 https://play.predr.ag/hackernews#?f=1&q=IyBDcm9zcyBBUEkgcXVl... Trustfall is designed for interoperation from day 1. It separates the queries from the data providers, allowing the infrastructure to evolve and change how it serves queries without any of the queries noticing anything except faster execution. In practice, that means you don't have to rewrite your product to make it run faster -- which makes both the product side and the infra side happier :) Here's a real-world example of that. The `cargo-semver-checks` Rust semantic versioning linter implements its lints as Trustfall queries over JSON files describing the package API, and I recently was able to speed up its execution by over 2000x without changing a single query -- just by changing how those queries execute under the hood. More details in my blog post here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... AMA, I guess :)

loveparade 4y ago

Isn't this the same as converting websites into a GraphQL API? Aren't there already dozens of projects and services that do the "convert websites into an API", at scale? What exactly is the innovation?

obi1kenobi 4y ago
No, it isn't: it's neither about websites nor GraphQL. The query language is substantially more powerful than GraphQL, much closer to SQL. And the idea is unifying and querying any kind of data, not just websites but also databases, raw files, ML models, anything at all. For example, the cargo-semver-checks semver linter for Rust uses Trustfall to abstract over the different JSON formats in which different Rust versions describe a library's API. The lints are written as Trustfall queries (GraphQL isn't expressive enough for this!) and don't care about how the underlying data is represented. Recently, I was able to tweak how those JSON files are used to serve data, and sped up cargo-semver-checks by over 2000x -- without changing a single query. More info here: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... If you want to see more of it in action, there's a 10min talk featuring demos and real-world use cases here: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin...
leobg 4y ago
I googled “convert website into a GraphQL API” and literally found no matches. Did you have any specific projects/services in mind that attempt to do that kind of thing?

haolez 4y ago

This looks a lot like what Apache Drill already does.

sdfhbdf 4y ago

Very interesting, sounds kinda like GraphQL counterpart to https://github.com/cube2222/octosql https://github.com/cube2222/octosql

obi1kenobi 4y ago
The syntax is reminiscent to GraphQL, but semantically it's a lot more expressive and not compatible. It's much closer to SQL in expressive power, actually: left joins, recursion, arbitrary filters, aggregation, etc. -- none of that stuff works in GraphQL but is supported in Trustfall.

dilawar 4y ago

obligatory, "written in Rust".

obi1kenobi 4y ago
It is! But it has Python and JS bindings, and the web playground uses Trustfall compiled to WASM: https://play.predr.ag/ https://play.predr.ag/

vsroy 4y ago

I imagine with these types of things the vast majority of the work is writing integrations. Could you explain how this makes writing integrations easier?

obi1kenobi 4y ago
Right now, to be perfectly honest, it doesn't yet. This is by design: the purely integrations-oriented side is a crowded space, and I'm not ready to jump in there yet. Stay tuned :) The focus instead has been in a place where it's easier to stand out: deeper query ability over the data providers than any competing solution, with stronger optimization guarantees. The goal is that any query optimization you could implement by hand, you should be able to implement within Trustfall -- while having the benefit of purely declarative type-checked queries, with integrated docs and autocomplete. If a query is too slow, you don't have to rewrite it -- you can just use Trustfall's APIs to tweak how it executes under the hood and speed it up by using caching, batching, prefetch, or an index. For a real-world demo of that, I wrote a blog post about speeding up cargo-semver-checks (a Rust linter for semantic versioning) by 2272x without changing any query, just by making some indexes available to the execution of the existing queries. This is awesome because it empowers anyone to write queries, knowing that either their query will either already be fast or it can be made fast in the future in a way that is entirely invisible to the query itself. Link: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov...
- cormacrelf 4y ago
  I read that yesterday and felt a bit tricked. You said there was a one line diff, which to me suggested you had made a query optimiser and added that to trustfall so that consumer applications could transparently benefit without writing any new code. But really, as it turned out, you just added APIs for for indexing a table and using those indexes, and then used those APIs to do select manual optimisations in a “fast path” in the trustfall-rustdoc-adapter crates. What does it matter that some crates are called trustfall adapters and some are not? You still had to optimise the execution of the query manually. I can see how it’s cool you didn’t need to change the text of the query, but people like SQL because the execution engines are smart enough to optimise for you. They will build a little hash table index on the fly if it saves a lot of full table scanning. The expectations re smart optimisation in the market you’re competing in are very high. If you say it was a one line upgrade to the trustfall executor then people will believe you. The net result is better than what most GraphQL infrastructure gives you. GraphQL doesn’t give you any tools to avoid full table scans, it just tells you “here’s the query someone wants, I will do the recursive queries with as many useless round trips as possible unless you implement all these things from scratch”. At least your API has the concept of an index now. But I think you’re trying to sell it as being as optimisable as SQL while trying to avoid telling users the bad news that it’s going to be them who has to write the optimiser, not you.

unshavedyak 4y ago

Could this be used to manage ingesting data from messier sources? Ala files (pdf/etc), web pages, etc? edit: Admittedly the website/video tends to talk about data sources a bit hand-wavy. I'd have loved some real world examples on how one goes about adding a data source. Also how we handle problems of scale.. ie passing filters to the data source, rather than drinking from a firehose and filtering after the fact. With that said, the idea is becoming interesting to me. At the very least i am liking the idea of a standardized query interface to "things". Just feels like edge cases might drown me.

obi1kenobi 4y ago
Yes, in at least two ways: - It could be used as the "ingestion process" itself: write the ingestion as a Trustfall query (or queries) over the data sources, and store the results in a traditional database or other system. - It could be used to make the "ingestion" a mere implementation detail: you could write a query over all the data sources, and that query could be executed against the raw data sources themselves, or some "ingested" format which would presumably be faster and more efficient. The query can't tell the difference, and this frees up the ingestion system to evolve independently of how it's being used. It sounds like you might have a specific use case in mind? I'd love to learn more about it if so! My Twitter DMs are open, or I could give you my email address if you'd prefer that.
obi1kenobi 4y ago
To be completely honest, I was planning on doing a "Show HN" in a month or two, but someone posted the link today and it caught me a bit off-guard -- that's why the docs and examples aren't there yet :) There's definitely a bit of a learning curve because the docs are few and far between, but in general, everything is much simpler than it seems. The people that have adopted it already have been able to do so within a couple of hours with minimal guidance from me. For passing filters to the data source, the API there is still very experimental (it was just an idea in my head until last week!) but I've been working on it recently. Here's a blog post showing a small bit of that new API and using it to dramatically optimize a workload: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... This is my full-time project right now, so stay tuned and expect a decently steep progress trajectory :)

wodenokoto 4y ago

Clicking through to the python repo and looking at the examples there, I guess the query language is GraphQL. Reading further, it seems that it is actually a variation of GraphQL. I've never used GraphQL before, maybe one can get used to it, but it doesn't look very nice for defining data queries.

obi1kenobi 4y ago
The language is syntactically similar to GraphQL, but the semantics are much richer -- in terms of expressiveness Trustfall is more like SQL than GraphQL. No language syntax makes everyone happy :) But it's definitely is something that isn't hard to get used to -- in my experience, people go from "never seen it before" to "can productively write their own queries" very quickly after trying a few examples. Try out the example queries over HackerNews in the Playground (ideally not on mobile) and see for yourself: https://play.predr.ag/hackernews https://play.predr.ag/hackernews And in the long run, Trustfall is more meant to be like an "LLVM for data sets," with pluggable language frontends. There's already been interest in plugging in other languages and making them generate Trustfall representation internally, to make use of the rest of its functionality without using the language itself. This is fine too!

nathanwallace 4y ago

Readers who like SQL may also enjoy Steampipe [1], an open source tool to live query 99+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc across the services. (Disclaimer - I'm a lead on the project.) 1 - https://github.com/turbot/steampipe https://github.com/turbot/steampipe

mafro 4y ago
+1 Steampipe's aggregators to combine multiple datasets into one are great
sztanko 4y ago
Very cool! Is it possible to connect to steampipe via jdbc connector?
- code_biologist 4y ago
  I presume so: "You can also use psql, pgcli, Metabase, Tableau, or any client that can connect to Postgres."
  - nathanwallace 4y ago
    Correct - it's just Postgres, so all your favorite clients / tools work as usual. We have docs for a number of example integrations [1] that work with Steampipe CLI (or cloud). 1 - https://steampipe.io/docs/cloud/integrations/overview https://steampipe.io/docs/cloud/integrations/overview
xrendan 4y ago
Is it possible to join those data sources against an existing database like MySQL or SQLServer?
- nathanwallace 4y ago
  Steampipe runs an embedded Postgres database [1]. One approach would be to connect that Postgres database with other databases using the usual (advanced) techniques like Foreign Data Wrappers [2]? With that connection in place, joins would work as normal. 1 - https://steampipe.io/docs/develop/overview https://steampipe.io/docs/develop/overview 2 - https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers

darkteflon 4y ago

This looks super cool. Going to put this together with Dagster tonight. @obi1kenobi, could you comment a bit on the motivation and background to this project?

obi1kenobi 4y ago
Might I direct you to a 10min talk I gave on the subject? https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin... The motivation is that queries that are trivial in a SQL database are painfully impractical over any other data source. Most query languages are tied way too closely to a data format or representation, and fail miserably at the boundaries to other systems or representations. Yes, we can e.g. plug SQL into APIs but there's a severe impedance mismatch there -- we end up forced into all sorts of unintuitive situations and poor UX because SQL was never meant for APIs. Based on experiences I had at work (some of them covered in that talk!), it felt like an "LLVM for data sources" could be useful -- that's what Trustfall is. Yes, it includes a query language. That's because SQL and GraphQL and the like just weren't a good fit to demonstrate true datasource-agnostic queries: we want a modern GraphQL-like user experience, with SQL-like expressiveness but without limitations that make it hard to plug in APIs and ML tool. But the query language is just one layer of the system, and it's entirely possible to replace it with another language parser while keeping the rest of the benefits. A key benefit of Trustfall is that you can write queries over any data sources, and then change the underlying implementation to add optimizations or to migrate to a new system without the queries needing to change. For example, cargo-semver-checks, a Rust semver linter I built on top of Trustfall, supports 4 separate format versions of the underlying data source it uses with the same set of queries, and I was recently able to speed up its execution by over 2000x with a tiny amount of code and effort and without even a single query needing to change -- just by tweaking how the data adapter executed queries to plug in some indexes where they mattered: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... Excited to see what you build, please post a link or email / DM me on Twitter if you'd rather not post publicly? The docs are still sparse (I wasn't planning to do a Show HN for another couple of months, but someone beat me to it today) but I'd be happy to help with anything. There are Python and Rust bindings, and feel free to open issues for anything that's confusing: https://pypi.org/project/trustfall/ https://pypi.org/project/trustfall/ https://crates.io/crates/trustfall_core https://crates.io/crates/trustfall_core

yevpats 4y ago

Also relevant - High Performance Open Source ELT Framework - https://github.com/cloudquery/cloudquery https://github.com/cloudquery/cloudquery

srcreigh 4y ago

What join algorithms does this support? How many data sources are integrated so far (what’s the MOM growth?) What’s the largest dataset this integrates with? And how many different people do you estimate have written queries in the language?

obi1kenobi 4y ago
As I wrote in reply to your other comment, the join algorithms are the adapter's choice. I wouldn't necessarily recommend using it as a SQL replacement, though. There's no measurable MoM growth yet. This is the first major look the broader community has had at the system outside of a relatively small conference where I gave a talk, and demos I've given to friends and colleagues. I wasn't planning on submitting to HN for another few months -- but someone else beat me to it today :) There are production-grade adapters for HackerNews APIs and for Rust's rustdoc JSON format, used by the Playground (https://play.predr.ag/hackernews https://play.predr.ag/hackernews) and the Rust semver-checking linter cargo-semver-checks, respectively. There are also a few more demo adapters in the Trustfall repo itself: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall I expect the number of adapters to grow significantly in the coming months, so please stay tuned and let me know if you have datasets you'd like to try it with! The "largest dataset" is a bit of a trick question: how big is the dataset of "all HackerNews data available via its Firebase and Algolia APIs"? Because that's what the Playground queries. The Rust semver linter has been used by dozens if not hundreds of crates, and the JSON payloads in question there are in the 100-400MB range. The example in this blog post runs 40 quite complex Trustfall queries (they express semver rules!) over 400MB across two JSON files in 8 seconds: https://predr.ag/blog/speeding-up-rust-semver-checking-by-over-2000x/ https://predr.ag/blog/speeding-up-rust-semver-checking-by-ov... You can also see more real-world use cases in this talk I gave last year: https://www.hytradboi.com/2022/how-to-query-almost-everything https://www.hytradboi.com/2022/how-to-query-almost-everythin... Probably around 200-300 people or so have written queries in the language, with experience levels ranging from seasoned engineers to Excel analysts and SQL analysts with no programming experience outside of that domain. The language itself is actually a refinement from an earlier open-source project I developed and open-sourced through my previous job. That open-source project was used to query everything from a multi-TB SQL cluster to APIs to ML models, and with Trustfall I've taken the opportunity to revisit and update design decisions that the previous project got to regret in retrospect.

alexisread 4y ago

Nice! I wrote something similar for my workplace a couple of years ago based around Rx (as Rx has many implementations in different languages - we had a multi-node browser/server requirement, and optimization in a streaming DSL is easier than SQL, as you've done, as you can hint+order lazy materializations) and libs like https://pypi.org/project/lquery/ https://pypi.org/project/lquery/ to do pushdown queries. Are you planning on doing reactive/live/materialized queries? You could use a config file to specify the live queries (in the Trustfall DSL) which can be fed to the engine on startup.

obi1kenobi 4y ago
Thanks! Reactive/live/materialized queries are certainly a possibility I've tried to keep open and I'd love to build, but there's so many possible things to build from here that I'm not sure when that might happen. In particular, while that tech is really cool, it's not clear to me how many use cases for it are actually ready to adopt it. I'm hesitant to build something that looks really cool, but is somewhat complex and doesn't get used a lot.

crosen99 4y ago

This is a nifty tool. It’s existence alongside the emerging LLMs reminds me of the two diametrically opposed approaches to harnessing it all: 1. Store the knowledge in a highly structured way and interrogate it with a precise and rigorous query language to extract the exact answer you want based on a well defined set of rules 2. Store the knowledge in whatever ad hoc way it’s produced, and then rely on a higher form of intelligence to take an equally ad hoc query, feed it through the entire universe of knowledge with some attention mechanism, and magically return a (statistically significant) response Both approaches are so satisfying when they work. Of course you also have everything in between and then you have tools like LangChain that start to bring it all together.

TickleSteve 4y ago
i.e. Approximate vs precise. But, there is another option. The tool (presumably AI/ML based) that takes the ad-hoc query and turns it into a precise query to the structured-data tool in order to return precise results. I hope that this will be Googles answer to ChatGPT... i.e. a chat interface to their existing search tool. This should then be able to "explain" the query its actually generated to the tool as a side-effect.
- crosen99 4y ago
  For sure, I just edited my comment to mention LangChain, which starts to get at that capability. Would be nice to see LangChain integrate with Trustfall similar to how it can integrate with Wolfram Alpha.