3 ms·

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreig

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.) 1 - https://github.com/turbot/steampipe https://github.com/turbot/steampipe

wvh 2y ago
We've been using Steampipe every once in a while and it works as advertised, though perhaps SQL isn't always the abstraction of choice mentally. Congrats on hitting v1!
- nathanwallace 2y ago
  thank you! We believe that SQL makes the simple stuff easy and the hard stuff doable :). Hopefully our repos & hubs have good enough examples to get you started most of the time!
oulipo 2y ago
Interesting! Can you tell me the typical use-case for this? Is it more for dev-ops stuff? Like to query all your servers to do some dashboards of your infra? Or can that be used also for ML / datascience etc?
- nathanwallace 2y ago
  Steampipe is most commonly used for DevSecOps. It works with any BI tool for dashboards (it's just Postgres), but is really great when paired with Powerpipe [1] for security benchmarks, visualizing infrastructure and much more [2]. It can definitely be used for a wide range of data scenarios, e.g. there is an OpenAI plugin [3]. 1 - https://github.com/turbot/powerpipe https://github.com/turbot/powerpipe 2 - https://hub.powerpipe.io https://hub.powerpipe.io 2 - https://github.com/turbot/steampipe-plugin-openai https://github.com/turbot/steampipe-plugin-openai
nlm 2y ago
That's a great piece of software indeed. I use it to join and analyze data from multiple APIs with materialized view, as most data analysis software only connects to databases, not APIs. It's also pretty easy to write custom plugins once you understand how it's done.
- nathanwallace 2y ago
  Thanks for leaning into Steampipe!
djbusby 2y ago
I think you mean disclosure. Why would you disclaim something cool?
- bbor 2y ago
  Ooo love this rabbit hole. I’ve always used “disclaimer” as “warning”, and would definitely use it for a conflict of interest, but technically the legal field uses it specifically for asserting a lack of liability (ish). Certainly interesting, but I think I’m gonna stubbornly stand my descriptivist ground here. We conquered “literally”, and “sneak peak” is soon to fall — I’ll add this to the list!
- nathanwallace 2y ago
  Fair point. I guess that I'm disclosing that I lead the project and disclaiming the self-interest in posting? :)
nkmnz 2y ago
the plugins link from your github repo goes to 404: https://hub.powerpipe.io/plugins https://hub.powerpipe.io/plugins :)
- nathanwallace 2y ago
  doh! thanks for the heads up :-(. Fixed to be https://hub.steampipe.io/plugins https://hub.steampipe.io/plugins
mind-blight 2y ago
This is super cool! Do you have any integrations with duckdb? It seems like these two could work really well together
- nathanwallace 2y ago
  Beyond the Steampipe CLI (which has an embedded Postgres), you can use the steampipe plugins as native Postgres FDWs [1], SQLite extensions [2] or command line export tools [3]. We're definitely interested in bringing the plugins to duckdb as well, but haven't had a chance to focus there yet! 1 - https://steampipe.io/blog/2023-12-postgres-extensions https://steampipe.io/blog/2023-12-postgres-extensions 2 - https://steampipe.io/blog/2023-12-sqlite-extensions https://steampipe.io/blog/2023-12-sqlite-extensions 3 - https://steampipe.io/blog/2023-12-steampipe-export https://steampipe.io/blog/2023-12-steampipe-export
pestaa 2y ago
This is super cool. Last time I searched only https://fdw.dev https://fdw.dev came up (from Supabase).
adhamsalama 2y ago
Looks pretty cool. Will try it out.

wvh 2y ago

We've been using Steampipe every once in a while and it works as advertised, though perhaps SQL isn't always the abstraction of choice mentally. Congrats on hitting v1!

nathanwallace 2y ago
thank you! We believe that SQL makes the simple stuff easy and the hard stuff doable :). Hopefully our repos & hubs have good enough examples to get you started most of the time!

oulipo 2y ago

Interesting! Can you tell me the typical use-case for this? Is it more for dev-ops stuff? Like to query all your servers to do some dashboards of your infra? Or can that be used also for ML / datascience etc?

nathanwallace 2y ago
Steampipe is most commonly used for DevSecOps. It works with any BI tool for dashboards (it's just Postgres), but is really great when paired with Powerpipe [1] for security benchmarks, visualizing infrastructure and much more [2]. It can definitely be used for a wide range of data scenarios, e.g. there is an OpenAI plugin [3]. 1 - https://github.com/turbot/powerpipe https://github.com/turbot/powerpipe 2 - https://hub.powerpipe.io https://hub.powerpipe.io 2 - https://github.com/turbot/steampipe-plugin-openai https://github.com/turbot/steampipe-plugin-openai

nlm 2y ago

That's a great piece of software indeed. I use it to join and analyze data from multiple APIs with materialized view, as most data analysis software only connects to databases, not APIs. It's also pretty easy to write custom plugins once you understand how it's done.

nathanwallace 2y ago
Thanks for leaning into Steampipe!

djbusby 2y ago

I think you mean disclosure. Why would you disclaim something cool?

bbor 2y ago
Ooo love this rabbit hole. I’ve always used “disclaimer” as “warning”, and would definitely use it for a conflict of interest, but technically the legal field uses it specifically for asserting a lack of liability (ish). Certainly interesting, but I think I’m gonna stubbornly stand my descriptivist ground here. We conquered “literally”, and “sneak peak” is soon to fall — I’ll add this to the list!
nathanwallace 2y ago
Fair point. I guess that I'm disclosing that I lead the project and disclaiming the self-interest in posting? :)

nkmnz 2y ago

the plugins link from your github repo goes to 404: https://hub.powerpipe.io/plugins https://hub.powerpipe.io/plugins :)

nathanwallace 2y ago
doh! thanks for the heads up :-(. Fixed to be https://hub.steampipe.io/plugins https://hub.steampipe.io/plugins

mind-blight 2y ago

This is super cool! Do you have any integrations with duckdb? It seems like these two could work really well together

nathanwallace 2y ago
Beyond the Steampipe CLI (which has an embedded Postgres), you can use the steampipe plugins as native Postgres FDWs [1], SQLite extensions [2] or command line export tools [3]. We're definitely interested in bringing the plugins to duckdb as well, but haven't had a chance to focus there yet! 1 - https://steampipe.io/blog/2023-12-postgres-extensions https://steampipe.io/blog/2023-12-postgres-extensions 2 - https://steampipe.io/blog/2023-12-sqlite-extensions https://steampipe.io/blog/2023-12-sqlite-extensions 3 - https://steampipe.io/blog/2023-12-steampipe-export https://steampipe.io/blog/2023-12-steampipe-export

pestaa 2y ago

This is super cool. Last time I searched only https://fdw.dev https://fdw.dev came up (from Supabase).

adhamsalama 2y ago

Looks pretty cool. Will try it out.