4 ms·

Cloudquery: Easily ask questions about your cloud infrastructure

75 points by yevpats 6y ago

leg100 6y ago
Looks like there are two steps to this tool: 1) Fetch data and put into a sql db 2) Query sql db Wouldn't it be better, performance aside, if there was just one step, i.e. querying the cloud with SQL, essentially an SQL-to-cloud-APIs interface. Otherwise you're always operating on out-of-date data.
- yevpats 6y ago
  It's a cool idea for a feature! The idea/advantage to have the data in regular SQL database - is to be able to work with all the standard tools and analytics dashboard you have and familiar with (AWS quicksight, Google DataStudio, sqlite shell, etc...)
- jacques_chester 6y ago
  The flipside is that performing joins in a process across many document databases (which is what this approach would resemble) is wildly non-performant compared to doing it in a relational database.
- Aperocky 6y ago
  But now every time you run a query without a where clause you're firing API calls to all AWS services. And then maybe recursive API calls on top of that. This is fine if you're a personal account, but then you wouldn't need this tool in the first place.
brodouevencode 6y ago
This is very interesting. If you're managing hundreds of accounts pulling data even from Config from the master payer level is tedious and time consuming. If that data can be pulled out and queried with a real query language (config ql just doesn't cut it) it would make mass management so much easier.
luizfelberti 6y ago
Damn this looks really good. Data-first SQL integrated interfaces always get a +1 from me, managing resources without unified abstractions is always painful, especially when you have to do this at scale, standard CLIs always end up falling short in some way Also, relevant tool for similar applications (albeit not SQL based): https://github.com/cloud-custodian/cloud-custodian https://github.com/cloud-custodian/cloud-custodian
- yevpats 6y ago
  Thanks! This is definitely one of the use-cases to have an SQL abstraction of many resources. Also the idea is to be able to define rules in cross-account, cross-apps - For example: making a rule/notification that depends on information in Okta and in GCP/AWS (This is pretty simple with SQL join). Regarding Custodian - This is definitely a great tool with it's own advantages. Some of the dis-advantages that lead me to start cloudquery is that it has a proprietary query language that you have to learn and as such is more limited than SQL (The example above won't be possible out-of-the-box).
ianpurton 6y ago
Really cool. If I could create resources with INSERT statements, even better.
- yevpats 6y ago
  Thanks! This is on the roadmap, I'm not sure it will be via Insert statements but there will be an option of action/change via some level of abstraction that is not via API/writing code.
- outworlder 6y ago
  Ouch. That's a tall order. There's a lot of stuff that needs to happen for that, and error-handling would be quite tricky. This would require either having a logic to "converge" the infrastructure – meaning implementing a big chunk of Terraform and adding some features that would make this thing look like a Kubernetes cousin – or having the insert statements blocking for minutes at a time.
  - yevpats 6y ago
    Yes, a complete support for insert will definitely look like k8s/terraform cousin. As mentioned in the comment below we probably will support some specific pre-defined actions. Anyway, currently we only support query and as we gather more feedback we will add the action functionality which has it's set of nuances some of which you described very well.
- nodesocket 6y ago
  I get the request, but it seems like the outcome is better achieved with Terraform or Pulumi.
  - vvram 6y ago
    Yup, you can create mock resources with TF/Pulumi
StratusBen 6y ago
I work on a product named http://vantage.sh/ http://vantage.sh/ that provides something similar as a managed service specifically for AWS. I respect the effort that went into this!
- iampims 6y ago
  Unsolicited feedback: your homepage could use a few more screenshots for us to see what vantage really is.
  - StratusBen 6y ago
    Very fair. We're about to ship a new marketing site with a features page that has more screenshots and feature explanations as well as a video demo on the home page.
    - sudofox 6y ago
      Excited to see the launch
mdaniel 6y ago
Obviously a quite popular project given the number of times it's been submitted this month: https://news.ycombinator.com/from?site=github.com/cloudquery https://news.ycombinator.com/from?site=github.com/cloudquery

75 points by yevpats 6y ago

Looks like there are two steps to this tool: 1) Fetch data and put into a sql db 2) Query sql db Wouldn't it be better, performance aside, if there was just one step, i.e. querying the cloud with SQL, essentially an SQL-to-cloud-APIs interface. Otherwise you're always operating on out-of-date data.

yevpats 6y ago
It's a cool idea for a feature! The idea/advantage to have the data in regular SQL database - is to be able to work with all the standard tools and analytics dashboard you have and familiar with (AWS quicksight, Google DataStudio, sqlite shell, etc...)
jacques_chester 6y ago
The flipside is that performing joins in a process across many document databases (which is what this approach would resemble) is wildly non-performant compared to doing it in a relational database.
Aperocky 6y ago
But now every time you run a query without a where clause you're firing API calls to all AWS services. And then maybe recursive API calls on top of that. This is fine if you're a personal account, but then you wouldn't need this tool in the first place.

brodouevencode 6y ago

This is very interesting. If you're managing hundreds of accounts pulling data even from Config from the master payer level is tedious and time consuming. If that data can be pulled out and queried with a real query language (config ql just doesn't cut it) it would make mass management so much easier.

luizfelberti 6y ago

Damn this looks really good. Data-first SQL integrated interfaces always get a +1 from me, managing resources without unified abstractions is always painful, especially when you have to do this at scale, standard CLIs always end up falling short in some way Also, relevant tool for similar applications (albeit not SQL based): https://github.com/cloud-custodian/cloud-custodian https://github.com/cloud-custodian/cloud-custodian

yevpats 6y ago
Thanks! This is definitely one of the use-cases to have an SQL abstraction of many resources. Also the idea is to be able to define rules in cross-account, cross-apps - For example: making a rule/notification that depends on information in Okta and in GCP/AWS (This is pretty simple with SQL join). Regarding Custodian - This is definitely a great tool with it's own advantages. Some of the dis-advantages that lead me to start cloudquery is that it has a proprietary query language that you have to learn and as such is more limited than SQL (The example above won't be possible out-of-the-box).

ianpurton 6y ago

Really cool. If I could create resources with INSERT statements, even better.

yevpats 6y ago
Thanks! This is on the roadmap, I'm not sure it will be via Insert statements but there will be an option of action/change via some level of abstraction that is not via API/writing code.
outworlder 6y ago
Ouch. That's a tall order. There's a lot of stuff that needs to happen for that, and error-handling would be quite tricky. This would require either having a logic to "converge" the infrastructure – meaning implementing a big chunk of Terraform and adding some features that would make this thing look like a Kubernetes cousin – or having the insert statements blocking for minutes at a time.
- yevpats 6y ago
  Yes, a complete support for insert will definitely look like k8s/terraform cousin. As mentioned in the comment below we probably will support some specific pre-defined actions. Anyway, currently we only support query and as we gather more feedback we will add the action functionality which has it's set of nuances some of which you described very well.
nodesocket 6y ago
I get the request, but it seems like the outcome is better achieved with Terraform or Pulumi.
- vvram 6y ago
  Yup, you can create mock resources with TF/Pulumi

StratusBen 6y ago

I work on a product named http://vantage.sh/ http://vantage.sh/ that provides something similar as a managed service specifically for AWS. I respect the effort that went into this!

iampims 6y ago
Unsolicited feedback: your homepage could use a few more screenshots for us to see what vantage really is.
- StratusBen 6y ago
  Very fair. We're about to ship a new marketing site with a features page that has more screenshots and feature explanations as well as a video demo on the home page.
  - sudofox 6y ago
    Excited to see the launch

mdaniel 6y ago

Obviously a quite popular project given the number of times it's been submitted this month: https://news.ycombinator.com/from?site=github.com/cloudquery https://news.ycombinator.com/from?site=github.com/cloudquery