3 ms·

How We Search for EC2 Instances in a Multi-account AWS Infrastructure

2 points by scapecast 5y ago

scapecast 5y ago
Co-founder of author here. The post describes how we developed a search syntax to solve the problem of finding resources running in a multi-account infrastructure. While the post is about AWS, the search works also for GCP. The (not so) secret sauce behind the search syntax is an open source project that we developed out of necessity of taming a multi-cloud infrastructure (AWS and GCP). We quickly realized that it's not just about understand WHAT resources are running, but also HOW these resources dependent on each other. Behind the scenes, Resoto collects your cloud resources and puts their metadata and dependencies into a large indexed graph that can be searched. The search syntax is how. In a way, parts of our product are an open source alternative to e.g. AWS Config and GCP Config Connector. Would love to hear any reactions from the community how they're currently solving the problem of keeping inventory of what's running in their cloud accounts.
wizwit999 5y ago
Technically cool but what's a real world problem that this solves? I've worked on pretty large AWS projects and if your engineers can't even reason about what resources exist, I feel like theres perhaps an organizational smell there.
- scapecast 5y ago
  Things break all the time - CI/CD pipelines, Terraform deployments, etc. - and leave behind orphaned resources. We worked on an infrastructure with 400K+ active resources. It’s just not possible for a human to track that without any help from tooling. If you don’t do something about those resources - at some point they will become a problem. We had cases where we ran into quota limits, and it would stop the entire operation. Next to search, we also offer ways for automation. Say you wanted to find all unused load balancers and mark them for clean up. You can persist that search as a Job and let it run on a schedule.

3 ms·

How We Search for EC2 Instances in a Multi-account AWS Infrastructure

2 points by scapecast 5y ago

scapecast 5y ago
Co-founder of author here. The post describes how we developed a search syntax to solve the problem of finding resources running in a multi-account infrastructure. While the post is about AWS, the search works also for GCP. The (not so) secret sauce behind the search syntax is an open source project that we developed out of necessity of taming a multi-cloud infrastructure (AWS and GCP). We quickly realized that it's not just about understand WHAT resources are running, but also HOW these resources dependent on each other. Behind the scenes, Resoto collects your cloud resources and puts their metadata and dependencies into a large indexed graph that can be searched. The search syntax is how. In a way, parts of our product are an open source alternative to e.g. AWS Config and GCP Config Connector. Would love to hear any reactions from the community how they're currently solving the problem of keeping inventory of what's running in their cloud accounts.
wizwit999 5y ago
Technically cool but what's a real world problem that this solves? I've worked on pretty large AWS projects and if your engineers can't even reason about what resources exist, I feel like theres perhaps an organizational smell there.
- scapecast 5y ago
  Things break all the time - CI/CD pipelines, Terraform deployments, etc. - and leave behind orphaned resources. We worked on an infrastructure with 400K+ active resources. It’s just not possible for a human to track that without any help from tooling. If you don’t do something about those resources - at some point they will become a problem. We had cases where we ran into quota limits, and it would stop the entire operation. Next to search, we also offer ways for automation. Say you wanted to find all unused load balancers and mark them for clean up. You can persist that search as a Job and let it run on a schedule.