5 ms·

Infrastructure is made up of many parts. A single S3 bucket may have multiple resources (a key, a policy, a notification queue). How is having several “insert i

by orf 5y ago

Infrastructure is made up of many parts. A single S3 bucket may have multiple resources (a key, a policy, a notification queue). How is having several “insert into” statements for each of these any different from “resource” blocks in terraform? If anything it would be much worse, because you either write some ungodly huge sql statement to create multiple resources or you loose the ability to know what resources depend on each other and form a graph of dependencies. This results in much slower plans, as you don’t have a dag and you need to potentially refresh state much more often, or something that looks like terraform but with way way more boilerplate and irrelevant syntax.

rubiquity 5y ago
We need to be creative and make types relevant to the resources being modeled. If the infrastructure database is just a bunch of string identifier fields then it isn't very helpful. You have to find good abstractions and model the resources in types that bring meaning to the data.
- orf 5y ago
  Sure, but that’s got nothing to do with SQL and could be modelled in terraform. Or better yet, indirectly using terraform via IAC providers in languages like Typescript. Show me a proper example of creating an actual s3 bucket that you’d use in production. KMS key, inventory configuration, resource policy, lifecycle policy, logging enabled. Created via SQL. Now show me how you’d take this and make it a reusable module so we can have a standard template for buckets that everyone can use.
  - rubiquity 5y ago
    You’re focusing too much on the initial creation rather than on going maintenance and evolution. SQL and relations are much better suited to handle evolution by enforcing constraints than a graph of stitched together pseudo JSON.
    - orf 5y ago
      No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. You describe your state, something reconciles that. How you describe your state and the dependencies between resources is absolutely key, and on the face of it it looks like this interface is totally inadequate. So, again, show me even a brief sketch of how you would describe what I said above with this model, and you’ll see it quickly falls apart.
      - rubiquity 5y ago
        You're coming off a bit worked up, but I'll humor you anyway. I'm not going to type out a bunch of SQL as an example. SQL vs HCL isn't the point and they basically break even on expressiveness. After you've typed out your pseudo-JSON, what exactly are the existing tools saving you? From having to use some wrapper around the cloud API? That's the easy part. By overly focusing on SQL you're missing the forest for the trees. The point is relations and RDBMS features such as constraints, triggers, stored procedures. Such a platform would be always online rather than just a tfstate file waiting for humans to munge it. It's also time to stop thinking about the cloud as literal resources like current tools do and start moving to more abstract concepts (more on that later). > No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. I run very critical infrastructure for a living and there absolutely is a difference. Creating new resources is easy -- they aren't being used and the world doesn't have any expectations for their performance or reliability. The bacon is made in evolving existing infrastructure without impacting user experience negatively. Terraform and other such generators give you very little in guard rails or help and silly outages happen all of the time because of it. Database engineers have been creating sophisticated execution engines for decades. The creator of SQLite cites treating every SQL query as its own program to be ran by a byte code VM as a key design decision. Writing Terraform or what have you is like programming in an AST (sorry Lispers). To date query execution engines figure out how to manage on-disk structures. There is no reason they couldn't be creating smart plans for infrastructure changes at a much higher abstraction level than "glue this KMS key to this bucket."
        5y ago

by orf 5y ago

rubiquity 5y ago

We need to be creative and make types relevant to the resources being modeled. If the infrastructure database is just a bunch of string identifier fields then it isn't very helpful. You have to find good abstractions and model the resources in types that bring meaning to the data.

orf 5y ago
Sure, but that’s got nothing to do with SQL and could be modelled in terraform. Or better yet, indirectly using terraform via IAC providers in languages like Typescript. Show me a proper example of creating an actual s3 bucket that you’d use in production. KMS key, inventory configuration, resource policy, lifecycle policy, logging enabled. Created via SQL. Now show me how you’d take this and make it a reusable module so we can have a standard template for buckets that everyone can use.
- rubiquity 5y ago
  You’re focusing too much on the initial creation rather than on going maintenance and evolution. SQL and relations are much better suited to handle evolution by enforcing constraints than a graph of stitched together pseudo JSON.
  - orf 5y ago
    No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. You describe your state, something reconciles that. How you describe your state and the dependencies between resources is absolutely key, and on the face of it it looks like this interface is totally inadequate. So, again, show me even a brief sketch of how you would describe what I said above with this model, and you’ll see it quickly falls apart.
    - rubiquity 5y ago
      You're coming off a bit worked up, but I'll humor you anyway. I'm not going to type out a bunch of SQL as an example. SQL vs HCL isn't the point and they basically break even on expressiveness. After you've typed out your pseudo-JSON, what exactly are the existing tools saving you? From having to use some wrapper around the cloud API? That's the easy part. By overly focusing on SQL you're missing the forest for the trees. The point is relations and RDBMS features such as constraints, triggers, stored procedures. Such a platform would be always online rather than just a tfstate file waiting for humans to munge it. It's also time to stop thinking about the cloud as literal resources like current tools do and start moving to more abstract concepts (more on that later). > No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. I run very critical infrastructure for a living and there absolutely is a difference. Creating new resources is easy -- they aren't being used and the world doesn't have any expectations for their performance or reliability. The bacon is made in evolving existing infrastructure without impacting user experience negatively. Terraform and other such generators give you very little in guard rails or help and silly outages happen all of the time because of it. Database engineers have been creating sophisticated execution engines for decades. The creator of SQLite cites treating every SQL query as its own program to be ran by a byte code VM as a key design decision. Writing Terraform or what have you is like programming in an AST (sorry Lispers). To date query execution engines figure out how to manage on-disk structures. There is no reason they couldn't be creating smart plans for infrastructure changes at a much higher abstraction level than "glue this KMS key to this bucket."
      - 5y ago