3 ms·

We need to be creative and make types relevant to the resources being modeled. If the infrastructure database is just a bunch of string identifier fields then i

We need to be creative and make types relevant to the resources being modeled. If the infrastructure database is just a bunch of string identifier fields then it isn't very helpful. You have to find good abstractions and model the resources in types that bring meaning to the data.

orf 5y ago
Sure, but that’s got nothing to do with SQL and could be modelled in terraform. Or better yet, indirectly using terraform via IAC providers in languages like Typescript. Show me a proper example of creating an actual s3 bucket that you’d use in production. KMS key, inventory configuration, resource policy, lifecycle policy, logging enabled. Created via SQL. Now show me how you’d take this and make it a reusable module so we can have a standard template for buckets that everyone can use.
- rubiquity 5y ago
  You’re focusing too much on the initial creation rather than on going maintenance and evolution. SQL and relations are much better suited to handle evolution by enforcing constraints than a graph of stitched together pseudo JSON.
  - orf 5y ago
    No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. You describe your state, something reconciles that. How you describe your state and the dependencies between resources is absolutely key, and on the face of it it looks like this interface is totally inadequate. So, again, show me even a brief sketch of how you would describe what I said above with this model, and you’ll see it quickly falls apart.
    - rubiquity 5y ago
      You're coming off a bit worked up, but I'll humor you anyway. I'm not going to type out a bunch of SQL as an example. SQL vs HCL isn't the point and they basically break even on expressiveness. After you've typed out your pseudo-JSON, what exactly are the existing tools saving you? From having to use some wrapper around the cloud API? That's the easy part. By overly focusing on SQL you're missing the forest for the trees. The point is relations and RDBMS features such as constraints, triggers, stored procedures. Such a platform would be always online rather than just a tfstate file waiting for humans to munge it. It's also time to stop thinking about the cloud as literal resources like current tools do and start moving to more abstract concepts (more on that later). > No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. I run very critical infrastructure for a living and there absolutely is a difference. Creating new resources is easy -- they aren't being used and the world doesn't have any expectations for their performance or reliability. The bacon is made in evolving existing infrastructure without impacting user experience negatively. Terraform and other such generators give you very little in guard rails or help and silly outages happen all of the time because of it. Database engineers have been creating sophisticated execution engines for decades. The creator of SQLite cites treating every SQL query as its own program to be ran by a byte code VM as a key design decision. Writing Terraform or what have you is like programming in an AST (sorry Lispers). To date query execution engines figure out how to manage on-disk structures. There is no reason they couldn't be creating smart plans for infrastructure changes at a much higher abstraction level than "glue this KMS key to this bucket."
      - orf 5y ago
        I think we're arguing two different points here. I'm not arguing that a theoretical better system exists that can plan infrastructure on a much more intelligent level than current public tooling does, or that you can't muck up terraform applies and cause downtime. I'm arguing that version controlling a bunch of SQL statements to describe your infrastructure feels wrong, outside of a few snappy short examples, and that this more intelligent system you're describing is in no way tied or related to using SQL as a DSL. You have some infrastructure that you want to exist in a certain state. I used a bucket to describe something that seems simple but often needs a lot of other related infrastructure to exist that is configured via potentially a lot of different API calls. These need to be created in a particular order, and any changes to those resources need to be reconciled in a particular order. Replace "bucket" with anything else that fits that, which is absolutely everything. Ok, so how do we describe that with SQL? INSERT INTO aws_kms_key VALUES (nothing?) RETURNING something? INSERT INTO aws_bucket (bucket_name, kms_key_id) VALUES ("bucket-name", (SELECT kms_key_arn FROM aws_kms_key WHERE what = ....?)) INSERT INTO aws_bucket_policy (bucket_arn, policy) VALUES (???, make_policy_somehow?()) Looks terrible, not least of all because naively you'd assume it would create buckets every time this is "run" or because you'd need to add variables and suddenly you're writing TSQL. Great. Now the terraform would be quite verbose, sure, but you'd encapsulate it in a module so all you'd need is: module "s3_bucket" { name = "bucket-name" kms = true } And that's way more descriptive than a large number of SQL statements to do the same thing. Sure, you could argue that you could write the following magic statement: INSERT INTO cool_custom_bucket (name, kms) VALUES ("bucket-name", true) But how would a user create their own abstractions? What if I want a custom policy for all our buckets? Would we write some trigger that updates a policy for all buckets that are inserted? Would we write some incomprehensible stored procedure that does some of this? Ok, but lets forget all that. Imagine it works and it's fluid and it's basically terraform but in SQL etc etc. So now we want to make some changes and roll them out. How does SQL help us here? Do we open a merge request and say "UPDATE s3_bucket WHERE name = "bucket-name" SET kms = false"? Do we edit the insert statement above? Do we just ignore versioned infrastructure entirely and open some pseudo-terminal and smash in "update production set down=true"? Seems horrible. If you're imagining something entirely different level of infrastructure management needed only by google-scale companies that can only be done with SQL then you might need to elaborate on why exactly that is, because I can't see the connection. Why couldn't this theoretical system work with infrastructure managed by a DSL like Terraform? Why couldn't you fork Terraform to make more intelligent decisions about large-scale updates, or manage state in a different way whilst keeping the DSL? And, as a side note, I rather appreciate the statefile being a plaintext file. It's much more flexible. But it can store state in a database as well.

orf 5y ago

Sure, but that’s got nothing to do with SQL and could be modelled in terraform. Or better yet, indirectly using terraform via IAC providers in languages like Typescript. Show me a proper example of creating an actual s3 bucket that you’d use in production. KMS key, inventory configuration, resource policy, lifecycle policy, logging enabled. Created via SQL. Now show me how you’d take this and make it a reusable module so we can have a standard template for buckets that everyone can use.

rubiquity 5y ago
You’re focusing too much on the initial creation rather than on going maintenance and evolution. SQL and relations are much better suited to handle evolution by enforcing constraints than a graph of stitched together pseudo JSON.
- orf 5y ago
  No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. You describe your state, something reconciles that. How you describe your state and the dependencies between resources is absolutely key, and on the face of it it looks like this interface is totally inadequate. So, again, show me even a brief sketch of how you would describe what I said above with this model, and you’ll see it quickly falls apart.
  - rubiquity 5y ago
    You're coming off a bit worked up, but I'll humor you anyway. I'm not going to type out a bunch of SQL as an example. SQL vs HCL isn't the point and they basically break even on expressiveness. After you've typed out your pseudo-JSON, what exactly are the existing tools saving you? From having to use some wrapper around the cloud API? That's the easy part. By overly focusing on SQL you're missing the forest for the trees. The point is relations and RDBMS features such as constraints, triggers, stored procedures. Such a platform would be always online rather than just a tfstate file waiting for humans to munge it. It's also time to stop thinking about the cloud as literal resources like current tools do and start moving to more abstract concepts (more on that later). > No, I’m not. There isn’t a difference between creating and ongoing maintenance- it’s the same thing. I run very critical infrastructure for a living and there absolutely is a difference. Creating new resources is easy -- they aren't being used and the world doesn't have any expectations for their performance or reliability. The bacon is made in evolving existing infrastructure without impacting user experience negatively. Terraform and other such generators give you very little in guard rails or help and silly outages happen all of the time because of it. Database engineers have been creating sophisticated execution engines for decades. The creator of SQLite cites treating every SQL query as its own program to be ran by a byte code VM as a key design decision. Writing Terraform or what have you is like programming in an AST (sorry Lispers). To date query execution engines figure out how to manage on-disk structures. There is no reason they couldn't be creating smart plans for infrastructure changes at a much higher abstraction level than "glue this KMS key to this bucket."
    - orf 5y ago
      I think we're arguing two different points here. I'm not arguing that a theoretical better system exists that can plan infrastructure on a much more intelligent level than current public tooling does, or that you can't muck up terraform applies and cause downtime. I'm arguing that version controlling a bunch of SQL statements to describe your infrastructure feels wrong, outside of a few snappy short examples, and that this more intelligent system you're describing is in no way tied or related to using SQL as a DSL. You have some infrastructure that you want to exist in a certain state. I used a bucket to describe something that seems simple but often needs a lot of other related infrastructure to exist that is configured via potentially a lot of different API calls. These need to be created in a particular order, and any changes to those resources need to be reconciled in a particular order. Replace "bucket" with anything else that fits that, which is absolutely everything. Ok, so how do we describe that with SQL? INSERT INTO aws_kms_key VALUES (nothing?) RETURNING something? INSERT INTO aws_bucket (bucket_name, kms_key_id) VALUES ("bucket-name", (SELECT kms_key_arn FROM aws_kms_key WHERE what = ....?)) INSERT INTO aws_bucket_policy (bucket_arn, policy) VALUES (???, make_policy_somehow?()) Looks terrible, not least of all because naively you'd assume it would create buckets every time this is "run" or because you'd need to add variables and suddenly you're writing TSQL. Great. Now the terraform would be quite verbose, sure, but you'd encapsulate it in a module so all you'd need is: module "s3_bucket" { name = "bucket-name" kms = true } And that's way more descriptive than a large number of SQL statements to do the same thing. Sure, you could argue that you could write the following magic statement: INSERT INTO cool_custom_bucket (name, kms) VALUES ("bucket-name", true) But how would a user create their own abstractions? What if I want a custom policy for all our buckets? Would we write some trigger that updates a policy for all buckets that are inserted? Would we write some incomprehensible stored procedure that does some of this? Ok, but lets forget all that. Imagine it works and it's fluid and it's basically terraform but in SQL etc etc. So now we want to make some changes and roll them out. How does SQL help us here? Do we open a merge request and say "UPDATE s3_bucket WHERE name = "bucket-name" SET kms = false"? Do we edit the insert statement above? Do we just ignore versioned infrastructure entirely and open some pseudo-terminal and smash in "update production set down=true"? Seems horrible. If you're imagining something entirely different level of infrastructure management needed only by google-scale companies that can only be done with SQL then you might need to elaborate on why exactly that is, because I can't see the connection. Why couldn't this theoretical system work with infrastructure managed by a DSL like Terraform? Why couldn't you fork Terraform to make more intelligent decisions about large-scale updates, or manage state in a different way whilst keeping the DSL? And, as a side note, I rather appreciate the statefile being a plaintext file. It's much more flexible. But it can store state in a database as well.