4 ms·

Why are Terraform modules convenient while stored procs are horrible though? They're doing the same job? E.g. call s3_bucket('bucket-name', true); Is not

Why are Terraform modules convenient while stored procs are horrible though? They're doing the same job? E.g. call s3_bucket('bucket-name', true); Is not really all that different from a Terraform module. And the rest of your objections boil down to 'use in-code database migrations', which app developers have been doing as a best practice for a long time now.

orf 5y ago
There are two decoupled parts to IAC - describing the state you want and reconciling that with the actual state of the world. Reconciling can be advanced as you want: you could take into account SLAs, perform progressive updates, or just ignore that and remove/create resources at will. So even if you used SQL and inserted it into a database, it's just a representation of state that needs to be reconciled in a separate step. The problem is that "use in-code database migrations" is very very different to this problem. With database migrations you have a previous state and you manually code the steps to progress that state: add a column, then copy the data, then delete the old column, then do this, then do that. You do the reconciliation yourself. And this is obvious when you think about it - imagine you did structure IAC like migrations, when you wanted to bring up some fresh infrastructure what happens? You would create all these resources, then modify them, then delete them, then create some new resources as a linear history of operations. Not good, very brittle. So you need to decouple "what you want" from "how you get it". You could indeed use SQL to describe "what you want", but what does that really give you over using a language like Python/TypeScript/HCL to do it? It's not expressive enough and even things like conditionals are hard, and lets not even talk about string templating. Lets try and model the `s3_bucket` procedure with Postgres: CREATE PROCEDURE s3_bucket(bucket_name, bucket) LANGUAGE plpgsql AS $$ DECLARE kms_arn string; bucket_arn string; BEGIN INSERT INTO aws_kms_key VALUES (nothing?) RETURNING arn INTO kms_arn; INSERT INTO aws_bucket (bucket_name, kms_key_id) VALUES ("bucket-name", kms_arn) RETURNING arn INTO bucket_arn; INSERT INTO aws_bucket_policy (bucket_arn, policy) VALUES (bucket_arn, make_policy_somehow?()) END $$; Of course, because you're describing a graph these returned arn's wouldn't actually _be_ arns, just placeholders that need to be reconciled later. And therein lies a problem - how do you ensure each node in your graph has a unique order-independent identifier? How do you know that you shouldn't need to create the `aws_kms_key` because it already exists, but you do need to create the `aws_bucket_policy` because you just added that `INSERT INTO` line? So you'd need to add a unique identifier to every `INSERT` statement: INSERT INTO aws_kms_key (node_id) VALUES ("this has to be unique else everything goes to hell") RETURNING arn INTO kms_arn; Ugly. What's better about this than: resource "aws_kms_key" key {} resource "aws_s3_bucket" bucket { name = "bucket-name" kms_key_id = aws_kms_key.key.arn } resource "aws_bucket_policy" policy { bucket_id = aws_s3_bucket.bucket.id policy = jsonencode({...}) # or use the IAM document resource } You could also model it with Pulumi: const key = new aws.kms.Key() const bucket = new aws.s3.Bucket("my-bucket"); const policy = new aws.s3.BucketPolicy(bucket, make_policy()) Both of these are _much_ easier to grok and more flexible than the stored-procedure soup you will immediately get into if you use SQL. And at the end of the day... it doesn't matter. You're not even _using_ a database to store your state, you're just using SQL to produce some graph of things you need to reconcile. You could write some huge stored procedure to turn your database state into a terraform/pulumi plan and get it to apply it with no problems. Given that... what's the point of using SQL when there are better alternatives? Or, if you really hate yourself, go write it in COBOL. Equally as suitable.
- yawaramin 5y ago
  > You're not even _using_ a database to store your state, you're just using SQL to produce some graph of things you need to reconcile. Well, that's one way of doing it. The better way, imho, is to actually have a database to store the state. That also solves: > You would create all these resources, then modify them, then delete them, then create some new resources as a linear history of operations. Not good, very brittle. When you have an actual database backing your SQL queries, the migrations would all run in this DB, produce a final state of the DB, then this final state would get applied to bring up your system in one shot. > So you'd need to add a unique identifier to every `INSERT` statement: Sure, and that's fairly easy to derive, because we already have a unique identifier, the bucket name: INSERT INTO aws_kms_key (node_id) VALUES (concat('aws_kms_key', bucket_name)) RETURNING arn INTO kms_arn;
  - orf 5y ago
    > Well, that's one way of doing it. The better way, imho, is to actually have a database to store the state It doesn’t matter how the state is stored, you still need to reconcile it. So yeah you can write your target state into a database if you wish, but you need something else to compare that with the state of the actual world and progress that towards the state you requested. > When you have an actual database backing your SQL queries, the migrations would all run in this DB, produce a final state of the DB, then this final state would get applied to bring up your system in one shot. Yes, what you are describing is how Terraform works. Without SQL. Except it’s never “one shot” because it’s a DAG of requirements where each resource creation can fail. > Sure, and that's fairly easy to derive, because we already have a unique identifier, the bucket name In this specific case, sure. But for a KMS key or in the general case, no you wouldn’t. You’d also be hard-coding provider specific information into your system - “a bucket ARN can be derived from the name attribute”. This is not always the case - consider any REST api that accepts a POST request to create something, but returns only an incrementing integer primary key to uniquely identify it.

orf 5y ago

There are two decoupled parts to IAC - describing the state you want and reconciling that with the actual state of the world. Reconciling can be advanced as you want: you could take into account SLAs, perform progressive updates, or just ignore that and remove/create resources at will. So even if you used SQL and inserted it into a database, it's just a representation of state that needs to be reconciled in a separate step. The problem is that "use in-code database migrations" is very very different to this problem. With database migrations you have a previous state and you manually code the steps to progress that state: add a column, then copy the data, then delete the old column, then do this, then do that. You do the reconciliation yourself. And this is obvious when you think about it - imagine you did structure IAC like migrations, when you wanted to bring up some fresh infrastructure what happens? You would create all these resources, then modify them, then delete them, then create some new resources as a linear history of operations. Not good, very brittle. So you need to decouple "what you want" from "how you get it". You could indeed use SQL to describe "what you want", but what does that really give you over using a language like Python/TypeScript/HCL to do it? It's not expressive enough and even things like conditionals are hard, and lets not even talk about string templating. Lets try and model the `s3_bucket` procedure with Postgres: CREATE PROCEDURE s3_bucket(bucket_name, bucket) LANGUAGE plpgsql AS $$ DECLARE kms_arn string; bucket_arn string; BEGIN INSERT INTO aws_kms_key VALUES (nothing?) RETURNING arn INTO kms_arn; INSERT INTO aws_bucket (bucket_name, kms_key_id) VALUES ("bucket-name", kms_arn) RETURNING arn INTO bucket_arn; INSERT INTO aws_bucket_policy (bucket_arn, policy) VALUES (bucket_arn, make_policy_somehow?()) END $$; Of course, because you're describing a graph these returned arn's wouldn't actually _be_ arns, just placeholders that need to be reconciled later. And therein lies a problem - how do you ensure each node in your graph has a unique order-independent identifier? How do you know that you shouldn't need to create the `aws_kms_key` because it already exists, but you do need to create the `aws_bucket_policy` because you just added that `INSERT INTO` line? So you'd need to add a unique identifier to every `INSERT` statement: INSERT INTO aws_kms_key (node_id) VALUES ("this has to be unique else everything goes to hell") RETURNING arn INTO kms_arn; Ugly. What's better about this than: resource "aws_kms_key" key {} resource "aws_s3_bucket" bucket { name = "bucket-name" kms_key_id = aws_kms_key.key.arn } resource "aws_bucket_policy" policy { bucket_id = aws_s3_bucket.bucket.id policy = jsonencode({...}) # or use the IAM document resource } You could also model it with Pulumi: const key = new aws.kms.Key() const bucket = new aws.s3.Bucket("my-bucket"); const policy = new aws.s3.BucketPolicy(bucket, make_policy()) Both of these are _much_ easier to grok and more flexible than the stored-procedure soup you will immediately get into if you use SQL. And at the end of the day... it doesn't matter. You're not even _using_ a database to store your state, you're just using SQL to produce some graph of things you need to reconcile. You could write some huge stored procedure to turn your database state into a terraform/pulumi plan and get it to apply it with no problems. Given that... what's the point of using SQL when there are better alternatives? Or, if you really hate yourself, go write it in COBOL. Equally as suitable.

yawaramin 5y ago
> You're not even _using_ a database to store your state, you're just using SQL to produce some graph of things you need to reconcile. Well, that's one way of doing it. The better way, imho, is to actually have a database to store the state. That also solves: > You would create all these resources, then modify them, then delete them, then create some new resources as a linear history of operations. Not good, very brittle. When you have an actual database backing your SQL queries, the migrations would all run in this DB, produce a final state of the DB, then this final state would get applied to bring up your system in one shot. > So you'd need to add a unique identifier to every `INSERT` statement: Sure, and that's fairly easy to derive, because we already have a unique identifier, the bucket name: INSERT INTO aws_kms_key (node_id) VALUES (concat('aws_kms_key', bucket_name)) RETURNING arn INTO kms_arn;
- orf 5y ago
  > Well, that's one way of doing it. The better way, imho, is to actually have a database to store the state It doesn’t matter how the state is stored, you still need to reconcile it. So yeah you can write your target state into a database if you wish, but you need something else to compare that with the state of the actual world and progress that towards the state you requested. > When you have an actual database backing your SQL queries, the migrations would all run in this DB, produce a final state of the DB, then this final state would get applied to bring up your system in one shot. Yes, what you are describing is how Terraform works. Without SQL. Except it’s never “one shot” because it’s a DAG of requirements where each resource creation can fail. > Sure, and that's fairly easy to derive, because we already have a unique identifier, the bucket name In this specific case, sure. But for a KMS key or in the general case, no you wouldn’t. You’d also be hard-coding provider specific information into your system - “a bucket ARN can be derived from the name attribute”. This is not always the case - consider any REST api that accepts a POST request to create something, but returns only an incrementing integer primary key to uniquely identify it.