3 ms·

Fantastic! Congratulations on the launch. Is there a way to version control the sync configurations? Any thoughts on putting that in the roadmap? I'd love to

Fantastic! Congratulations on the launch. Is there a way to version control the sync configurations? Any thoughts on putting that in the roadmap? I'd love to be able to put my 'Castled config' in the same repo as my dbt project, for example.

aruntdharan 5y ago
You mean the warehouse/app credentials, when you say sync configurations? If so, yes, that seems like a great idea. Infact I think your warehouse credentials are already there in your dbt repo in a specific format. Castled can directly read those credentials from there.
- amcaskill 5y ago
  That is not what I meant, but also pretty interesting. I actually mean the 'definition' of the syncs themselves. I am picturing JSON or YAML that describes the source fields, their mapping to the destination fields, and any other meta about the sync: frequency, number of retries, whatever else that you could configure in the UI So when I go and update my dbt model to modify one of the tables that I am syncing from, I can make the corresponding changes to my Castled settings file, and release it all as one atomic update to my data infrastructure. It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team.
  - aruntdharan 5y ago
    We can definitely consider that. But I feel its a lot of config and can be error prone. For instance, source-destination field mapping configs might be complex and have various issues like data type mismatches, typos in field names etc and a user interface is better suited to guide you through the entire process. But I see value in exporting the config to a github repo after the pipeline is created and thereafter future edits can be done via the github repo. Does that make sense?
    - amcaskill 5y ago
      Yes 100% -- you could also imagine just syncing from the UI to a repo, rather than trying to make the config human-editable. Toggle into a branch in the UI, make edits, and have those committed to the repo by the tool. Looks awesome, I am rooting for you guys!
      - aruntdharan 5y ago
        Thanks for the input!
  - mason55 5y ago
    > It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team. Yeah this one certainly depends on the target customer. For me, any tool that didn't have source control integration for configuration would be a non-starter. But it's quite possible that the target audience for this tool doesn't even understand the term "source control".
  - bleonard 5y ago
    Congrats to the Castled team on the launch. At Grouparoo, this is a primary use case. We have a UI that engineers use locally. This helps gets things right. It outputs a JSON configuration that is checked in. When that is deployed, it does all the syncing.

aruntdharan 5y ago

You mean the warehouse/app credentials, when you say sync configurations? If so, yes, that seems like a great idea. Infact I think your warehouse credentials are already there in your dbt repo in a specific format. Castled can directly read those credentials from there.

amcaskill 5y ago
That is not what I meant, but also pretty interesting. I actually mean the 'definition' of the syncs themselves. I am picturing JSON or YAML that describes the source fields, their mapping to the destination fields, and any other meta about the sync: frequency, number of retries, whatever else that you could configure in the UI So when I go and update my dbt model to modify one of the tables that I am syncing from, I can make the corresponding changes to my Castled settings file, and release it all as one atomic update to my data infrastructure. It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team.
- aruntdharan 5y ago
  We can definitely consider that. But I feel its a lot of config and can be error prone. For instance, source-destination field mapping configs might be complex and have various issues like data type mismatches, typos in field names etc and a user interface is better suited to guide you through the entire process. But I see value in exporting the config to a github repo after the pipeline is created and thereafter future edits can be done via the github repo. Does that make sense?
  - amcaskill 5y ago
    Yes 100% -- you could also imagine just syncing from the UI to a repo, rather than trying to make the config human-editable. Toggle into a branch in the UI, make edits, and have those committed to the repo by the tool. Looks awesome, I am rooting for you guys!
    - aruntdharan 5y ago
      Thanks for the input!
- mason55 5y ago
  > It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team. Yeah this one certainly depends on the target customer. For me, any tool that didn't have source control integration for configuration would be a non-starter. But it's quite possible that the target audience for this tool doesn't even understand the term "source control".
- bleonard 5y ago
  Congrats to the Castled team on the launch. At Grouparoo, this is a primary use case. We have a UI that engineers use locally. This helps gets things right. It outputs a JSON configuration that is checked in. When that is deployed, it does all the syncing.