ryan_green

3 ms·

1. 1.▲
   by ryan_green 3y ago

   exactly!!!
2. 2.▲
   by ryan_green 3y ago

   apologies for not getting into more detail--wanted to start by covering things at a high level. There are a few key concepts that might be helpful. * data state - this is contents of both your data and metadata at a given point in time. if
3. 3.▲
   by ryan_green 3y ago

   sort of. the main issues we've found with each developer having their own DB on a complex data pipeline are 1) if that DB contains petabytes of data, creating it one for each developer is non-trivial from a time and cost perspective 2
4. 4.▲
   by ryan_green 3y ago

   saltcured, find these comments super insightful! > Yeah, there's a lot of hidden magic/assumptions in having a "writable snapshot of a specific version" of production data. That's absolutely a huge assumption. T
5. 5.▲
   by ryan_green 3y ago

   NBJack, your point about the difficult of managing external dependencies is well taken. That said, our data pipeline uses cloud storage and multiple external services and the scenarios you're describing haven't materialized so fa
6. 6.▲
   by ryan_green 3y ago

   totally agree. and creating ephemeral environments for data pipelines is quite a bit more challenging than systems with a less complicated data state. nonetheless, this has paid off for us already many times over.