4 ms·

The esteemed Google monorepo. Piper is a beautiful thing. Trunk-based development with rapid CI/CD. Everyone with access is essentially a knowledge worker and

by hbarka 5y ago

The esteemed Google monorepo. Piper is a beautiful thing. Trunk-based development with rapid CI/CD. Everyone with access is essentially a knowledge worker and can create DAG-like data pipelines with their SQL code. There is no data team—they are all data team. They also have access to SQL macros and functions that have a global scope and evolves over time as more and more people build. Couples with a high performing database called F1, this is the secret to their success.

he0001 5y ago
I think that many people forget that version control is part of your delivery pipeline, not just a way to “store and move code”. I think there are possibilities to use the version control tools to much more than just that. GitOps touches that but I believe there’s a lot more to it. I think there’s some of it in what you describe here.
- mountainriver 5y ago
  Yes! VCS hasn’t been fully utilized yet. All pipelines need versioning and most things are a pipeline. This is an exciting area of exploration
dijit 5y ago
I’ve been looking for something like piper, is there only perforce?
- hbarka 5y ago
  dbt + dagster + GitHub
- fragmede 5y ago
  Microsoft chose to extend git for their internal source control, and Facebook went all-in on Mercurial+extensions. Google also has a Mercurial+extensions interface to their monorepo. The sheer size of their monorepos mean that tools that don't work for them will work elsewhere. In particular, Git has made improvements recently working with giant (as opposed to merely bit or still-too-humongous-to-manage) repos so it's now workable after massaging the config. The other alternative to Perforce is SVN. There have been some improvements with it that make it more workable in today's git/"decentralized" world. Not knowing your particular use case it's hard to say if that's at all appropriate.

by hbarka 5y ago

he0001 5y ago

I think that many people forget that version control is part of your delivery pipeline, not just a way to “store and move code”. I think there are possibilities to use the version control tools to much more than just that. GitOps touches that but I believe there’s a lot more to it. I think there’s some of it in what you describe here.

mountainriver 5y ago
Yes! VCS hasn’t been fully utilized yet. All pipelines need versioning and most things are a pipeline. This is an exciting area of exploration

dijit 5y ago

I’ve been looking for something like piper, is there only perforce?

hbarka 5y ago
dbt + dagster + GitHub
fragmede 5y ago
Microsoft chose to extend git for their internal source control, and Facebook went all-in on Mercurial+extensions. Google also has a Mercurial+extensions interface to their monorepo. The sheer size of their monorepos mean that tools that don't work for them will work elsewhere. In particular, Git has made improvements recently working with giant (as opposed to merely bit or still-too-humongous-to-manage) repos so it's now workable after massaging the config. The other alternative to Perforce is SVN. There have been some improvements with it that make it more workable in today's git/"decentralized" world. Not knowing your particular use case it's hard to say if that's at all appropriate.