4 ms·

> In an ideal world, a version control system should be strictly internal facing and be able to fall over without impacting live traffic; however the Piper team

• ablatt89 2y ago
  In addition to the benefits above, by storing storing config in version control is 2 party control, presubmit linting, and easy rollbacks. I suppose the trade off is slower rollouts of config and some delay time in pushing out a config but hey, there's no free lunch :-)
  • daemin 2y ago
    Nothing preventing you from building a versioned configuration system using a database or similar, especially if you build tools to actually change the configuration rather than hand editing text files.
    • rockwotj 2y ago
      Google has this, IIRC it’s called CDPush
      • underdeserver 2y ago
        CDPush does versioning, canaried rollouts of... hand edited text files.
      • ablatt89 2y ago
        I believe CDPush still has some latency involved which would be slower than reading raw config data from a DB.
      • kccqzy 2y ago
        CDPush does the pushing. It's not a database.
    • acac10 2y ago
      The key to this is machine AND human readable formats. That is why the text format of protobufs is such a good approach: - structured formats - declared schema - the parser to read the file is also a validity checker (at least for data types) There are tools that do similar with JSON but none are as simple and unobtrusive as protobuf tool chain.
  • acac10 2y ago
    When SOX compliance became a thing (remember Sarbannes-Oxley?), many moons ago, I was able to send EY consultants away during their audit for Unix controls, just by showing them that all our (Autodesk Unix team) configs were in SVN, logged, with authors and full change control. The Windows team had to spend months figuring out something that was not even remotely close to control or auditability. Frankly, I just assumed that it was the std practice today and I am surprised people are still not doing it.
• mingyizhao 2y ago
  Google's source code system isn't only used to store config: some teams also use it to store data like ML models. There's a limit on the max file size, causing all sorts of trouble in the LLM era. So it's a bit like Git LFS, but worse (why have the limit at all?).
  • DEADMINCE 2y ago
    > why have the limit at all? Because it's probably buggy and not tested with commits past a certain file size. Better question, why no use a more appropriate, instead of convenient, tool for the job?
    • Jensson 2y ago
      > Better question, why no use a more appropriate, instead of convenient, tool for the job? Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes. There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things. Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.
      • DEADMINCE 2y ago
        > Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes. That's not an answer so much as an excuse. > There is no tool that does that except for the version control systems that programmers use. It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.
      • WEIHAO97 2y ago
• inopinatus 2y ago
  Almost everything anyone ever wants to do with production is put files on a server and start a process. The rest comes down to your tolerance for baroque scaffolding and someone else’s janky shell scripts