5 ms·

Google's source code system isn't only used to store config: some teams also use it to store data like ML models. There's a limit on the max file size, causing

Google's source code system isn't only used to store config: some teams also use it to store data like ML models. There's a limit on the max file size, causing all sorts of trouble in the LLM era. So it's a bit like Git LFS, but worse (why have the limit at all?).

DEADMINCE 2y ago
> why have the limit at all? Because it's probably buggy and not tested with commits past a certain file size. Better question, why no use a more appropriate, instead of convenient, tool for the job?
- Jensson 2y ago
  > Better question, why no use a more appropriate, instead of convenient, tool for the job? Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes. There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things. Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.
  - DEADMINCE 2y ago
    > Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes. That's not an answer so much as an excuse. > There is no tool that does that except for the version control systems that programmers use. It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.
  - WEIHAO97 2y ago
    Our team just built it, appearing in ICML 24': https://www.microsoft.com/en-us/research/uploads/prodnew/2024/06/mgit_icml_2024.pdf https://www.microsoft.com/en-us/research/uploads/prodnew/202...
summerlight 2y ago
There are a number of really good internal tools for ML model management. It'd be really weird to rely on Piper to manage production ML models since this does not provide critical functionalities (monitoring, deployment, dependency management etc etc)
- mingyizhao 2y ago
  That team has some uncommon testing and deployment requirements, and they don't really run a service, so their use of Piper makes some sense to me. Although I agree it's still very weird. They are trying to migrate right now.
mattnewton 2y ago
When I was there models weights were pretty much universally stored in colossus, either directly or through some bare bones tooling. I’m surprised the team you were working with had requirements that lead them to using piper directly for that. We did abuse piper for all sorts of things though, most notably “golden” screenshots for diffing ui code changes across multiple search results pages.
deleted 2y ago
[deleted]
tylerhou 2y ago
Data in Piper (actually, even in SrcFS…?) is stored forever. So the cost of a “small” amount of data like few gigabytes can add up over time (cost for storage, replication, transmission, etc.)

DEADMINCE 2y ago

> why have the limit at all? Because it's probably buggy and not tested with commits past a certain file size. Better question, why no use a more appropriate, instead of convenient, tool for the job?

Jensson 2y ago
> Better question, why no use a more appropriate, instead of convenient, tool for the job? Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes. There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things. Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.
- DEADMINCE 2y ago
  > Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes. That's not an answer so much as an excuse. > There is no tool that does that except for the version control systems that programmers use. It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.
- WEIHAO97 2y ago
  Our team just built it, appearing in ICML 24': https://www.microsoft.com/en-us/research/uploads/prodnew/2024/06/mgit_icml_2024.pdf https://www.microsoft.com/en-us/research/uploads/prodnew/202...

summerlight 2y ago

There are a number of really good internal tools for ML model management. It'd be really weird to rely on Piper to manage production ML models since this does not provide critical functionalities (monitoring, deployment, dependency management etc etc)

mingyizhao 2y ago
That team has some uncommon testing and deployment requirements, and they don't really run a service, so their use of Piper makes some sense to me. Although I agree it's still very weird. They are trying to migrate right now.

mattnewton 2y ago

When I was there models weights were pretty much universally stored in colossus, either directly or through some bare bones tooling. I’m surprised the team you were working with had requirements that lead them to using piper directly for that. We did abuse piper for all sorts of things though, most notably “golden” screenshots for diffing ui code changes across multiple search results pages.

deleted 2y ago

[deleted]

tylerhou 2y ago

Data in Piper (actually, even in SrcFS…?) is stored forever. So the cost of a “small” amount of data like few gigabytes can add up over time (cost for storage, replication, transmission, etc.)