4 ms·

protobuf has a serialized file limit of 2gb? or like a definition file? I haven't touched PB in ages and certainly not something that would be transmitting that

by scrame 3y ago

protobuf has a serialized file limit of 2gb? or like a definition file? I haven't touched PB in ages and certainly not something that would be transmitting that much in a single message. is your complaint that the whole context/weights needs to be sent through the whole file? I'm asking from a position of ignorance, I'm surpised to see a serialized transport of that size, and wondering why it's specifically limited at 2gb. like to be able to mmap on 32-bit hardware?

Roark66 3y ago
>protobuf has a serialized file limit of 2gb I'm not sure if this is a file size limit too or just an object memory representation size limit. For me using a library designed for message passing to save/read your AI models is a bad design decision. >is your complaint that the whole context/weights needs to be sent through the whole file? I store large onnx models with "external" weights, but even so many operations fail with the dreaded "ModelProto exceeds maximum protobuf size of 2GB: 3385275542". So the complaint is that you simply can't do a lot of stuff with models over 2gb. You can create a session to execute the model, you can run the vanilla optimisation over it. But trying to run transformer specific optimisation errors out as well as making any attempts at slicing the model. Additionaly some conversion processes. It appears the code that does that looks up the model size and just errors out if over 2GB.it doesn't even try loading it
- snnn 3y ago
  > I'm not sure if this is a file size limit too or just an object memory representation size limit. Message. Each message cannot be bigger than 2GB(and usually should be not this big), but a file could contain multiple messages. This limitation helps them prevent integer overflows, since every length is 32-bit but every application process is 64-bit, you can convert the length numbers to 64-bit before doing any arithmetic operation. Therefore, fundamentally, there is no way to make it secure on 32-bit platforms, or no way to support more than 2GB on 64-bit platforms without totally rewriting the code.

by scrame 3y ago

Roark66 3y ago

>protobuf has a serialized file limit of 2gb I'm not sure if this is a file size limit too or just an object memory representation size limit. For me using a library designed for message passing to save/read your AI models is a bad design decision. >is your complaint that the whole context/weights needs to be sent through the whole file? I store large onnx models with "external" weights, but even so many operations fail with the dreaded "ModelProto exceeds maximum protobuf size of 2GB: 3385275542". So the complaint is that you simply can't do a lot of stuff with models over 2gb. You can create a session to execute the model, you can run the vanilla optimisation over it. But trying to run transformer specific optimisation errors out as well as making any attempts at slicing the model. Additionaly some conversion processes. It appears the code that does that looks up the model size and just errors out if over 2GB.it doesn't even try loading it

snnn 3y ago
> I'm not sure if this is a file size limit too or just an object memory representation size limit. Message. Each message cannot be bigger than 2GB(and usually should be not this big), but a file could contain multiple messages. This limitation helps them prevent integer overflows, since every length is 32-bit but every application process is 64-bit, you can convert the length numbers to 64-bit before doing any arithmetic operation. Therefore, fundamentally, there is no way to make it secure on 32-bit platforms, or no way to support more than 2GB on 64-bit platforms without totally rewriting the code.