4 ms·
Can someone who knows more about neural networks tell me how this works? Apparently the model's weights don't need to be sent over the network. The only thing w
by brian_cloutier 9y ago
Can someone who knows more about neural networks tell me how this works? Apparently the model's weights don't need to be sent over the network. The only thing which is shared is some seed used to initialize the model with random weights. While the network encodes data it simultaneously updates its weights. To decode, somehow you can run the same process and magically get back the original data?
The paper doesn't explain how this is possible, how would I go about implementing something like this?
- john_moscow 9y agoThe naive solution would be to split the input data into the batches. Compress the first batch using a random model, then train a new model based on it. Compress the second batch using the model trained on the first batch and so on. Depending on your data structure, you can use a model trained on all the previous batches (or just N previous batches) when compressing the i-th batch. This can be likely optimized (e.g. 7-Zip's arithmetical coder doesn't split data into batches, but rather uses exponentially filtered conditional probabilities that are instantaneously updated each time a new sample is processed). I'm no expert in RNNs, but my hunch is that you can adjust the model incrementally using backpropagation right after it processes another sample instead of completely retraining it.
- brian_cloutier 9y agoInteresting. So, to decode you would start with the unoptimized model and decode the first batch. You then train using that first batch to get a better model and use your improved model to decode the next batch. Okay, that makes sense; thanks for the response!