3 ms·

I randomly clicked on repeatability and am still curious about how it's achieved with distributed training. Wouldn't deterministic synchronization make things s

I randomly clicked on repeatability and am still curious about how it's achieved with distributed training. Wouldn't deterministic synchronization make things slow? But I have heard that at least in a couple of big companies, their training is repeatable.

eru 3y ago
You would want to make training updates commutative as much as possible. That way it doesn't matter which order you apply the updates in.