3 ms·
Oh wow... I didn't even know these limitations (and in fact ,they are not very Google-able). Can you talk about these limitations and your experience?
by sandGorgon 10y ago
Oh wow... I didn't even know these limitations (and in fact ,they are not very Google-able).
Can you talk about these limitations and your experience?
- snnn 10y agohttps://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-6235 https://issues.apache.org/jira/browse/SPARK-6235 You'll hit this bug when your model size is larger than 2GB. BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that. https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-5490 https://issues.apache.org/jira/browse/SPARK-5490