3 ms·
Reynold from Databricks and the Apache Spark project here. Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you r
by rxin 10y ago
Reynold from Databricks and the Apache Spark project here.
Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?
- snnn 10y agoHi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.
- sandGorgon 10y agoOh wow... I didn't even know these limitations (and in fact ,they are not very Google-able). Can you talk about these limitations and your experience?
- snnn 10y agohttps://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-139 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-1476 https://issues.apache.org/jira/browse/SPARK-6235 https://issues.apache.org/jira/browse/SPARK-6235 You'll hit this bug when your model size is larger than 2GB. BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that. https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-732 https://issues.apache.org/jira/browse/SPARK-5490 https://issues.apache.org/jira/browse/SPARK-5490