4 ms·

The Distributed Tensor Algebra Compiler (2022)

40 points by yeesian 3y ago

grepLeigh 3y ago
Would anyone be interested in discussing this paper together, especially through the lens of "how can we schedule algebraic expressions and checkpoint computed progress across a heterogenous pool of consumer/donated machines?" I'm an infrastructure engineer, mostly focused on databases, data pipelines, ML infra for the past 10-15 years. Even when designing homogenous compute clusters, I had to dig in and understand compiler-level implementations in MLIR and LLVM. I'm not a compiler expert by any measure, but know just enough to be dangerous and curious about (safely) scheduling computations across a pool of volunteers machines. Seems especially important to chew on now, with training of foundational LLM weights costing 7-9 figures.
- yeesian 3y ago
  (This is more of a link-dump than a paper discussion --) For the line of inquiry w.r.t tensor compilers and MLIR/LLVM (linalg, polyhedral, [sparse_]tensor, etc), I personally found the following really helpful: https://news.ycombinator.com/item?id=25545373 https://news.ycombinator.com/item?id=25545373 (links to a survey), https://github.com/merrymercy/awesome-tensor-compilers https://github.com/merrymercy/awesome-tensor-compilers I also have an interest in the community more widely associated with pandas/dataframes-like languages (e.g. modin/dask/ray/polars/ibis) with substrait/calcite/arrow their choice of IR. Some links: https://github.com/modin-project/modin https://github.com/modin-project/modin, https://github.com/dask/dask/issues/8980 https://github.com/dask/dask/issues/8980, https://news.ycombinator.com/item?id=16510610 https://news.ycombinator.com/item?id=16510610, https://news.ycombinator.com/item?id=35521785 https://news.ycombinator.com/item?id=35521785 I broadly classify them as such since the former has a stronger disposition towards linear/tensor-algebra, while the latter towards relational algebra, and it isn't yet clear (to me) how well innovations in one carry over to the other (if they do), and hence I'm also curious to hear more about proposals for a unified language across linalg and relational alg (e.g. https://news.ycombinator.com/item?id=36349015 https://news.ycombinator.com/item?id=36349015). I'm particularly interested in pandas precisely because it seems to be right at the intersection of both forms of algebra (and draws a strong reaction from people who are familiar/comfortable with one community and not the other). See e.g. https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol... and https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
- flashgordon 3y ago
  I would love to be a part of this if that's ok :). I love this mainly as someone interested in compilers and infra
erichocean 3y ago
This kind of stuff is the future, but it would be much better if they targeted custom MLIR dialects instead of creating yet another incompatible compiler stack.
- rohany 3y ago
  I agree! Much of this work was done as part of the overarching TACO project (https://github.com/tensor-compiler/taco https://github.com/tensor-compiler/taco), in an attempt to distribute sparse tensor computations (https://rohany.github.io/publications/sc2022-spdistal.pdf https://rohany.github.io/publications/sc2022-spdistal.pdf). MLIR recently (~mid 2022) began implementing the ideas from TACO into a "sparse tensor" dialect, so perhaps some of these ideas could make it into there. I'm working with MLIR these days, and if I could re-do the project now I would probably utilize and targetb the MLIR linalg infrastructure!
rohany 3y ago
I am the author of this paper -- kind of crazy to see it posted here on its own! I'll hang around to try and answer questions

40 points by yeesian 3y ago

Would anyone be interested in discussing this paper together, especially through the lens of "how can we schedule algebraic expressions and checkpoint computed progress across a heterogenous pool of consumer/donated machines?" I'm an infrastructure engineer, mostly focused on databases, data pipelines, ML infra for the past 10-15 years. Even when designing homogenous compute clusters, I had to dig in and understand compiler-level implementations in MLIR and LLVM. I'm not a compiler expert by any measure, but know just enough to be dangerous and curious about (safely) scheduling computations across a pool of volunteers machines. Seems especially important to chew on now, with training of foundational LLM weights costing 7-9 figures.

yeesian 3y ago
(This is more of a link-dump than a paper discussion --) For the line of inquiry w.r.t tensor compilers and MLIR/LLVM (linalg, polyhedral, [sparse_]tensor, etc), I personally found the following really helpful: https://news.ycombinator.com/item?id=25545373 https://news.ycombinator.com/item?id=25545373 (links to a survey), https://github.com/merrymercy/awesome-tensor-compilers https://github.com/merrymercy/awesome-tensor-compilers I also have an interest in the community more widely associated with pandas/dataframes-like languages (e.g. modin/dask/ray/polars/ibis) with substrait/calcite/arrow their choice of IR. Some links: https://github.com/modin-project/modin https://github.com/modin-project/modin, https://github.com/dask/dask/issues/8980 https://github.com/dask/dask/issues/8980, https://news.ycombinator.com/item?id=16510610 https://news.ycombinator.com/item?id=16510610, https://news.ycombinator.com/item?id=35521785 https://news.ycombinator.com/item?id=35521785 I broadly classify them as such since the former has a stronger disposition towards linear/tensor-algebra, while the latter towards relational algebra, and it isn't yet clear (to me) how well innovations in one carry over to the other (if they do), and hence I'm also curious to hear more about proposals for a unified language across linalg and relational alg (e.g. https://news.ycombinator.com/item?id=36349015 https://news.ycombinator.com/item?id=36349015). I'm particularly interested in pandas precisely because it seems to be right at the intersection of both forms of algebra (and draws a strong reaction from people who are familiar/comfortable with one community and not the other). See e.g. https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol... and https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/
flashgordon 3y ago
I would love to be a part of this if that's ok :). I love this mainly as someone interested in compilers and infra

erichocean 3y ago

This kind of stuff is the future, but it would be much better if they targeted custom MLIR dialects instead of creating yet another incompatible compiler stack.

rohany 3y ago
I agree! Much of this work was done as part of the overarching TACO project (https://github.com/tensor-compiler/taco https://github.com/tensor-compiler/taco), in an attempt to distribute sparse tensor computations (https://rohany.github.io/publications/sc2022-spdistal.pdf https://rohany.github.io/publications/sc2022-spdistal.pdf). MLIR recently (~mid 2022) began implementing the ideas from TACO into a "sparse tensor" dialect, so perhaps some of these ideas could make it into there. I'm working with MLIR these days, and if I could re-do the project now I would probably utilize and targetb the MLIR linalg infrastructure!

rohany 3y ago

I am the author of this paper -- kind of crazy to see it posted here on its own! I'll hang around to try and answer questions