4 ms·

This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. Has anyone converted stuff like gradien

This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. Has anyone converted stuff like gradient descent to set theory? https://support.unicomsi.com/manuals/soliddb/7/SQL_Guide/2_GettingStartedSQL.03.04.html https://support.unicomsi.com/manuals/soliddb/7/SQL_Guide/2_G... https://www.sqlshack.com/mathematics-sql-server-fast-introduction-set-theory/ https://www.sqlshack.com/mathematics-sql-server-fast-introdu... https://www.sqlshack.com/learn-sql-set-theory/ https://www.sqlshack.com/learn-sql-set-theory/ Right now AI algorithms kind of look imperative and stateful to me, like state machines. But there should be a functional or relational representation, where pattern matching would be used to derive the current state instead. It's trivial to go from functional to imperative representation, but often nearly impossible to go the other way. That's why monadic (sorry if I'm mincing terms) logic is so troublesome. Mutable variables, futures/promises, async, etc can't be statically analyzed, which is why most code today is difficult to parallelize and stuff like C++ optimizers don't even increase speed to the next order of magnitude. But functional languages have nearly infinite optimization potential through divide and conquer approaches like sharding and scatter-gather arrays that can run internally without side effects. In other words, we can throw hardware at SQL for linear speedup since it's embarrassingly parallel, but might have limited success optimizing something like Python.

dogcomplex 4y ago
Replying to this to remind myself to research later, as this is a very important question
hdespiritu 4y ago
Your question "Has anyone converted stuff like gradient descent to set theory?" doesn't make sense from the perspective that gradient descent uses differential calculus to find min/max points of an objective function and differential calculus requires a lot of additional assumptions on top of set theory. That being said, current deep learning libraries such as JAX and Pytorch use automatic differentiation to efficiently compute partial derivatives used for optimization algorithms such as gradient descent and it's not clear to me what the level of effort would be to convert that to something that could run efficiently in SQL?
- zackmorris 4y ago
  Thank you, I knew that derivatives were used in gradient descent, but automatic differentiation is new to me. The Wikipedia article is fairly opaque compared to what I learned in school, but this tidbit stood out: https://en.wikipedia.org/wiki/Automatic_differentiation#Implementation https://en.wikipedia.org/wiki/Automatic_differentiation#Impl... Source code transformation (SCT): the compiler processes source code so that the derivatives are calculated alongside each instruction. Operator overloading (OO): operators are overridden so that derivatives are calculated for numbers and vectors. Based on the state of software these days, I'm guessing that OO (the "bare hands" method) is what's mainstream. It would be far better IMHO to use SCT, since it's a universal solution that doesn't require manually refactoring programs. But stuff like SCT might be considered metaprogramming, which seems to have fallen out of fashion. I grew up with C++ macros and templates, so I feel that this is somewhat tragic, although readability and collaboration are much better today. A modern example might be something like aspect-oriented programming (AOP): https://en.wikipedia.org/wiki/Aspect-oriented_programming https://en.wikipedia.org/wiki/Aspect-oriented_programming I once used the AOP library AspectJ to trace a Java app's execution, since Java made the (unfortunate) choice to focus on objects rather than functions, which makes it generally a poor fit for data processing, due to its high use of mutable state within objects (mutable state is what limits most object-oriented projects to around 1 million lines). Meaning that I couldn't remember the program's context as I was stepping through it, and had to analyze traces instead. AspectJ allows one to hook into the code without modifying it, sort of like a debugger, so that stuff like function calls and variable mutations can be watched: https://en.wikipedia.org/wiki/AspectJ https://en.wikipedia.org/wiki/AspectJ https://www.eclipse.org/aspectj/doc/released/progguide/index.html https://www.eclipse.org/aspectj/doc/released/progguide/index... Looks like this might still be an open problem in Python: https://stackoverflow.com/questions/12356713/aspect-oriented-programming-aop-in-python https://stackoverflow.com/questions/12356713/aspect-oriented... https://docs.spring.io/spring-python/1.2.x/sphinx/html/aop.html#aspect-oriented-programming https://docs.spring.io/spring-python/1.2.x/sphinx/html/aop.h... But it seems like SQL would be a good candidate for AOP: https://stackoverflow.com/questions/12271588/aspect-oriented-sql-server https://stackoverflow.com/questions/12271588/aspect-oriented... https://technology.amis.nl/it/aspect-oriented-programming-aop-in-plsql-is-it-hot-is-it-cool-or-utterly-useless/ https://technology.amis.nl/it/aspect-oriented-programming-ao... If we had that, maybe we could automatically generate derivatives for the set operations. Then either access them as variables in stored procedures, or possibly as something like views or via metadata stored somewhere like MySQL's INFORMATION_SCHEMA. I don't really know, but maybe these breadcrumbs could be helpful.
teruakohatu 4y ago
> This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. I understand the sentiment but you can run ML on plain CPUs, in fact it is easier to do so, you just don't get the massive speed benefits of GPUs. Hugging face's public models are all run on CPUs to save costs. > Right now AI algorithms kind of look imperative and stateful to me, like state machines. The maths is not stateful or imperative, only our implementation. Much like how functional languages are executed on a stateful CPU. Research in parallelization is ongoing.
mr_toad 4y ago
Most ML algorithms are numerical approximations, so yes, they look procedural. You can try and find analytical solutions, but except for the simplest models they’re intractable.

dogcomplex 4y ago

Replying to this to remind myself to research later, as this is a very important question

hdespiritu 4y ago

Your question "Has anyone converted stuff like gradient descent to set theory?" doesn't make sense from the perspective that gradient descent uses differential calculus to find min/max points of an objective function and differential calculus requires a lot of additional assumptions on top of set theory. That being said, current deep learning libraries such as JAX and Pytorch use automatic differentiation to efficiently compute partial derivatives used for optimization algorithms such as gradient descent and it's not clear to me what the level of effort would be to convert that to something that could run efficiently in SQL?

zackmorris 4y ago
Thank you, I knew that derivatives were used in gradient descent, but automatic differentiation is new to me. The Wikipedia article is fairly opaque compared to what I learned in school, but this tidbit stood out: https://en.wikipedia.org/wiki/Automatic_differentiation#Implementation https://en.wikipedia.org/wiki/Automatic_differentiation#Impl... Source code transformation (SCT): the compiler processes source code so that the derivatives are calculated alongside each instruction. Operator overloading (OO): operators are overridden so that derivatives are calculated for numbers and vectors. Based on the state of software these days, I'm guessing that OO (the "bare hands" method) is what's mainstream. It would be far better IMHO to use SCT, since it's a universal solution that doesn't require manually refactoring programs. But stuff like SCT might be considered metaprogramming, which seems to have fallen out of fashion. I grew up with C++ macros and templates, so I feel that this is somewhat tragic, although readability and collaboration are much better today. A modern example might be something like aspect-oriented programming (AOP): https://en.wikipedia.org/wiki/Aspect-oriented_programming https://en.wikipedia.org/wiki/Aspect-oriented_programming I once used the AOP library AspectJ to trace a Java app's execution, since Java made the (unfortunate) choice to focus on objects rather than functions, which makes it generally a poor fit for data processing, due to its high use of mutable state within objects (mutable state is what limits most object-oriented projects to around 1 million lines). Meaning that I couldn't remember the program's context as I was stepping through it, and had to analyze traces instead. AspectJ allows one to hook into the code without modifying it, sort of like a debugger, so that stuff like function calls and variable mutations can be watched: https://en.wikipedia.org/wiki/AspectJ https://en.wikipedia.org/wiki/AspectJ https://www.eclipse.org/aspectj/doc/released/progguide/index.html https://www.eclipse.org/aspectj/doc/released/progguide/index... Looks like this might still be an open problem in Python: https://stackoverflow.com/questions/12356713/aspect-oriented-programming-aop-in-python https://stackoverflow.com/questions/12356713/aspect-oriented... https://docs.spring.io/spring-python/1.2.x/sphinx/html/aop.html#aspect-oriented-programming https://docs.spring.io/spring-python/1.2.x/sphinx/html/aop.h... But it seems like SQL would be a good candidate for AOP: https://stackoverflow.com/questions/12271588/aspect-oriented-sql-server https://stackoverflow.com/questions/12271588/aspect-oriented... https://technology.amis.nl/it/aspect-oriented-programming-aop-in-plsql-is-it-hot-is-it-cool-or-utterly-useless/ https://technology.amis.nl/it/aspect-oriented-programming-ao... If we had that, maybe we could automatically generate derivatives for the set operations. Then either access them as variables in stored procedures, or possibly as something like views or via metadata stored somewhere like MySQL's INFORMATION_SCHEMA. I don't really know, but maybe these breadcrumbs could be helpful.

teruakohatu 4y ago

> This is great! Moving away from the proprietary nature of GPUs and complex math gatekeeping should help democratize AI. I understand the sentiment but you can run ML on plain CPUs, in fact it is easier to do so, you just don't get the massive speed benefits of GPUs. Hugging face's public models are all run on CPUs to save costs. > Right now AI algorithms kind of look imperative and stateful to me, like state machines. The maths is not stateful or imperative, only our implementation. Much like how functional languages are executed on a stateful CPU. Research in parallelization is ongoing.

mr_toad 4y ago

Most ML algorithms are numerical approximations, so yes, they look procedural. You can try and find analytical solutions, but except for the simplest models they’re intractable.