4 ms·
I guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important th
by hermitcrab 7mo ago
I guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important thing is creating a set of composable operations/primitive that are meaningful and useful to your end user. We have ended up 73 distinct transforms in Easy Data Transform. Sure they overlap to an extent, but feel they are at the right semantic level for our users, who are not category theorists.
- mrlongroots 7mo agoAlgebras are also nice for implementations. If you can decompose a domain into a few algebraic primitives you can write nice SIMD/CUDA kernels for those primitives. To your point, I wonder if the 73 distinct transforms were just different defaults/usability wrappers over these. And you may also get into situations where kernels can be fused together or other batching constraints enable optimizations that nice algebraic primitives don't capture. But that's just systems---theory is useful in helping rethink API bloats and keeping us all honest.
- hermitcrab 7mo agoThey are effectively highly level wrappers over the most primitive operations. High enough level that they can be used from a GUI, rather than code. It is a balance. Too few transforms and they become to low level for my users. Too many and you struggle to find the transform you want.
- jimbokun 7mo agoYou don’t have to limit the transforms you offer users to just the core ones. But for your own sanity you can implement the none core ones in terms of the core ones.
- whattheheckheck 7mo agoHave you heard of the book Mathematics for Big data https://github.com/Accla/d4m https://github.com/Accla/d4m He says himself the ideas are more important than the software package
- hermitcrab 7mo agoD4M seems to be a library, not a book. Or am I missing something?
- esafak 7mo agohttps://mitpress.mit.edu/9780262038393/mathematics-of-big-data/ https://mitpress.mit.edu/9780262038393/mathematics-of-big-da...
- tikhonj 7mo agoYou can have both: you start with a small, mathematically inspired algebraic core, then you express the higher-level more user-friendly operations in terms of the algebraic core. As long as your core primitives are well designed (easier said than done!), this accomplishes two things: it makes your implementation simpler, and it helps guide and constrain your user-facing design. This latter aspect is a bit unintuitive (why would you want more constraints to work around?), but I've seen it lead to much better interface designs in multiple projects. By forcing yourself to express user-level affordances in terms of a small conceptual core, you end up with a user design that is more internally consistent and composable.
- jimbokun 7mo agoFor one thing it gives users of your library fewer concepts to learn.
- hermitcrab 7mo agoYes, but fewer concepts may not be simpler in practice. E.g. assembler is simpler than C++, but I wouldn't want to write a big program in assembler.