2 ms·
It's a good idea to write a book about a data-oriented development style (I'm working on a methodology, a way of working in data-intensive projects called Data2
by jll29 6y ago
It's a good idea to write a book about a data-oriented development style (I'm working on a methodology, a way of working in data-intensive projects called Data2Value).
However, JavaScript or Clojure are not ideal for demonstrating this methodology in the sense that industrial applications will more likely be built in C++, Java or Python. For example C++ and Java support Apache UIMA, which is an industry standard for data-oriented systems. UIMA (originally developed at IBM before open sourced, and use in their Watson system) manages data as immutable objects (e.g. text, videos) that are enriched with annotations (e.g. syntax graphs, topical tags, subtitles...).
Functional designs are often well-suited to data flow related processing, whereas in OOP, you end up with a pipeline object and various DataStream objects that it inputs and outputs.
In my experience, data-intensive systems often need to:
- cater for distributed processing due to large-scale (which calls for Apache Spark, and then PySpark or Scala);
- compute-intensive work like machine learning, which may require GPUs or other bespoke hardware (Tensorflow supports GPUs, Google's cloud has TPUs); and
- special purpose data structures (e.g. Bloom filters, huge persistent graphs, R* trees...) specific to the nature of the processing (this latter point, I guess, contradicts to the author's claims).
- dragandj 6y agoFYI Clojure supports GPU processing since 5 years ago. https://github.com/uncomplicate https://github.com/uncomplicate
- thom 6y agoData oriented programming is orthogonal to the type of biggish data tools you're talking about, although I agree that in the present day, the latter is actually a more interesting subject for methodological discussion.