5 ms·

Much of his thrust seems to boil down to being really offended by the cost of safety of allowing concurrent writes while reads are going on. When dealing with

Much of his thrust seems to boil down to being really offended by the cost of safety of allowing concurrent writes while reads are going on. When dealing with analysis workloads, his line of thinking can yield fast runtimes and doesn't bring that much of a programmer burden. However, it's killingly complex when it comes to live mutating business databases that need to scale. Even many job-based analysis workloads have a lot of concurrent mutation in trying to efficiently figure out what's been done and what needs to be done next when the actual workflow is not a straight pipeline. I do agree that 'for' loops should be replaced by broader declarations of what needs to be done, instead of these explicit instructions on how to iterate through the data.

brudgers 11y ago
I take the author's viewpoint as: when the data is actually big, big data operations are for all practical purposes going to be run in batch mode - it's just hard to squeeze enough new data through the pipeline in ten minutes to change the statistical analysis of many terabytes of big data...if the data is so unpredictable that a snapshot isn't going to be meaningful then slowing operations down to allow concurrent writes isn't going to provide better analysis and what is really needed is not big data but some sort of near real-time stream processing. All this comes with the assumption that much of what people consider big data isn't really big data.
wyager 11y ago
Could you expand on what you mean about for loops? In the functional world, there are all sorts of precise constructs to operate on lists (maps, folds, unfolds, scans, etc.), and many versions of these that support concurrency. That's what I imagine when you say "declarations of what needs to be done".
- white-flame 11y ago
  The standard "for" structure is necessarily ordered. Either an index is incremented/decremented, or an iterator object's next() comes in. Since there are side effects that can affect either the iterator or the index, and dependent side effects between loop body runs, obeying a "for" loop means strictly ordered (unless the iterator interface abstracts something else), serialized execution of the loop body. Even maps, folds, etc are often defined in terms of what order they process data. This is useful for early exit, and mutable state dependency between iterations. But again, this demands serialized, specifically ordered processing of elements. Lists themselves are ordered structures. Their generalized interface does not assume it's just a bag of items easily thrown around in sub-parts, especially if the linked list nodes are to be modified. Of course, the above applies to impure functional languages. Purely functional of course has far more freedoms, but even the Lisps and such ultimately demand sequential processing for most of their fundamental list operations.
  - taeric 11y ago
    Much of the functional world is concerned with defining structures that are heavily linked to themselves. Tagged data is essentially the norm. So, it isn't just that maps are defined in terms of car/cdr, it is that the list itself is defined in terms of car/cdr. The "aha" moment is when you can see that an array of items in c can efficiently represent a tree structure where the parent/child relationships are implied, not tagged. The performance gains from this are impressive.
    - reagency 11y ago
      Functional languages have arrays.
      - pjmlp 11y ago
        Pssst, let the developers that don't open their eyes beyond C and C++ think their languages are the only ones with such features. :) Every time I see a C or C++ talk discussing value types or arrays, I get to think those guys never saw other programming languages, e.g. Algol/Wirth family. Another example of the Lady Gaga culture, instead of learning about the world of programming languages.
      - taeric 11y ago
        I did not mean to claim that they do not. My claim is that "much of the functional..." So, referring to things like cons lists is easy, though that one is old. Look at the multitude of pointers in any of the purely functional data structures, though. There is a lot of overhead in building up these structures. Look into the Scala (and clojure?) vector sometime. Literally a 32-ary tree. Makes the algorithm somewhat pleasant to look at, but are we really shocked when something using a simple array can beat it in performance? Now, it is a perfectly fair and valid point that optimizing development time may be preferred. We are allowed contradictary valid points in situations that are trade off defined.
        justthistime_ 11y ago
        I think that no-one will argue that an imperative, transient data-structure will be faster than a functional, persistent data-structure if you use it imperatively. If you use persistent data-structures the way they are meant to be used, they can and will be faster than a simple array. Both have valid use-cases, but they are often distinct.

brudgers 11y ago

I take the author's viewpoint as: when the data is actually big, big data operations are for all practical purposes going to be run in batch mode - it's just hard to squeeze enough new data through the pipeline in ten minutes to change the statistical analysis of many terabytes of big data...if the data is so unpredictable that a snapshot isn't going to be meaningful then slowing operations down to allow concurrent writes isn't going to provide better analysis and what is really needed is not big data but some sort of near real-time stream processing. All this comes with the assumption that much of what people consider big data isn't really big data.

wyager 11y ago

Could you expand on what you mean about for loops? In the functional world, there are all sorts of precise constructs to operate on lists (maps, folds, unfolds, scans, etc.), and many versions of these that support concurrency. That's what I imagine when you say "declarations of what needs to be done".

white-flame 11y ago
The standard "for" structure is necessarily ordered. Either an index is incremented/decremented, or an iterator object's next() comes in. Since there are side effects that can affect either the iterator or the index, and dependent side effects between loop body runs, obeying a "for" loop means strictly ordered (unless the iterator interface abstracts something else), serialized execution of the loop body. Even maps, folds, etc are often defined in terms of what order they process data. This is useful for early exit, and mutable state dependency between iterations. But again, this demands serialized, specifically ordered processing of elements. Lists themselves are ordered structures. Their generalized interface does not assume it's just a bag of items easily thrown around in sub-parts, especially if the linked list nodes are to be modified. Of course, the above applies to impure functional languages. Purely functional of course has far more freedoms, but even the Lisps and such ultimately demand sequential processing for most of their fundamental list operations.
- taeric 11y ago
  Much of the functional world is concerned with defining structures that are heavily linked to themselves. Tagged data is essentially the norm. So, it isn't just that maps are defined in terms of car/cdr, it is that the list itself is defined in terms of car/cdr. The "aha" moment is when you can see that an array of items in c can efficiently represent a tree structure where the parent/child relationships are implied, not tagged. The performance gains from this are impressive.
  - reagency 11y ago
    Functional languages have arrays.
    - pjmlp 11y ago
      Pssst, let the developers that don't open their eyes beyond C and C++ think their languages are the only ones with such features. :) Every time I see a C or C++ talk discussing value types or arrays, I get to think those guys never saw other programming languages, e.g. Algol/Wirth family. Another example of the Lady Gaga culture, instead of learning about the world of programming languages.
    - taeric 11y ago
      I did not mean to claim that they do not. My claim is that "much of the functional..." So, referring to things like cons lists is easy, though that one is old. Look at the multitude of pointers in any of the purely functional data structures, though. There is a lot of overhead in building up these structures. Look into the Scala (and clojure?) vector sometime. Literally a 32-ary tree. Makes the algorithm somewhat pleasant to look at, but are we really shocked when something using a simple array can beat it in performance? Now, it is a perfectly fair and valid point that optimizing development time may be preferred. We are allowed contradictary valid points in situations that are trade off defined.
      - justthistime_ 11y ago
        I think that no-one will argue that an imperative, transient data-structure will be faster than a functional, persistent data-structure if you use it imperatively. If you use persistent data-structures the way they are meant to be used, they can and will be faster than a simple array. Both have valid use-cases, but they are often distinct.