5 ms·

One struggle I’ve had with wrapping my head around using FP and lisp like languages for a “real world” system is handling something like logging. Ideally that’s

One struggle I’ve had with wrapping my head around using FP and lisp like languages for a “real world” system is handling something like logging. Ideally that’s handled outside of the function that might be doing a data transformation but how do you build a lot message that outputs information about old and new values without contamination of your “pure” transducer? You could I guess have a “before” step that iterates your data stream and logs all the before values, and then an “after” step that iterates after and logs all the after and get something like: ``` (->> (map log-before data) (map transform-data) (map log-after-data)) ``` But doesn’t that cause you to iterate your data 2x more times than you “need” to and also split your logging into 2x as many statements (and thus 2x as much IO)

trealira 2y ago
So, do you mean like you have some big array, and you want to do something like this? (Below is not a real programming language.) for i in 0 to arr.len() { new_val = f(arr[i]); log("Changing {arr[i]} to {new_val}.\n"); arr[i] = new_val; } I haven't used Haskell in a long time, but here's a kind of pure way you might do it in that language, which I got after tinkering in the GHCi REPL for a bit. In Haskell, since you want to separate IO from pure logic as much as possible, functions that would do logging return instead a tuple of the log to print at the end, and the pure value. But because that's annoying and would require rewriting a lot of code manipulating tuples, there's a monad called the Writer monad which does it for you, and you extract it at the end with the `runWriter` function, which gives you back the tuple after you're done doing the computation you want to log. You shouldn't use Text or String as the log type, because using the Writer involves appending a lot of strings, which is really inefficient. You should use a Text Builder, because it's efficient to append Builder types together, and because they become Text at the end, which is the string type you're supposed to use for Unicode text in Haskell. So, this is it: import qualified Data.Text.Lazy as T import qualified Data.Text.Lazy.Builder as B import qualified Data.Text.Lazy.IO as TIO import Control.Monad.Writer mapWithLog :: (Traversable t, Show a, Show b) => (a -> b) -> t a -> Writer B.Builder (t b) mapWithLog f = mapM helper where helper x = do let x' = f x tell (make x <> B.fromString " becomes " <> make x' <> B.fromString ". ") pure x' make x = B.fromString (show x) theActualIOFunction list = do let (newList, logBuilder) = runWriter (mapWithLog negate list) let log = B.toLazyText logBuilder TIO.putStrLn log -- do something with the new list... So "theActualIOFunction [1,2,3]" would print: 1 becomes -1. 2 becomes -2. 3 becomes -3. And then it does something with the new list, which has been negated now.
- tpmoney 2y ago
  Does this imply that the logging doesn't happen until all the items have been processed though? If I'm processing a list of 10M items, I have to store up 10M*${num log statements} messages until the whole thing is done?
  - trealira 2y ago
    Yes, it does imply that, except since Haskell is lazy, you'll be holding onto a thunk until the IO function is evaluated, so you won't have a list of 10 million messages in memory up until you're printing, and even then, lists are lazy, too, so you won't ever have all entries of the list in memory at once, either, because list entries are also thunks, and once you're done printing it, you'll throw it away and evaluate a thunk to create the next cons cell in the list, and then you evaluate another thunk to get the item that the next cell points to and print it. Everything is implicitly interleaved. In the case above, where I constructed a really long string, it depends on the type of string you use. I used lazy Text, which is internally a lazy list of strict chunks of text, so that won't ever have to be in memory all at once to print it, but if I had used the strict version of Text, then it would have just been a really long string that had to be evaluated and loaded into memory all at once before being printed.
    - tpmoney 2y ago
      Sorry, I lack a lot of context for Haskell and its terms (my experience with FP is limited largely to forays into Lisp / Clojure), but if I'm understanding right, you're saying because the collection is being lazily evaluated, the whole process up to the point of re-combining the items back into their final collection will be happening in a parallel manner, so as long as the IO is ordered to occur before that final collection, it will occur while other items are still being processed? So if the program were running and the system crashed half way through, we'd still have logs for everything that was processed up to the point it crashed (modulo anything that was inflight at the time of the crash)? What happens if there are multiple steps with logging at each point? Say perhaps a program where we want to: 1) Read records from a file 2) Apply some transformations and log 3) Use the resulting transformations as keys to look up data from a database and log that interaction 4) Use the result from the database to transform the data further if the lookup returned a result, or drop the result otherwise (and log) 5) Write the result of the final transform to a different file and do all of the above while reporting progress information to the user. And to be very clear, I'm genuinely curious and looking to learn so if I'm asking too much from your personal time, or your own understanding, or the answer is "that's a task that FP just isn't well suited for" those answers are acceptable to me.

trealira 2y ago

So, do you mean like you have some big array, and you want to do something like this? (Below is not a real programming language.) for i in 0 to arr.len() { new_val = f(arr[i]); log("Changing {arr[i]} to {new_val}.\n"); arr[i] = new_val; } I haven't used Haskell in a long time, but here's a kind of pure way you might do it in that language, which I got after tinkering in the GHCi REPL for a bit. In Haskell, since you want to separate IO from pure logic as much as possible, functions that would do logging return instead a tuple of the log to print at the end, and the pure value. But because that's annoying and would require rewriting a lot of code manipulating tuples, there's a monad called the Writer monad which does it for you, and you extract it at the end with the `runWriter` function, which gives you back the tuple after you're done doing the computation you want to log. You shouldn't use Text or String as the log type, because using the Writer involves appending a lot of strings, which is really inefficient. You should use a Text Builder, because it's efficient to append Builder types together, and because they become Text at the end, which is the string type you're supposed to use for Unicode text in Haskell. So, this is it: import qualified Data.Text.Lazy as T import qualified Data.Text.Lazy.Builder as B import qualified Data.Text.Lazy.IO as TIO import Control.Monad.Writer mapWithLog :: (Traversable t, Show a, Show b) => (a -> b) -> t a -> Writer B.Builder (t b) mapWithLog f = mapM helper where helper x = do let x' = f x tell (make x <> B.fromString " becomes " <> make x' <> B.fromString ". ") pure x' make x = B.fromString (show x) theActualIOFunction list = do let (newList, logBuilder) = runWriter (mapWithLog negate list) let log = B.toLazyText logBuilder TIO.putStrLn log -- do something with the new list... So "theActualIOFunction [1,2,3]" would print: 1 becomes -1. 2 becomes -2. 3 becomes -3. And then it does something with the new list, which has been negated now.

tpmoney 2y ago
Does this imply that the logging doesn't happen until all the items have been processed though? If I'm processing a list of 10M items, I have to store up 10M*${num log statements} messages until the whole thing is done?
- trealira 2y ago
  Yes, it does imply that, except since Haskell is lazy, you'll be holding onto a thunk until the IO function is evaluated, so you won't have a list of 10 million messages in memory up until you're printing, and even then, lists are lazy, too, so you won't ever have all entries of the list in memory at once, either, because list entries are also thunks, and once you're done printing it, you'll throw it away and evaluate a thunk to create the next cons cell in the list, and then you evaluate another thunk to get the item that the next cell points to and print it. Everything is implicitly interleaved. In the case above, where I constructed a really long string, it depends on the type of string you use. I used lazy Text, which is internally a lazy list of strict chunks of text, so that won't ever have to be in memory all at once to print it, but if I had used the strict version of Text, then it would have just been a really long string that had to be evaluated and loaded into memory all at once before being printed.
  - tpmoney 2y ago
    Sorry, I lack a lot of context for Haskell and its terms (my experience with FP is limited largely to forays into Lisp / Clojure), but if I'm understanding right, you're saying because the collection is being lazily evaluated, the whole process up to the point of re-combining the items back into their final collection will be happening in a parallel manner, so as long as the IO is ordered to occur before that final collection, it will occur while other items are still being processed? So if the program were running and the system crashed half way through, we'd still have logs for everything that was processed up to the point it crashed (modulo anything that was inflight at the time of the crash)? What happens if there are multiple steps with logging at each point? Say perhaps a program where we want to: 1) Read records from a file 2) Apply some transformations and log 3) Use the resulting transformations as keys to look up data from a database and log that interaction 4) Use the result from the database to transform the data further if the lookup returned a result, or drop the result otherwise (and log) 5) Write the result of the final transform to a different file and do all of the above while reporting progress information to the user. And to be very clear, I'm genuinely curious and looking to learn so if I'm asking too much from your personal time, or your own understanding, or the answer is "that's a task that FP just isn't well suited for" those answers are acceptable to me.