3 ms·

I'd be interested to see a skeleton of your Haskell ETL scripts. I've been looking to clojure to clean up my current processes and I sure would like to see how

by ghotli 17y ago

I'd be interested to see a skeleton of your Haskell ETL scripts. I've been looking to clojure to clean up my current processes and I sure would like to see how you approach it in haskell.

jrockway 17y ago
Nothing special. Everything happens in some instance of MonadError and MonadState; error for aborting when there is some problem, and state for logging. (Why not writer for logging? After a week of playing with various levels of strictness over Writer, with the help of #haskell, I couldn't get any variant of writer to not use all my stack space. Dropping in State fixed the problem. I know why State fixed the problem, but I don't know why I couldn't make Writer strict enough. Oh well.) Unlike with most logging, I don't really care to see the results as the process is running; the log is just a list of failed sanity checks that may indicate problems with the data. It just gets printed at the end, and I can check out the datasets that are suspicious when I feel like it. Thanks to Catch and some wrappers I have that turn IO errors into Lefts, I know that my process isn't going to die halfway through. My only worry is possible data corruption because some assumption is violated, so if I see anything weird, I make a note and continue. If something is clearly wrong, then that dataset is Left instead of Right, and is skipped. I also have a function that downgrades Lefts to Right + a warning, so I can have a few failures in the ETL run and worry about those after the non-error sets are processed. The actual scripts just read config data from a file or the command-line, and run some function that actually does the work. These are named something like theFooProcess/Main.hs, and are built into executables like "theFooProcess.exe" with Cabal. (I also build the library code as a separate Cabal target, but I don't have any projects that refer to each other yet, so this really just wastes my time during compilation. Heh.) (Oh yeah, did I mention that my processes are all Windows-based because the third-party libraries are Windows-only? Yeah. But GHC handles it great! Emacs + ghci + cabal on Windows is exactly like Emacs + ghci + cabal on UNIX. With Haskell, Windows is not much of a problem... except for the security updates that reboot my machines in the middle of ETL runs :)
- ghotli 17y ago
  Thanks for this. As a man who only has a cursory understanding of Haskell your comment sheds light on how things actually fit together. It also provides a good starting point for expanding my Haskell understanding.

by ghotli 17y ago

I'd be interested to see a skeleton of your Haskell ETL scripts. I've been looking to clojure to clean up my current processes and I sure would like to see how you approach it in haskell.

jrockway 17y ago

Nothing special. Everything happens in some instance of MonadError and MonadState; error for aborting when there is some problem, and state for logging. (Why not writer for logging? After a week of playing with various levels of strictness over Writer, with the help of #haskell, I couldn't get any variant of writer to not use all my stack space. Dropping in State fixed the problem. I know why State fixed the problem, but I don't know why I couldn't make Writer strict enough. Oh well.) Unlike with most logging, I don't really care to see the results as the process is running; the log is just a list of failed sanity checks that may indicate problems with the data. It just gets printed at the end, and I can check out the datasets that are suspicious when I feel like it. Thanks to Catch and some wrappers I have that turn IO errors into Lefts, I know that my process isn't going to die halfway through. My only worry is possible data corruption because some assumption is violated, so if I see anything weird, I make a note and continue. If something is clearly wrong, then that dataset is Left instead of Right, and is skipped. I also have a function that downgrades Lefts to Right + a warning, so I can have a few failures in the ETL run and worry about those after the non-error sets are processed. The actual scripts just read config data from a file or the command-line, and run some function that actually does the work. These are named something like theFooProcess/Main.hs, and are built into executables like "theFooProcess.exe" with Cabal. (I also build the library code as a separate Cabal target, but I don't have any projects that refer to each other yet, so this really just wastes my time during compilation. Heh.) (Oh yeah, did I mention that my processes are all Windows-based because the third-party libraries are Windows-only? Yeah. But GHC handles it great! Emacs + ghci + cabal on Windows is exactly like Emacs + ghci + cabal on UNIX. With Haskell, Windows is not much of a problem... except for the security updates that reboot my machines in the middle of ETL runs :)

ghotli 17y ago
Thanks for this. As a man who only has a cursory understanding of Haskell your comment sheds light on how things actually fit together. It also provides a good starting point for expanding my Haskell understanding.