4 ms·

Great post, for this pipeline I would have probably used a makefile for the batch pipeline instead of airflow just to keep it simple. I would also make my sink

Great post, for this pipeline I would have probably used a makefile for the batch pipeline instead of airflow just to keep it simple. I would also make my sink a SQLite database so that you can easily search through it with a web interface using datasette. For the places where bash was used I would just use python and any cli tools you want to call I just use subprocess. It’s much simpler and I can run the scripts in a repl and execute cells in Jupyter or just normal pycharm so its quick and interactive. Love that you included something on building a data dictionary, I am honestly guilty of in the past not including a good data dictionary for the source data. I would just leave in the output of df.describe() or df.info() at the top of the jupyter notebook where you restructure the source data before processing it. I now think you should include and save as a CSV a data dictionary of the source data and the final data as it’s more maintainable or at least leave a comment in your script. Otherwise everything else is pretty similar to what I would do, I just went to my google takeout and apparently all my google play data and songs are gone so I guess I can’t try this myself…

wodenokoto 5y ago
My first thought was also "why not SQLite?", but the author says he already has a MariaDB running. So, using the tools you know. I guess it is the same for make vs airflow. I had no idea they could be used interchangeably for single machine workloads. While I've seen datasette mentioned a lot of places, I still don't really know what it is, but if it makes exploring sqlite databases easy, I should give it a try!
- faizshah 5y ago
  The makefile data pipeline is definitely an underrated technique a couple great HN comments on this technique: - https://news.ycombinator.com/item?id=22283368 https://news.ycombinator.com/item?id=22283368 - https://news.ycombinator.com/item?id=18896204 https://news.ycombinator.com/item?id=18896204 I personally learned it from bioinformaticians theres great coverage of this and other command line data skills in this book: https://www.oreilly.com/library/view/bioinformatics-data-skills/9781449367480/ https://www.oreilly.com/library/view/bioinformatics-data-ski... The SQLite, pandas, bash, make stack for quick data science projects is a great and maintainable one that doesn’t require too much specialized knowledge.
  - diarrhea 5y ago
    Great read on Jupyter Notebooks. I've always had a strong dislike for them as well - just feel fuzzy, dirty, often in weird state - and the user put it into words nicely. I might just be bad with Jupyter, but I've been willing to mold these notebooks into saner forms (modularization, keeping state, deterministic cell order/content/execution) and came up empty. At some point you got to ask if it's not the tool's fault. Modularization especially is infuriating -- it seems there is no reliable way to rerun `import myownmodule` when work is done on the latter. It won't detect changed code. A kernel restart is necessary (`reload` and friends didn't help), losing all state. It's punishment for saner software engineering practices.
    - faizshah 5y ago
      You are absolutely right I have taken to referring to that as “notebook code.” I love jupyter because its a great interactive programming environment that speeds up building scripts. But jupyter notebooks always end up being…notebooks. They are records of past work and thoughts that I had but I wouldn't present my paper notes to a coworker as finished documentation. I always end up extracting those jupyter snippets into runnable scripts and then I end up doing a lot of the work all over again as I parameterize things. I have taken to using pycharm scientific mode and adding “cells” into my scripts: https://blog.jetbrains.com/pycharm/2018/04/pycharm-scientific-mode-with-code-cells/ https://blog.jetbrains.com/pycharm/2018/04/pycharm-scientifi... You get the interactivity of jupyter with the cells, the scientific view for plots and data, but you’re also writing a real script so you can still maintain some sanity in your code and good git history (and hopefully add some tests). Then I combine these scripts together into a makefile and voila quick, easy and maintainable scripts. If you just stick to make, python, pandas and bash any programmer can modify your pipeline without needing specialist skills. I have written scrapers + ETL in <1.5 hour (journalist needed some data quick) with this method. God I wish I still worked in data science I’m currently a front end dev and don’t get to do any of this at work. It’s such fun and offers so much room for creativity and problem solving.
      - wodenokoto 5y ago
- otter-in-a-suit 5y ago
  I use SQLite quite a bit and I think it's fantastic, especially since Hipp seems to be a wonderful guy (judging by The Changelog podcast). And you're right - use the tools you know and have running. I have all sorts of schemas and tables on that old instance, since I tend to use it if I need "anything SQL" - when I'm at home, is it as easy as using SQLite. My latest article used Trino and Hadoop-adjacent stuff... while fascinating it its own right, sometimes it's nice to just say "jdcb:// ..." :)

wodenokoto 5y ago

My first thought was also "why not SQLite?", but the author says he already has a MariaDB running. So, using the tools you know. I guess it is the same for make vs airflow. I had no idea they could be used interchangeably for single machine workloads. While I've seen datasette mentioned a lot of places, I still don't really know what it is, but if it makes exploring sqlite databases easy, I should give it a try!

faizshah 5y ago
The makefile data pipeline is definitely an underrated technique a couple great HN comments on this technique: - https://news.ycombinator.com/item?id=22283368 https://news.ycombinator.com/item?id=22283368 - https://news.ycombinator.com/item?id=18896204 https://news.ycombinator.com/item?id=18896204 I personally learned it from bioinformaticians theres great coverage of this and other command line data skills in this book: https://www.oreilly.com/library/view/bioinformatics-data-skills/9781449367480/ https://www.oreilly.com/library/view/bioinformatics-data-ski... The SQLite, pandas, bash, make stack for quick data science projects is a great and maintainable one that doesn’t require too much specialized knowledge.
- diarrhea 5y ago
  Great read on Jupyter Notebooks. I've always had a strong dislike for them as well - just feel fuzzy, dirty, often in weird state - and the user put it into words nicely. I might just be bad with Jupyter, but I've been willing to mold these notebooks into saner forms (modularization, keeping state, deterministic cell order/content/execution) and came up empty. At some point you got to ask if it's not the tool's fault. Modularization especially is infuriating -- it seems there is no reliable way to rerun `import myownmodule` when work is done on the latter. It won't detect changed code. A kernel restart is necessary (`reload` and friends didn't help), losing all state. It's punishment for saner software engineering practices.
  - faizshah 5y ago
    You are absolutely right I have taken to referring to that as “notebook code.” I love jupyter because its a great interactive programming environment that speeds up building scripts. But jupyter notebooks always end up being…notebooks. They are records of past work and thoughts that I had but I wouldn't present my paper notes to a coworker as finished documentation. I always end up extracting those jupyter snippets into runnable scripts and then I end up doing a lot of the work all over again as I parameterize things. I have taken to using pycharm scientific mode and adding “cells” into my scripts: https://blog.jetbrains.com/pycharm/2018/04/pycharm-scientific-mode-with-code-cells/ https://blog.jetbrains.com/pycharm/2018/04/pycharm-scientifi... You get the interactivity of jupyter with the cells, the scientific view for plots and data, but you’re also writing a real script so you can still maintain some sanity in your code and good git history (and hopefully add some tests). Then I combine these scripts together into a makefile and voila quick, easy and maintainable scripts. If you just stick to make, python, pandas and bash any programmer can modify your pipeline without needing specialist skills. I have written scrapers + ETL in <1.5 hour (journalist needed some data quick) with this method. God I wish I still worked in data science I’m currently a front end dev and don’t get to do any of this at work. It’s such fun and offers so much room for creativity and problem solving.
    - wodenokoto 5y ago
otter-in-a-suit 5y ago
I use SQLite quite a bit and I think it's fantastic, especially since Hipp seems to be a wonderful guy (judging by The Changelog podcast). And you're right - use the tools you know and have running. I have all sorts of schemas and tables on that old instance, since I tend to use it if I need "anything SQL" - when I'm at home, is it as easy as using SQLite. My latest article used Trino and Hadoop-adjacent stuff... while fascinating it its own right, sometimes it's nice to just say "jdcb:// ..." :)