3 ms·
Yes. I've worked a couple of jobs that had a lakehouse with transforms written in notebooks. Usually sitting on a data lake, with processing handled by a Spark
by z3c0 5y ago
Yes. I've worked a couple of jobs that had a lakehouse with transforms written in notebooks. Usually sitting on a data lake, with processing handled by a Spark cluster. I've much preferred it to the ETL tools of yore, even if Spark can throw some weird curveballs.
That's on the warehousing side, however. On the reporting side, I've only seen notebooks catch on with small research teams, a la Jupyter Lab or something similar. Most of the hype today is around self-service BI tools, not tools that require hosting and a working knowledge of Python, Scala, or whatever.