3 ms·
Seems like a great idea. When you look at many ML frameworks half the code and learning overhead is data schlepping code and table like structures that "reinven
by zmmmmm 5y ago
Seems like a great idea. When you look at many ML frameworks half the code and learning overhead is data schlepping code and table like structures that "reinvent" the schema that already exists inside a database. Not to mention, there can be security concerns from dumping large amounts of data out of the primary store (how are you going to GDPR delete that stuff later on?). So why not use it natively where the data already is?
For anything substantive it seems like a bad idea to run this on your primary store since the last thing you want to do is eat up precious CPU and RAM needed by your OLTP database. But in a data warehouse or similar replicated setup, it seems like a really neat idea.
- jiocrag 5y agoYeah — seems like all you need is Snowflake-esque separation of storage and compute and bob’s your uncle.
- coatue 5y agoLooks like a good fit to run on Hydras.io (YC W22) the Postgres data warehouse that separates the PG query layer from compute / storage. Disclaimer, I'm the co-founder.
- montanalow 5y agoNice! Feel free to file any issues on Github if something doesn't work. I'd love to understand more about the internals of the hydra architecture.
- mr_toad 5y agoIt looks like this is running Python to do the actual ML? There’s nothing stopping you from reading database table structures directly into memory in Python or R now. You don’t need an intermediate data store. I agree that running training on production instances would be a bad idea. First, you need to denormalise data for ML, and secondly you typically don’t want your training data to be constantly changing.
- montanalow 5y agoOutside of academia, you need to be able to constantly adapt to changing training data. That is a constant pitfall for many projects that try to transition from offline to online.
- mr_toad 5y agoOutside academia you need to do a lot of data cleaning and feature engineering, and if you’re constantly changing the model as well as the data you’ll never be able to attribute changes to either. Chances are a DBA wouldn’t consider letting you do data engineering in a live production database anyway, so this really is all academic.
- chaps 5y agoHaven't played with it yet, but this tooling is exactly what I'm looking for for my SQL-first-python-second "data lake" workflows. It doesn't exactly matter to me if data in a table gets rewritten in-place since it's built on a model meant to be destroyed at any time from an original source file - generally data from FOIA or gov sources that's write-once. In a way, having to pull things out of the DB and into python is something that requires change attribution on its own since it's a conversion between abstractions which I tend to think of as a lossy process (even if it isn't). This sort of tooling keeps the abstractions localized, so it's much easier to maintain a mental model of models of changes.
- code_biologist 5y agoOutside academia you need to do a lot of data cleaning and feature engineering, and if you’re constantly changing the model as well as the data you’ll never be able to attribute changes to either. I get the concern but sometimes I really just do want a black box regressor or classifier. Model performance monitoring is important, but I don't care about attribution. Chances are a DBA wouldn’t consider letting you do data engineering in a live production database anyway, so this really is all academic. Maybe it isn't data engineering, but I'm curious what you'd call using Google's BigQuery ML? "BigQuery ML enables users to create and execute machine learning models in BigQuery by using standard SQL queries." I haven't used it in production, but I'd use it in a heartbeat if I was on BigQuery.