3 ms·
I was the main data science engineer at one of my previous companies. We used tools like airflow for running python scripts to import data, clean/transform it,
by remmargorp64 6y ago
I was the main data science engineer at one of my previous companies. We used tools like airflow for running python scripts to import data, clean/transform it, train models, and even test various models against datasets. We also used Azure for similar things.
It's easy to do "dev ops" for machine learning. Basically, just automate everything and implement gatekeeping mechanisms along with active monitoring.
It's true, though. I had to cobble together a lot of custom things at the time, but it wasn't that hard to do.
- nik_s 6y agoI'm the CTO at a data science company, and this has been my experience too. I've been lucky enough to have quite a few engineers go from zero practical experience to being able to train and deploy complex ml solutions, and the most successful solutions have always involved a combination of just a couple of tools: - airflow and/or celery for running data extraction and transformation jobs - pandas and numpy for data wrangling - sklearn, xgboost, lightgbm, pytorch or tensorflow for training/inference - flask or Django to serve results It's a handful of technologies, but they're (generally) mature, battle tested, and well documented.
- softwaredoug 6y agoGenerally true. Though I will say that in larger orgs, you will occasionally get someone doing some ML they read a paper on that's not well supported by major tooling. I mean it's the same trend chasing you see in engineering...