6 ms·
The part of the ETL you'd want to "crystallize" is just the transformation from raw data to feature vector. Beyond that, you already "change in ETL" when you e
by agibsonccc 9y ago
The part of the ETL you'd want to "crystallize" is just the transformation from raw data to feature vector.
Beyond that, you already "change in ETL" when you experiment. I'm not sure how this changes anything.
Out of nowhere you've sprinkled in "multi gpu support" which I'm not sure is relevant here. Do you mean as part of training?
We handle the gpu bits for you. All you do is define your transform logic, it runs on one of our backends like spark and then when you go to allocate a tensor boom gpu.
There's no special compilation or process needed to make this happen.
Dl4j supports multi gpu training out of the box. All you need to do is uses our parallelwrapper module.
We can also do distributed training with gpus on spark as well (yes this includes cudnn).
You've also for some reason decided to attach an open ended coding library where you can do whatever you want to a database?
Baked in ML in the database servers is already notoriously bad. The whole point of what we're doing is to provide a middle ground.
Data engineers have to do this anyways.
- deepGem 9y agoAdam, sorry guess I was not clear. What I said was that the MS Sql server + R strategy of baking in ML in PLSQL is appalling and doing something like what you are doing with DL4J is perhaps the right approach. The multi GPU support rant was for SQL Server + R, not DL4J. "You've also for some reason decided to attach an open ended coding library where you can do whatever you want to a database?" I didn't catch this part.
- agibsonccc 9y agoRight so I think we were agreeing that how database servers bake in the ML is a bit weird and not the way to go. The "open ended coding library" is datavec here. Comparing them here is only really semi valid. The processes there are definitely brittle.
- stolsvik 9y agoI think he agreed with you..!
- agibsonccc 9y agoYeah sorry about that..I was a bit confused on how to read it. There was a lot mixed in there. I think we're clear now :D.