5 ms·
Is it possible to move data around between languages / engines? E.g. would it be possible to run a query against Red Shift (step 1), then cleanse it in Python
by siganakis 13y ago
Is it possible to move data around between languages / engines?
E.g. would it be possible to run a query against Red Shift (step 1), then cleanse it in Python (step 2), run R scripts over the Python output (step 3) then dump the results back to Red Shift (step 4)?
If I then decide I need to change the Red Shift query (step 1), can I then re-run the whole pipeline?
Munging data between different tools is what I seem to spend most of my time doing, so anything that helps that would be a big productivity boost.
- apatil 13y agoAt the moment, not really. In your scenario, you could have a Python dashboard launch a Redshift dashboard with startup code that runs the initial query, then clean the result, then launch an R dashboard and pass the clean data to it either over the shared filesystem or a messaging system such as 0MQ or Redis, then save the results to S3 for consumption by Redshift. You're probably looking for something smoother, though. We definitely intend to have a good solution for workflows like you describe in the future.
- ironchef 13y agoI think you may want to think about treating the analyses like an ETL pipeline (think dependent jobs in chronos or some such) using some intermediary (S3, whatevs). That would probably be useful to a lot of analysts.