4 ms·
Pretty neat! I'm guessing that concord isn't limited to just map/reduce... correct? I think building in integrations to other systems in the stream processing e
by reubano 10y ago
Pretty neat! I'm guessing that concord isn't limited to just map/reduce... correct? I think building in integrations to other systems in the stream processing ecosystem is key. First on the list is to integrate with popular workflow schedulers so that you can design topologies which riko would then parse.
Next up I think would be supporting custom sources/sinks such as Twitter, HDFS, RDMS, etc. What exactly would be involved in "porting" riko to concord and what would the advantages be for doing so?
- agallego 10y agoInteresting, will keep an eye out as you mature it, looks like an awesome DSL for us to fill the runtime. Advantages are: 1. mesos integration, with that comes containerization support, multi tenancy, QoS, proper pipeline supervision, etc 2. Scheduling of pipelines. i.e.: Schedule them on 100 computers. 3. the outputs of your DAG could be consumed by other systems immediately and even written in different programming languages. So your python DSL could be the source to the Scala DSL at some point. so language interop 4. Available KV storage 5. Tracing (Zipkin) - ala Google Dapper. 6. Fast networking - C++ backed runtime is 1 order of magnitude faster than the python one. What would be involved, is not much from what I can tell. Each concord 'operator' is like a networked function. so given a DAG, you could generate many operators internally, or literally write them to a file, i.e.: operator_one.py etc. The code generation or internal scheduling would be the glue that's needed. if you ever become interested, ping me! would love to collab alex@concord.io