3 ms·
I have developed a somewhat similar system for Python / C / C++. I declare my data structures in YAML. From this I generate code for memory allocation; data va
by w_t_payne 9y ago
I have developed a somewhat similar system for Python / C / C++.
I declare my data structures in YAML. From this I generate code for memory allocation; data validation; serialization and de-serialisation routines.
I also declare the topology of the data flow graph in a separate YAML file, decorating each edge with the data type.
Finally, I have a separate YAML file that specifies which computers/processes each node will run on.
Nodes communicate by passing messages. For nodes in the same process, this is simply shared memory (zero overhead). For nodes in different processes that share a memory bus, this is a shared memory queue. For nodes on different machines, the data goes over the network.
Nodes can be python, C or C++.
Very nice. Very declarative. The bare minimum of duplication and boilerplate. Changing from MIL to SIL to HIL is (mostly) just a configuration change. Logging, measurement and replay is built into the framework, as is support for parameter tuning.
Within a single process, the model of computation is synchronous data flow. Across processes, it is asynchronous data flow, but because we have queues with blocking reads and nonblocking writes, we still keep the ability to replay data through the system in a deterministic manner. (Modulo nodes exceeding their time budget).
- corey_moncure 9y agoThink this could be ported to .NET Core? I'd love to take a look at it. I'm building something on similar lines with Redis and Protobuf, but I'm in over my head a little bit
- w_t_payne 9y agoIt is mostly written in Python: Most of the work is done at 'compile-time' -- reading configuration files and generating code for the runtime. It would not be too difficult to port the runtime, I suspect.
- Game_Ender 9y agoSome quick questions: - How do you version your messages? - What problem domain are you working on? - What kind of process orchestration system do you use?
- tmpmov 9y agoSame questions. These would be good to know.
- w_t_payne 9y agoProblem domain: Embedded machine vision / Embedded sensor data processing / Autonomous systems. Process orchestration: Currently very simple. (Python multiprocessing). May change to something more complex in future. Message versioning: It depends. My configuration management / deployment model means that most of the time backwards compatibility is not required. (The entire system is deployed as a single unit). When messages are persisted, the data recording system files the messages in a configuration management system that knows the system configuration that generated those messages -- so that when they are replayed back -- e.g. to test a new configuration -- the appropriate reading/de-serialization logic can be selected.
- amorphid 9y agoFor Python, say I'm building a Docker container in Jenkins for Python app that includes your code. Does the YAML get evaluated when pushing the Jenkins "build now" or when executing "docker run <args>"?
- w_t_payne 9y agoThe YAML is evaluated at 'compile time'. I.e. the equivalent of the 'build now' step. It is worth noting: this is part of a custom code-generation/build/test/parameter-tuning system that I developed for embedded machine-vision/sensor-data-processing systems. It is not intended to be used by third-parties; or distributed as a library -- it is very much integrated into my scripted workflow.