4 ms·
Flattered by your interest but now I'm afraid the words "DSL" and "Lisp-like" might've given the impression of far more complexity than the project actually was
by skytreader 5y ago
Flattered by your interest but now I'm afraid the words "DSL" and "Lisp-like" might've given the impression of far more complexity than the project actually was. Keep in mind the core engine is ~200LoC. Maybe twice that at most. But really, there were far more rules than engine logic in the end.
The project was started for legal compliance so I don't want to go into specifics in case I make a slip one way or another. It did not run in the production cluster per se but was important nonetheless. (Actually, gods forbid it run in production!)
Very vaguely, it simplified structured data. A rule is recursively defined, base cases being simple rules that just returned primitive data. The original data could be an object with deep properties but most rules just simplified it to one with less fields.
I call it "Lisp-like" because after writing a handful of recursive rules, I had flashbacks of when I studied (actually) Scheme in uni. The tree-structure and meta-rules certainly gave that impression. I did not parse parens though, rather the rule syntax is JSON dictionaries. I hope this aided in readability---it should be self-explanatory what each rule did---and so my team did not find it too esoteric.
I don't remember the exact volume of data we processed but it was in the tens of GB, if not low hundreds. We parallelized it and had maybe 20 runners working concurrently. You can claim mapreduce (ha!) at this approach but honestly I think we could've parallelized it as easily if I did not go with this DSL-approach. :)