7 ms·

Years ago I was at a trading firm where I needed to write a test suite for our order entry system (the software that sends orders to a stock exchange). Tests co

Years ago I was at a trading firm where I needed to write a test suite for our order entry system (the software that sends orders to a stock exchange). Tests could become quite large because I had to track the sequence of acknowledgments and executions from the test exchange. The tests would have been a real pain to write in C++, so I created my own language! It was inspired by both Expect and Cucumber. https://en.wikipedia.org/wiki/Expect https://en.wikipedia.org/wiki/Expect https://cucumber.io/docs https://cucumber.io/docs That allowed me to write a simple script of expected behavior, like: test "Cross own order" -> new buy IBM 100 $141.01 1 <- ack 1 -> new sell IBM 200 $141.00 2 <- ack 2 <- trade 1 100 $141.01 <- trade 2 100 $141.01 My interpreter would run the above script against the exchange's UAT environment. The tests could get pretty sophisticated. I could investigate swanky exchange-specific order types by verifying queue position, liquidity flags, etc. By writing these test scripts, I could verify (1) that my order entry code was correct and (2) that I actually understood what the exchanges were doing with my orders. These test scripts had another added benefit in that traders now had executable documentation. So whenever a hypothetical scenario came up, I could write a script to verify that our assumptions were correct and then email the script's code as a precise set of steps that the exchange was taking. I ended up writing the whole language in Boost.Spirit, which admittedly was a beast. But I still believe that creating a testing language was the right thing to do.

random314 7y ago
I am curious. Was there any particular feature that your language had, which was lacking in C++, that helped in simplifying the test script?
- PaulRobinson 7y ago
  Not OP, but looking at his example, conciseness. DSLs are feared by many, but actually when you're doing the same thing over and over again and it's hugely verbose, they're great!
- ratww 7y ago
  Also not OP but: Having a limited language that looks more like data than code provides great opportunities to analyze it in ways that you can't analyze Turing-complete languages (halting problem and all). An example: A friend works for a game translation studio and one of their games has a tool that read the game scripts and is able to detect missing/unused external resources, overlapping dialogue, unmatched mouth animation/voiceovers, etc... without even running the game.
- pwm 7y ago
  In essence as you gain expressive power you lose analytical/introspective/reasoning power. Being more powerful on a concrete level costs power on the meta-level. There is a great talk about this called “Constraints Liberate, Liberties Constrain” from Rúnar Bjarnason.
- xorcist 7y ago
  These are things you typically want to do run time, and not require a compile cycle to change. There's also the risk of breaking changes to the scripts, if writing in C++ exposes too much of the project's object model.
- Jtsummers 7y ago
  Almost every DSL's capabilities are expressible in almost any language. The value is in the expression, not in what it expresses (usually). I can describe (off-the-cuff example) a regular expression using any programming language's basic constructs: Regex r = new Character('a').then(new Repetition(new Character('b')); Or I can write that as: ab* Which is clearer? You can probably imagine a simpler example in most languages, but you still have to design and construct an API, and then train people to that API. The DSL offers (in theory, not all do) a more natural representation of the problem, which can then be converted to an internal structure or directly executed. That more natural representation is what brings you most of the value. Another aspect of this, and this is where some have argued benefits in more algebraic-notation languages (the MLs) over others: You can often write DSLs that allow reasoning to be clearer. Back to that regex example, you can view regular expressions as an algebra. You can manipulate them like algebraic expressions. You cannot manipulate that first example algebraically, so if you decide you want to change it in some way or conduct some analysis on it, those activities are much harder. In the first API example, how would I alter it to indicate that a should be repeated 0 or more times? I have to insert a new new Repetition(...) element around the first Character element. It's tedious and error prone. For the regex, I add an asterisk after the a. It's clearer, more concise, more amenable to reflection (by the author and others), and more maintainable in the end. Additionally, if I want, I can change the implementation of that DSL to reflect new capabilities or new backends. With the API version, that's harder to accomplish. ========================================================= At work, we also have test DSLs. One of the biggest benefits besides the more natural expression of test cases, was that it was reimplemented and all the old tests continued to run without modification. From an old Fortran implementation to a more modern C# implementation. The ability to migrate 20+ years worth of test cases was a great boon. Every one of them could've been done in Fortran directly before, or C# now. But in 10-20 years we may have motivation to reimplement again. The one reimplementation was because: Original dev retired, poor documentation of code base, inability to port to modern operating systems due to graphics API used and code design, inability to extend to support newer systems being tested. I believe the C# implementation is better in all those ways, but limitations may be discovered in the future that require another rewrite (I do know it was successfully handed off to new developers, so it is more comprehensible and maintainable, if nothing else).
ratww 7y ago
Me and some co-workers did a mini-language a long time ago too, but it was for a BPM workflow software that was so in vogue ten years ago [1]. It only had a super buggy graphical interface that customers hated. Some of the largest customers had workflows with more than 100 steps/conditions, and we were losing money because we needed to have a full-time coder on-site for each customer just to program the workflows. The database structure however was quite good, and I noticed that it was being deleted and recreated each time the workflow was edited. Adding a text mode was just a matter of parsing individual lines and persisting them on the database. No parser, just regex matching. Even an extra blank space was a syntax error. Something like that: q1 = Question("Submit Request?") a1 = Answer("Submit") a2 = Answer("Reject") Connect(q1, a1) Connect(a1, a2) c1 = Cancel("Cancel") f2 = Finish("Finish") Connect(a1, f1) Connect(a2, c1) It was implemented as an easter egg. Using text mode disabled the graphical part, since we lost the positions of each box. But nobody ever complained about that, of course. [1] Looked similar to this: https://archibus.ro/wp-content/uploads/2017/07/GWE.png https://archibus.ro/wp-content/uploads/2017/07/GWE.png
- flukus 7y ago
  Many years ago I was working on a complex screen scraper (the good kind) app and every site we scraped was ~1000 lines of error prone code, plus all the usual problems of sites changing randomly under us and the need to respond quickly. They wanted to scale up to scrape dozens of sites and we weren't given the resources so I came up with a little xpath like language "//content-area/table/tbody/@foreach/tr/@get-value" or something like that, after that the largest scraper was a dozen lines of code. I think it came down on the right side of reducing complexity rather than creating more but I was let go shortly after for the heinous crime of not being at my desk at 9am.
  - specialist 7y ago
    This is a Most Righteous Answer, kissing cousin to The Correct Answer. Repeating myself, sorry: I've done a lot of electronic medical records. At the time, SOAP, WSDL, HL7 3.x (XML format) reigned. Techniques like yours is how our two person team was able to run circles around our much larger partners. In other words, while they were stuck trying to update and compile the XSDs, we just treated inbound data as screen scrapping. Sharing a tip of my own: while xpath expressions are mostly great, we migrated to globbing expressions. The wildcarding allowed our scrappers to be a lot more flexible, robust in regards to all the chaotic mutant data we were receiving. https://en.wikipedia.org/wiki/Glob_%28programming%29 https://en.wikipedia.org/wiki/Glob_%28programming%29 PS- IMHO, The Correct Answer is to have path expression intrinsics, built right into the language. What LINQ should have been. Imagine if Java had regex intrinsics like Perl, instead of that weird library. Some idea.
reacweb 7y ago
Why not using expect directly ? For one of my most complex setup, I have a perl module (using Expect) that is used by all my test scenarios (perl scripts).
Erlich_Bachman 7y ago
I believe it's called Domain-Specific Language (DSL) and there are systems that excel at quickly, easily and reliably creating these languages, for example Groovy. You could write a language like in your example, and it would be usable and stable in a couple of hours. Then you could script all kinds programs in it quickly, while also being able to leverage the rest of the flexibility of the Groovy language, interchaning its code with DSL statements.
- jmkni 7y ago
  Jetbrains have a new tool specifically for this purpose - https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/ I haven't played with it yet, it looks interesting!
  - snuxoll 7y ago
    MPS is hardly new, but it’s certainly an interesting tool if you are targeting the JVM.
twic 7y ago
Okay but why not: test("Cross own order"); send(newBuy("IBM", 100, 141.01, 1)); expect(ack(1)); send(newSell("IBM", 200, 141.00, 2)); expect(ack(2)); expect(trade(1, 100, 141.01)); expect(trade(2, 100, 141.01)); Because then you'd just need to write a few helper functions in C++, not an entire interpreter.
- Crinus 7y ago
  They already wrote > The tests would have been a real pain to write in C++ so i guess what the script piece above shows wasn't the entire deal. When you have a custom language you have full control over its syntax you can make it do all sorts of things that are hard to do in something like C++. As an example you can build a tree of key-value nodes using C++ calls just fine, but writing a JSON file is usually much simpler and concise (especially when deep nesting comes into place). And a custom syntax could be even simpler than JSON.
  - lugg 7y ago
    You just described functions.
- stestagg 7y ago
  One reason, from our similar experience is that showing non-Technical business people (in some environments) text with brackets, semicolons and the general code annotations you show here effectively “scares” them. This isn’t always true. But it tends to be institutional when it happens. In some places people learn deeply that code is the realm of the tech teams and they should not engage with it. Getting them to engage with the executable tests can be fantastically useful but it can also be a delicate sales game, where simple things like using a DSL can be key
  - phtrivier 7y ago
    That's assuming the "non-tech businness people" are actually going to spend time writing and reading such scripts. I've never seen a situation where that ended up happening (because they're going to write the script wrong, they're going to need help debugging, etc...) Their time is probably better spend doing something else. So they will write english in mails and powerpoint, and a tech will write code in a programming language. The closest thing I can imagine to having biz people "write scripts", is if it's done in a GUI with a lot of assistance and a quick visual feedback loop (think Excel macros.) Maybe writing the GUI is a better investment than writing a compiler.
stestagg 7y ago
Interestingly we’re doing something very similar, but using cucumber syntax. So the tests are much more chatty In our situation we probably have more concepts (middle office) but the interactions we’re modelling are simpler. Going down this route is quite interesting because initially we got a lot of scepticism from other groups, but over time the solution has evolved with more and more documentation / traceability, and they ended up pretty complete documentation. Every time someone decides that the development teams have life too easy and try to impose specific extra paperwork or documentation burdens on the day to day work, giving them that test pack seems to make them go quiet :)
lozzo 7y ago
Excellent. Your solution is a common pattern after all. It's creating a DSL domain specific language and then obviously you need to write some kind of interpreter for it. For all it matters, I found myself in similar situations but what I decided to do is to use an existing language (in my case it was JavaScript but nothing to do with the Javascript you run on node or on the web)
greggyb 7y ago
Vaguely related, I came across this talk the other day, and it is interesting to see this sort of thing come up twice in short succession for me. The talk is about building an exchange from Jane Street. Leaving it here in case anyone is interested. https://www.youtube.com/watch?v=b1e4t2k2KJY https://www.youtube.com/watch?v=b1e4t2k2KJY

random314 7y ago

I am curious. Was there any particular feature that your language had, which was lacking in C++, that helped in simplifying the test script?

PaulRobinson 7y ago
Not OP, but looking at his example, conciseness. DSLs are feared by many, but actually when you're doing the same thing over and over again and it's hugely verbose, they're great!
ratww 7y ago
Also not OP but: Having a limited language that looks more like data than code provides great opportunities to analyze it in ways that you can't analyze Turing-complete languages (halting problem and all). An example: A friend works for a game translation studio and one of their games has a tool that read the game scripts and is able to detect missing/unused external resources, overlapping dialogue, unmatched mouth animation/voiceovers, etc... without even running the game.
pwm 7y ago
In essence as you gain expressive power you lose analytical/introspective/reasoning power. Being more powerful on a concrete level costs power on the meta-level. There is a great talk about this called “Constraints Liberate, Liberties Constrain” from Rúnar Bjarnason.
xorcist 7y ago
These are things you typically want to do run time, and not require a compile cycle to change. There's also the risk of breaking changes to the scripts, if writing in C++ exposes too much of the project's object model.
Jtsummers 7y ago
Almost every DSL's capabilities are expressible in almost any language. The value is in the expression, not in what it expresses (usually). I can describe (off-the-cuff example) a regular expression using any programming language's basic constructs: Regex r = new Character('a').then(new Repetition(new Character('b')); Or I can write that as: ab* Which is clearer? You can probably imagine a simpler example in most languages, but you still have to design and construct an API, and then train people to that API. The DSL offers (in theory, not all do) a more natural representation of the problem, which can then be converted to an internal structure or directly executed. That more natural representation is what brings you most of the value. Another aspect of this, and this is where some have argued benefits in more algebraic-notation languages (the MLs) over others: You can often write DSLs that allow reasoning to be clearer. Back to that regex example, you can view regular expressions as an algebra. You can manipulate them like algebraic expressions. You cannot manipulate that first example algebraically, so if you decide you want to change it in some way or conduct some analysis on it, those activities are much harder. In the first API example, how would I alter it to indicate that a should be repeated 0 or more times? I have to insert a new new Repetition(...) element around the first Character element. It's tedious and error prone. For the regex, I add an asterisk after the a. It's clearer, more concise, more amenable to reflection (by the author and others), and more maintainable in the end. Additionally, if I want, I can change the implementation of that DSL to reflect new capabilities or new backends. With the API version, that's harder to accomplish. ========================================================= At work, we also have test DSLs. One of the biggest benefits besides the more natural expression of test cases, was that it was reimplemented and all the old tests continued to run without modification. From an old Fortran implementation to a more modern C# implementation. The ability to migrate 20+ years worth of test cases was a great boon. Every one of them could've been done in Fortran directly before, or C# now. But in 10-20 years we may have motivation to reimplement again. The one reimplementation was because: Original dev retired, poor documentation of code base, inability to port to modern operating systems due to graphics API used and code design, inability to extend to support newer systems being tested. I believe the C# implementation is better in all those ways, but limitations may be discovered in the future that require another rewrite (I do know it was successfully handed off to new developers, so it is more comprehensible and maintainable, if nothing else).

ratww 7y ago

Me and some co-workers did a mini-language a long time ago too, but it was for a BPM workflow software that was so in vogue ten years ago [1]. It only had a super buggy graphical interface that customers hated. Some of the largest customers had workflows with more than 100 steps/conditions, and we were losing money because we needed to have a full-time coder on-site for each customer just to program the workflows. The database structure however was quite good, and I noticed that it was being deleted and recreated each time the workflow was edited. Adding a text mode was just a matter of parsing individual lines and persisting them on the database. No parser, just regex matching. Even an extra blank space was a syntax error. Something like that: q1 = Question("Submit Request?") a1 = Answer("Submit") a2 = Answer("Reject") Connect(q1, a1) Connect(a1, a2) c1 = Cancel("Cancel") f2 = Finish("Finish") Connect(a1, f1) Connect(a2, c1) It was implemented as an easter egg. Using text mode disabled the graphical part, since we lost the positions of each box. But nobody ever complained about that, of course. [1] Looked similar to this: https://archibus.ro/wp-content/uploads/2017/07/GWE.png https://archibus.ro/wp-content/uploads/2017/07/GWE.png

flukus 7y ago
Many years ago I was working on a complex screen scraper (the good kind) app and every site we scraped was ~1000 lines of error prone code, plus all the usual problems of sites changing randomly under us and the need to respond quickly. They wanted to scale up to scrape dozens of sites and we weren't given the resources so I came up with a little xpath like language "//content-area/table/tbody/@foreach/tr/@get-value" or something like that, after that the largest scraper was a dozen lines of code. I think it came down on the right side of reducing complexity rather than creating more but I was let go shortly after for the heinous crime of not being at my desk at 9am.
- specialist 7y ago
  This is a Most Righteous Answer, kissing cousin to The Correct Answer. Repeating myself, sorry: I've done a lot of electronic medical records. At the time, SOAP, WSDL, HL7 3.x (XML format) reigned. Techniques like yours is how our two person team was able to run circles around our much larger partners. In other words, while they were stuck trying to update and compile the XSDs, we just treated inbound data as screen scrapping. Sharing a tip of my own: while xpath expressions are mostly great, we migrated to globbing expressions. The wildcarding allowed our scrappers to be a lot more flexible, robust in regards to all the chaotic mutant data we were receiving. https://en.wikipedia.org/wiki/Glob_%28programming%29 https://en.wikipedia.org/wiki/Glob_%28programming%29 PS- IMHO, The Correct Answer is to have path expression intrinsics, built right into the language. What LINQ should have been. Imagine if Java had regex intrinsics like Perl, instead of that weird library. Some idea.

reacweb 7y ago

Why not using expect directly ? For one of my most complex setup, I have a perl module (using Expect) that is used by all my test scenarios (perl scripts).

Erlich_Bachman 7y ago

I believe it's called Domain-Specific Language (DSL) and there are systems that excel at quickly, easily and reliably creating these languages, for example Groovy. You could write a language like in your example, and it would be usable and stable in a couple of hours. Then you could script all kinds programs in it quickly, while also being able to leverage the rest of the flexibility of the Groovy language, interchaning its code with DSL statements.

jmkni 7y ago
Jetbrains have a new tool specifically for this purpose - https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/ I haven't played with it yet, it looks interesting!
- snuxoll 7y ago
  MPS is hardly new, but it’s certainly an interesting tool if you are targeting the JVM.

twic 7y ago

Okay but why not: test("Cross own order"); send(newBuy("IBM", 100, 141.01, 1)); expect(ack(1)); send(newSell("IBM", 200, 141.00, 2)); expect(ack(2)); expect(trade(1, 100, 141.01)); expect(trade(2, 100, 141.01)); Because then you'd just need to write a few helper functions in C++, not an entire interpreter.

Crinus 7y ago
They already wrote > The tests would have been a real pain to write in C++ so i guess what the script piece above shows wasn't the entire deal. When you have a custom language you have full control over its syntax you can make it do all sorts of things that are hard to do in something like C++. As an example you can build a tree of key-value nodes using C++ calls just fine, but writing a JSON file is usually much simpler and concise (especially when deep nesting comes into place). And a custom syntax could be even simpler than JSON.
- lugg 7y ago
  You just described functions.
stestagg 7y ago
One reason, from our similar experience is that showing non-Technical business people (in some environments) text with brackets, semicolons and the general code annotations you show here effectively “scares” them. This isn’t always true. But it tends to be institutional when it happens. In some places people learn deeply that code is the realm of the tech teams and they should not engage with it. Getting them to engage with the executable tests can be fantastically useful but it can also be a delicate sales game, where simple things like using a DSL can be key
- phtrivier 7y ago
  That's assuming the "non-tech businness people" are actually going to spend time writing and reading such scripts. I've never seen a situation where that ended up happening (because they're going to write the script wrong, they're going to need help debugging, etc...) Their time is probably better spend doing something else. So they will write english in mails and powerpoint, and a tech will write code in a programming language. The closest thing I can imagine to having biz people "write scripts", is if it's done in a GUI with a lot of assistance and a quick visual feedback loop (think Excel macros.) Maybe writing the GUI is a better investment than writing a compiler.

stestagg 7y ago

Interestingly we’re doing something very similar, but using cucumber syntax. So the tests are much more chatty In our situation we probably have more concepts (middle office) but the interactions we’re modelling are simpler. Going down this route is quite interesting because initially we got a lot of scepticism from other groups, but over time the solution has evolved with more and more documentation / traceability, and they ended up pretty complete documentation. Every time someone decides that the development teams have life too easy and try to impose specific extra paperwork or documentation burdens on the day to day work, giving them that test pack seems to make them go quiet :)

lozzo 7y ago

Excellent. Your solution is a common pattern after all. It's creating a DSL domain specific language and then obviously you need to write some kind of interpreter for it. For all it matters, I found myself in similar situations but what I decided to do is to use an existing language (in my case it was JavaScript but nothing to do with the Javascript you run on node or on the web)

greggyb 7y ago

Vaguely related, I came across this talk the other day, and it is interesting to see this sort of thing come up twice in short succession for me. The talk is about building an exchange from Jane Street. Leaving it here in case anyone is interested. https://www.youtube.com/watch?v=b1e4t2k2KJY https://www.youtube.com/watch?v=b1e4t2k2KJY