3 ms·

Ask HN: Best Language for testing embedded Linux hardware

2 points by manuf_eng 11y ago

I work for a company that designs and manufactures embedded single board computers, primarily ARM based. Currently we use bash scripts and NFS filesystems for testing and programming. It's worked well to date, but I can't help but think Go, Python or some other object oriented language would be a better approach(code reuse / common architecture that can be shared). My concerns are we tend to implement the same boiler plate logic over and over - call a test which passes or fails. Log low level details to a server for analysis(NFS plain text file), and display a high level pass / fail message. Most scripts are tightly coupled, so one script is used for multiple boards. This results in surprises when someone makes a change for one product and breaks another related product that shares the same bash script. The boards themselves vary widely in that the kernels could be anywhere from 2.4 to 3.1x, oabi, eabi... busybox, ubuntu, debian for filesystems... This makes me think an interpreted language such as Python would be the way to go. My concern is this is run in a manufacturing environment so interpreted languages(e.g. Python) that are slow would cost money on every board processed. Any thoughts / suggestions on a language and / or architecture? Googling around hasn't turned up much...

manuf_eng 11y ago
Should have mentioned these are high end embedded systems. 512MB or more of RAM and 400MHz or faster CPUs.
- dottrap 11y ago
  It sounds like you have more fundamental architecture issues. Language is mostly orthogonal to this and switching languages won't necessarily make anything better, though I personally find bash to be clumsy for writing anything large and kind of slow too. So first, don't dismiss C. C still dominates in embedded environments for good reasons. Every processor has a C compiler, C is fast, C gives you a degree of compiler checking, C is small, and writing re-usable libraries in C is a well-understood thing (this is the basis of every Linux/Unix distro). And yes, you can write tests for C. Look at CTest for one. If you really want a scripting language, then Lua is generally the top choice for embedded. Lua is tiny, fast, 100% ANSI C (which means you can port it anywhere because as mentioned, everybody has a C compiler), and Lua is extremely extensible for custom needs. Lua is also designed to interoperate with C so if you are really concerned with performance, you can pretty easily split up the performance sensitive stuff into C and write the rest in Lua.
  - dottrap 11y ago
    And LuaJIT is also available for your platform too if you want blazing fast performance.
  - dottrap 11y ago
    One more thing. Be careful of "object oriented" design if you are thinking about shared libraries and re-usability. Depending on your definition of "object oriented" and the language you pick, it actually causes tighter coupling between components because the type system often encourages everything to be related. Since you mentioned Go, Go's approach to object-oriented tends to use interfaces which tends to avoid this problem. But your milage will vary depending on the popular semantics of whatever language you choose and also the conventions library writers use if you choose to use pre-existing libraries.
  - manuf_eng 11y ago
    Great point about architecture, this is most definitely a major issue and is not specific to what language is used. I've just been reassigned to oversee development in this area and am hoping to fix both these issues. Over time the manufacturing software has evolved. Originally everything was written in C, it was a single C file to test all products. It was fast but from a maintenance standpoint it grew to be too large and this approach was abandoned as a result. In practice we've found writing low level tests in C that only do one thing and then use bash as glue logic seemed to work well. Some hardware tests must be implemented in C(low level register accesses where timing is critical). We currently interface with USB printers(the board under test) to print a serial number corresponding to a MAC. address. I found maintaining this utility on all boards became a major issue(udev, devfs, busybox - eabi, oabi...) so I implemented it in Python and from a maintenance standpoint it's been a huge improvement. The problem is it's slow on some products(30 seconds or more to print a label and log to a file via NFS) and I have to admit I haven't yet dug in to understand why.
    - dottrap 11y ago
      Sounds like a good job for Lua. You can replace the glue logic with Lua. And then you can augment or replace the tests on a case-by-case basis. For tests that must be done in C, you can either leave them as is and drive them externally, or create a binding so you can invoke the C portions from Lua. LuaJIT's FFI feature makes it very easy to experiment with bridging because it avoids writing binding code.
wyldfire 11y ago
python's not necessarily slow -- are your tests typically CPU bound or I/O bound? If the latter, I suspect the difference between Python and other languages will be insignificant. More powerful ARM CPUs (ARM7 or better IIRC) can take advantage of pypy.
drallison 11y ago
The particular programming language you use for your test framework does not matter. Execution efficiency of the framework is not likely to be a problem. A carefully designed test framework that manages the board testing in an automated fashion does. Thinking through and formalizing the testing process (and the analysis to test results) will be much more fruitful than worrying about whether to program the framework in BASH, Python, Golang, or LISP.
pfalcon 11y ago
If you currently use bash, I wouldn't be much concerned that Python would be slow - in general, a general-purpose language would have more advanced interpreter than a shell. Python is also pretty high-level language with "batteries included", definitely more high-level than Go, Lua mentioned here. And with implementations like MicroPython (https://github.com/micropython/micropython https://github.com/micropython/micropython) you can be sure that you can scale down if needed (e.g. to a system with just 2-4MB of flash with Linux, or 128KB+ with bare-metal).

2 points by manuf_eng 11y ago

manuf_eng 11y ago

Should have mentioned these are high end embedded systems. 512MB or more of RAM and 400MHz or faster CPUs.

dottrap 11y ago
It sounds like you have more fundamental architecture issues. Language is mostly orthogonal to this and switching languages won't necessarily make anything better, though I personally find bash to be clumsy for writing anything large and kind of slow too. So first, don't dismiss C. C still dominates in embedded environments for good reasons. Every processor has a C compiler, C is fast, C gives you a degree of compiler checking, C is small, and writing re-usable libraries in C is a well-understood thing (this is the basis of every Linux/Unix distro). And yes, you can write tests for C. Look at CTest for one. If you really want a scripting language, then Lua is generally the top choice for embedded. Lua is tiny, fast, 100% ANSI C (which means you can port it anywhere because as mentioned, everybody has a C compiler), and Lua is extremely extensible for custom needs. Lua is also designed to interoperate with C so if you are really concerned with performance, you can pretty easily split up the performance sensitive stuff into C and write the rest in Lua.
- dottrap 11y ago
  And LuaJIT is also available for your platform too if you want blazing fast performance.
- dottrap 11y ago
  One more thing. Be careful of "object oriented" design if you are thinking about shared libraries and re-usability. Depending on your definition of "object oriented" and the language you pick, it actually causes tighter coupling between components because the type system often encourages everything to be related. Since you mentioned Go, Go's approach to object-oriented tends to use interfaces which tends to avoid this problem. But your milage will vary depending on the popular semantics of whatever language you choose and also the conventions library writers use if you choose to use pre-existing libraries.
- manuf_eng 11y ago
  Great point about architecture, this is most definitely a major issue and is not specific to what language is used. I've just been reassigned to oversee development in this area and am hoping to fix both these issues. Over time the manufacturing software has evolved. Originally everything was written in C, it was a single C file to test all products. It was fast but from a maintenance standpoint it grew to be too large and this approach was abandoned as a result. In practice we've found writing low level tests in C that only do one thing and then use bash as glue logic seemed to work well. Some hardware tests must be implemented in C(low level register accesses where timing is critical). We currently interface with USB printers(the board under test) to print a serial number corresponding to a MAC. address. I found maintaining this utility on all boards became a major issue(udev, devfs, busybox - eabi, oabi...) so I implemented it in Python and from a maintenance standpoint it's been a huge improvement. The problem is it's slow on some products(30 seconds or more to print a label and log to a file via NFS) and I have to admit I haven't yet dug in to understand why.
  - dottrap 11y ago
    Sounds like a good job for Lua. You can replace the glue logic with Lua. And then you can augment or replace the tests on a case-by-case basis. For tests that must be done in C, you can either leave them as is and drive them externally, or create a binding so you can invoke the C portions from Lua. LuaJIT's FFI feature makes it very easy to experiment with bridging because it avoids writing binding code.

wyldfire 11y ago

python's not necessarily slow -- are your tests typically CPU bound or I/O bound? If the latter, I suspect the difference between Python and other languages will be insignificant. More powerful ARM CPUs (ARM7 or better IIRC) can take advantage of pypy.

drallison 11y ago

The particular programming language you use for your test framework does not matter. Execution efficiency of the framework is not likely to be a problem. A carefully designed test framework that manages the board testing in an automated fashion does. Thinking through and formalizing the testing process (and the analysis to test results) will be much more fruitful than worrying about whether to program the framework in BASH, Python, Golang, or LISP.

pfalcon 11y ago

If you currently use bash, I wouldn't be much concerned that Python would be slow - in general, a general-purpose language would have more advanced interpreter than a shell. Python is also pretty high-level language with "batteries included", definitely more high-level than Go, Lua mentioned here. And with implementations like MicroPython (https://github.com/micropython/micropython https://github.com/micropython/micropython) you can be sure that you can scale down if needed (e.g. to a system with just 2-4MB of flash with Linux, or 128KB+ with bare-metal).