4 ms·

Show HN: Torque – A lightweight meta-assembler for any processor

79 points by benbridle 1y ago

Hello everyone, I've been working on this project for the past few months. Torque is a meta-assembler: instead of having an instruction set built into the assembler, you use macros to build up a small language that decribes an instruction set and then you use that to write your program. It's designed to work for any microcontroller/processor architecture, you build from the bit level upwards so there aren't any assumptions around word widths, instruction formats, or endianness. I created Torque initially to write programs for a PIC microcontroller, after running into difficulties with the official assembler. I've also used it to write programs for the Z80 processor inside an old TRS-80 computer. Let me know if you try it out or have any questions!

sitkack 1y ago
Super fun site! Did you get inspiration from other assemblers or macro processors? You have it running on a TRS-80, how does that work? I had no idea Rust could target a TRS-80. I am getting hints of Forth, Lisp and TCL. How would you go about laying out structs in memory? I am sure you considered an internal DSL, what caused you go with something stand alone? Any thoughts on adding a constraint solver, like Z3 and allowing end users to set constraints on things like the size of a jump. I could see taking this an growing it into a compiler by making macro(macro(macros txt))) Is there an internal IR? Projects for inspiration https://github.com/mattbierner/Template-Assembly https://github.com/mattbierner/Template-Assembly Specifying representations of machine instructions https://dl.acm.org/doi/pdf/10.1145/256167.256225 https://dl.acm.org/doi/pdf/10.1145/256167.256225 https://www.semanticscholar.org/paper/Specifying-representations-of-machine-instructions-Ramsey-Fern%C3%A1ndez/b5775a11496422c2dfcaa2e31cc10f52c01cbce7 https://www.semanticscholar.org/paper/Specifying-representat... Typed Assembly Language (TAL) https://www.cs.cornell.edu/talc/ https://www.cs.cornell.edu/talc/ And you haven't come across it, you are in for a treat https://en.wikipedia.org/wiki/META_II https://en.wikipedia.org/wiki/META_II has spawned a whole trove of clones https://en.wikipedia.org/wiki/OMeta https://en.wikipedia.org/wiki/OMeta https://github.com/DalekBaldwin/clometa https://github.com/DalekBaldwin/clometa
- benbridle 1y ago
  Thank you! My main inspiration was the Uxn assembly language [0], which is itself heavily inspired by Forth. I loved how easy it was to build something that looks like a high-level language by just stacking up macros, and I wanted to have that with embedded development too. Rust isn't involved past implementing the Torque executable; you write your program with the Torque language and then run the assembler on it to convert it to machine code. You can see the whole process of running code on the TRS-80 from start to finish here [1]. For laying out structs, I'd build a macro that expands to the memory representation of the struct. If I wanted a struct representing, say, a 2D point with signed 16-bit little-endian integers for the x and y coords, I would build it from scratch like this (this is a valid program, you can assemble it with Torque): %BYTE:n #nnnn_nnnn ; %LOW:n BYTE:[n 0xff <and>] ; %HIGH:n BYTE:[n 8 <shr>] ; %16LE:n LOW:n HIGH:n ; %POINT:x:y 16LE:x 16LE:y ; @point-1 POINT:50:-7 @point-2 POINT:20:-45 @point-3 POINT:0:0 If I want the address of a field, I can add an offset to the struct address, using macros to name the offset values: %POINT.X 0 ; %POINT.Y 2 ; SET:[point-3 POINT.X +]:15 SET:[point-3 POINT.Y +]:32 Creating a DSL for an existing language wasn't something I'd ever considered. By being a standalone executable it's really easy to use and share, people don't have to install a whole language toolchain in order to use it. Regarding constraints solving and jumping, Torque already throws an error if you try to pack too large a value into too small a field. This works really well for things like relative jumps, because jumping too far will create a value that can't fit in the instruction. I'm planning on adding an error-throwing token to the language that could be used alongside expressions and conditions to further constrain the values accepted by a macro, but I'm really happy with the simplicity of the language so far. The actual internal representation isn't what I'd call an 'IR' per se, nothing like with a C compiler. It's all very pedestrian; the syntax tree is baked down across multiple passes, with macros acting as a glorified copy-paste system. Thanks for the interest and the links, every one of those linked projects is new to me. [0] https://wiki.xxiivv.com/site/uxn.html https://wiki.xxiivv.com/site/uxn.html [1] https://benbridle.com/articles/torque-programming-the-trs-80.html https://benbridle.com/articles/torque-programming-the-trs-80...
zadkey 1y ago
Excellent work man! I know some low level guys who would really appreciate this.
- benbridle 1y ago
  Thanks, that's great to hear! What platforms do they develop for?
anonzzzies 1y ago
Well done! I write quite a bit of z80 for msx and this seems a nice addition.
curtisszmania 1y ago
[dead]
2ton_jeff 1y ago
Very cool and I like the idea of a "meta-assembler." The most-recent version of flatassembler (fasm 2) is built with fasmg which is also a "meta-assembler" of sorts, in that it also doesn't directly support a specific instruction set and instead is a very powerful macro assembler. I'm keen to check out functionality overlaps between the two implementations. https://board.flatassembler.net/topic.php?t=19389 https://board.flatassembler.net/topic.php?t=19389 https://flatassembler.net/download.php https://flatassembler.net/download.php
- benbridle 1y ago
  Oh neat! Thanks for the link, I hadn't heard of fasmg before. It looks like fasmg builds up from the byte level, so it would only work for architectures that use 8-bit words. Torque builds up from the bit level, so it can assemble code for architectures like in PIC microcontrollers, using word sizes of 12 or 14 bits. However, fasmg does allow a lot more control over the syntax of the language. The documentation shows some pretty powerful string manipulation that's used to parse real x86 assembler code, which makes sense given the purpose of it. Torque doesn't allow overriding the syntax to that degree, the macro invocation syntax is baked into the assembler. One thing that intrigues me about fasmg is how it handles circular dependencies in expressions [0] (search for 'circular'). Currently in Torque it isn't possible to use a label reference inside a predicate, because the predicate evaluating one way could insert additional code, moving the label and causing the predicate to evaluate the other way [1]. But in fasmg it's possible to use the result of an expression as part of its own calculation. [0] https://flatassembler.net/docs.php?article=fasmg https://flatassembler.net/docs.php?article=fasmg [1] https://benbridle.com/projects/torque/roadmap.html https://benbridle.com/projects/torque/roadmap.html
  - 2ton_jeff 1y ago
    I reached out to the author of fasmg WRT your post and circular dependency interest and he pointed me toward two posts that he wrote very specifically to explain what he believes is unique to fasm/fasmg and allows to handle circular dependencies of many kinds. [0] Types of multi-pass assembly, and [1] related pitfalls. [0] https://board.flatassembler.net/topic.php?t=20249 https://board.flatassembler.net/topic.php?t=20249 [1] https://board.flatassembler.net/topic.php?t=21060 https://board.flatassembler.net/topic.php?t=21060
angelmm 1y ago
Any time I read about Z80, my mind wants to try to run something on a GameBoy emulator. Pretty cool project!
eterps 1y ago
Nice! Would be interesting to target the RISC CPU of https://www.projectoberon.net https://www.projectoberon.net with it.
userbinator 1y ago
Reminds me of TDASM from the turn of the century: https://web.archive.org/web/20230906054935/http://www.penguin.cz/~niki/tdasm/ https://web.archive.org/web/20230906054935/http://www.pengui... I remember there were a few other meta-assemblers I came across in the 80s-90s, so this is definitely not "unchartered territory", but it's good to see another one show up. Of course, in the other direction there are meta-disassemblers used for analysis in tools like Ghidra.
kunley 1y ago
Need to address this point: "Assemblers tend to be poorly documented" I wish everything in programming was as good documented as assemblers and ISAs.
- benbridle 1y ago
  I should clarify, I was thinking of microcontroller toolchains when I wrote this. The documentation I've seen for Intel processors is solid.
  - kunley 1y ago
    Not to mention ARM64... And 6502 ;-) (and probably most of the ancient ones)
bjackman 1y ago
At first I thought it was useless: "but each ISA will still end up having different effective syntax because the underlying macro systems will not be designed the same". But then I reread it and realised I was not paying attention to the usecase. It's about making it easy to write assemblers. So this isn't for your Arms and RISC-Vs it's for your random niche microcontrollers where the vendor-provided toolchain kinda sucks. Seems cool! I've experienced a couple of under-documented assemblers in my time. In neither case did this turn out to be that much of a problem in practice, but I guess I just don't write that much assembly.
MathMonkeyMan 1y ago
I'm reminded of a 2016 [talk][1] where Rob Pike describes the common-denominator assembly language that the Go compiler generates. Then that assembly is translated into machine-specific code via table lookups. See the 11 minute mark. [1]: https://www.youtube.com/watch?v=KINIAgRpkDA https://www.youtube.com/watch?v=KINIAgRpkDA
roetlich 1y ago
This looks very cool, maybe I'll try this for risc-v if I have some time. I guess you could also use this to write RAW images, or other binary data.
zzo38computer 1y ago
Is there any support for non-Unicode text? Is there any support for octal numbers? These things should be corrected, hopefully. Also, some of the links in the table of contents of the documentation does not seems to work.
- benbridle 1y ago
  With character sets, I was initially going to support non-Unicode text by adding a --char-set flag to the assembler, but I decided that the character set should be defined somehow inside each program. My thought was that they could be defined as large table-like macros, something like the following: %BYTE:n #nnnn_nnnn ; %CHAR:n ?[n 'A' ==] BYTE:0x01 ?[n 'B' ==] BYTE:0x02 ?[n 'C' ==] BYTE:0x03 ?[n 'D' ==] BYTE:0x04 ; CHAR:"ABCDABCD" This is, admittedly, quite unweildy for character sets exceeding a few hundred characters, but it would work passably for small character sets like those used for HD44780-style LCD screens. What character sets did you have in mind? Octal was another feature I couldn't make up my mind about, just because I wasn't familiar with any architectures that require it. It'll be trivial to tack on though. For the Z80 instruction set, since the instruction encoding tends to cleave along octal lines, I used the following macro to pack octal digits into bytes, which has the advantage of allowing variables to be passed into each digit (the ADDr macro shows how it's used): %XYZ:x:y:z #xxyyyzzz ; %ADDr:r XYZ:2:0:r ; Thanks for the heads-up about the table of contents, the links should all work now.
  - zzo38computer 1y ago
    I think the "0" prefix for octal (as used in the C programming language) is not so good and that "0o" is better; so, if octal literals are implemented then "0o" is better. For non-Unicode text, probably the simplest thing would be to treat the input as a sequence of bytes instead of Unicode characters; or equivalently to treat it as ISO-8859-1 (although programming it to use ISO-8859-1 may be less efficient then just using bytes, possibly; I don't know much about the working of Rust programming, so I don't actually know if it is or not). By "non-Unicode text", I did not mean character mapping, although character mapping is another feature that would be useful to implement, similar to what you mentioned although it could be made more efficient (like you mention). Some way to map a input sequence of bytes (whether or not it is valid UTF-8) to a output character code, would work, probably.
vanderZwan 1y ago
I shared Torque elsewhere and people asked what license it has
- benbridle 1y ago
  Thanks for the interest! Torque is public domain, feel free to use it or change it or share it however you want. I'd love to hear how people use it, but no obligation. If you shared it someplace public I'd love to pop by, if you have a link.
  - vanderZwan 1y ago
    I shared it on the catlang discord, Devine immediately replied "wow, that looks so much like uxn 0_o" hahaha. It's not the most active server but the response has been very positive! A few people thought it would probably be fun to use it to build a custom IR as part of a compiler pipeline, that was the context of the question. I personally really love the utf8 and the "z80 for loop" examples, those are beautiful and make me nostalgic for the days I was writing stuff for the TI-83+ in Z80 assembly.
  - benbridle 1y ago
    I've since been informed that public domain isn't really a proper term, legally speaking, so to clarify things I've released Torque under the MIT license.
2ro 1y ago
someone should embed Prolog or microKanren

79 points by benbridle 1y ago

sitkack 1y ago

Super fun site! Did you get inspiration from other assemblers or macro processors? You have it running on a TRS-80, how does that work? I had no idea Rust could target a TRS-80. I am getting hints of Forth, Lisp and TCL. How would you go about laying out structs in memory? I am sure you considered an internal DSL, what caused you go with something stand alone? Any thoughts on adding a constraint solver, like Z3 and allowing end users to set constraints on things like the size of a jump. I could see taking this an growing it into a compiler by making macro(macro(macros txt))) Is there an internal IR? Projects for inspiration https://github.com/mattbierner/Template-Assembly https://github.com/mattbierner/Template-Assembly Specifying representations of machine instructions https://dl.acm.org/doi/pdf/10.1145/256167.256225 https://dl.acm.org/doi/pdf/10.1145/256167.256225 https://www.semanticscholar.org/paper/Specifying-representations-of-machine-instructions-Ramsey-Fern%C3%A1ndez/b5775a11496422c2dfcaa2e31cc10f52c01cbce7 https://www.semanticscholar.org/paper/Specifying-representat... Typed Assembly Language (TAL) https://www.cs.cornell.edu/talc/ https://www.cs.cornell.edu/talc/ And you haven't come across it, you are in for a treat https://en.wikipedia.org/wiki/META_II https://en.wikipedia.org/wiki/META_II has spawned a whole trove of clones https://en.wikipedia.org/wiki/OMeta https://en.wikipedia.org/wiki/OMeta https://github.com/DalekBaldwin/clometa https://github.com/DalekBaldwin/clometa

benbridle 1y ago
Thank you! My main inspiration was the Uxn assembly language [0], which is itself heavily inspired by Forth. I loved how easy it was to build something that looks like a high-level language by just stacking up macros, and I wanted to have that with embedded development too. Rust isn't involved past implementing the Torque executable; you write your program with the Torque language and then run the assembler on it to convert it to machine code. You can see the whole process of running code on the TRS-80 from start to finish here [1]. For laying out structs, I'd build a macro that expands to the memory representation of the struct. If I wanted a struct representing, say, a 2D point with signed 16-bit little-endian integers for the x and y coords, I would build it from scratch like this (this is a valid program, you can assemble it with Torque): %BYTE:n #nnnn_nnnn ; %LOW:n BYTE:[n 0xff <and>] ; %HIGH:n BYTE:[n 8 <shr>] ; %16LE:n LOW:n HIGH:n ; %POINT:x:y 16LE:x 16LE:y ; @point-1 POINT:50:-7 @point-2 POINT:20:-45 @point-3 POINT:0:0 If I want the address of a field, I can add an offset to the struct address, using macros to name the offset values: %POINT.X 0 ; %POINT.Y 2 ; SET:[point-3 POINT.X +]:15 SET:[point-3 POINT.Y +]:32 Creating a DSL for an existing language wasn't something I'd ever considered. By being a standalone executable it's really easy to use and share, people don't have to install a whole language toolchain in order to use it. Regarding constraints solving and jumping, Torque already throws an error if you try to pack too large a value into too small a field. This works really well for things like relative jumps, because jumping too far will create a value that can't fit in the instruction. I'm planning on adding an error-throwing token to the language that could be used alongside expressions and conditions to further constrain the values accepted by a macro, but I'm really happy with the simplicity of the language so far. The actual internal representation isn't what I'd call an 'IR' per se, nothing like with a C compiler. It's all very pedestrian; the syntax tree is baked down across multiple passes, with macros acting as a glorified copy-paste system. Thanks for the interest and the links, every one of those linked projects is new to me. [0] https://wiki.xxiivv.com/site/uxn.html https://wiki.xxiivv.com/site/uxn.html [1] https://benbridle.com/articles/torque-programming-the-trs-80.html https://benbridle.com/articles/torque-programming-the-trs-80...

zadkey 1y ago

Excellent work man! I know some low level guys who would really appreciate this.

benbridle 1y ago
Thanks, that's great to hear! What platforms do they develop for?

anonzzzies 1y ago

Well done! I write quite a bit of z80 for msx and this seems a nice addition.

curtisszmania 1y ago

[dead]

2ton_jeff 1y ago

Very cool and I like the idea of a "meta-assembler." The most-recent version of flatassembler (fasm 2) is built with fasmg which is also a "meta-assembler" of sorts, in that it also doesn't directly support a specific instruction set and instead is a very powerful macro assembler. I'm keen to check out functionality overlaps between the two implementations. https://board.flatassembler.net/topic.php?t=19389 https://board.flatassembler.net/topic.php?t=19389 https://flatassembler.net/download.php https://flatassembler.net/download.php

benbridle 1y ago
Oh neat! Thanks for the link, I hadn't heard of fasmg before. It looks like fasmg builds up from the byte level, so it would only work for architectures that use 8-bit words. Torque builds up from the bit level, so it can assemble code for architectures like in PIC microcontrollers, using word sizes of 12 or 14 bits. However, fasmg does allow a lot more control over the syntax of the language. The documentation shows some pretty powerful string manipulation that's used to parse real x86 assembler code, which makes sense given the purpose of it. Torque doesn't allow overriding the syntax to that degree, the macro invocation syntax is baked into the assembler. One thing that intrigues me about fasmg is how it handles circular dependencies in expressions [0] (search for 'circular'). Currently in Torque it isn't possible to use a label reference inside a predicate, because the predicate evaluating one way could insert additional code, moving the label and causing the predicate to evaluate the other way [1]. But in fasmg it's possible to use the result of an expression as part of its own calculation. [0] https://flatassembler.net/docs.php?article=fasmg https://flatassembler.net/docs.php?article=fasmg [1] https://benbridle.com/projects/torque/roadmap.html https://benbridle.com/projects/torque/roadmap.html
- 2ton_jeff 1y ago
  I reached out to the author of fasmg WRT your post and circular dependency interest and he pointed me toward two posts that he wrote very specifically to explain what he believes is unique to fasm/fasmg and allows to handle circular dependencies of many kinds. [0] Types of multi-pass assembly, and [1] related pitfalls. [0] https://board.flatassembler.net/topic.php?t=20249 https://board.flatassembler.net/topic.php?t=20249 [1] https://board.flatassembler.net/topic.php?t=21060 https://board.flatassembler.net/topic.php?t=21060

angelmm 1y ago

Any time I read about Z80, my mind wants to try to run something on a GameBoy emulator. Pretty cool project!

eterps 1y ago

Nice! Would be interesting to target the RISC CPU of https://www.projectoberon.net https://www.projectoberon.net with it.

userbinator 1y ago

Reminds me of TDASM from the turn of the century: https://web.archive.org/web/20230906054935/http://www.penguin.cz/~niki/tdasm/ https://web.archive.org/web/20230906054935/http://www.pengui... I remember there were a few other meta-assemblers I came across in the 80s-90s, so this is definitely not "unchartered territory", but it's good to see another one show up. Of course, in the other direction there are meta-disassemblers used for analysis in tools like Ghidra.

kunley 1y ago

Need to address this point: "Assemblers tend to be poorly documented" I wish everything in programming was as good documented as assemblers and ISAs.

benbridle 1y ago
I should clarify, I was thinking of microcontroller toolchains when I wrote this. The documentation I've seen for Intel processors is solid.
- kunley 1y ago
  Not to mention ARM64... And 6502 ;-) (and probably most of the ancient ones)

bjackman 1y ago

At first I thought it was useless: "but each ISA will still end up having different effective syntax because the underlying macro systems will not be designed the same". But then I reread it and realised I was not paying attention to the usecase. It's about making it easy to write assemblers. So this isn't for your Arms and RISC-Vs it's for your random niche microcontrollers where the vendor-provided toolchain kinda sucks. Seems cool! I've experienced a couple of under-documented assemblers in my time. In neither case did this turn out to be that much of a problem in practice, but I guess I just don't write that much assembly.

MathMonkeyMan 1y ago

I'm reminded of a 2016 [talk][1] where Rob Pike describes the common-denominator assembly language that the Go compiler generates. Then that assembly is translated into machine-specific code via table lookups. See the 11 minute mark. [1]: https://www.youtube.com/watch?v=KINIAgRpkDA https://www.youtube.com/watch?v=KINIAgRpkDA

roetlich 1y ago

This looks very cool, maybe I'll try this for risc-v if I have some time. I guess you could also use this to write RAW images, or other binary data.

zzo38computer 1y ago

Is there any support for non-Unicode text? Is there any support for octal numbers? These things should be corrected, hopefully. Also, some of the links in the table of contents of the documentation does not seems to work.

benbridle 1y ago
With character sets, I was initially going to support non-Unicode text by adding a --char-set flag to the assembler, but I decided that the character set should be defined somehow inside each program. My thought was that they could be defined as large table-like macros, something like the following: %BYTE:n #nnnn_nnnn ; %CHAR:n ?[n 'A' ==] BYTE:0x01 ?[n 'B' ==] BYTE:0x02 ?[n 'C' ==] BYTE:0x03 ?[n 'D' ==] BYTE:0x04 ; CHAR:"ABCDABCD" This is, admittedly, quite unweildy for character sets exceeding a few hundred characters, but it would work passably for small character sets like those used for HD44780-style LCD screens. What character sets did you have in mind? Octal was another feature I couldn't make up my mind about, just because I wasn't familiar with any architectures that require it. It'll be trivial to tack on though. For the Z80 instruction set, since the instruction encoding tends to cleave along octal lines, I used the following macro to pack octal digits into bytes, which has the advantage of allowing variables to be passed into each digit (the ADDr macro shows how it's used): %XYZ:x:y:z #xxyyyzzz ; %ADDr:r XYZ:2:0:r ; Thanks for the heads-up about the table of contents, the links should all work now.
- zzo38computer 1y ago
  I think the "0" prefix for octal (as used in the C programming language) is not so good and that "0o" is better; so, if octal literals are implemented then "0o" is better. For non-Unicode text, probably the simplest thing would be to treat the input as a sequence of bytes instead of Unicode characters; or equivalently to treat it as ISO-8859-1 (although programming it to use ISO-8859-1 may be less efficient then just using bytes, possibly; I don't know much about the working of Rust programming, so I don't actually know if it is or not). By "non-Unicode text", I did not mean character mapping, although character mapping is another feature that would be useful to implement, similar to what you mentioned although it could be made more efficient (like you mention). Some way to map a input sequence of bytes (whether or not it is valid UTF-8) to a output character code, would work, probably.

vanderZwan 1y ago

I shared Torque elsewhere and people asked what license it has

benbridle 1y ago
Thanks for the interest! Torque is public domain, feel free to use it or change it or share it however you want. I'd love to hear how people use it, but no obligation. If you shared it someplace public I'd love to pop by, if you have a link.
- vanderZwan 1y ago
  I shared it on the catlang discord, Devine immediately replied "wow, that looks so much like uxn 0_o" hahaha. It's not the most active server but the response has been very positive! A few people thought it would probably be fun to use it to build a custom IR as part of a compiler pipeline, that was the context of the question. I personally really love the utf8 and the "z80 for loop" examples, those are beautiful and make me nostalgic for the days I was writing stuff for the TI-83+ in Z80 assembly.
- benbridle 1y ago
  I've since been informed that public domain isn't really a proper term, legally speaking, so to clarify things I've released Torque under the MIT license.

2ro 1y ago

someone should embed Prolog or microKanren