13 ms·

Embedded Rust in Production?

205 points by michidk 2y ago

ngcc_hk 2y ago
If you “embed” ulisp on that Platform, the serial interface seems have issues. Wonder whether this is the case of embedded rust?
kahlonel 2y ago
[dead]
cozzyd 2y ago
Debugging on embedded is not THAT hard, if it's just program logic (as opposed to things like interrupts or DMA). Just use gdb on the programmer like you normally would...
dorristim6 2y ago
[dead]
nadyatolica 2y ago
[dead]
bpbp-mango 2y ago
nice pop-up that scrolls you to the top of the article
- 6r17 2y ago
  I was using brave and did not see that popup
sshine 2y ago
Reposted: 2024-02-20 (3 comments): https://news.ycombinator.com/item?id=39446699 https://news.ycombinator.com/item?id=39446699 2024-02-04 (0 comments): https://news.ycombinator.com/item?id=39251131 https://news.ycombinator.com/item?id=39251131
baq 2y ago
As expected, the people problem is the biggest factor. Turns out getting C folks to learn Rust is a difficult proposition (hello, lkml) but the other way around it isn't too much of a problem. I wonder how much of it is low-level experienced developers only ever using C fail to see that C is not the universally best tool (or, 'if all you have is a hammer, everything looks like a nail' question).
- dunefox 2y ago
  > I wonder how much of it is low-level experienced developers only ever using C fail to see that C is not the universally best tool I'd say that's it. From my own experience with software developers, convincing some of them to learn something new is practically impossible.
  - lawn 2y ago
    It's surprising because I personally jump at an opportunity to try something new. Well, with some exceptions...
  - marcyb5st 2y ago
    I generally speaking agree, but I would be more specific: If the person in question has a true passion for the craft, you can regardless of the age/seniority of the developer (at least in my experience). In fact, learning something like a new programing language is a big undertaking and if your work doesn't offer incentives/rewards the will has to come from the person him/herself and so that's why the passion bit I mentioned above. In my experience I also notice that more senior/older devs are more reluctant to learn new things, but I am unsure if that's due having their passion destroyed by many years of bullshit companies politics, pointless meetings/trainings, and adherence to the latest flavor of agile development every quarter or simply an age thing (I'm not there yet and so I can't tell first hand).
    - Lanolderen 2y ago
      Would this not also be partially connected to not wanting to throw away your 15 years experience with C for 1 year experience in Rust assuming the company actually ditches C? (or any other direct replacement language/tool scenario) Sounds like a way to replace yourself by 2 low pay students who also have 1 year of Rust experience.
RandomThoughts3 2y ago
The article is really light on actual details. Basically, this is a company who uses ESP32 to read serial data from batteries using UART and retransmits it in json over MQTT. They apparently had buggy C code to do that (for unspecified reasons) and successfully rewrote that in Rust. Conclusion, you can write ESP32 code in Rust. No information on what that actually entails sadly.
- sgt 2y ago
  Yes I wonder what was wrong with the C code. Was it fixable, was it just a single loop code that ran or could they have used something like FreeRTOS to handle multiple tasks with less chance of bugs?
  - RandomThoughts3 2y ago
    I would have liked to have more details because it's easy to have memory related issues when using a serial protocol in a memory constrained environment and I was wondering if somehow some feature of Rust helped them. I would guess that the more expressive type system is very nice but I don't expect memory safery features to be that useful in this context. Would like to hear from someone with real world experience on that.
    - lsllc 2y ago
      If you're parsing a serial protocol on an embedded system where you're only getting a byte-at-a-time (from a register), or possibly up to 8 at a time, then a parser combinator library such as nom should make it very easy to create a bulletproof parser. Definitely much easier than in C, where I've resorted to using Ragel for safely parsing serial protocols (and also in Go!) since it generates a single small dependency free source file containing the state machine and your "actions", but nom is much more expressive.
      - RandomThoughts3 2y ago
        Deeply doubt so. Parser combinators are notoriously poor performance and memory-wise. They are somewhat nice to write if you like the style and don’t mind the tax but we are talking about ESP32 here. Unless they are inexperienced, the issue is unlikely to be the state machine anyway and more likely to be in how they manage buffering of long message to avoid running out of RAM.
marmaduke 2y ago
I’m doing some C on esp32 currently, and wondered about switching to rust, and I understand that simple cases with Rust are solid. I’m definitely interested in trying it out. However we need to work with a lot of the hardware apis like twai, ble, lte, ota, etc. It seems like support is spotty or it’s DIY. It’s also worth keeping mind the remark about developer availability. I can generally answer questions about the C to someone who’s not experienced in C, while Rust might be another story.
- selfmodruntime 2y ago
  > It seems like support is spotty or it’s DIY I can somewhat attest to this! I have a pretty extensive prototype of using Rust on ESP32 in production. I wrote a reference implementation for ESP32 platforms deployed via BRSKI, a secure remote key bootstrapping protocol which builds upon attestation keys and public key infrastructure. I'm largely using `esp-idf-svc` for most things, which is a Rust-native wrapper around ESP-IDF code. For anything closer to the hardware, I'm using `esp-idf-sys` which is just a thin wrapper around the the `ESP-IDF` SDK. You can see the project here: https://github.com/hm-seclab/open-brski https://github.com/hm-seclab/open-brski Hope this helps! I'm by no means an expert but I sometimes help contribute to the Rust ESP32 ecosystem, so ask away if you want :)
  - marmaduke 2y ago
    Hi! Thanks for your reply. First time hearing about Brski, I’ll have a look I guess your takeaway is that despite spotty support, the benefits of rust are net positive? I guess learning to call C from Rust is gonna be part of it too right ?
    - selfmodruntime 2y ago
      I would say that the benefits are definitely worth it. I make a point to only never use `unwrap` or even `expect` because the panic handler can be a bit spotty on uncommon hardware. I throw an `anyhow` error (a common crate) to a top level handler, log the error there and then crash. Saves extensive logging and I never get the seemingly random crashes I got with C hardware. I almost never use the debugger anymore. The only system crash I get are when I accidentally flood the stack space. > I guess learning to call C from Rust is gonna be part of it too right Depends on the platform, but for ESP32, all native C SDK functions have thin Rust wrappers.
dazzawazza 2y ago
Access to competant Rust developers can be a challenge even for large companies. I recently finished a contract at a (very large game dev) company where some tools were written in Rust. The tools were a re-write of python scripts and added no new functionality but were slightly faster in Rust. The reality was that these tools were unmaintainable by the rest of the company. Only the author "knew" Rust and it was hard to justify a new hire Rust developer to maintain this small set of tools. The only reason these tools were written in Rust was because the dev wanted to learn Rust (a big but common mistake). I pointed out to the Technical Director that this was a big mistake and the teams had taken on a large amount of technical debt for no reason other than the ego of the wanna-be-rust-developer. Since I "knew" Rust he wanted me to maintain it. My advice was to go back to the Python scripts and I left.
- tokinonagare 2y ago
  > some tools were written in Rust. The tools were a re-write Some memes just write themselves.
- ram_rar 2y ago
  > The reality was that these tools were unmaintainable by the rest of the company. Only the author "knew" Rust and it was hard to justify a new hire Rust developer to maintain this small set of tools. Only If I had a dollar for everytime this happens. Choosing a language for the company is a business decision. we have a similar issue with Elixir and now the entire team is spending a quarter just to get rid of it. For anyone building startup on LLMs , this is one killer application I would pay for. The generated code doesn't event have to 100% correct, just right enough for devs to tweek would go a long way.
- serial_dev 2y ago
  While I get most of your points, I believe some companies sometimes can or maybe even should experiment with new languages, and see what benefits it brings them. The important part is evaluating the experiment and having the courage to say “well that didn’t work out, let’s go back to the original version”. They should have evaluated what is “slightly faster”, how much money it saves and how much it will cost extra to maintain it.
cesaref 2y ago
The problem I have with articles like this is that if we were to substitute anything in for 'Rust', it would read the same. I imagine if they had re-written it in C it would also be better than before. If it was an advert for anything, it might be chucking away a dodgy prototype and starting again (which is surprisingly rare). Anyhow, on the other side of the coin, it's good to see newer languages getting a proper outing in real world situations. Proving stuff is 'up to it' can be a bit of a long haul, so every data point is useful.
- simon_void 2y ago
  > Rust takes longer to write than C > but spent basically zero time debugging that's the difference to rewriting it in C
  - nicce 2y ago
    One could argue that when the program is finished without bugs, then it is maybe completed. So does it take actully longer to write with Rust?
    - pornel 2y ago
      It depends how proficient you're in Rust. Once Rust itself is not a difficulty for you, there's a lot of productivity gained from having a modern language with many conveniences, and great tooling. Rust moves more bugs to compile time, so you will technically spend more time getting the code to compile, but in my experience in 99% of cases it's a time saving. And it lets you be more confident about a program correct by construction, rather than merely fuzzed and not observed to crash. The 1% of counter-examples is trying to be too clever with generic interfaces and hitting Rust's limits.
culebron21 2y ago
The author complains there are few developers available to maintain Rust code. I see very few job postings, and almost all of them are either cryptocurrencies (I don't want to waste my life on this), or "3 years professional Rust development in production" (disqualifies self-learners). Given that nowadays most applications are not replied, it makes little sense to spend time even browsing the postings.
- EasyMark 2y ago
  I see postings occasionally for FAANG and Microsoft jobs, and very randomly and rarely from small shops.
- rapsey 2y ago
  There really is a strange dichotomy. Plenty of Rust developers have trouble finding jobs and apparently companies have trouble finding Rust developers.
  - pytness 2y ago
    For being a relatively new language, there are almost no "entry level" positions. Just take a look at https://rustjobs.dev/ https://rustjobs.dev/. Most of them are well paid remote jobs, but they are asking for +3 years of "professional experience" with rust, "with a proven track record of building and deploying production-quality code", and more. Hell, iirc, i saw one asking for proof of contributions to the rust repo (eg, being a core maintainer). edit: to be fair, i saw one position a while ago asking to be willing to learn rust
    - FridgeSeal 2y ago
      > For being a relatively new language, there are almost no "entry level" positions. Isn’t this a fairly well-known phenomenon though? - new language makes waves. The people who picked it up early do some impressive stuff. - early early adopter companies either snap them, or internally make the choice to learn it too. In turn, they do some stuff with it. - gets a reputation for being the hot new thing. Other companies “want in on it”- they want to be able to do the fancy cool things the other places did, but they don’t have the patience, time or culture to grow it themselves so they aim to hire out seniors and everyone with lots of prior experience. <——- many orgs are here. - proliferates enough that it’s well and truly mainstream. Also known as the “hire 1000 Java devs and throw bodies at stuff” stage of hiring and availability. Python, Node, Java, PHP, etc are here.
blae 2y ago
I hate how prevalent AI "art" has become on articles like these.
- augustk 2y ago
  https://www.reddit.com/r/Phobia/comments/webj9b/ai_generated_images_make_me_physically_uneasy_and/ https://www.reddit.com/r/Phobia/comments/webj9b/ai_generated...
- ndndjdjdn 2y ago
  It is soo cheesy. The crab is even pulling a YT thumbnail face.
myrmidon 2y ago
A very important point that the article neglects is that for a lot of embedded platforms, you have to heavily rely on the hardware manufacturers libraries and tooling. All of that stuff is typically gonna be targeted at C. Without this, using even very simple hardware interfaces (like an SPI/I2C bus) is gonna be a huge pain, because you'll have to comb through reference manuals and register descriptions for your processor and piece everything together yourself instead of just calling a few API functions (this is also very error-prone and using Rust is not really gonna help one bit). The only chance to get even halfway decent rust integration is to pick one of the like 3 most popular hardware platforms among hobby enthusiasts (think ESP32, raspberry pico), which is simply not viable for a LOT of embedded applications. So I still think its probably a really bad idea for a typical embedded-shop to fully go for rust right now-- the downsides from lacking tooling/libraries, reduced developer pool and the need to train extant devs appear very hard to overcome to me.
- vvanders 2y ago
  Even with the vendor libraries last time I had to poke at a CAN bus I ended up having to go to that same level because the C helper libraries omitted key features of the interface. There's a number of micros these days where the same libraries are provided using SVD[1] that will generate interfaces which is handy. [1] https://docs.rs/svd2rust/latest/svd2rust/ https://docs.rs/svd2rust/latest/svd2rust/
- wyager 2y ago
  Rust solved this by autogenning code from mfgr published device xml descriptors. Eg https://embassy.dev/ https://embassy.dev/ Better than any C(++) embedded hal I've used
- EasyMark 2y ago
  I think it’s a pretty mixed bag, but yeah no one really has a lot of support for rust, but c++ support is quite common and “good enough” for lots of things where you’d like more complicated but zero-cost abstractions. RAII, getting rid of pointers, maps, vectors, etc
- alex_suzuki 2y ago
Do123 2y ago
Didn't they just have a shitty implementation written in C (could have been any other language...it's not the language!) and than learned from the past mistakes and written a new implementation in Rust(could have been any other language)? And now the author tries to attribute it's success to Rust?
- alex_suzuki 2y ago
  Maybe. But them the dev wouldn’t have been able to add „Completed a migration of legacy C codebase to Rust with zero issues in production“ to their resume. Also might have been easier to get budget for „build something new to replace old thing“ than „fix pesky bugs in somewhat-working existing thing“.
  - zifpanachr23 2y ago
    Given the difference in availability of jobs and the amount of code out there, unless you are working in startup world or on web dev, this would be a major red flag on a resume in a lot of cases. Even the use of the word "legacy" is a problem. Like, this isn't "legacy" you asshole, this is our companies highest revenue product with a decade of hard work behind it and we don't want somebody that is going to be constantly trying to shill rust when we hired you to help maintain an important C codebase that is part of the foundation of the business. We need you to get good at C, despite all its flaws and issues, not complain. This kind of attitude only works in an exceedingly small part of the software world that just happens to be disproportionately represented on sites like HN. Elsewhere, it's not a good luck to be using words like "legacy" on a resume without a lot of explanatory text about why exactly it really was legacy and deserved a full rewrite.
    - ryandrake 2y ago
      “Legacy” is just the tech industry’s passive aggressive euphemism that means “it’s old and I haven’t learned how it works.” I have the same sense as you: when someone tells me they rewrote their company’s “legacy” code, my spidey sense perks up. Now I’m wondering if there was really a good reason or if he just refuses to learn an existing code base.
skwee357 2y ago
As someone who is using Rust in production for a year now [0], albeit in a different industry -- webdev, I really like the language. Sure, the first steps were rough, but eventually DX became decent, and the safety guarantees of Rust allow me to have a safe mind when developing and deploying (something I can't say about other popular dynamic languages). Having said that, I agree with one of the commenters in this thread: Rust is essentially a solution looking for a problem. It is a great language, but it fails to find its niche. Rust developers are nowhere to be found, companies are not hiring Rust developers (except if you want to work in crypto). [0]https://yieldcode.blog/post/one-year-of-rust-in-production/ https://yieldcode.blog/post/one-year-of-rust-in-production/
- wyager 2y ago
  The niche is anywhere C or C++ is used, which it excels at
- wanderingbit 2y ago
  It has definitely found a niche in the crypto space, specifically with the node clients and the underlying new cryptography libraries they use. For instance, the more efficient Ethereum devs can make their node clients, the cheaper it is to run and the more people around the world can run it, which increases decentralization. Rust makes this possible without compromising on stability and maintainability.
  - skwee357 2y ago
    The problem is that I still see all crypto as "scam", thus choosing Rust makes a non-viable choice for career progression. But I don't care about career progression anymore, so :shrug.png:
    - physicsguy 2y ago
      There’s a guy who I work with who went to work for a crypto place that then exploded (as they all tend to do). He had quite a difficult time finding a new position, and was asked multiple times about working for a scammy company. So perhaps don’t underestimate the negative effect it can have career wise…
nsoonhui 2y ago
I'm ignorant about Rust, but to me it's just static type language akin to C#. And C# has IOT library which seems to target Rust most usual use case, namely on embedded platform. C# also has memory safety just like Rust. So why do we need Rust at all? What's the use case for it? Anything that I'm missing?
- fwip 2y ago
  I think the thing you're missing here is that "Internet of Things" usually means machines way beefier than "embedded." IoT is roughly equivalent to a Raspberry Pi - the thing will usually have an operating system that you're running on top of, and most of your existing knowledge about computers will port over. Embedded is the chip in a happy meal toy, or your microwave in 1995. There is no "operating system." Your code is the only code running on the machine.
  - buescher 2y ago
    That kind of embedded work, like connect-the-dots pcb layout work, went overseas 15-20 years ago. The typical new embedded project in the US today has minimally a not-particularly-resource-constrained 8-bit processor with gobs of fancy peripherals, something like the PIC Q10 series or the ST equivalent. The median would be a 32-bit ARM Cortex M. Typical IoT SoCs are a step up from that but it’s not huge. The ESP32 is pretty representative but there are significant other options - see what Amazon supports in FreeRTOS as a starter. It’s a big leap from small IoT SoCs to SBCs like the Pi.
- sn9 2y ago
  It's kinda like getting the benefits of F#'s type system but with compilation to native code without garbage collection or data races.
- 15155 2y ago
  > Anything that I'm missing? C# is garbage collected. This is a no-go in many/most embedded software applications. C# also grants you poor explicit control over heap/stack allocation: this is essential for embedded development.
goodpoint 2y ago
Really? https://join.com/companies/stabl/12642327-initiative-application-full-time?widgetv2=true&pid=d73d1a20e99ab4ced633 https://join.com/companies/stabl/12642327-initiative-applica...
goodpoint 2y ago
> Since Rust, and especially embedded Rust (lots of FFI & unsafe), is quite hard to learn, it is not viable (for us) to retrain a C developer to Rust. Rust does not need a phd in quantum physics. Anybody can learn it with a bit of patience.
- tialaramex 2y ago
  In particular, you could teach it to Physics PhDs, which might be useful as today they are mostly taught Python and so they end up leaning on libraries to go faster because the Python isn't fast enough. Once upon a time a specialist in Computational Chemistry or Physics or whatever would be expected to learn Fortran, and it makes lots of sense to use Rust for the same purpose in the interim (eventually a WUFFS-like special purpose language seems like a better fit to be able to deliver absolute safety and better performance by not needing to care about generality)
deterministic 2y ago
Maybe just maybe hire better C developers? Just an idea.
pixelfarmer 2y ago
I remember looking into Rust for a personal project, on embedded, in 2016. After poking through all of it I decided against doing that because it was clear I'd be spending a lot of time getting Rust working at all instead of doing anything for the project itself. So C it was. The thing I have to say in the context of the article is this: There is no way to know whether a complete rewrite in C would have yielded similar results to Rust. The phrase "C prototype" made me squirm, even more so when reading that in the context of critical infrastructure. It is known by now (or should be) that such prototypes live on like zombies, so unless it is really some throwaway (from the point of architecture!), these things tend to live on for longer than most feel comfortable with. And, being so critical in function, maintainability is one of the primary concerns. Yes, Rust will, eventually, at some point, maybe? the go-to language we use in the embedded field, but we are talking not just about a language replacement, we are talking ecosystem replacement. That is not going to happen overnight. That said, as some mentioned Java, Perl, and such things: I revived a personal Perl 5 project not too long ago that was more than 20 years old by that point. Needed a small change because the latest installment of Perl 5 is a bit more restrictive with some borderline syntax things (good), but other than that it just worked. In the larger context of the project there is also some C code for binary file processing, also >20 years old. Needed a renaming of a POSIX function (arguments and functionality all the same, though), and then it worked, even compiled as native 64 bit code. Granted, there are not that many dependencies beside POSIX, and the code was even back then written to a level of quality that allowed it to run on all sorts of (POSIX) platforms already. Which is why "C prototype" sounds to me like "we cobbled something together", and all sorts of bugs are no surprise then. You can cut only so many corners before it becomes an issue, especially in software that is used all the time and in a critical place of a system. This needs to be done right, else you will waste a lot of time (and money!) afterwards.
- lnsru 2y ago
  Imho Rust will be not much further in embedded world in 2026 than it was in 2016. I managed to get in the role where I have hiring decisions to make. From this perspective I need somebody to be able to work with existing projects in C from the 2006 instead of knowing cool new language. There is no single advantage for a company selling products to change the language used. Transition will only cost money. Regarding C prototype. I wrote it. 6000 lines of code, works nicely, 4200 lines of code were Xilinx driver calls to move data between the hardware blocks. So changing the language will not really bring any benefit. Maybe even opposite - one must study the register calls and read data sheets to create equivalent functions in other language. The code wasn’t beautiful, was created as “prototype” and was at the end the version shipped to a client.
jnordwick 2y ago
TL;DR we had some buggy C code and fixed the bugs then rewrote it in Rust and wow we didn't have as many bugs... that we already fixed. Rewriting something is not the same as the first effort. Try green fielding see how long it can she develop it once you already had the first basic C code it's pretty trivial to convert it to rust most of the time especially just using other libraries. And imagine that. You fixed all the bugs in the prototype and the second rewrite didn't have as many bugs. That's his nothing about the second versions language it just says you fixed all the bugs in the prototype.
dextrous 2y ago
I am a C/C++ dev learning Rust on my own, and enjoying it. I am finally starting to enjoy the jiu jitsu match with the compiler/borrow-checker and the warm “my code is safe” afterglow … but I have a question for the more experienced Rust devs out there, particularly in light of the OP’s observation about “lots of unsafe” in the Rust embedded realm (which makes sense). If your Rust project leans heavily on unsafe code and/or many libraries that use lots of unsafe, then aren’t you fooling yourself to some degree; i.e. trusting that the unsafe code you write or that written by the 10 other people who wrote the unsafe libs you’re using is ok? Seems like that tosses some cold water on the warm afterglow.
- dannymi 2y ago
  >If your Rust project leans heavily on unsafe code and/or many libraries that use lots of unsafe, then aren’t you fooling yourself to some degree; i.e. trusting that the unsafe code you write or that written by the 10 other people who wrote the unsafe libs you’re using is ok? Seems like that tosses some cold water on the warm afterglow. It's true is that you have to trust your dependencies (unsafe or not). Not needing to trust at all that developers know what they are doing was never a thing a programming language could provide. We can only carve out some specific properties that we can machine-check in a limited way. There are limits on what a type system can do (Rice's theorem, Gödel's incompleteness theorem), and in addition there are limits on what a non-dependent type system can do. Therefore, you either need unsafe (something that adds operations that the type system doesn't model) or you can't write some perfectly OK programs. Basically, the Rust type system is a toy model of your computer's abilities and the domain you want to model. And so is any other type system. The type systems of systems languages at least have some inkling of the actual machine--which is not necessarily the case in non-systems languages. Ask a computer engineer what he thinks about this toy model's misconceptions, like that reading and writing from the same location via the memory bus affect the same thing, or reading the same memory location twice in a row when there's only one cpu is guaranteed to give you the same value, or that reading a memory location can't change it, or that writing to some memory location can't automatically change some other aliased memory location, or that writing some memory location from cpu 1 means cpu 2 can immediately read that new value out etcetc. I could go on (memory barriers, cache coherency, paging, ...). This is not specific to Rust. I'm not sure why we are having new "unsafe" discussions lately. Java and .NET have unsafe as well. Didn't we have that discussion already in ca. year 2000 and everyone arranged themselves with it? What changed? Are there new arguments? If you want to have some empirical tests if the unsafe blocks are broken, run your program under miri. Now you could say that you could just make better and better type systems that encompass everything as it really is. To that I say (1) you can't do that in principle and (2) if you could, humans wouldn't be able to practically use it anymore and (3) It would be too much effort for something that only a tiny minority of programs need in some places. The toy model is pretty good 95% of the time!
n8henrie 2y ago
ESP32. ESP-IDF, not no_std.
Dowwie 2y ago
If you're going to try using rust for esp32 development, you'll use the wrapper libraries that Espressif maintains. When things don't work correctly, you'll be dealing with a larger problem space as you try to figure out whether the problem is your implementation / logic, the lower-level libraries, or the wrapper libraries. I don't know what benefits you're gaining using rust if you're not using a native rust (to the metal) stack of libraries. If only Oxide found a need for esp32 modules, perhaps they would be up for the task?
priio 2y ago
Has anyone tried Nim in embedded systems? I wonder how it went.
- archargelod 2y ago
  I don't have any first-hand experience with embedded myself, but I've seen positive comments on different forums about using Nim in production for embedded devices. For example, see a couple pages of comments by girvo [1]. Also there's a video on writing keyboard driver in Nim [2]. [1] - https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=girvo%20nim&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... [2] - https://youtu.be/dcHEhO4J29U https://youtu.be/dcHEhO4J29U
  - cb321 2y ago
    You might also find interesting Nim on embedded material on the back-commentary of elcritch (https://news.ycombinator.com/user?id=elcritch https://news.ycombinator.com/user?id=elcritch ). E.g.: https://news.ycombinator.com/item?id=40964296 https://news.ycombinator.com/item?id=40964296

205 points by michidk 2y ago

If you “embed” ulisp on that Platform, the serial interface seems have issues. Wonder whether this is the case of embedded rust?

kahlonel 2y ago

[dead]

cozzyd 2y ago

Debugging on embedded is not THAT hard, if it's just program logic (as opposed to things like interrupts or DMA). Just use gdb on the programmer like you normally would...

dorristim6 2y ago

[dead]

nadyatolica 2y ago

[dead]

bpbp-mango 2y ago

nice pop-up that scrolls you to the top of the article

6r17 2y ago
I was using brave and did not see that popup

sshine 2y ago

Reposted: 2024-02-20 (3 comments): https://news.ycombinator.com/item?id=39446699 https://news.ycombinator.com/item?id=39446699 2024-02-04 (0 comments): https://news.ycombinator.com/item?id=39251131 https://news.ycombinator.com/item?id=39251131

baq 2y ago

As expected, the people problem is the biggest factor. Turns out getting C folks to learn Rust is a difficult proposition (hello, lkml) but the other way around it isn't too much of a problem. I wonder how much of it is low-level experienced developers only ever using C fail to see that C is not the universally best tool (or, 'if all you have is a hammer, everything looks like a nail' question).

dunefox 2y ago
> I wonder how much of it is low-level experienced developers only ever using C fail to see that C is not the universally best tool I'd say that's it. From my own experience with software developers, convincing some of them to learn something new is practically impossible.
- lawn 2y ago
  It's surprising because I personally jump at an opportunity to try something new. Well, with some exceptions...
- marcyb5st 2y ago
  I generally speaking agree, but I would be more specific: If the person in question has a true passion for the craft, you can regardless of the age/seniority of the developer (at least in my experience). In fact, learning something like a new programing language is a big undertaking and if your work doesn't offer incentives/rewards the will has to come from the person him/herself and so that's why the passion bit I mentioned above. In my experience I also notice that more senior/older devs are more reluctant to learn new things, but I am unsure if that's due having their passion destroyed by many years of bullshit companies politics, pointless meetings/trainings, and adherence to the latest flavor of agile development every quarter or simply an age thing (I'm not there yet and so I can't tell first hand).
  - Lanolderen 2y ago
    Would this not also be partially connected to not wanting to throw away your 15 years experience with C for 1 year experience in Rust assuming the company actually ditches C? (or any other direct replacement language/tool scenario) Sounds like a way to replace yourself by 2 low pay students who also have 1 year of Rust experience.

RandomThoughts3 2y ago

The article is really light on actual details. Basically, this is a company who uses ESP32 to read serial data from batteries using UART and retransmits it in json over MQTT. They apparently had buggy C code to do that (for unspecified reasons) and successfully rewrote that in Rust. Conclusion, you can write ESP32 code in Rust. No information on what that actually entails sadly.

sgt 2y ago
Yes I wonder what was wrong with the C code. Was it fixable, was it just a single loop code that ran or could they have used something like FreeRTOS to handle multiple tasks with less chance of bugs?
- RandomThoughts3 2y ago
  I would have liked to have more details because it's easy to have memory related issues when using a serial protocol in a memory constrained environment and I was wondering if somehow some feature of Rust helped them. I would guess that the more expressive type system is very nice but I don't expect memory safery features to be that useful in this context. Would like to hear from someone with real world experience on that.
  - lsllc 2y ago
    If you're parsing a serial protocol on an embedded system where you're only getting a byte-at-a-time (from a register), or possibly up to 8 at a time, then a parser combinator library such as nom should make it very easy to create a bulletproof parser. Definitely much easier than in C, where I've resorted to using Ragel for safely parsing serial protocols (and also in Go!) since it generates a single small dependency free source file containing the state machine and your "actions", but nom is much more expressive.
    - RandomThoughts3 2y ago
      Deeply doubt so. Parser combinators are notoriously poor performance and memory-wise. They are somewhat nice to write if you like the style and don’t mind the tax but we are talking about ESP32 here. Unless they are inexperienced, the issue is unlikely to be the state machine anyway and more likely to be in how they manage buffering of long message to avoid running out of RAM.

marmaduke 2y ago

I’m doing some C on esp32 currently, and wondered about switching to rust, and I understand that simple cases with Rust are solid. I’m definitely interested in trying it out. However we need to work with a lot of the hardware apis like twai, ble, lte, ota, etc. It seems like support is spotty or it’s DIY. It’s also worth keeping mind the remark about developer availability. I can generally answer questions about the C to someone who’s not experienced in C, while Rust might be another story.

selfmodruntime 2y ago
> It seems like support is spotty or it’s DIY I can somewhat attest to this! I have a pretty extensive prototype of using Rust on ESP32 in production. I wrote a reference implementation for ESP32 platforms deployed via BRSKI, a secure remote key bootstrapping protocol which builds upon attestation keys and public key infrastructure. I'm largely using `esp-idf-svc` for most things, which is a Rust-native wrapper around ESP-IDF code. For anything closer to the hardware, I'm using `esp-idf-sys` which is just a thin wrapper around the the `ESP-IDF` SDK. You can see the project here: https://github.com/hm-seclab/open-brski https://github.com/hm-seclab/open-brski Hope this helps! I'm by no means an expert but I sometimes help contribute to the Rust ESP32 ecosystem, so ask away if you want :)
- marmaduke 2y ago
  Hi! Thanks for your reply. First time hearing about Brski, I’ll have a look I guess your takeaway is that despite spotty support, the benefits of rust are net positive? I guess learning to call C from Rust is gonna be part of it too right ?
  - selfmodruntime 2y ago
    I would say that the benefits are definitely worth it. I make a point to only never use `unwrap` or even `expect` because the panic handler can be a bit spotty on uncommon hardware. I throw an `anyhow` error (a common crate) to a top level handler, log the error there and then crash. Saves extensive logging and I never get the seemingly random crashes I got with C hardware. I almost never use the debugger anymore. The only system crash I get are when I accidentally flood the stack space. > I guess learning to call C from Rust is gonna be part of it too right Depends on the platform, but for ESP32, all native C SDK functions have thin Rust wrappers.

dazzawazza 2y ago

Access to competant Rust developers can be a challenge even for large companies. I recently finished a contract at a (very large game dev) company where some tools were written in Rust. The tools were a re-write of python scripts and added no new functionality but were slightly faster in Rust. The reality was that these tools were unmaintainable by the rest of the company. Only the author "knew" Rust and it was hard to justify a new hire Rust developer to maintain this small set of tools. The only reason these tools were written in Rust was because the dev wanted to learn Rust (a big but common mistake). I pointed out to the Technical Director that this was a big mistake and the teams had taken on a large amount of technical debt for no reason other than the ego of the wanna-be-rust-developer. Since I "knew" Rust he wanted me to maintain it. My advice was to go back to the Python scripts and I left.

tokinonagare 2y ago
> some tools were written in Rust. The tools were a re-write Some memes just write themselves.
ram_rar 2y ago
> The reality was that these tools were unmaintainable by the rest of the company. Only the author "knew" Rust and it was hard to justify a new hire Rust developer to maintain this small set of tools. Only If I had a dollar for everytime this happens. Choosing a language for the company is a business decision. we have a similar issue with Elixir and now the entire team is spending a quarter just to get rid of it. For anyone building startup on LLMs , this is one killer application I would pay for. The generated code doesn't event have to 100% correct, just right enough for devs to tweek would go a long way.
serial_dev 2y ago
While I get most of your points, I believe some companies sometimes can or maybe even should experiment with new languages, and see what benefits it brings them. The important part is evaluating the experiment and having the courage to say “well that didn’t work out, let’s go back to the original version”. They should have evaluated what is “slightly faster”, how much money it saves and how much it will cost extra to maintain it.

cesaref 2y ago

The problem I have with articles like this is that if we were to substitute anything in for 'Rust', it would read the same. I imagine if they had re-written it in C it would also be better than before. If it was an advert for anything, it might be chucking away a dodgy prototype and starting again (which is surprisingly rare). Anyhow, on the other side of the coin, it's good to see newer languages getting a proper outing in real world situations. Proving stuff is 'up to it' can be a bit of a long haul, so every data point is useful.

simon_void 2y ago
> Rust takes longer to write than C > but spent basically zero time debugging that's the difference to rewriting it in C
- nicce 2y ago
  One could argue that when the program is finished without bugs, then it is maybe completed. So does it take actully longer to write with Rust?
  - pornel 2y ago
    It depends how proficient you're in Rust. Once Rust itself is not a difficulty for you, there's a lot of productivity gained from having a modern language with many conveniences, and great tooling. Rust moves more bugs to compile time, so you will technically spend more time getting the code to compile, but in my experience in 99% of cases it's a time saving. And it lets you be more confident about a program correct by construction, rather than merely fuzzed and not observed to crash. The 1% of counter-examples is trying to be too clever with generic interfaces and hitting Rust's limits.

culebron21 2y ago

The author complains there are few developers available to maintain Rust code. I see very few job postings, and almost all of them are either cryptocurrencies (I don't want to waste my life on this), or "3 years professional Rust development in production" (disqualifies self-learners). Given that nowadays most applications are not replied, it makes little sense to spend time even browsing the postings.

EasyMark 2y ago
I see postings occasionally for FAANG and Microsoft jobs, and very randomly and rarely from small shops.
rapsey 2y ago
There really is a strange dichotomy. Plenty of Rust developers have trouble finding jobs and apparently companies have trouble finding Rust developers.
- pytness 2y ago
  For being a relatively new language, there are almost no "entry level" positions. Just take a look at https://rustjobs.dev/ https://rustjobs.dev/. Most of them are well paid remote jobs, but they are asking for +3 years of "professional experience" with rust, "with a proven track record of building and deploying production-quality code", and more. Hell, iirc, i saw one asking for proof of contributions to the rust repo (eg, being a core maintainer). edit: to be fair, i saw one position a while ago asking to be willing to learn rust
  - FridgeSeal 2y ago
    > For being a relatively new language, there are almost no "entry level" positions. Isn’t this a fairly well-known phenomenon though? - new language makes waves. The people who picked it up early do some impressive stuff. - early early adopter companies either snap them, or internally make the choice to learn it too. In turn, they do some stuff with it. - gets a reputation for being the hot new thing. Other companies “want in on it”- they want to be able to do the fancy cool things the other places did, but they don’t have the patience, time or culture to grow it themselves so they aim to hire out seniors and everyone with lots of prior experience. <——- many orgs are here. - proliferates enough that it’s well and truly mainstream. Also known as the “hire 1000 Java devs and throw bodies at stuff” stage of hiring and availability. Python, Node, Java, PHP, etc are here.

blae 2y ago

I hate how prevalent AI "art" has become on articles like these.

augustk 2y ago
https://www.reddit.com/r/Phobia/comments/webj9b/ai_generated_images_make_me_physically_uneasy_and/ https://www.reddit.com/r/Phobia/comments/webj9b/ai_generated...
ndndjdjdn 2y ago
It is soo cheesy. The crab is even pulling a YT thumbnail face.

myrmidon 2y ago

A very important point that the article neglects is that for a lot of embedded platforms, you have to heavily rely on the hardware manufacturers libraries and tooling. All of that stuff is typically gonna be targeted at C. Without this, using even very simple hardware interfaces (like an SPI/I2C bus) is gonna be a huge pain, because you'll have to comb through reference manuals and register descriptions for your processor and piece everything together yourself instead of just calling a few API functions (this is also very error-prone and using Rust is not really gonna help one bit). The only chance to get even halfway decent rust integration is to pick one of the like 3 most popular hardware platforms among hobby enthusiasts (think ESP32, raspberry pico), which is simply not viable for a LOT of embedded applications. So I still think its probably a really bad idea for a typical embedded-shop to fully go for rust right now-- the downsides from lacking tooling/libraries, reduced developer pool and the need to train extant devs appear very hard to overcome to me.

vvanders 2y ago
Even with the vendor libraries last time I had to poke at a CAN bus I ended up having to go to that same level because the C helper libraries omitted key features of the interface. There's a number of micros these days where the same libraries are provided using SVD[1] that will generate interfaces which is handy. [1] https://docs.rs/svd2rust/latest/svd2rust/ https://docs.rs/svd2rust/latest/svd2rust/
wyager 2y ago
Rust solved this by autogenning code from mfgr published device xml descriptors. Eg https://embassy.dev/ https://embassy.dev/ Better than any C(++) embedded hal I've used
EasyMark 2y ago
I think it’s a pretty mixed bag, but yeah no one really has a lot of support for rust, but c++ support is quite common and “good enough” for lots of things where you’d like more complicated but zero-cost abstractions. RAII, getting rid of pointers, maps, vectors, etc
alex_suzuki 2y ago

Do123 2y ago

Didn't they just have a shitty implementation written in C (could have been any other language...it's not the language!) and than learned from the past mistakes and written a new implementation in Rust(could have been any other language)? And now the author tries to attribute it's success to Rust?

alex_suzuki 2y ago
Maybe. But them the dev wouldn’t have been able to add „Completed a migration of legacy C codebase to Rust with zero issues in production“ to their resume. Also might have been easier to get budget for „build something new to replace old thing“ than „fix pesky bugs in somewhat-working existing thing“.
- zifpanachr23 2y ago
  Given the difference in availability of jobs and the amount of code out there, unless you are working in startup world or on web dev, this would be a major red flag on a resume in a lot of cases. Even the use of the word "legacy" is a problem. Like, this isn't "legacy" you asshole, this is our companies highest revenue product with a decade of hard work behind it and we don't want somebody that is going to be constantly trying to shill rust when we hired you to help maintain an important C codebase that is part of the foundation of the business. We need you to get good at C, despite all its flaws and issues, not complain. This kind of attitude only works in an exceedingly small part of the software world that just happens to be disproportionately represented on sites like HN. Elsewhere, it's not a good luck to be using words like "legacy" on a resume without a lot of explanatory text about why exactly it really was legacy and deserved a full rewrite.
  - ryandrake 2y ago
    “Legacy” is just the tech industry’s passive aggressive euphemism that means “it’s old and I haven’t learned how it works.” I have the same sense as you: when someone tells me they rewrote their company’s “legacy” code, my spidey sense perks up. Now I’m wondering if there was really a good reason or if he just refuses to learn an existing code base.

skwee357 2y ago

As someone who is using Rust in production for a year now [0], albeit in a different industry -- webdev, I really like the language. Sure, the first steps were rough, but eventually DX became decent, and the safety guarantees of Rust allow me to have a safe mind when developing and deploying (something I can't say about other popular dynamic languages). Having said that, I agree with one of the commenters in this thread: Rust is essentially a solution looking for a problem. It is a great language, but it fails to find its niche. Rust developers are nowhere to be found, companies are not hiring Rust developers (except if you want to work in crypto). [0]https://yieldcode.blog/post/one-year-of-rust-in-production/ https://yieldcode.blog/post/one-year-of-rust-in-production/

wyager 2y ago
The niche is anywhere C or C++ is used, which it excels at
wanderingbit 2y ago
It has definitely found a niche in the crypto space, specifically with the node clients and the underlying new cryptography libraries they use. For instance, the more efficient Ethereum devs can make their node clients, the cheaper it is to run and the more people around the world can run it, which increases decentralization. Rust makes this possible without compromising on stability and maintainability.
- skwee357 2y ago
  The problem is that I still see all crypto as "scam", thus choosing Rust makes a non-viable choice for career progression. But I don't care about career progression anymore, so :shrug.png:
  - physicsguy 2y ago
    There’s a guy who I work with who went to work for a crypto place that then exploded (as they all tend to do). He had quite a difficult time finding a new position, and was asked multiple times about working for a scammy company. So perhaps don’t underestimate the negative effect it can have career wise…

nsoonhui 2y ago

I'm ignorant about Rust, but to me it's just static type language akin to C#. And C# has IOT library which seems to target Rust most usual use case, namely on embedded platform. C# also has memory safety just like Rust. So why do we need Rust at all? What's the use case for it? Anything that I'm missing?

fwip 2y ago
I think the thing you're missing here is that "Internet of Things" usually means machines way beefier than "embedded." IoT is roughly equivalent to a Raspberry Pi - the thing will usually have an operating system that you're running on top of, and most of your existing knowledge about computers will port over. Embedded is the chip in a happy meal toy, or your microwave in 1995. There is no "operating system." Your code is the only code running on the machine.
- buescher 2y ago
  That kind of embedded work, like connect-the-dots pcb layout work, went overseas 15-20 years ago. The typical new embedded project in the US today has minimally a not-particularly-resource-constrained 8-bit processor with gobs of fancy peripherals, something like the PIC Q10 series or the ST equivalent. The median would be a 32-bit ARM Cortex M. Typical IoT SoCs are a step up from that but it’s not huge. The ESP32 is pretty representative but there are significant other options - see what Amazon supports in FreeRTOS as a starter. It’s a big leap from small IoT SoCs to SBCs like the Pi.
sn9 2y ago
It's kinda like getting the benefits of F#'s type system but with compilation to native code without garbage collection or data races.
15155 2y ago
> Anything that I'm missing? C# is garbage collected. This is a no-go in many/most embedded software applications. C# also grants you poor explicit control over heap/stack allocation: this is essential for embedded development.

goodpoint 2y ago

Really? https://join.com/companies/stabl/12642327-initiative-application-full-time?widgetv2=true&pid=d73d1a20e99ab4ced633 https://join.com/companies/stabl/12642327-initiative-applica...

goodpoint 2y ago

> Since Rust, and especially embedded Rust (lots of FFI & unsafe), is quite hard to learn, it is not viable (for us) to retrain a C developer to Rust. Rust does not need a phd in quantum physics. Anybody can learn it with a bit of patience.

tialaramex 2y ago
In particular, you could teach it to Physics PhDs, which might be useful as today they are mostly taught Python and so they end up leaning on libraries to go faster because the Python isn't fast enough. Once upon a time a specialist in Computational Chemistry or Physics or whatever would be expected to learn Fortran, and it makes lots of sense to use Rust for the same purpose in the interim (eventually a WUFFS-like special purpose language seems like a better fit to be able to deliver absolute safety and better performance by not needing to care about generality)

deterministic 2y ago

Maybe just maybe hire better C developers? Just an idea.

pixelfarmer 2y ago

I remember looking into Rust for a personal project, on embedded, in 2016. After poking through all of it I decided against doing that because it was clear I'd be spending a lot of time getting Rust working at all instead of doing anything for the project itself. So C it was. The thing I have to say in the context of the article is this: There is no way to know whether a complete rewrite in C would have yielded similar results to Rust. The phrase "C prototype" made me squirm, even more so when reading that in the context of critical infrastructure. It is known by now (or should be) that such prototypes live on like zombies, so unless it is really some throwaway (from the point of architecture!), these things tend to live on for longer than most feel comfortable with. And, being so critical in function, maintainability is one of the primary concerns. Yes, Rust will, eventually, at some point, maybe? the go-to language we use in the embedded field, but we are talking not just about a language replacement, we are talking ecosystem replacement. That is not going to happen overnight. That said, as some mentioned Java, Perl, and such things: I revived a personal Perl 5 project not too long ago that was more than 20 years old by that point. Needed a small change because the latest installment of Perl 5 is a bit more restrictive with some borderline syntax things (good), but other than that it just worked. In the larger context of the project there is also some C code for binary file processing, also >20 years old. Needed a renaming of a POSIX function (arguments and functionality all the same, though), and then it worked, even compiled as native 64 bit code. Granted, there are not that many dependencies beside POSIX, and the code was even back then written to a level of quality that allowed it to run on all sorts of (POSIX) platforms already. Which is why "C prototype" sounds to me like "we cobbled something together", and all sorts of bugs are no surprise then. You can cut only so many corners before it becomes an issue, especially in software that is used all the time and in a critical place of a system. This needs to be done right, else you will waste a lot of time (and money!) afterwards.

lnsru 2y ago
Imho Rust will be not much further in embedded world in 2026 than it was in 2016. I managed to get in the role where I have hiring decisions to make. From this perspective I need somebody to be able to work with existing projects in C from the 2006 instead of knowing cool new language. There is no single advantage for a company selling products to change the language used. Transition will only cost money. Regarding C prototype. I wrote it. 6000 lines of code, works nicely, 4200 lines of code were Xilinx driver calls to move data between the hardware blocks. So changing the language will not really bring any benefit. Maybe even opposite - one must study the register calls and read data sheets to create equivalent functions in other language. The code wasn’t beautiful, was created as “prototype” and was at the end the version shipped to a client.

jnordwick 2y ago

TL;DR we had some buggy C code and fixed the bugs then rewrote it in Rust and wow we didn't have as many bugs... that we already fixed. Rewriting something is not the same as the first effort. Try green fielding see how long it can she develop it once you already had the first basic C code it's pretty trivial to convert it to rust most of the time especially just using other libraries. And imagine that. You fixed all the bugs in the prototype and the second rewrite didn't have as many bugs. That's his nothing about the second versions language it just says you fixed all the bugs in the prototype.

dextrous 2y ago

I am a C/C++ dev learning Rust on my own, and enjoying it. I am finally starting to enjoy the jiu jitsu match with the compiler/borrow-checker and the warm “my code is safe” afterglow … but I have a question for the more experienced Rust devs out there, particularly in light of the OP’s observation about “lots of unsafe” in the Rust embedded realm (which makes sense). If your Rust project leans heavily on unsafe code and/or many libraries that use lots of unsafe, then aren’t you fooling yourself to some degree; i.e. trusting that the unsafe code you write or that written by the 10 other people who wrote the unsafe libs you’re using is ok? Seems like that tosses some cold water on the warm afterglow.

dannymi 2y ago
>If your Rust project leans heavily on unsafe code and/or many libraries that use lots of unsafe, then aren’t you fooling yourself to some degree; i.e. trusting that the unsafe code you write or that written by the 10 other people who wrote the unsafe libs you’re using is ok? Seems like that tosses some cold water on the warm afterglow. It's true is that you have to trust your dependencies (unsafe or not). Not needing to trust at all that developers know what they are doing was never a thing a programming language could provide. We can only carve out some specific properties that we can machine-check in a limited way. There are limits on what a type system can do (Rice's theorem, Gödel's incompleteness theorem), and in addition there are limits on what a non-dependent type system can do. Therefore, you either need unsafe (something that adds operations that the type system doesn't model) or you can't write some perfectly OK programs. Basically, the Rust type system is a toy model of your computer's abilities and the domain you want to model. And so is any other type system. The type systems of systems languages at least have some inkling of the actual machine--which is not necessarily the case in non-systems languages. Ask a computer engineer what he thinks about this toy model's misconceptions, like that reading and writing from the same location via the memory bus affect the same thing, or reading the same memory location twice in a row when there's only one cpu is guaranteed to give you the same value, or that reading a memory location can't change it, or that writing to some memory location can't automatically change some other aliased memory location, or that writing some memory location from cpu 1 means cpu 2 can immediately read that new value out etcetc. I could go on (memory barriers, cache coherency, paging, ...). This is not specific to Rust. I'm not sure why we are having new "unsafe" discussions lately. Java and .NET have unsafe as well. Didn't we have that discussion already in ca. year 2000 and everyone arranged themselves with it? What changed? Are there new arguments? If you want to have some empirical tests if the unsafe blocks are broken, run your program under miri. Now you could say that you could just make better and better type systems that encompass everything as it really is. To that I say (1) you can't do that in principle and (2) if you could, humans wouldn't be able to practically use it anymore and (3) It would be too much effort for something that only a tiny minority of programs need in some places. The toy model is pretty good 95% of the time!

n8henrie 2y ago

ESP32. ESP-IDF, not no_std.

Dowwie 2y ago

If you're going to try using rust for esp32 development, you'll use the wrapper libraries that Espressif maintains. When things don't work correctly, you'll be dealing with a larger problem space as you try to figure out whether the problem is your implementation / logic, the lower-level libraries, or the wrapper libraries. I don't know what benefits you're gaining using rust if you're not using a native rust (to the metal) stack of libraries. If only Oxide found a need for esp32 modules, perhaps they would be up for the task?

priio 2y ago

Has anyone tried Nim in embedded systems? I wonder how it went.

archargelod 2y ago
I don't have any first-hand experience with embedded myself, but I've seen positive comments on different forums about using Nim in production for embedded devices. For example, see a couple pages of comments by girvo [1]. Also there's a video on writing keyboard driver in Nim [2]. [1] - https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=girvo%20nim&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... [2] - https://youtu.be/dcHEhO4J29U https://youtu.be/dcHEhO4J29U
- cb321 2y ago
  You might also find interesting Nim on embedded material on the back-commentary of elcritch (https://news.ycombinator.com/user?id=elcritch https://news.ycombinator.com/user?id=elcritch ). E.g.: https://news.ycombinator.com/item?id=40964296 https://news.ycombinator.com/item?id=40964296