Well, what's your comfort zone?
I wish I had a recommendation based on experience for one of these really strange operating systems like EUMEL, Guardian, OS/400, and L3. But I don't. I've used CP/M and MS-DOS, but those are just really limited, not really interesting. Although, with ZCPR and 4DOS, you could make them reasonably usable, it was like coming out of Plato's cave when I switched my primary operating environment from 4DOS to csh on Ultrix.
Squeak is a pretty different operating environment that isn't simply primitive. Oberon is another. They can both run as user processes on top of Linux, as well as on bare metal. Both of them are somewhat alien.
Are you comfortable with embedded development? If not, try Arduino. It starts out easy, since you program the boards in C++, but you have the opportunity to build things that will run for months on a AA battery with submicrosecond interrupt response time — because there's no OS. (It's routine for even programming novices to write their own interrupt handlers.) Arduino instantly gives you the ability to measure things on microsecond timescales, a thousand times faster than you can normally see. Modern boards like the Blue Pill have response latencies in the 100-nanosecond range when they're awake. That's the time it takes light to go 30 meters, as you're probably aware.
In retrocomputing land, VMS was the first OS I used that was really usable. The OpenVMS Hobbyist Program still exists, and it's actually possible to run old versions of Mozilla on it. F-83 was an interactive Forth IDE that provided higher-order programming, virtual memory, and multithreading under MS-DOS, in 1983 — without syntax or types. Turbo Pascal was also an IDE, in a way the first modern IDE, around the same time; the first versions ran on CP/M and MS-DOS. But I think that you kind of had to be grappling with the limitations of BASIC on those systems to appreciate that.
There are Pick systems that still have enthusiastic users: https://www.pickwiki.com/index.php/Pick_Operating_System https://www.pickwiki.com/index.php/Pick_Operating_System but they don't sound appealing to me. Other systems with cult fanbases include FileMaker, HyperCard, and Lotus Agenda, which last I think you can run successfully under FreeDOS. Agenda is interesting in part because it's so alien. (It's easy to forget that it was normal at the time to have to use the program manual to figure out how to exit.)
There are a bunch of modern specialized development environments that can do strange things. Radare2 is an environment focused on reverse engineering. Emacs is focused on text editing, but for some reason it's also the main user interface for interactive proof assistants like Coq and Lean, which are shaping up to be pretty interesting. R is focused on statistics. Jupyter is sort of focused on data visualization, although not really. (Now I see you've been doing deep learning for 10 years, so I guess Jupyter is your best friend.) LibreOffice Calc is focused on rectangular arrays of mostly numerical data (although in many cases their most advanced users use Excel instead). You can develop applications in all of them.
How about math? It's one thing to invoke a Runge-Kutta integration method; it's another to be able to prove convergence bounds on it. And machine-checked formal proof is shaping up to be an interesting thing, like I said.
How about cryptography? That has the advantage that there are right answers and wrong answers, so you can test your code.
How about shaders? Shadertoy is accessible and super fun. Maybe that's too similar to HPC, but the shader parallelism model (similar to ispc) is pretty different from both AVX and MPI.
How about mobile development? SIGCHI papers are full of experimental user interface ideas to explore, and Android Studio is free and relatively usable, if clumsy. Have you seen Onyx Ashanti's Beatjazz?
In the neighborhood of beatjazz, there's livecoding. It's a thrill to get a nightclub full of people dancing to your code, and there are a bunch of different environments.
GNU Radio with an RTL-SDR makes it possible for you to run DSP algorithms on RF signals over a pretty wide frequency range, with applications in communications and sensing. Maybe if you've been doing HPC, DSP is already second nature, but if not it might be rewarding. And DSP has close connections to control theory and image processing, as well as the more obvious applications.
How about alternative programming paradigms? If you're comfortable in procedural and OO programming, how about extreme alternatives — answer-set programming like miniKANREN, constraint-logic programming (as supported by modern Prologs https://www.metalevel.at/prolog/clpz https://www.metalevel.at/prolog/clpz not just Mozart/Oz), Erlang-style fault-tolerance-focused programming, APL-style array programming (though maybe you're familiar enough with that to take it for granted), or Forth? How about strongly typed programming like Haskell, Rust, or OCaml? (And of course Haskell is purely functional, and OCaml is mostly so.)
And STM solvers like Z3 can easily solve problems now that were infeasible only a few years ago.
Also, wasm.
Or maybe try hacking together some games in Godot.
I don't know, myself I find that it's hard to avoid getting out of my comfort zone in some direction, just because the world is so big and my knowledge is so small. Deep learning is the out-of-my-comfort-zone programming thing I want to try next!
Holy moly. Over the past hour I've come back to your response and read it several times. There's so much to unpack there.
I started writing several replies but felt I wasn't able to give you the praise you deserved, but the passage of time compels me to respond so please know you have my full gratitude
Thank you so much for such a detailed reply
*SMT solvers
STM is either a scanning tunneling microscope or software transactional memory, both of which are pretty interesting, but Z3 is neither.
Also, it occurred to me that I didn't even mention browsers, probably because I myself am so comfortable with them. The big software platforms right now are POSIX, Android, Java, browsers, and whatever Microsoft is doing now — it's enough effort to reuse code written for one environment for another that people often just rewrite it. Browsers have by far the best GUI library — not that the DOM is a great or even acceptable interface, but things like React and D3 are — and the major advantage that if you write stuff in a browser you can immediately show it to many people. The development tools are insane, and HTML5 supports most cellphone peripherals in a cross-platform way.