3 ms·

Low level development is often fun and somewhat easy on mature hardware where someone else has already discovered the issues. But on new hardware, you're deali

Low level development is often fun and somewhat easy on mature hardware where someone else has already discovered the issues. But on new hardware, you're dealing with gremlins. Physical factors can affect hardware, and sometimes you just can't be sure if it's your code, Cthulhu or position of the moon. For example higher temperature can both decrease or increase crystal frequency. Firmware/software can affect hardware through power consumption and other routes. It can be really hard to know if it's hardware or software, except in hindsight. Catching something that happens frequently is one thing, you have oscilloscopes and stuff. But good luck hunting down the issue that happens once per month with 20 test devices. Sometimes these things can take more than half a year to catch and fix. When you mess up in kernel driver, result is often something pretty bad. Like crashing or freezing the system, or worse, silent corruption. Freezes and crashes are great, if you can use kernel debugger or get the dumps. Not so great when it happens on the other side of the world without any low level data to work with. You also have to really understand kernel interfaces, OS power management, driver life-cycle issues and performance ramifications to mention some. I don't mean low level is necessarily hard, but I do mean it really, really depends!

ajdlinux 7y ago
Additionally, when bringing up new hardware, the knowledge you need in order to understand the state of a malfunctioning device and debug a crash is often stored in the head of the logic designer who wrote the HDL, not exactly something you can look up on StackOverflow.
- taneq 7y ago
  It's the R&D equivalent of "contact your network administrator." "But... but I AM the hardware vendor! sob"
  - ajdlinux 7y ago
    It's always great when you need to figure out what a particular register means or the architecture of a particular error handling process or something like that, and you open the hardware spec and find "TBD" because it's all still unreleased...
- vardump 7y ago
  Yeah... just need to have a ton of debug registers included. Especially if it's ASIC. And yes, Stack Overflow is really not going to help you with pretty much any of this. When doing low level programming, better get comfortable with that there's often no one in the world who can answer your question. Find it out yourself.
taneq 7y ago
All of this is true, and you haven't even started getting into prototype hardware where a minor coding error could cause your hardware to literally catch fire. When you're playing with mechatronics, you often get into fun games of "was it mechanical, electrical or software that let the magic smoke out". :)
- jammygit 7y ago
  I heard the term “smoke test” came from the hardware expression “if there’s no smoke, it will probably work”
  - jholman 7y ago
    If you consider the various meaning of "smoke test" at https://en.wikipedia.org/wiki/Smoke_testing https://en.wikipedia.org/wiki/Smoke_testing , I suspect you'll agree that the mechanical meaning is almost certainly older than the electrical or the software. That's where you have some vessel that should be air-tight, and you blow some smoke into it to see if/where the smoke leaks. You could do that for plumbing, you could do that for musical instruments, you could do that for all sorts of things. Maybe the electrical engineers came up with the term "smoke testing" all own their own, but I think it's a lot likelier that they heard it from some plumbers or something, liked it, and then it gained _additional_ cachet from the association with smoke-means-failure.
    - taneq 7y ago
      I don't think it's too much of a stretch to imagine EEs coming up with "smoke test" independently. I've never heard it actually used in the mechanical sense (although I think I was aware of the term) but I've heard many people talk about magic smoke and electrical smoke tests. And the usage isn't quite congruent - an electrical smoke test (as I understand the term, anyway) is very specifically the act of applying full voltage to a freshly assembled and untested piece of equipment and being ready to shut off the power if anything starts to smoke (or less dramatically, seeing if any circuit breakers trip.)
    - cafard 7y ago
jammygit 7y ago
Is low level programming as lucrative? I love it, but I don’t hear many people talking about the jobs
- AnimalMuppet 7y ago
  There isn't as much work in that area, but there aren't as many people that do it, either. It balances out to pay pretty well, at least in my neck of the woods.
- vardump 7y ago
  Lucrative? Do you mean pay? Some low level jobs pay really bad, some very well. So all I can say it might be, or it might not.
Fr0styMatt88 7y ago
This brings back memories.... At a previous employer we had a product of ours that would just randomly reboot. Standard Windows on an industrial PC, nothing you’d think would be that esoteric. It’d look like a sudden power failure. No crashes, nothing in the logs to indicate anything at all. Similar hardware platform to some other products we had where this issue never arose. It literally took months to find the cause, while in the meantime the people that didn’t want us to succeed (think big-company politics) were scoring points against us. We did everything we could think of, fixed a lot of bugs, tried stress-testing the system, things like that. Nothing worked. We were met with skepticism when we started saying it was the hardware. They said it can’t be the hardware.... Except this time it was — and we found out almost purely by good luck. One of the hardware guys happened to have an oscilloscope hooked up to a system when the reboot happened. The reboot was preceded by ‘something strange’ on the scope (I’m likely missing stuff here since this is memory from a few years back). It turned out the motherboard had a bug which would manifest when the system went into a certain lower power state. That’s why stress-testing never caused it. In fact, the testing we did was actively preventing us from catching it!! Lesson well and truly learned. On a related note - is it just me or did Intel have a lot of trouble with their power-management implementation a few years back? I’ve since worked on another product that would randomly hard-freeze and sure enough after checking the Intel errata doc for the CPU/SoC in question (an Atom something-or-other), sure enough there were issues with S-states that meant we needed to limit them (to S1 I believe) in the BIOS, which did indeed fix the freeze issue. To anyone that wants to avoid the long road here, realise that modern Windows, Linux and macOS systems never freeze or kernel panic without a special reason. Part of what I had to unlearn many years ago was the notion that on Windows, freezes and blue-screens were things that ‘just happened’ in the course of otherwise regular operation. Low-level can get you, even if you aren’t doing low-level work.
- vardump 7y ago
  Problems like that are fairly common when you're developing hardware products. Even if you're using products / modules someone else made. > On a related note - is it just me or did Intel have a lot of trouble with their power-management implementation a few years back? Everyone seems to have troubles with their power management stuff! Power state transitions are tricky in hardware, firmware and software. There are a ton of corner cases and assumptions. For example, you have to be careful (in software!) that you don't cause voltage dips by switching on chips and their peripherals too quickly. See [0]. CPUs, chipsets and peripherals have pages and pages of errata that BIOS, microcode/firmware updates and operating systems work around. [0]: https://en.wikipedia.org/wiki/Inrush_current https://en.wikipedia.org/wiki/Inrush_current

ajdlinux 7y ago

Additionally, when bringing up new hardware, the knowledge you need in order to understand the state of a malfunctioning device and debug a crash is often stored in the head of the logic designer who wrote the HDL, not exactly something you can look up on StackOverflow.

taneq 7y ago
It's the R&D equivalent of "contact your network administrator." "But... but I AM the hardware vendor! sob"
- ajdlinux 7y ago
  It's always great when you need to figure out what a particular register means or the architecture of a particular error handling process or something like that, and you open the hardware spec and find "TBD" because it's all still unreleased...
vardump 7y ago
Yeah... just need to have a ton of debug registers included. Especially if it's ASIC. And yes, Stack Overflow is really not going to help you with pretty much any of this. When doing low level programming, better get comfortable with that there's often no one in the world who can answer your question. Find it out yourself.

taneq 7y ago

All of this is true, and you haven't even started getting into prototype hardware where a minor coding error could cause your hardware to literally catch fire. When you're playing with mechatronics, you often get into fun games of "was it mechanical, electrical or software that let the magic smoke out". :)

jammygit 7y ago
I heard the term “smoke test” came from the hardware expression “if there’s no smoke, it will probably work”
- jholman 7y ago
  If you consider the various meaning of "smoke test" at https://en.wikipedia.org/wiki/Smoke_testing https://en.wikipedia.org/wiki/Smoke_testing , I suspect you'll agree that the mechanical meaning is almost certainly older than the electrical or the software. That's where you have some vessel that should be air-tight, and you blow some smoke into it to see if/where the smoke leaks. You could do that for plumbing, you could do that for musical instruments, you could do that for all sorts of things. Maybe the electrical engineers came up with the term "smoke testing" all own their own, but I think it's a lot likelier that they heard it from some plumbers or something, liked it, and then it gained _additional_ cachet from the association with smoke-means-failure.
  - taneq 7y ago
    I don't think it's too much of a stretch to imagine EEs coming up with "smoke test" independently. I've never heard it actually used in the mechanical sense (although I think I was aware of the term) but I've heard many people talk about magic smoke and electrical smoke tests. And the usage isn't quite congruent - an electrical smoke test (as I understand the term, anyway) is very specifically the act of applying full voltage to a freshly assembled and untested piece of equipment and being ready to shut off the power if anything starts to smoke (or less dramatically, seeing if any circuit breakers trip.)
  - cafard 7y ago

jammygit 7y ago

Is low level programming as lucrative? I love it, but I don’t hear many people talking about the jobs

AnimalMuppet 7y ago
There isn't as much work in that area, but there aren't as many people that do it, either. It balances out to pay pretty well, at least in my neck of the woods.
vardump 7y ago
Lucrative? Do you mean pay? Some low level jobs pay really bad, some very well. So all I can say it might be, or it might not.

Fr0styMatt88 7y ago

This brings back memories.... At a previous employer we had a product of ours that would just randomly reboot. Standard Windows on an industrial PC, nothing you’d think would be that esoteric. It’d look like a sudden power failure. No crashes, nothing in the logs to indicate anything at all. Similar hardware platform to some other products we had where this issue never arose. It literally took months to find the cause, while in the meantime the people that didn’t want us to succeed (think big-company politics) were scoring points against us. We did everything we could think of, fixed a lot of bugs, tried stress-testing the system, things like that. Nothing worked. We were met with skepticism when we started saying it was the hardware. They said it can’t be the hardware.... Except this time it was — and we found out almost purely by good luck. One of the hardware guys happened to have an oscilloscope hooked up to a system when the reboot happened. The reboot was preceded by ‘something strange’ on the scope (I’m likely missing stuff here since this is memory from a few years back). It turned out the motherboard had a bug which would manifest when the system went into a certain lower power state. That’s why stress-testing never caused it. In fact, the testing we did was actively preventing us from catching it!! Lesson well and truly learned. On a related note - is it just me or did Intel have a lot of trouble with their power-management implementation a few years back? I’ve since worked on another product that would randomly hard-freeze and sure enough after checking the Intel errata doc for the CPU/SoC in question (an Atom something-or-other), sure enough there were issues with S-states that meant we needed to limit them (to S1 I believe) in the BIOS, which did indeed fix the freeze issue. To anyone that wants to avoid the long road here, realise that modern Windows, Linux and macOS systems never freeze or kernel panic without a special reason. Part of what I had to unlearn many years ago was the notion that on Windows, freezes and blue-screens were things that ‘just happened’ in the course of otherwise regular operation. Low-level can get you, even if you aren’t doing low-level work.

vardump 7y ago
Problems like that are fairly common when you're developing hardware products. Even if you're using products / modules someone else made. > On a related note - is it just me or did Intel have a lot of trouble with their power-management implementation a few years back? Everyone seems to have troubles with their power management stuff! Power state transitions are tricky in hardware, firmware and software. There are a ton of corner cases and assumptions. For example, you have to be careful (in software!) that you don't cause voltage dips by switching on chips and their peripherals too quickly. See [0]. CPUs, chipsets and peripherals have pages and pages of errata that BIOS, microcode/firmware updates and operating systems work around. [0]: https://en.wikipedia.org/wiki/Inrush_current https://en.wikipedia.org/wiki/Inrush_current