10 ms·

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However wi

by pkaye 8y ago

I worked on SSD firmware for quite a long time and here is my perspective. Early flash used to fairly reliable with almost minimal error correction. However with increasing density, smaller processes and multi level cells, it has gone progressively less reliable and slower. Here are some of the things that we need to worry about: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/20160808_PreConfH_Parnell.pdf https://www.flashmemorysummit.com/English/Collaterals/Procee... To compensate for all these deficiencies, the SSD architecture and hence the entire FTL becomes very complicated because any part of it can become damaged at any time. We always have to have backup algorithms to recovery from any scenario. Its difficult to build algorithms that can recovery from arbitrary failures in a reasonable time. I cannot have a drive sitting around for 20 minutes trying to fsck itself. Another problem is that the job while rewarding is not very lucrative. The chance of a multi million dollar payoff for an employee is low. I have a higher chance working on a web connected gadget to become a millionaire. So that means it is really hard to recruit those who are top notch programmers who known how to figure out the algorithms, write the code, debug the hardware. Most new grads these days are interested in python, javascript and machine learning.

loeg 8y ago
Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not very lucrative. Do you mean it's lower paying than typical bigcorp software jobs outside of FAANG, or just that there aren't a lot of startups with astronomical valuations in the media FTL space?
- godzillabrennus 8y ago
  As someone that identified a bug in a Drobo firmware once and was offered a job on the spot I think the problem with attracting talent is two fold. The first problem is really two parts, not only is it rare to find people who have passion for storage related technologies but very few will gain exposure to these technologies to develop that passion. Kids don't routinely grow up with a SAN in the house. They do tend to grow up with lots of internet connected consumer caliber devices and can easily gain exposure to working with these technologies. I was fortunately able to explore this type of technology in depth because a family owned business let me tinker with their server equipment in high school. After college I then co-founded a startup back before the cloud became big. That meant we needed to make use of old hardware to provide service to our customers at a price point that made our service profitable. Old drives were not a reliable way to do that. New drives were extremely expensive for old servers back in the day when SCSI was the interface that you expected for a server. We had to get creative and play with JBOD devices. ZFS was an amazing tool for us in those days, and it still is for anyone who wants to tinker. The other aspect is that while these skills are valuable for creating a "job" they do not have potential for creating "massive wealth". Why learn about storage if you aren't going to be part of the first 10 employees at a company that has a $10B exit? Let Amazon and the other cloud vendors worry about that stuff. Knowledge is power though. I recently came across an AI startup that I'm now helping. They were spending significant money using GPU computational power to provide artificial intelligence training through a cloud provider. They blew through about $300k in credits within the first year to give you an idea of how much money that type of power can cost. I am now helping them cut over to their own co-location facility. The first year alone they will save so much money it will pay for the next three years. Then you read articles like this: https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge https://www.newyorker.com/magazine/2018/12/10/the-friendship... Reading that helps reinforce the idea that no matter what path you are on in the field there is potential that some random thing you learned about an SSD firmware helps you optimize some growth stage companies product and ultimately that helps you build wealth.
  - loeg 8y ago
    I'm mostly curious because I work in a storage-adjacent field (NAS) for a BigCorp and the pay is pretty good, if not quite FAANG level. It's not a startup by any means, but I will easily become a multi-millionaire in a handful of years. I was curious about the other side of the fence.
    - apetresc 8y ago
      Huh? If you're going to "easily become a multi-millionaire in a handful of years", then your pay is more than pretty good, and certainly not worse than FAANG level.
      - mandelbrotwurst 8y ago
        Maybe his hands are larger than ours ;)
        loeg 8y ago
        Only the ten digits ;-)
        mandelbrotwurst 8y ago
        Even at that pedestrian level, if you pack them tightly enough I could imagine fitting a few billion years at least into the volume of an average-sized handful!
      - loeg 8y ago
        Sorry, handful of years from today. I've been working for 7 years now. FAANG comp would probably be 20%-25% higher; I'm mostly good at keeping my expenses down and saving a high proportion of my income. I've also had the good fortune of the bull market working in my favor for the entire time I've been employed.
    - hueving 8y ago
      What is a handful of years? That sounds like FAANG pay to me.
- xyzzy_plugh 8y ago
  Both, and usually significantly. Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer. Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazing. I left the embedded space and have never looked back.
  - loeg 8y ago
    > Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer. Yeah, yikes. That does not sound appealing :-).
  - howard941 8y ago
    So $SALT_MINE, a highly profitable privately held Fortune 500 just decided to revamp their pay scales. They've now decided that they want to be at the 50 percentile remuneration-wise in the durable goods sector. That is the white goods sector, washing machines and such. I predict we will lose all of our engineers - embedded dudes included.
    - deleted 8y ago
      [deleted]
    - chrisbennet 8y ago
      Yeah, but their numbers will look good for a year or two and by then they (CEO, etc) will have moved on to another company before things crash.
  - Rinzler89 8y ago
    May I ask how did you manage to leave the embedded world and where did you go after? I'm asking since after investing 6 years into this field, which I love, and jumping a couple of companies I realized the market(Europe) is really bad for this gig. Not only is our work highly challenging it's also poorly paid while at the same time our CEO is crying to the local press they can't find devs(to work for peanuts) and is forced to look for them in Asia.
- klaasvakie 8y ago
  > Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. If the drive part of a RAID setup I would actually prefer it just reports itself failed and doesn't slow down access to the array by scanning itself for 20 minutes. As far as I know, that is one of the main differences when buying enterprise or nas drives compared to consumer drives. With nas drives, the firmware gives up very quickly since the drive is assumed to be part of an array with redundancy. Consumer drives will retry reads for a very long time before reporting i/o error.
  - loeg 8y ago
    Potato, potato. If your array doesn't kick out a drive that stalls for 30 seconds, much less 20 minutes, why not?
russdill 8y ago
This is a bit of a misrepresentation. The only flash that never had ECC was NOR. Some embedded systems had NOR, but I'd be incredibly rare to find a consumer SSD with NOR. When designing a NAND memory product, you aim for some max allowed error rate. You choose the error correction algorithm based on that target. Error rates for NAND products are precisely what the designer intended. Because SSDs are so large and there is such a large number of them, errors can still occur (at a known rate). FTLs should take that into account. Critical data structures can add checksum redundancy which can reduce the error rate for those to an even lower value, which is usually necessary anyway since power disturbances during erase or program can cause programming errors. There are of course a number of patterns that increase error rates that FTLs have to be programmed to prevent. Encrypting is the first important step, since it makes the data unlikely to be uniform. The second is mitigating read disturb.
- russdill 8y ago
  Oh, and as someone noted below, spinning disks have also long had error correction in order to reach some target error rate.
- pkaye 8y ago
  I don't mean to imply NAND flash never had ECC. Just that the early flash had needed maybe 1-3 bits of ECC and I would never see an 1-bit error until maybe a few months of use. Things were a lot easier back then. Scrambling and read disturbs were not a issue until later on. All the things you suggest can be done. Just so many opportunities to screw up in the implementation despite the best efforts.
Matthias247 8y ago
> Another problem is that the job while rewarding is not very lucrative. The chance of a multi million dollar payoff for an employee is low. I have a higher chance working on a web connected gadget to become a millionaire. So that means it is really hard to recruit those who are top notch programmers who known how to figure out the algorithms, write the code, debug the hardware. Most new grads these days are interested in python, javascript and machine learning. That seems unfortunately true for most low-level infrastructure software. I am very good at those things, and have a medium amount of experience of building web software. Yet I can find a lot more jobs on the latter domain, and they probably won't pay worse (most likely even better). Besides pay the non-rewarding things in this domain are mostly that the high complexity is often not properly understood by management, rewarded or taken into account in planning. Which then leads to suboptimal products that have been delivered under time pressure.
- agumonkey 8y ago
  I don't know how the system could / should interface with stupid trends sucking resources (high complexity projects, smart individuals, long term).
  - dgzl 8y ago
    The business would have to offer employees something lucrative they otherwise wouldn't earn. I'm not sure what that would be however.
    - the8472 8y ago
      open source all the things, get to collaborate with smart people all over the world.
  - snazz 8y ago
    Infrastructure almost always becomes a commodity, or at least moves behind the scenes. For it to be lucrative, interesting technical position, you have to have a large-scale enterprise (like Google), serving millions of people, and you have to promote it as a worthwhile career path (also done at Google as the SRE position).
howard941 8y ago
It's not any harder to recruit for that role than any other bare metal, embedded position where new grads are expected to be top notch programmers who know how to figure out the algorithms, write the code, debug the hardware, and bring decades of experience. IOW you're recruiting for unicorns and the pickings are slim.
deleted 8y ago
[deleted]
ezoe 8y ago
But why is that the pay is not so high? Manufacturing the hardware requires a lot of capitol to begin with. Why these SSD vendors can't afford the high salary to lure the top talents?
- ams6110 8y ago
  Hardware is a ruthlessly competitive business and profit margins are very thin.
  - mcv 8y ago
    Even so, I'd imagine more reliable SSDs would be able to fetch a higher price.
ams6110 8y ago
Let's be honest, the chance of a multi-million dollar payoff for an employee is low regardless. Young entry-level devs are optimistic and also vulnerable to believing a line of bullshit on how much their options might be worth some day. I do agree it is a "higher" chance in web/mobile technology, sort of like how your chance of winning the lottery is "higher" if you buy 10 tickets instead of 1.
- woolvalley 8y ago
  You still get paid a lot more working at google working on generic backend protobuf shuffling than you will working on SSD firmware at a hardware company or intel's C++ compiler.
  - deleted 8y ago
    [deleted]
  - monocasa 8y ago
    For those doubting you, the going rate for embedded engineers out here in the Denver area where a lot of these SSD controllers are designed is ~$90k. Embedded engineers get peanuts for some reason.
    - vermilingua 8y ago
      ~$90k is now peanuts. Interesting.
      - monocasa 8y ago
        That's less than 30k in 1980 dollars. Yeah, that's next to nothing.
        checkyoursudo 8y ago
        For example: http://www.pewresearch.org/fact-tank/2018/08/07/for-most-us-workers-real-wages-have-barely-budged-for-decades/ http://www.pewresearch.org/fact-tank/2018/08/07/for-most-us-...
sebazzz 8y ago
Are you saying that TLC SSDs are essentially unreliable and it is a miracle we don't see higher failure rates? Regarding your point regarding languages: I would be interested and motivated, but I probably don't have the skill (probably, only did some x86 ASM/C++) nor the location (Europe). Usually someone doesn't just start with C++ but with a managed language, and once someone lands a job becomes demotivated or just doesn't have enough time.
SolarNet 8y ago
I'm surprised no one has mentioned the real difference here. In most startups employees are granted equity. They are closer to worker collectives than they are corporations. In corporations only the capitalist gets any return, they are extracting value from their employees. The point being, the solution to this sort of systemic problem is for more corporations to become worker owned. If the guy writing the SSD algorithms has a say in governance and a cut in the profits they will want to stick around. It's the stable version of startups.
Vadoff 8y ago
You can even offer 0 equity, but pretty sure if you offer a $180k salary you'll have a flock of new grads apply. The reason you're not getting enough new grads is probably because your compensation isn't competitive enough.
feelix 8y ago
Sorry to spin off topic, but perhaps having the chance to speak to someone that's worked on SSD firmware for the first time... is there any feasible way to recover data after a TRIM command has been issued that you can think of? Is there any way to trick the firmware into not returning 0's when reading the blocks of a deleted file? Mostly interested in doing so for Apple TRIM destroying the entire data recovery and forensics market seems like such a big deal, I still can't believe it although it started years ago
deleted 8y ago
[deleted]
jiveturkey 8y ago
> Another problem is that the job while rewarding is not very lucrative. Why is this a problem? Let's not forget, we live in a capitalistic society. The job of the capitalists is to exploit labor. The cost of labor is a direct result of the market demand (or not) for a more complex or, by your point, a more robust product. It need not simply be a market reaction, though. A vendor can create the market for a more complex/robust product. But still, their job is to exploit the available labor so if the people capable of such are available at a lower cost, so be it. > I have a higher chance working on a web connected gadget to become a millionaire. As others have noted, perhaps true but the numbers are so small it may as well also be zero. However, as you should now be pointedly aware, the perception doesn't match reality. Question: do new grads (or let's say up to 3 years experience) working on SSD firmware, making $90k, have to clock in 60+ hours / week and live in insanely high cost regions where their $120k startup salary qualifies them for housing assistance programs and they have to live with at least one if not more roommates? Or is $90k (per another comment) quite good in relation to the total picture? IOW, my guess is that you are not competing for talent on salary or total comp. My guess is you are competing on general industry attractiveness. The entire mindset around embedded vs web/consumer programming is different. So I think to frame it as a compensation problem is wrong, and by misframing it you will never "solve" it.

by pkaye 8y ago

loeg 8y ago

Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. Host operating systems should probably flag an IO as failed long before 20 minutes and then you know 1) nothing made it to disk and 2) have some chance to avoid introducing additional corruption, if e.g., the OS is smart enough to kick out the drive when this happens. > Another problem is that the job while rewarding is not very lucrative. Do you mean it's lower paying than typical bigcorp software jobs outside of FAANG, or just that there aren't a lot of startups with astronomical valuations in the media FTL space?

godzillabrennus 8y ago
As someone that identified a bug in a Drobo firmware once and was offered a job on the spot I think the problem with attracting talent is two fold. The first problem is really two parts, not only is it rare to find people who have passion for storage related technologies but very few will gain exposure to these technologies to develop that passion. Kids don't routinely grow up with a SAN in the house. They do tend to grow up with lots of internet connected consumer caliber devices and can easily gain exposure to working with these technologies. I was fortunately able to explore this type of technology in depth because a family owned business let me tinker with their server equipment in high school. After college I then co-founded a startup back before the cloud became big. That meant we needed to make use of old hardware to provide service to our customers at a price point that made our service profitable. Old drives were not a reliable way to do that. New drives were extremely expensive for old servers back in the day when SCSI was the interface that you expected for a server. We had to get creative and play with JBOD devices. ZFS was an amazing tool for us in those days, and it still is for anyone who wants to tinker. The other aspect is that while these skills are valuable for creating a "job" they do not have potential for creating "massive wealth". Why learn about storage if you aren't going to be part of the first 10 employees at a company that has a $10B exit? Let Amazon and the other cloud vendors worry about that stuff. Knowledge is power though. I recently came across an AI startup that I'm now helping. They were spending significant money using GPU computational power to provide artificial intelligence training through a cloud provider. They blew through about $300k in credits within the first year to give you an idea of how much money that type of power can cost. I am now helping them cut over to their own co-location facility. The first year alone they will save so much money it will pay for the next three years. Then you read articles like this: https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge https://www.newyorker.com/magazine/2018/12/10/the-friendship... Reading that helps reinforce the idea that no matter what path you are on in the field there is potential that some random thing you learned about an SSD firmware helps you optimize some growth stage companies product and ultimately that helps you build wealth.
- loeg 8y ago
  I'm mostly curious because I work in a storage-adjacent field (NAS) for a BigCorp and the pay is pretty good, if not quite FAANG level. It's not a startup by any means, but I will easily become a multi-millionaire in a handful of years. I was curious about the other side of the fence.
  - apetresc 8y ago
    Huh? If you're going to "easily become a multi-millionaire in a handful of years", then your pay is more than pretty good, and certainly not worse than FAANG level.
    - mandelbrotwurst 8y ago
      Maybe his hands are larger than ours ;)
      - loeg 8y ago
        Only the ten digits ;-)
        mandelbrotwurst 8y ago
        Even at that pedestrian level, if you pack them tightly enough I could imagine fitting a few billion years at least into the volume of an average-sized handful!
    - loeg 8y ago
      Sorry, handful of years from today. I've been working for 7 years now. FAANG comp would probably be 20%-25% higher; I'm mostly good at keeping my expenses down and saving a high proportion of my income. I've also had the good fortune of the bull market working in my favor for the entire time I've been employed.
  - hueving 8y ago
    What is a handful of years? That sounds like FAANG pay to me.
xyzzy_plugh 8y ago
Both, and usually significantly. Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer. Embedded also attracts a certain type of engineer, usually very smart and able to manage extreme complexity with attention to detail but at the cost of anything resembling readable, let alone maintainable, software. The fact that anything at all works in the modern world is amazing. I left the embedded space and have never looked back.
- loeg 8y ago
  > Last I checked it was nearly twice as lucrative to be a Ruby-on-rails developer than an embedded engineer. Yeah, yikes. That does not sound appealing :-).
- howard941 8y ago
  So $SALT_MINE, a highly profitable privately held Fortune 500 just decided to revamp their pay scales. They've now decided that they want to be at the 50 percentile remuneration-wise in the durable goods sector. That is the white goods sector, washing machines and such. I predict we will lose all of our engineers - embedded dudes included.
  - deleted 8y ago
    [deleted]
  - chrisbennet 8y ago
    Yeah, but their numbers will look good for a year or two and by then they (CEO, etc) will have moved on to another company before things crash.
- Rinzler89 8y ago
  May I ask how did you manage to leave the embedded world and where did you go after? I'm asking since after investing 6 years into this field, which I love, and jumping a couple of companies I realized the market(Europe) is really bad for this gig. Not only is our work highly challenging it's also poorly paid while at the same time our CEO is crying to the local press they can't find devs(to work for peanuts) and is forced to look for them in Asia.
klaasvakie 8y ago
> Users and administrators almost certainly prefer a 20 minute IO latency over data corruption. If the drive part of a RAID setup I would actually prefer it just reports itself failed and doesn't slow down access to the array by scanning itself for 20 minutes. As far as I know, that is one of the main differences when buying enterprise or nas drives compared to consumer drives. With nas drives, the firmware gives up very quickly since the drive is assumed to be part of an array with redundancy. Consumer drives will retry reads for a very long time before reporting i/o error.
- loeg 8y ago
  Potato, potato. If your array doesn't kick out a drive that stalls for 30 seconds, much less 20 minutes, why not?

russdill 8y ago

This is a bit of a misrepresentation. The only flash that never had ECC was NOR. Some embedded systems had NOR, but I'd be incredibly rare to find a consumer SSD with NOR. When designing a NAND memory product, you aim for some max allowed error rate. You choose the error correction algorithm based on that target. Error rates for NAND products are precisely what the designer intended. Because SSDs are so large and there is such a large number of them, errors can still occur (at a known rate). FTLs should take that into account. Critical data structures can add checksum redundancy which can reduce the error rate for those to an even lower value, which is usually necessary anyway since power disturbances during erase or program can cause programming errors. There are of course a number of patterns that increase error rates that FTLs have to be programmed to prevent. Encrypting is the first important step, since it makes the data unlikely to be uniform. The second is mitigating read disturb.

russdill 8y ago
Oh, and as someone noted below, spinning disks have also long had error correction in order to reach some target error rate.
pkaye 8y ago
I don't mean to imply NAND flash never had ECC. Just that the early flash had needed maybe 1-3 bits of ECC and I would never see an 1-bit error until maybe a few months of use. Things were a lot easier back then. Scrambling and read disturbs were not a issue until later on. All the things you suggest can be done. Just so many opportunities to screw up in the implementation despite the best efforts.

Matthias247 8y ago

> Another problem is that the job while rewarding is not very lucrative. The chance of a multi million dollar payoff for an employee is low. I have a higher chance working on a web connected gadget to become a millionaire. So that means it is really hard to recruit those who are top notch programmers who known how to figure out the algorithms, write the code, debug the hardware. Most new grads these days are interested in python, javascript and machine learning. That seems unfortunately true for most low-level infrastructure software. I am very good at those things, and have a medium amount of experience of building web software. Yet I can find a lot more jobs on the latter domain, and they probably won't pay worse (most likely even better). Besides pay the non-rewarding things in this domain are mostly that the high complexity is often not properly understood by management, rewarded or taken into account in planning. Which then leads to suboptimal products that have been delivered under time pressure.

agumonkey 8y ago
I don't know how the system could / should interface with stupid trends sucking resources (high complexity projects, smart individuals, long term).
- dgzl 8y ago
  The business would have to offer employees something lucrative they otherwise wouldn't earn. I'm not sure what that would be however.
  - the8472 8y ago
    open source all the things, get to collaborate with smart people all over the world.
- snazz 8y ago
  Infrastructure almost always becomes a commodity, or at least moves behind the scenes. For it to be lucrative, interesting technical position, you have to have a large-scale enterprise (like Google), serving millions of people, and you have to promote it as a worthwhile career path (also done at Google as the SRE position).

howard941 8y ago

It's not any harder to recruit for that role than any other bare metal, embedded position where new grads are expected to be top notch programmers who know how to figure out the algorithms, write the code, debug the hardware, and bring decades of experience. IOW you're recruiting for unicorns and the pickings are slim.

deleted 8y ago

[deleted]

ezoe 8y ago

But why is that the pay is not so high? Manufacturing the hardware requires a lot of capitol to begin with. Why these SSD vendors can't afford the high salary to lure the top talents?

ams6110 8y ago
Hardware is a ruthlessly competitive business and profit margins are very thin.
- mcv 8y ago
  Even so, I'd imagine more reliable SSDs would be able to fetch a higher price.

ams6110 8y ago

Let's be honest, the chance of a multi-million dollar payoff for an employee is low regardless. Young entry-level devs are optimistic and also vulnerable to believing a line of bullshit on how much their options might be worth some day. I do agree it is a "higher" chance in web/mobile technology, sort of like how your chance of winning the lottery is "higher" if you buy 10 tickets instead of 1.

woolvalley 8y ago
You still get paid a lot more working at google working on generic backend protobuf shuffling than you will working on SSD firmware at a hardware company or intel's C++ compiler.
- deleted 8y ago
  [deleted]
- monocasa 8y ago
  For those doubting you, the going rate for embedded engineers out here in the Denver area where a lot of these SSD controllers are designed is ~$90k. Embedded engineers get peanuts for some reason.
  - vermilingua 8y ago
    ~$90k is now peanuts. Interesting.
    - monocasa 8y ago
      That's less than 30k in 1980 dollars. Yeah, that's next to nothing.
      - checkyoursudo 8y ago
        For example: http://www.pewresearch.org/fact-tank/2018/08/07/for-most-us-workers-real-wages-have-barely-budged-for-decades/ http://www.pewresearch.org/fact-tank/2018/08/07/for-most-us-...

sebazzz 8y ago

Are you saying that TLC SSDs are essentially unreliable and it is a miracle we don't see higher failure rates? Regarding your point regarding languages: I would be interested and motivated, but I probably don't have the skill (probably, only did some x86 ASM/C++) nor the location (Europe). Usually someone doesn't just start with C++ but with a managed language, and once someone lands a job becomes demotivated or just doesn't have enough time.

SolarNet 8y ago

I'm surprised no one has mentioned the real difference here. In most startups employees are granted equity. They are closer to worker collectives than they are corporations. In corporations only the capitalist gets any return, they are extracting value from their employees. The point being, the solution to this sort of systemic problem is for more corporations to become worker owned. If the guy writing the SSD algorithms has a say in governance and a cut in the profits they will want to stick around. It's the stable version of startups.

Vadoff 8y ago

You can even offer 0 equity, but pretty sure if you offer a $180k salary you'll have a flock of new grads apply. The reason you're not getting enough new grads is probably because your compensation isn't competitive enough.

feelix 8y ago

Sorry to spin off topic, but perhaps having the chance to speak to someone that's worked on SSD firmware for the first time... is there any feasible way to recover data after a TRIM command has been issued that you can think of? Is there any way to trick the firmware into not returning 0's when reading the blocks of a deleted file? Mostly interested in doing so for Apple TRIM destroying the entire data recovery and forensics market seems like such a big deal, I still can't believe it although it started years ago

deleted 8y ago

[deleted]

jiveturkey 8y ago

> Another problem is that the job while rewarding is not very lucrative. Why is this a problem? Let's not forget, we live in a capitalistic society. The job of the capitalists is to exploit labor. The cost of labor is a direct result of the market demand (or not) for a more complex or, by your point, a more robust product. It need not simply be a market reaction, though. A vendor can create the market for a more complex/robust product. But still, their job is to exploit the available labor so if the people capable of such are available at a lower cost, so be it. > I have a higher chance working on a web connected gadget to become a millionaire. As others have noted, perhaps true but the numbers are so small it may as well also be zero. However, as you should now be pointedly aware, the perception doesn't match reality. Question: do new grads (or let's say up to 3 years experience) working on SSD firmware, making $90k, have to clock in 60+ hours / week and live in insanely high cost regions where their $120k startup salary qualifies them for housing assistance programs and they have to live with at least one if not more roommates? Or is $90k (per another comment) quite good in relation to the total picture? IOW, my guess is that you are not competing for talent on salary or total comp. My guess is you are competing on general industry attractiveness. The entire mindset around embedded vs web/consumer programming is different. So I think to frame it as a compensation problem is wrong, and by misframing it you will never "solve" it.