6 ms·

Firms Struggle to Find Programmers Who Can Write Parallel Processing Applications

18 points by adammichaelc 18y ago

thorax 18y ago
This is where companies like Cilk Arts are going to get things going, hopefully: http://youtube.com/watch?v=FjGKLYINGsg http://youtube.com/watch?v=FjGKLYINGsg Edit: To provide some more context, they work on tools and extensions for C++. Things like race detection tools and new language keywords that make it easier to parallelize algorithms. Based on this work at MIT: http://supertech.csail.mit.edu/cilk/ http://supertech.csail.mit.edu/cilk/
- seiji 18y ago
  I'll see your ten minute video with annoying sound and raise you an hour long video about Erlang's background, how current parallel architectures work, and how Erlang nicely fits current systems requirements: http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world http://www.infoq.com/presentations/erlang-software-for-a-con... (plug: we also offer Erlang training http://blags.genges.com/itsgenetic/Bay_Area_Erlang_Training http://blags.genges.com/itsgenetic/Bay_Area_Erlang_Training )
  - reitzensteinm 18y ago
    Fold! Er, no pun intended. Thanks for the links!
- mleonhard 18y ago
  Cilk looks like a useful abstraction on top of threading libraries. It will be quite useful for people who already know how to write proper parallel code. Concurrency-challenged programmers will need a language that better protects them from race conditions. I think they'll need a strongly-typed language combined with something like Cilk.
  - ibster 18y ago
    A new company has been spun out of MIT, with the goal of commercializing the Cilk technology. multicore programming blog: http://www.cilk.com/multicore-blog/ http://www.cilk.com/multicore-blog/
- ibster 18y ago
  we think the key to successful adoption of a multicore programming paradigm is the ability to maintain the serial semantics of the original app, rather than having to restructure code. http://www.cilk.com/multicore-blog/ http://www.cilk.com/multicore-blog/
jaydub 18y ago
Last semester I took a course on parallel computing and one of the things the Professor stressed throughout the course was that historically, parallel machines were built focusing first only on the hardware challenges and software/programmability was a mere afterthought. As John Hennessy related in an interview with the ACM: “Many of the early ideas were motivated by observations of what was easy to implement in the hardware rather than what was easy to use [vis-a-vis programming].” We are going to see the situation change when there is a shift towards a more abstract/simplified approach to parallelism, where the minutia of parallelism is handled under the hood. I.e a return to the foundations of PRAM theory of the 70s. Otherwise, I don't see it getting much easier to find that kind of talent.
astrec 18y ago
Wall Street will solve this exactly as they've solved every other talent problem - offer such ungodly amounts of money that every man and his dog wants to enter the field; skim the cream.
henning 18y ago
I don't work in this industry (thankfully), but why work at a megafirm like Goldman Sachs or even the less elite ones like Morgan Stanley when you could work for a cool boutique rocket scientist company like D. E. Shaw or Jane Street? (Jane St uses OCaml very heavily.) You'll have trouble finding good software people as long as you make them work deathmarch hours and treat them like glorified typists while acting like hypermacho traders are gods. They will have problems as long as they view technology as a depressing cost instead of an asset.
- rms 18y ago
  It's pretty hard to get a job at D.E. Shaw...
- deleted 18y ago
  [deleted]
- deleted 18y ago
  [deleted]
- jaydub 18y ago
  Great point. Having been an intern at Wall Street firm in technology, sometimes I got the feeling that that the revenue generating branches are viewed as the champions of the firm, while the technologists are essentially (well-paid) stilts that hold up the business.
nazgulnarsil 18y ago
soon we'll need a dedicated processor just to handle which data gets sent to the other processors.
- iamelgringo 18y ago
  It's here. Have you looked at the Cell processor architecture?
Tichy 18y ago
Are they unable to find programmers capable of doing that, or unable to find programmers who have "5 years experience writing parallel processing applications" on their CV? I suspect the latter.
- pmjordan 18y ago
  Parallel programming has been around for a long time. It just hasn't been mainstream, and, apparently it still isn't. The people who are really good at it are on the bleeding edge, and presumably don't fancy working for some bank.
  - Tichy 18y ago
    Not saying that nobody could have 5 years experience or more with it. But I think if you put some faith into your engineers, they can also learn a new technology. Surely if you hire a decent developer, he or she will be able to master parallel programming within a reasonable timeframe.
1gor 18y ago
Hm... seems to be an opportunity. What is the best language to learn to take advantage of this market demand? Criteria: 1) manages complexity involving multiple processors 2) has utility libraries (or easy adapters can be written for existing C libraries 3) either has existing mindshare in the industry or at least is not too esoteric and can be sold there. Any other criteria? What languages can we select?
- randombit 18y ago
  What languages can we select? Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
- randombit 18y ago
  What languages can we select? Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
- randombit 18y ago
  What languages can we select? Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
- randombit 18y ago
  _What languages can we select?_ Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
- ComputerGuru 18y ago
  If you don't mind being restricted to mostly-Microsoft platforms, C# is the race horse you'll want to place your best on. It's multi-threading support is great (the best I've seen so far w/ regards to the balance between control & simplicity). The way its delegates (callbacks), thread pool, and invokes work is quite impressive & powerful. And, of course, C# can use any C++ library you throw at it w/ P/Invoke & a lot of marshalling.
iamelgringo 18y ago
There's going to be a lot of interest in Erlang in the next couple of years. It is going to take a couple of years for CIO's to realize what a nightmare shared state concurrency is to engineer. After a while, they will be forced to look at different languages to solve the problem. Or, perhaps, they won't see the problem, and startups willing to use languages with a different concurrency model will come to dominate those markets.
ComputerGuru 18y ago
Writing parallel-processing applications isn't easy. I run a non-profit tech R&D organization; and we have internships where under-grad students can try their hand at coding in "a corporate environment" with joint development, toolchains, and the whole shebang. We gauge the students by how much they accomplish in how much/little time and how elegant their solution is. And at towards the end of the internship, we invariably ask the best students to add some form of multi-threading or parallel-processing to the application they're working on (in C++, C#, or Java mostly) and even our best students have trouble with getting proper parallel-processing code working (and these guys had done brilliantly elsewhere). I personally learned it the hard way, scouring the 'net for all the different methods until I built my own "templates" that work great for me.
gaius 18y ago
We'll see a schism form in the developer community. The elite (the people who were doing Java in 1996, i.e. those who actively pursue language evolution) will make the transition smoothly to Erlang, Scala, OCaml/F# and similar. In fact these people have already done so inside their own heads, they're just waiting for the job market to catch up with them. The masses (the "Java enterprise solutions architects", who just learn what they need to pass the test/get a job) will be stuck there churning out the same old same old.

18 points by adammichaelc 18y ago

This is where companies like Cilk Arts are going to get things going, hopefully: http://youtube.com/watch?v=FjGKLYINGsg http://youtube.com/watch?v=FjGKLYINGsg Edit: To provide some more context, they work on tools and extensions for C++. Things like race detection tools and new language keywords that make it easier to parallelize algorithms. Based on this work at MIT: http://supertech.csail.mit.edu/cilk/ http://supertech.csail.mit.edu/cilk/

seiji 18y ago
I'll see your ten minute video with annoying sound and raise you an hour long video about Erlang's background, how current parallel architectures work, and how Erlang nicely fits current systems requirements: http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world http://www.infoq.com/presentations/erlang-software-for-a-con... (plug: we also offer Erlang training http://blags.genges.com/itsgenetic/Bay_Area_Erlang_Training http://blags.genges.com/itsgenetic/Bay_Area_Erlang_Training )
- reitzensteinm 18y ago
  Fold! Er, no pun intended. Thanks for the links!
mleonhard 18y ago
Cilk looks like a useful abstraction on top of threading libraries. It will be quite useful for people who already know how to write proper parallel code. Concurrency-challenged programmers will need a language that better protects them from race conditions. I think they'll need a strongly-typed language combined with something like Cilk.
- ibster 18y ago
  A new company has been spun out of MIT, with the goal of commercializing the Cilk technology. multicore programming blog: http://www.cilk.com/multicore-blog/ http://www.cilk.com/multicore-blog/
ibster 18y ago
we think the key to successful adoption of a multicore programming paradigm is the ability to maintain the serial semantics of the original app, rather than having to restructure code. http://www.cilk.com/multicore-blog/ http://www.cilk.com/multicore-blog/

jaydub 18y ago

Last semester I took a course on parallel computing and one of the things the Professor stressed throughout the course was that historically, parallel machines were built focusing first only on the hardware challenges and software/programmability was a mere afterthought. As John Hennessy related in an interview with the ACM: “Many of the early ideas were motivated by observations of what was easy to implement in the hardware rather than what was easy to use [vis-a-vis programming].” We are going to see the situation change when there is a shift towards a more abstract/simplified approach to parallelism, where the minutia of parallelism is handled under the hood. I.e a return to the foundations of PRAM theory of the 70s. Otherwise, I don't see it getting much easier to find that kind of talent.

astrec 18y ago

Wall Street will solve this exactly as they've solved every other talent problem - offer such ungodly amounts of money that every man and his dog wants to enter the field; skim the cream.

henning 18y ago

I don't work in this industry (thankfully), but why work at a megafirm like Goldman Sachs or even the less elite ones like Morgan Stanley when you could work for a cool boutique rocket scientist company like D. E. Shaw or Jane Street? (Jane St uses OCaml very heavily.) You'll have trouble finding good software people as long as you make them work deathmarch hours and treat them like glorified typists while acting like hypermacho traders are gods. They will have problems as long as they view technology as a depressing cost instead of an asset.

rms 18y ago
It's pretty hard to get a job at D.E. Shaw...
deleted 18y ago
[deleted]
deleted 18y ago
[deleted]
jaydub 18y ago
Great point. Having been an intern at Wall Street firm in technology, sometimes I got the feeling that that the revenue generating branches are viewed as the champions of the firm, while the technologists are essentially (well-paid) stilts that hold up the business.

nazgulnarsil 18y ago

soon we'll need a dedicated processor just to handle which data gets sent to the other processors.

iamelgringo 18y ago
It's here. Have you looked at the Cell processor architecture?

Tichy 18y ago

Are they unable to find programmers capable of doing that, or unable to find programmers who have "5 years experience writing parallel processing applications" on their CV? I suspect the latter.

pmjordan 18y ago
Parallel programming has been around for a long time. It just hasn't been mainstream, and, apparently it still isn't. The people who are really good at it are on the bleeding edge, and presumably don't fancy working for some bank.
- Tichy 18y ago
  Not saying that nobody could have 5 years experience or more with it. But I think if you put some faith into your engineers, they can also learn a new technology. Surely if you hire a decent developer, he or she will be able to master parallel programming within a reasonable timeframe.

1gor 18y ago

Hm... seems to be an opportunity. What is the best language to learn to take advantage of this market demand? Criteria: 1) manages complexity involving multiple processors 2) has utility libraries (or easy adapters can be written for existing C libraries 3) either has existing mindshare in the industry or at least is not too esoteric and can be sold there. Any other criteria? What languages can we select?

randombit 18y ago
What languages can we select? Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
randombit 18y ago
What languages can we select? Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
randombit 18y ago
What languages can we select? Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
randombit 18y ago
_What languages can we select?_ Realistically (ie, if you're asking what to learn to get a job at one of these places), C++, Java, and C# because (outside of handful of places like Jane St) it seems like that is what everybody in these parts uses. (It's 'industry standard', after all [sigh])
ComputerGuru 18y ago
If you don't mind being restricted to mostly-Microsoft platforms, C# is the race horse you'll want to place your best on. It's multi-threading support is great (the best I've seen so far w/ regards to the balance between control & simplicity). The way its delegates (callbacks), thread pool, and invokes work is quite impressive & powerful. And, of course, C# can use any C++ library you throw at it w/ P/Invoke & a lot of marshalling.

iamelgringo 18y ago

There's going to be a lot of interest in Erlang in the next couple of years. It is going to take a couple of years for CIO's to realize what a nightmare shared state concurrency is to engineer. After a while, they will be forced to look at different languages to solve the problem. Or, perhaps, they won't see the problem, and startups willing to use languages with a different concurrency model will come to dominate those markets.

ComputerGuru 18y ago

Writing parallel-processing applications isn't easy. I run a non-profit tech R&D organization; and we have internships where under-grad students can try their hand at coding in "a corporate environment" with joint development, toolchains, and the whole shebang. We gauge the students by how much they accomplish in how much/little time and how elegant their solution is. And at towards the end of the internship, we invariably ask the best students to add some form of multi-threading or parallel-processing to the application they're working on (in C++, C#, or Java mostly) and even our best students have trouble with getting proper parallel-processing code working (and these guys had done brilliantly elsewhere). I personally learned it the hard way, scouring the 'net for all the different methods until I built my own "templates" that work great for me.

gaius 18y ago

We'll see a schism form in the developer community. The elite (the people who were doing Java in 1996, i.e. those who actively pursue language evolution) will make the transition smoothly to Erlang, Scala, OCaml/F# and similar. In fact these people have already done so inside their own heads, they're just waiting for the job market to catch up with them. The masses (the "Java enterprise solutions architects", who just learn what they need to pass the test/get a job) will be stuck there churning out the same old same old.