4 ms·

I am with you on the attractiveness of the data scientist role. I know many people with that title who work on distributed systems, compiler hacks for GPU code,

I am with you on the attractiveness of the data scientist role. I know many people with that title who work on distributed systems, compiler hacks for GPU code, etc. It seems to mean "software engineer who is good at math and therefore smart enough to attack the most interesting projects". If you're a software engineer, there are a million jerks out there who don't understand your job but think they could do it just as well given a couple "21 days" books. If you're a data scientist, you get a lot more autonomy and dibs on the interesting work.

kyllo 14y ago
If you're a software engineer, there are a million jerks out there who don't understand your job but think they could do it just as well given a couple "21 days" books. If you're a data scientist, you get a lot more autonomy and dibs on the interesting work. Well, to paraphrase something you've said in your blog, successful convex work tends to become concave over time. The first business CRUD app was a huge breakthrough. But today, making CRUD apps, which is still how a whole lot of programmers make their living, is becoming an increasingly concave task, as the technically difficult parts are being abstracted away. We won't really need any more LAMP developers soon--Rails is really only one layer of abstraction below the point where non-programmers will be able to generate functioning CRUD apps with a few mouseclicks, to collect their business data. So, those "million jerks" are getting closer to being right every day, if your job as a software engineer is just making CRUD apps. So, now "data science" (aka massively parallel processing of large distributed datasets with statistical modeling and machine learning techniques) is where the convex work is. And of course you get a lot more control and autonomy when you're doing convex work, because it's uncharted territory and no one really understands it yet, so there are no standards or best practices to manage your performance to.

kyllo 14y ago

If you're a software engineer, there are a million jerks out there who don't understand your job but think they could do it just as well given a couple "21 days" books. If you're a data scientist, you get a lot more autonomy and dibs on the interesting work. Well, to paraphrase something you've said in your blog, successful convex work tends to become concave over time. The first business CRUD app was a huge breakthrough. But today, making CRUD apps, which is still how a whole lot of programmers make their living, is becoming an increasingly concave task, as the technically difficult parts are being abstracted away. We won't really need any more LAMP developers soon--Rails is really only one layer of abstraction below the point where non-programmers will be able to generate functioning CRUD apps with a few mouseclicks, to collect their business data. So, those "million jerks" are getting closer to being right every day, if your job as a software engineer is just making CRUD apps. So, now "data science" (aka massively parallel processing of large distributed datasets with statistical modeling and machine learning techniques) is where the convex work is. And of course you get a lot more control and autonomy when you're doing convex work, because it's uncharted territory and no one really understands it yet, so there are no standards or best practices to manage your performance to.