5 ms·

So true. At my old job, higher-ups insisted we use Spark on a data-set my iPhone could handle. Blows my mind to this day. On a personal note, I loathe having t

by dvt 9y ago

So true. At my old job, higher-ups insisted we use Spark on a data-set my iPhone could handle. Blows my mind to this day. On a personal note, I loathe having to do interviews (and I also dislike interviewing people), but if none of my projects take off, I'll have to head back into the maw of that soul-crushing corporate machine all over again.

maxxxxx 9y ago
Same in my compnay. We have a dataset of 1 Terabyte that needs to be queried once a week. IT has talked management into installing a Hadoop cluster for this and we have to pay an analytics team. I told my boss that we can probably run this on a SQL database on a USB drive attached to his laptop without problems.
- tqkxzugoaupvwqr 9y ago
  Suggest to your boss to do a test. If your off-the-shelf solution performs equally well, axe the cluster and analytics team. Maybe go to him with an estimated cost comparison.
  - maxxxxx 9y ago
    We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away. They have convinced upper management that we are doing "big data" and upper management likes the sound of this. It's the good old problem that when you do something that's "cool" at the moment you get more credibility than using tried and true tech like SQL databases. I'll probably also jump on the Hadoop bandwagon because it's good for the resume and more exposure inside the company. Total waste of money and we lose a lot of flexibility but who gives a f..k these days. Gotta go with the flow.
    - yeukhon 9y ago
      > We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away. So true, but in my experience, once you spread the words, other business owners will jump in like an infectious disease and ask for a ticket to use your system. Now your USB won't work anymore. So people "higher-up" want to bet on "fail big" than "fail short", very counter-intuitive, but that's exactly how some enterprise IT spend their budget.
    - murukesh_s 9y ago
      I would advice you to jump into Hadoop and also spark. It's a bloody mean world out there who wouldn't care what you have done or how you have done but just wanted to know if you can configure the most popular tool in the market.. you may explain them you didn't get a chance etc, but most of the time interviewers are like your current management.
alfiedotwtf 9y ago
> At my old job, higher-ups insisted we use Spark... Saw the same thing, but with Kubernetes. They spent so much making the platform infinitely scalable, that they ran out of budget to build the actual product that was supposed to sit on top of that infrastructure (╯°□°）╯︵ ┻━┻
- dvt 9y ago
  What's really screwed up is that several people that made that god-awful decision have now moved on to different companies (with pay raises I'm sure). So not only did they make an objectively terrible decision, but they also left the mess to whomever it ended up falling in the lap of, and not only that, but they got rewarded for it, too! Corporate software engineering is insanity.
  - maxxxxx 9y ago
    Always jump on the cool stuff even if it makes no sense for the business. In most places as SW engineer you are not rewarded for creating business value or saving money. You are being rewarded for knowing technologies.
    - mrep 9y ago
      For lower positions/companies, I would probably agree. If you want to make a lot of stable liquid money (400,000$+ for principal engineers at big companies and 7 figures for the top individual contributors), then you need to join a big company and deliver high value results consistently to get those offers/promotions (probably a little luck too).
      - lovich 9y ago
        True, but if you're not one of those principle engineers already then you likely have no ability influence what tech choices are being made and you should just learn the new tech so you can jump ship more easily in the future
itronitron 9y ago
I hope they also set up a Raspberry Pi cluster on which to run Spark.

by dvt 9y ago

maxxxxx 9y ago

Same in my compnay. We have a dataset of 1 Terabyte that needs to be queried once a week. IT has talked management into installing a Hadoop cluster for this and we have to pay an analytics team. I told my boss that we can probably run this on a SQL database on a USB drive attached to his laptop without problems.

tqkxzugoaupvwqr 9y ago
Suggest to your boss to do a test. If your off-the-shelf solution performs equally well, axe the cluster and analytics team. Maybe go to him with an estimated cost comparison.
- maxxxxx 9y ago
  We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away. They have convinced upper management that we are doing "big data" and upper management likes the sound of this. It's the good old problem that when you do something that's "cool" at the moment you get more credibility than using tried and true tech like SQL databases. I'll probably also jump on the Hadoop bandwagon because it's good for the resume and more exposure inside the company. Total waste of money and we lose a lot of flexibility but who gives a f..k these days. Gotta go with the flow.
  - yeukhon 9y ago
    > We already have done tests for in-house testing but nobody wants to stick his neck out and tell IT to go away. So true, but in my experience, once you spread the words, other business owners will jump in like an infectious disease and ask for a ticket to use your system. Now your USB won't work anymore. So people "higher-up" want to bet on "fail big" than "fail short", very counter-intuitive, but that's exactly how some enterprise IT spend their budget.
  - murukesh_s 9y ago
    I would advice you to jump into Hadoop and also spark. It's a bloody mean world out there who wouldn't care what you have done or how you have done but just wanted to know if you can configure the most popular tool in the market.. you may explain them you didn't get a chance etc, but most of the time interviewers are like your current management.

alfiedotwtf 9y ago

> At my old job, higher-ups insisted we use Spark... Saw the same thing, but with Kubernetes. They spent so much making the platform infinitely scalable, that they ran out of budget to build the actual product that was supposed to sit on top of that infrastructure (╯°□°）╯︵ ┻━┻

dvt 9y ago
What's really screwed up is that several people that made that god-awful decision have now moved on to different companies (with pay raises I'm sure). So not only did they make an objectively terrible decision, but they also left the mess to whomever it ended up falling in the lap of, and not only that, but they got rewarded for it, too! Corporate software engineering is insanity.
- maxxxxx 9y ago
  Always jump on the cool stuff even if it makes no sense for the business. In most places as SW engineer you are not rewarded for creating business value or saving money. You are being rewarded for knowing technologies.
  - mrep 9y ago
    For lower positions/companies, I would probably agree. If you want to make a lot of stable liquid money (400,000$+ for principal engineers at big companies and 7 figures for the top individual contributors), then you need to join a big company and deliver high value results consistently to get those offers/promotions (probably a little luck too).
    - lovich 9y ago
      True, but if you're not one of those principle engineers already then you likely have no ability influence what tech choices are being made and you should just learn the new tech so you can jump ship more easily in the future

itronitron 9y ago

I hope they also set up a Raspberry Pi cluster on which to run Spark.