5 ms·

If you're putting Excel-sized data in Hadoop, something has gone seriously wrong in your IT department's decision-making process. It does make for a nice parabl

If you're putting Excel-sized data in Hadoop, something has gone seriously wrong in your IT department's decision-making process. It does make for a nice parable about the dangers of shiny-new-thing-syndrome though.

maxxxxx 9y ago
Yep. this is going on everywhere. How many solid but boring Java applications have been replaced by a microservice, cloud, NoSQL, NoieJS monster?
- sogen 9y ago
  http://www.zdnet.com/article/disgruntled-developer-breaks-thousands-of-javascript-node-js-apps/ http://www.zdnet.com/article/disgruntled-developer-breaks-th...
jordanb 9y ago
Not really. They could be trying to build a company-wide data lake. Some files in the lake will be very small, others will be very large. Because it's in HDFS it won't matter if the very small ones suddenly become very large. End users groan because people groan every time anything changes and they have to learn something new.
- ghostly_s 9y ago
  If users were previously empowered to do a task themselves, and now need to request another team do that task on their behalf, that's a very valid reason to be groaning IMO.
  - wil421 9y ago
    You have to understand that enterprise class companies don’t want Access databases under people desks. It might be valid reason for the sales or ordering guy who loses control to complain. For the IT professionals at the company it allows them to create a scalable solution that can be used at an enterprise level.
    - maxxxxx 9y ago
      And suddenly every addition that could be done in 5 minutes now takes months or gets rejected by the devs.
      - jordanb 9y ago
        Hadoop isn't Oracle. If you need to write a file to the data lake, you write it. If you need to read something out you do that too. There's no schema, it's actually a file system with actual files in it, like your shared drive. Except that it's distributed, can handle arbitrarily large files and lets you perform queries on those files instead of downloading all 2TB to access a single row.
- klodolph 9y ago
  I mean, end users also groan because the user experience for spreadsheets is far better than writing map reduce jobs. One is a skill that a large percentage of BS graduates have, another is a skill that a small percentage of CS graduates have.
  - jordanb 9y ago
    You don't need to write map reduce jobs to access data in Hadoop. In fact, you can do it from excel: https://support.office.com/en-us/article/Connect-to-a-Hadoop-Distributed-File-System-HDFS-Power-Query-8feeb417-e430-4c5b-b136-64b043a00fc2 https://support.office.com/en-us/article/Connect-to-a-Hadoop...
sokoloff 9y ago
They're practicing RDD -> "Résumé Driven Development"
- maxxxxx 9y ago
  Agreed. The problem is that RDD works extremely well if you want to get ahead.
- JustAnotherPat 9y ago
  you laugh, but a guy who knows Hadoop is worth more than a guy who doesn't all else being equal
  - oldandtired 9y ago
    One must disagree with this. It doesn't matter what the technology is, if one has the appropriate mindset and enough years under the belt, it will be picked up. What matters in terms of better value is whether or not one can understand the problems space and find an appropriate solution for it. All sorts of solution space solutions can be used, some of which are better than others. The point is to try and keep it simple enough for the user, irrespective of how difficult it might be for the IT guy. The knowledge of Hadoop in and of itself is not as valuable as one would suggest it is. [Edit fix wording]
    - maxxxxx 9y ago
      Unfortunately that's not how the job market works in most cases.
      - chii 9y ago
        That's only true if the management level can't see that they are screwing themselves over, either willingly or ignorantly. Hiring a hadoop guy for what used to work in excel is a bad decision in most ceases...
sgt101 9y ago
Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense. Also putting the data into a secured multi user environment with redundancy and also business continuity. as opposed to fecking sharepoint
- Aloha 9y ago
  Sharepoint is where good information goes to die. It's a write only wiki - you put stuff in and you can never find it again.
  - muxator 9y ago
    Mervellous. Maybe this is true for a lot of DMSes, as well. But marvellous. :)
- gaius 9y ago
  Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense Yes but https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
  - sgt101 9y ago
    Sometimes, but it depends on what you are doing with a data set - even if it's 1.75Gb : from the link... "Since the problem is basically just to look at the result lines of each file and aggregate the different results" which is why this works on this data for this problem. Many problems are not efficient to solve in this way, for example searches through large products and expansions, especially where the evaluation function is costly. Some problems are data bound - you are handling 100'sTb of data, hadoop can be a good fit. Some problems are CPU bound, and embarrassingly parallel - you require massive parallelism to get through the core seconds that you need to solve the problem. If you need 1 million core seconds and you have 1000 cores, and a way of getting the 1000 cores to do 1/1000th of the work... all is good. Some problems are memory bound. You can build single space machines with many TB memory now, also you can cache with SSD's and optanes more than in the past, but Hadoop does offer a utility for these problems. But in a corporate all problems are admin and discovery bound. If your dataset is what I need and I don't know about it, can't find it and anyway if I did then you won't give it to me and I can't unlock it legitimately; I am screwed. If it's a proper corporate asset then I've got a fighting chance.

maxxxxx 9y ago

Yep. this is going on everywhere. How many solid but boring Java applications have been replaced by a microservice, cloud, NoSQL, NoieJS monster?

sogen 9y ago
http://www.zdnet.com/article/disgruntled-developer-breaks-thousands-of-javascript-node-js-apps/ http://www.zdnet.com/article/disgruntled-developer-breaks-th...

jordanb 9y ago

Not really. They could be trying to build a company-wide data lake. Some files in the lake will be very small, others will be very large. Because it's in HDFS it won't matter if the very small ones suddenly become very large. End users groan because people groan every time anything changes and they have to learn something new.

ghostly_s 9y ago
If users were previously empowered to do a task themselves, and now need to request another team do that task on their behalf, that's a very valid reason to be groaning IMO.
- wil421 9y ago
  You have to understand that enterprise class companies don’t want Access databases under people desks. It might be valid reason for the sales or ordering guy who loses control to complain. For the IT professionals at the company it allows them to create a scalable solution that can be used at an enterprise level.
  - maxxxxx 9y ago
    And suddenly every addition that could be done in 5 minutes now takes months or gets rejected by the devs.
    - jordanb 9y ago
      Hadoop isn't Oracle. If you need to write a file to the data lake, you write it. If you need to read something out you do that too. There's no schema, it's actually a file system with actual files in it, like your shared drive. Except that it's distributed, can handle arbitrarily large files and lets you perform queries on those files instead of downloading all 2TB to access a single row.
klodolph 9y ago
I mean, end users also groan because the user experience for spreadsheets is far better than writing map reduce jobs. One is a skill that a large percentage of BS graduates have, another is a skill that a small percentage of CS graduates have.
- jordanb 9y ago
  You don't need to write map reduce jobs to access data in Hadoop. In fact, you can do it from excel: https://support.office.com/en-us/article/Connect-to-a-Hadoop-Distributed-File-System-HDFS-Power-Query-8feeb417-e430-4c5b-b136-64b043a00fc2 https://support.office.com/en-us/article/Connect-to-a-Hadoop...

sokoloff 9y ago

They're practicing RDD -> "Résumé Driven Development"

maxxxxx 9y ago
Agreed. The problem is that RDD works extremely well if you want to get ahead.
JustAnotherPat 9y ago
you laugh, but a guy who knows Hadoop is worth more than a guy who doesn't all else being equal
- oldandtired 9y ago
  One must disagree with this. It doesn't matter what the technology is, if one has the appropriate mindset and enough years under the belt, it will be picked up. What matters in terms of better value is whether or not one can understand the problems space and find an appropriate solution for it. All sorts of solution space solutions can be used, some of which are better than others. The point is to try and keep it simple enough for the user, irrespective of how difficult it might be for the IT guy. The knowledge of Hadoop in and of itself is not as valuable as one would suggest it is. [Edit fix wording]
  - maxxxxx 9y ago
    Unfortunately that's not how the job market works in most cases.
    - chii 9y ago
      That's only true if the management level can't see that they are screwing themselves over, either willingly or ignorantly. Hiring a hadoop guy for what used to work in excel is a bad decision in most ceases...

sgt101 9y ago

Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense. Also putting the data into a secured multi user environment with redundancy and also business continuity. as opposed to fecking sharepoint

Aloha 9y ago
Sharepoint is where good information goes to die. It's a write only wiki - you put stuff in and you can never find it again.
- muxator 9y ago
  Mervellous. Maybe this is true for a lot of DMSes, as well. But marvellous. :)
gaius 9y ago
Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense Yes but https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
- sgt101 9y ago
  Sometimes, but it depends on what you are doing with a data set - even if it's 1.75Gb : from the link... "Since the problem is basically just to look at the result lines of each file and aggregate the different results" which is why this works on this data for this problem. Many problems are not efficient to solve in this way, for example searches through large products and expansions, especially where the evaluation function is costly. Some problems are data bound - you are handling 100'sTb of data, hadoop can be a good fit. Some problems are CPU bound, and embarrassingly parallel - you require massive parallelism to get through the core seconds that you need to solve the problem. If you need 1 million core seconds and you have 1000 cores, and a way of getting the 1000 cores to do 1/1000th of the work... all is good. Some problems are memory bound. You can build single space machines with many TB memory now, also you can cache with SSD's and optanes more than in the past, but Hadoop does offer a utility for these problems. But in a corporate all problems are admin and discovery bound. If your dataset is what I need and I don't know about it, can't find it and anyway if I did then you won't give it to me and I can't unlock it legitimately; I am screwed. If it's a proper corporate asset then I've got a fighting chance.