5 ms·
I can see this in my company. IT has for some reason decided to put some data into Hadoop. Now the users need to write spec documents for each little thing and
by maxxxxx 9y ago
I can see this in my company. IT has for some reason decided to put some data into Hadoop. Now the users need to write spec documents for each little thing and every trivial report takes months to get done. Before they had Excel and could create whatever they needed in minutes. Us programmers don't always make life easier for end users. Often we create a level of bureaucracy.
EDIT: I like a lot of the responses here. They clearly illustrate the disconnect between end users and IT/devs in many companies.
- adambard 9y agoIf you're putting Excel-sized data in Hadoop, something has gone seriously wrong in your IT department's decision-making process. It does make for a nice parable about the dangers of shiny-new-thing-syndrome though.
- maxxxxx 9y agoYep. this is going on everywhere. How many solid but boring Java applications have been replaced by a microservice, cloud, NoSQL, NoieJS monster?
- sogen 9y agohttp://www.zdnet.com/article/disgruntled-developer-breaks-thousands-of-javascript-node-js-apps/ http://www.zdnet.com/article/disgruntled-developer-breaks-th...
- jordanb 9y agoNot really. They could be trying to build a company-wide data lake. Some files in the lake will be very small, others will be very large. Because it's in HDFS it won't matter if the very small ones suddenly become very large. End users groan because people groan every time anything changes and they have to learn something new.
- ghostly_s 9y agoIf users were previously empowered to do a task themselves, and now need to request another team do that task on their behalf, that's a very valid reason to be groaning IMO.
- wil421 9y agoYou have to understand that enterprise class companies don’t want Access databases under people desks. It might be valid reason for the sales or ordering guy who loses control to complain. For the IT professionals at the company it allows them to create a scalable solution that can be used at an enterprise level.
- maxxxxx 9y agoAnd suddenly every addition that could be done in 5 minutes now takes months or gets rejected by the devs.
- jordanb 9y agoHadoop isn't Oracle. If you need to write a file to the data lake, you write it. If you need to read something out you do that too. There's no schema, it's actually a file system with actual files in it, like your shared drive. Except that it's distributed, can handle arbitrarily large files and lets you perform queries on those files instead of downloading all 2TB to access a single row.
- klodolph 9y agoI mean, end users also groan because the user experience for spreadsheets is far better than writing map reduce jobs. One is a skill that a large percentage of BS graduates have, another is a skill that a small percentage of CS graduates have.
- jordanb 9y agoYou don't need to write map reduce jobs to access data in Hadoop. In fact, you can do it from excel: https://support.office.com/en-us/article/Connect-to-a-Hadoop-Distributed-File-System-HDFS-Power-Query-8feeb417-e430-4c5b-b136-64b043a00fc2 https://support.office.com/en-us/article/Connect-to-a-Hadoop...
- sokoloff 9y agoThey're practicing RDD -> "Résumé Driven Development"
- maxxxxx 9y agoAgreed. The problem is that RDD works extremely well if you want to get ahead.
- JustAnotherPat 9y agoyou laugh, but a guy who knows Hadoop is worth more than a guy who doesn't all else being equal
- oldandtired 9y agoOne must disagree with this. It doesn't matter what the technology is, if one has the appropriate mindset and enough years under the belt, it will be picked up. What matters in terms of better value is whether or not one can understand the problems space and find an appropriate solution for it. All sorts of solution space solutions can be used, some of which are better than others. The point is to try and keep it simple enough for the user, irrespective of how difficult it might be for the IT guy. The knowledge of Hadoop in and of itself is not as valuable as one would suggest it is. [Edit fix wording]
- sgt101 9y agoWell, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense. Also putting the data into a secured multi user environment with redundancy and also business continuity. as opposed to fecking sharepoint
- Aloha 9y agoSharepoint is where good information goes to die. It's a write only wiki - you put stuff in and you can never find it again.
- muxator 9y agoMervellous. Maybe this is true for a lot of DMSes, as well. But marvellous. :)
- gaius 9y agoWell, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense Yes but https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://aadrake.com/command-line-tools-can-be-235x-faster-th...
- sgt101 9y agoSometimes, but it depends on what you are doing with a data set - even if it's 1.75Gb : from the link... "Since the problem is basically just to look at the result lines of each file and aggregate the different results" which is why this works on this data for this problem. Many problems are not efficient to solve in this way, for example searches through large products and expansions, especially where the evaluation function is costly. Some problems are data bound - you are handling 100'sTb of data, hadoop can be a good fit. Some problems are CPU bound, and embarrassingly parallel - you require massive parallelism to get through the core seconds that you need to solve the problem. If you need 1 million core seconds and you have 1000 cores, and a way of getting the 1000 cores to do 1/1000th of the work... all is good. Some problems are memory bound. You can build single space machines with many TB memory now, also you can cache with SSD's and optanes more than in the past, but Hadoop does offer a utility for these problems. But in a corporate all problems are admin and discovery bound. If your dataset is what I need and I don't know about it, can't find it and anyway if I did then you won't give it to me and I can't unlock it legitimately; I am screwed. If it's a proper corporate asset then I've got a fighting chance.
- michaelmrose 9y agoAnything that can be done in minutes with Excel could be done in relatively short order via an actual programming language.
- cafard 9y agoBy accountants and financial analysts?
- maxxxxx 9y agoWith Excel any reasonably capable business analyst can do sophisticated calculations quickly without needing anything else. With a web app, SQL database and whatever the ramp up cost is huge in comparison.
- michaelmrose 9y agoThe ramp up cost isn't months vs minutes. That is just pure nonsense.
- turc1656 9y agoI disagree. I am working on something right now at my employer where we are taking spreadsheets that have been used for quite a long time (and were created fairly quickly) and they are converting all the business logic to code. Each workbook is taking weeks for them to code and test and then send back to us to sign off on.
- pointytrees 9y agoI'm a coder DOING some of these conversions. I see both sides of the coin here. One additional point I wanted to mention is that once we have done one or two, or a dozen of these conversions, we start to see common themes. More often than expected, there are mistakes and misunderstandings in some of the spreadsheets. There are inconsistencies in the spreadsheets. In IT we also get calls that a spreadsheet is "locked by someone." Or that someone deleted a spreadsheet. With code, and databases, we can ensure consistency between operators. We can also leverage the data across other applications since it lives in a database now. We can pull reports for management more easily, and with more automation. As some other comments mentioned, spreadsheets are great for prototyping, but also have their drawbacks.
- curun1r 9y agoUs programmers also think about things like failure scenarios and disaster recovery. It may be trendy to criticize us for overcomplicating designs--and many do--but non-tech people using Excel are often doing more than just getting the job done...they're creating a disaster waiting to happen.
- never_immune 9y agoI wouldn't generalize too much. I once took a job where a $50 million in revenue a year business was running on a single AWS server that included everything (web, database, cache, etc.) with no backup of the data and the server was hand-maintained. It was truly a disaster waiting to happen.
- oldandtired 9y agoI have been on both sides of the divide and my view is that the overcomplicated designs are more hazardous. If one understands the end user and their requirements, one is generally able to steer them in the appropriate direction. However, the general view of IT departments is that they just dictate what they want and not help with what is needed. policy is set and then demanded to be followed with little or no justification to the end user why and how it will benefit the end user. The number of times that I as IT professional (designer, developer, etc) have had to fight the IT department over some problem to get it solved is far too many. Instead of fixing or changing something that will allow end users to actually get on with their word, IT departments are seen as dictatorial time wasters who don't get back to you to sort out what is needed. Part of our job is to think of failure scenarios and disaster recovery. This should not get in the road of those using the systems being put in place. The end user has a job to do, we as professionals should be providing good solution to enhance as well as protecting them from any disasters.
- christophilus 9y ago> Us programmers don't always make life easier for end users. I'd go so far and say: We programmers often make life more difficult for end users.
- oldandtired 9y agoOne would agree with this. Most IT groups I have come across seem to go out of their way to make life difficult for the end user. This has been a source of work for me over the years, providing a means to bypass the IT department to get end user work done.