22 ms·

I have seen so so so many projects get bogged down by the need to use a "big data" stack. I think my favorite example, was a team that spent six months trying

I have seen so so so many projects get bogged down by the need to use a "big data" stack. I think my favorite example, was a team that spent six months trying to build a system to take in files, parse them, and store them. Files came through a little less than one per second, which translated to about 100kb. This translated to about 2.5Gb a day of data. The data only needed to be stored for a year, and could easily be compressed. They felt the need to setup a cluster, with 1Tb of RAM to handle processing the documents, they had a 25 Kafka instances, etc. It was just insane. I just pulled out a python script, and combine that with Postgres and within an afternoon I had completed the project (albeit not production ready). This is so typical within companies it makes me gag. They were easily spending $100k a month just on infrastructure, my solution cost ~$400 ($1200 with replication). The sad part, is that convincing management to use my solution was the hardest part. Basically, I had to explain how my system was more robust, faster, cheaper, etc. Even side-by-side comparisons didn't seem to convince them, they just felt the other solution was better some how... Eventually, I convinced them, after about a month of debates, and an endless stream of proof.

deleted 9y ago
[deleted]
quantumhobbit 9y ago
The problem you had was that you were trying to convince management to act in the best interest of the company rather than their own best interest. They would much rather be in charge of a $100k a month project instead of a $1k a month project. Also the big solution required a bunch of engineers working full time which puts them higher up the ladder compared to your 1 engineer working part time solution. You were saving the company money but hurting their resumes.
- brango 9y ago
  Yes. Don't forget "CV-first" development is a thing.
  - maxxxxx 9y ago
    The sad thing is that resume driven development seems to be one of the best strategies for developers these days.
    - 8ytecoder 9y ago
      Anecdote: At the startup (now acquired) that I work for, we simplified the architecture so much that we spend under $3k a month in infra for tens of millions in revenues a year while having 99.9% availability for 5+ years now. I don't have any big name tech to add to my resume. The fact that we ran a e-commerce site with a low budget and few engineers was never recognized by recruiters. They called me based on my experience with "distributed systems" 8 years ago. All recruiters (well, most of them at least) can look for is keywords. On the other hand, we also had difficulties hiring "good" engineers. People chose either a company with a brand name recognition or one that's working on "exciting" technology. As engineers, we fail to appreciate that we are there to serve business first and foremost. As leaders, we fail to put our companies first. This is an industry wide problem. If past trends of other major industries are any indicator, the whole "meritocracy" of tech industry will disappear within the next decade.
      - guelo 9y ago
        > As engineers, we fail to appreciate that we are there to serve business first and foremost The company has zero loyalty to you and will screw you over if it makes business sense to do so. There is absolutely no reason to put the company above your career interests.
        ralmeida 9y ago
        This only holds if you believe that deliberately making decisions that will cost the company more than they should is in your career interests. As said in this topic - it's true that in some places, resume-driven development pays off for the developers. It's not the same everywhere and to me it looks more like a symptom of a dysfunctional structure than par for the course in business. This means it is in your career interests to increase company revenue and/or reduce costs, because this will make you more attractive to many companies (the ones we would likely want to work on) when you move on.
- mattjaynes 9y ago
  Great point. Ultimately, you need to create a situation where they will also be winning career-wise. I recently wrote about this. The TLDR is that 20% time is a great investment and can ultimately save the company a ton of time and money. It gives the engineers some playtime in order to build their CV's and "get their wiggles out". Ultimately, if done right it can protect your production systems from a lot of madness. https://hvops.com/articles/how-to-keep-passion-from-hugging-your-systems-to-death/ https://hvops.com/articles/how-to-keep-passion-from-hugging-...
  - bsg75 9y ago
    > Ultimately, you need to create a situation where they will also be winning career-wise. The message could be that the goal is about the destination not the journey.
- komali2 9y ago
  Sounds like a good time to go over their head. Depending on the size of the company, straight to someone that deals with money - CFO? Dick move but so is rejecting an objectively better solution to save one's own middle management ass.
  - cookiecaper 9y ago
    This is terrible advice. A C-level will see you as a rogue engineer. He will trust his managerial layers over a peon. I know this as I spent far too long being that rogue, peon engineer, screaming into the void. I just didn't get it. It's not about the tech used, and it never has been. It's about being enterprisey. The grandparent is correct. This is 100% a political problem. We have a bad habit of discounting such problems in tech. We shouldn't do that anymore. Life is much better when you cooperate, instead of fighting an uphill battle. Compact, reasonable solutions are the domain of startups. Bloating the engineering layer beyond any reasonable limit is an inherent cost of growing the company, and we shouldn't try to counteract that. We must operate within the framework we're given. The political concerns extend beyond the company's own internals. They must appear enterprisey if they expect to be treated enterprisey. Today, enterprises have "data science" departments and blow $100k/mo on useless crap. If they're not doing that, it's a liability for the whole company, not just from the petty territorial perspective of one individual. It doesn't matter that it's possible to accomplish the same thing with one one-hundredth of the monthly expense. The enterprise signaling is worth the cost.
    - jimbokun 9y ago
      What is the concrete ROI on being "enterprisey"? Google, Facebook, Amazon, Apple are definitely not enterprisey, and I don't think it has hurt their profits. I think they tend to be pretty frugal, overall, when spending their own money on hardware and infrastructure. And I bet if the "enterprisey" company ever tries to compete with a company like Google, Facebook, Amazon, or Apple, they will be destroyed in that market.
      - cookiecaper 9y ago
- osrec 9y ago
  Reminds me of almost every technology project I've seen in finance! It's all about building complex empires, rather than simple, functional solutions. Sometimes, it's also just about using up the assigned technology budget so that it's not scaled down the following year!
  - ythn 9y ago
    Seems like you could spin up a side project business that has a simple solution to their problem but charges enough to soak up their whole budget. Everybody is happy!
    - cookiecaper 9y ago
      Have to provide enough bits and twiddles to allow the dept head to hire his favorite people and you're right. This is basically what Tableau et al have done. There are enough buttons that you can have a few "Tableau guys", and you don't have to hire custom grapher guys.
  - rhizome 9y ago
    Funny, I had a phone screen today and in the midst of asking about how I saw my future and what I wanted to work in, the recruiter was tiptoeing around a situation where someone at the company had suggested React or something and the devs had pushed back (successfully) on the basis that the site didn't need it. I got the feeling he was trying not to step on my dreams of being a cutting-edge JS frontend trailblazer, but it is really a point in the company's favor (to me) that they were able to resist the urge. Basically, they are looking for web developers but it seems like they have to filter out all the frontend ninja rockstars.
    - Rapzid 9y ago
      Why didn't they "need" it? How do they decide what they "need"? This sounds like typical anti-change pushback, which I have learned can actually be a good thing. However, this anecdote is severely lacking in insight; much like most people's support of, or opposition to, change. Further, like the widespread belief that sentences shouldn't start with conjunctions; much less conjunctive adverbs.
- vlunkr 9y ago
  I'm sad about how right you probably are.
- treebeard901 9y ago
  > You were saving the company money but hurting their resumes. To take it a step further, Management has to own up to their original failure and try to explain to their bosses how they could spend so much time and money unnecessarily. Another psychological problem here is the perception by people who do not understand the technology that the higher priced solution is better because it costs more.
  - hinkley 9y ago
    You have the machine that goes 'ping'! You're very lucky. This is the most expensive piece of equipment in the hospital. - Monty Python, Meaning of Life
  - mowenz 9y ago
    And a step further: a 100k project had commissions, meetings, expenses all associated with a 100k project. If people are cheating enough to waste 100k on useless crap expect them to cheat some of that for themselves. That means hotel rooms, airline tickets, meals, etc and all, and any commission the sales guy shares with you for making it happen. And it does happen...
- conjectures 9y ago
  I would love to agree with you but there is another interpretation: they are clueless. Suppose management in a firm uncritically contracted the big data revolution meme. Then they believe they are in the position of a city mayor trying to build a bridge and some wiseacre comes along and says they can do it in an afternoon with two pieces of string and a stick of gum. The problem is that the analogy doesn't hold, but they don't know that.
- vivekd 9y ago
  Add to that, if anything goes wrong, they would have a much easier time cya-ing and convincing their superiors they did all they can do to prevent it if they build a large, complex solution than if they just went to the quickest, simplest one. In fact justifying anything going wrong at all is much easier with large, complex undertakings than small, simple ones. Plus people jump on fads, big data is trendy right now and when you read and hear about something a lot your mind tends to go to it first.
- 1290cc 9y ago
  This. If you've ever heard the story about the 1 man engineer with the better, faster cheaper solution that competed with IBM (slower, thousands of $$$ more) for business? IBM won every time because the buyers wanted to be associated with big impressive projects(and have IBM on their resume). This is actually a sales pricing tactic that every engineer should learn so that they dont price themselves out of the market - by being too cheap.
phd514 9y ago
I've worked in a similar situation -- a transaction processing system with ~1M tx/day and ~2TB total data. They used 15 Cassandra nodes and a monster Redshift cluster (>64 CPU cores, ~512GB RAM, 30+TB disk) for the OLTP and OLAP queries. I almost can't put Cassandra and Redshift on my resume because when more knowledgeable people ask about my experience with them, that pathetic use case makes me look bad by association.
- scapecast 9y ago
  good lord, why did they need a cluster with 30TB when all they had was 2TB? you could downsize that cluster by a factor of 10x.
pcarolan 9y ago
'they had a 25 Kafka instances' < that is absolutely insane
aub3bhat 9y ago
I fail to understand the hype behind Kafka, my guess is that 99% of Kafka use cases can be handled by a simple RabbitMQ/Celery etc. But maybe its not considered cool enough?
- monstrado 9y ago
  Perhaps you haven't spent enough time reading into Kafka? It solves very different problems than the RabbitMQ/Clery "worker queue" pattern you're referring to. That doesn't necessarily mean people don't use Kafka for the wrong reasons.
  - aub3bhat 9y ago
    I don't deny that Kafka solves different problems and has excellent performance at high loads (100K msg/second). >> That doesn't necessarily mean people don't use Kafka for the wrong reason. Exactly what I meant.
  - andy_ppp 9y ago
    This is the best explanation I've found about Kafka and Samza: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
- bsg75 9y ago
  I too would be very interested in understanding more about the Kafka hype. Its everywhere lately.
  - monstrado 9y ago
    Why not just go to their documentation? http://kafka.apache.org/documentation/#introduction http://kafka.apache.org/documentation/#introduction
    - bsg75 9y ago
      I can understand the product from the docs, but not necessarily why Kafka-everything seems to be approaching Hadoop-everything.
rcurry 9y ago
I should know better by now, but I'm still surprised by the relentless pursuit of the shiny that some teams get mired in and even more so that their management allows them to do it. There are an awful lot of problems that can be solved with a simple Postgres instance running on RDS.
mattjaynes 9y ago
Yes! Had a startup client with a 4 person full-time "Big Data" team. They brought me into help them automate their server infrastructure. As part of the process I had them show me the Hadoop jobs they were running. 100% of the jobs could have been done as rudimentary SQL queries to their DB. Welcome to the land of engineering, where we will fine-tune an algorithm to save milliseconds, yet waste man-years of engineering time to look like the big kids. So many engineering choices are based on fashion rather than need. Gotta get that hot new bag from Prada to show off to your engineering friends! :D
- arenaninja 9y ago
  Yep. This is why I'm not sure that I want to do a deeper dive into algorithms. It seems like I could learn to implement all this awesome stuff with the least number of CPU cycles, but it's probably best to learn to communicate these things extremely well just to keep my sanity
  - ktRolster 9y ago
    It's worth it, don't let that stop you.
- ves 9y ago
  They're an RDD shop (Resume Driven Development).
  - humanrebar 9y ago
    To be fair, your employer is not concerned with your resume. They certainly don't pay you a premium for working on things that look boring or harmful on a resume. If companies were serious about providing realistic advancement opportunities (in salary, in tech, in responsibility) there wouldn't be as much of an impulse to work on "cool" (i.e., marketable) projects.
  - scarmig 9y ago
    What's the alternative? Many dev jobs can be accomplished by pretty tried and true technology. For a startup and even most mid sized companies, MySQL and a web framework will get you pretty much everything you need. But any kid a year out of school can do that. So those positions are not really well paid, certainly not enough for a family in the Bay Area. To become a mid-level engineer and make more money, you've got to implement some kind of complicated distributed system. Incorrectly. And then you can "fix" it (or someone else's) to get the next promotion! This is stylized and not 100% true to reality, but I think it gets at a core truth.
badthingfactory 9y ago
Do you have any tips on how to convince management they don't have big data? My company's largest database has been in production for 10 years and could fit into the RAM on my dev machine, yet I'm constantly pushing back against "big data" stacks. This post made me laugh because "big data" and Hadoop were mentioned in our standup meeting yesterday morning.
- pps43 9y ago
  A good definition of "big data" is "data that won't fit on one machine". Corresponding rule of thumb is that you don't need big data tools unless you have big data.
  - cookiecaper 9y ago
    I don't think that's a good definition. "One machine" of data is highly variable; everyone has a different impression of the size of "one machine". Does "fit" mean fit in memory or on disk? Why is "Big Data" automatically superior to sharding with a traditional RDBMS, or a clustered document database?
    - aurelianito 9y ago
      I usually change "one machine" with "my notebook" and "fit" with "can analyze". It is big data if I cannot analyze it using my notebook. So it depends both on the size of the data (a petabyte is big data), the performance requirements (10GB/s is big data even if I keep 1 minute of data in the system) and also depends on the kind of analysis (doing TSP on a 1000000-node graph is big data, even if it fits my notebook memory). I also define "small data" as anything that can be analyzed using Excel. It is usually dramatic enough to get buy-in :).
    - marcosdumay 9y ago
      It's the best definition. It makes "big data" the name of the problem you have when your data can not be worked in a coherent way, what must be solved by distributed tools. If you can buy a bigger machine, you can make "big data" bigger, and maybe evade this problem; if you must access it a lot of times, fitting on disk is useless and "big data" just got smaller; etc.
Spooky23 9y ago
One bunch of yahoos I ran into was building a Hadoop cluster to do some bullshit to user and PC configuration data before loading it into a big helpdesk system like Remedy or ServiceNow. They were 6 months in and made no progress other than building 30 servers. I wrote an awk script to process the CSVs and did the rest in Excel in about 30 minutes. I had an intern automate the process with a perl script, which took 3-4 days! :) The program management was very upset, mostly because they looked like a pack of clowns.
- unixhero 9y ago
  As a client I would have been comically outraged
- ktRolster 9y ago
  The program management was very upset, mostly because they looked like a pack of clowns. How did they respond to that? Any kind of retaliation?
  - Spooky23 9y ago
    Not happy! Pulling a move like that requires extensive, pre-positioned aircover.
    - lucidguppy 9y ago
      Why not just output "HADOOP CLUSTER OUTPUT" at the top of the awk script...
ende 9y ago
Their solution is too hot, but yours is maybe too cold. Somewhere is an approach that is just right. While I too am weary of those needing to use the latest craze on any simple problem, I've become equally weary of the "tried and true" orthodox. The Python/Flask-Postgres stack can be great for rapidly prototyping a functioning application, but sometimes this solution is unable to evolve or scale to address irregular needs. It's almost always fine for your very typical web app. It can struggle with more complicated data processing, especially where irregular resource utilization and complex workflow orchestration are concerned. Celery workers can only address those problems to a degree. Home grown ETL is easy at version 1 and usually a nightmare by version 2. It's a hard problem with lots of wheel reinventing, so it's good that there are some emerging common platforms (particularly things like Airflow). A full on hadoop stack is rarely warranted, but I can understand the reasoning behind wanting a flexible enough processing capacity to accommodate any anticipated load regardless of frequency.
- jimbokun 9y ago
  Yes, but if you try to design for what you anticipate to be the bottlenecks as you scale, you are almost guaranteed to discover the real bottlenecks are in a entirely different part of the architecture than you anticipated. So there is still a good argument to be made for developing the Minimum Viable Product in whatever technologies are most productive for your developers, and figure out how to scale as you grow. It's a difficult balancing act.
ChuckMcM 9y ago
Some of this however is 'cookbook coding'. Too many people learn recipes rather than ingredients. Consider your programmer that goes to a 'big data' class and is taught how to use the stack. They are taught this generally on a 'toy' application because it would take to long to set up a real application. They are there to learn the stack, so they either ignore or get only lip service to the 'when would this be appropriate' slide. Now they come back to work and they know the recipe for using this big data stack. The boss gives them their task, they apply their recipe, and voila, they have a solution. All win right? If it is any consolation the type of engineering situation you are experiencing does (in my experience at least) eventually correct with the manager being moved out.
joyofdata 9y ago
okay - but what if frequency increases to three files and beyond within a year, the file size doubles and the expected latency of processing needs to be reduced? Also compressing doesn't go well with ad hoc analysis. Sounds like maybe a Hadoop setup is not the worst idea to be ready for the future.
- jononor 9y ago
  If you cannot engineer to accommodate changing requirements, you are not ready for the future.
z3t4 9y ago
The requirements, hopes and dreams was probably bigger then one document per second.
ajhurliman 9y ago
I was interviewing for a company that had "big data" once; they were looking for someone with Hadoop experience. It turns out 8GB is big data, and I told them they might want to explore other options, since they could do that in-memory with Spark or Pandas. Never got an email back.
- _wmd 9y ago
  Similar story. It became clear over the course of a (contracting) interview that the cloth-eared PM who according to LinkedIn had just completed some big data course was using every imaginable buzzword he could fit on the whiteboard, and seemed less than impressed at my modelling attempts interfering with his drawing the most ridiculous architectural diagram to power.. an executive management dashboard. I just flat out told him I didn't want anything to do with that kind of project and asked to be shown the exit.
maxxxxx 9y ago
I am going through this right now. We are producing 1 TB per year and have to run a few SQL queries per week. Some people in corporate IT have sold our management on Hadoop and now the whole project has turned from super simple storing into a SQL database into this crazy cycle of writing requirements for IT for every little thing. We are not even allowed to write directly into Hadoop but have to go through their badly defined API. I have run a SQL database on a USB disk with the same data without problems but some people are just attached to the idea of "big data" so Hadoop it.
pavanred 9y ago
This seems to be a pretty common sentiment here. But, here's my question, if you start up doesn't have big data now, then should you assume it will not have big data tomorrow. I work with a start up which currently doesn't have "big data", but perhaps "medium data". And, I can perhaps manage without a big data stack in production. But, if I look at the company/sales targets then in the next 6-12 months we will be working with clients that assures "big data". Now, here are my choices - 1. Stick to python scripts and/or large aws instances because it works for now. If the sales team closes deals in the next few months after working tirelessly for months, then though the sales team sold the client a great solution, but in reality we can't scale and we fail. 2. Plan for your start up to succeed, plan according to the company targets. Try and strike a balance for your production stack that isn't an huge overkill now but isn't under planned/worked either. It's easy to say we shouldn't use big data stack till we have big data, but its too late (specially for a startup) to start building a big data stack after you have big data.
- amenod 9y ago
  Been there. Believe me: stick to Python scripts! Always. And when you finally land that first customer and you have trouble scaling, first scale vertically (buy better machine) and work days and nights to build a scalable solution. But no sooner. Why? Because your problem is not technical, it is business related. You have no idea why your startup will fail or why you will need to pivot. Because if you did, it wouldn't be a startup. Or you would have had that client already. You might need to throw away your solution because it is not solving the right problem. Actually, it is almost certain that it is solving a problem nobody is prepared to pay for. So stick to Python until people start throwing money at you - because you don't have a product-market fit yet. And your fancy Big Data solution will be worth nothing, because it will be so damn impossible to adapt it to new requirements. I wish I could send this comment back in time to myself... :-/ But since I can't, how about at least you learn from my mistakes and not yours? EDIT: good luck!
  - knz 9y ago
    > first scale vertically (buy better machine) With tools like message queues and Docker making it so easy to scale horizontally you don't even have to go vertically. We just won an industry award at work for a multi billion data point spatial analysis project that was all done with Python scripts + Docker on EC2 and PostgreSQL/PostGIS on RDS. A consultant was working in parallel with Hadoop etc and we kept up just fine. Use what works not what is "best". Edit: A dumbed down version of the Python/Docker piece for anyone interested (https://medium.com/@mbaker/horizontally-scaling-gis-python-analysis-with-docker-65eedf3d75ef https://medium.com/@mbaker/horizontally-scaling-gis-python-a...). It's really easy to scale horizontally with Docker...
marcosdumay 9y ago
Your manager is certainly certain that "you get what you pay for", and you are paying much more for the other solution.
dwhitena 9y ago
Thanks for sharing this story. My experience has been that data scientists and analysts aren't able to efficiently use Hadoop/Spark even in cases where it is warranted. These individuals don't generally like working with Java/Scala and/or haven't spend time understanding the underlying structures used (e.g., RDDs, caching, etc.). As a result, they either don't put their sophisticated modeling or analyses into production, or they hand off their application to other engineers to implement for production size "big data." This produces all sorts of problems and inefficiencies, not the least of which is the fact that the engineers don't understand the analyses and the data scientists don't understand the implementation. My (biased, as I work for them) opinion is that something like Pachyderm (http://pachyderm.io/ http://pachyderm.io/) will ease some of these struggles. The philosophy of those who work on this open source project is that data people should be able to use the tooling and frameworks they like/need and be able to push there analyses to production pipelines without re-writing, lots of frictions, or worrying about things like data sharding and parallelism. For example, in Pachyderm you can create a nice, simple Python/R script that is single threaded and runs nicely on your laptop. You can then put the exact same script into pachyderm and run it in a distributed way across many workers on a cluster. Thus, keeping your code simple and approachable, while still allowing people to push things into infrastructure and create value.
SatvikBeri 9y ago
I work in data science. Before interviewing at a company, I try to find an engineer and ask them two things: 1. What's your tech stack? 2. Why? This blatantly disqualifies ~90% of startups which are doing crazy things like using Hadoop for 10gb of data. OTOH, I get really impressed when someone describes effectively using "old" technologies for large amounts of data, or can pinpoint precisely why they use something with reasons other than data size. One good example: "We use Kafka because having a centralized difference log is the best way we've found for several different data stores to read from one source of truth, and we started doing this years ago. If we were starting today, we might use Kinesis on AWS, but the benefits are small compared to the amount of specific infrastructure we've built at this point."
mrweasel 9y ago
Not quit the same, but years back I was to take on the title of "BizTalk consultant". My company and another consulting company had decided that what our customer needed was a BizTalk server. When the BizTalk implementation broke down and I first needed to look into how it worked and what it did, I found that it just moved a small XML file to an SFTP server and wrote a log entry. So I replaced the entire setup with 50 lines of C#. Luckily my boss was onboard, arguing that we didn't really have the qualifications to do BizTalk. The idea had originally been that the customer, a harbour, would in the long run need to do a ton of data exchanges with the government organisations and shipping companies. The thing is that they where planing for a future that never really happened.
RandyRanderson 9y ago
I'm no fan of big data but this is completely unbelievable. And by that i mean i don't believe you. The industry standard is ~100-200 debugged lines of code a day. (If your team tracks hours, look this up on a big project; taking into account all the hours one spends, not just coding.) So your claim, being generous, is this team spent 6 months to not produce the equivalent of 100 lines of code? Even if completely true, this comes off as a humblebrag.
- gregw134 9y ago
  Sounds believable to me. If you don't know what you're doing you can easily waste six months setting up a hadoop cluster with ingestion.
gopalv 9y ago
> I just pulled out a python script, and combine that with Postgres and within an afternoon I had completed the project (albeit not production ready) There are two problems here - one is that people prototype their architecture using massively over-engineered systems and the second is that a rough prototype makes it way into production. So, as a Hadoop perf engineer, I deal with both issues - "We have a Kafka stream hooked up to a Storm pipeline and it is always breaking and we can't debug ... what is this shit?" or "Postgres is stuck on BYTEA vaccuum loops and you can fix it with Hadoop, right?". There are significant advantages in prototyping with an easy to debug architecture, until all the business requirements stabilize. Sometimes the right answer is to use Postgres better (specifically, table inheritance for deletion of old data + indexes instead of delete from with <=), sometimes the right answer is a big data system designed for cold data storage & deep scans. This all comes back to the "We made a plan, now it is a death march" problem - the iterative failure process of building a prototype and having it actually fail is very important, but most people feel like they'll get fired if their work fails in the real world. Over-engineering is usually career insulation and somewhat defense against an architecture council.
victor106 9y ago
It is difficult to get a man to understand something, when his salary depends on his not understanding it. -Upton Sinclair
BurningFrog 9y ago
I've tried to argue at two different jobs that if the data fits on my phone, it is not Big Data.
FridgeSeal 9y ago
I had a job interview for a Data Science position, in which I was talking about data science stuff I'd done in my current position. I mentioned that I started with R, but it would keep choking and running out of memory, so I moved to Python and largely fixed the issue. To which one of the interviewers asked why I didn't use a spark cluster. I was almost too stunned to answer. Their solution, was not to just use something more efficient, or even to just rent a single, more powerful EC2 instance or something, but to go to the effort of setting up and running a cluster. The dataset wasn't even that big: a few gb on disk.
- lmm 9y ago
  There's no a priori reason setting up a cluster has to be more effort than setting up a non-cluster. I mean, a computer is more complicated than a breadboard of custom-wired components, but most people find it easier to solve their problems with a computer. And Spark is usually much more efficient than R or Python (and gives you a nicer language to work with IMO, though that's very subjective). It's entirely possible a 1-node cluster would have outperformed your non-cluster approach, and while a 1-node cluster seems stupid it's useful if you know you're eventually going to have to scale, because it ensures that you test your scalability assumptions and don't embed any non-distributable steps into your logic.
__coaxialcabal 9y ago
Hadoop doesn't take six months for the project you described. It sounds like your team was learning Hadoop, not executing on Hadoop. Are you concerned that you fought a utility in favor of a tactical solution that now requires its own independent maintenance maintenancelifecycle?
est 9y ago
I recently upgraded ElasticSearch to 5.x I cant use a static HTML plugin called head, because port 9200 was already taken and site plugins were disabled. Wanna use head plugin? Install 300 megabytes docker image and setup CORS.
lloydjatkinson 9y ago
I have no experience in this area but $100k a month seems like an insane amount of money.

deleted 9y ago

[deleted]

quantumhobbit 9y ago

The problem you had was that you were trying to convince management to act in the best interest of the company rather than their own best interest. They would much rather be in charge of a $100k a month project instead of a $1k a month project. Also the big solution required a bunch of engineers working full time which puts them higher up the ladder compared to your 1 engineer working part time solution. You were saving the company money but hurting their resumes.

brango 9y ago
Yes. Don't forget "CV-first" development is a thing.
- maxxxxx 9y ago
  The sad thing is that resume driven development seems to be one of the best strategies for developers these days.
  - 8ytecoder 9y ago
    Anecdote: At the startup (now acquired) that I work for, we simplified the architecture so much that we spend under $3k a month in infra for tens of millions in revenues a year while having 99.9% availability for 5+ years now. I don't have any big name tech to add to my resume. The fact that we ran a e-commerce site with a low budget and few engineers was never recognized by recruiters. They called me based on my experience with "distributed systems" 8 years ago. All recruiters (well, most of them at least) can look for is keywords. On the other hand, we also had difficulties hiring "good" engineers. People chose either a company with a brand name recognition or one that's working on "exciting" technology. As engineers, we fail to appreciate that we are there to serve business first and foremost. As leaders, we fail to put our companies first. This is an industry wide problem. If past trends of other major industries are any indicator, the whole "meritocracy" of tech industry will disappear within the next decade.
    - guelo 9y ago
      > As engineers, we fail to appreciate that we are there to serve business first and foremost The company has zero loyalty to you and will screw you over if it makes business sense to do so. There is absolutely no reason to put the company above your career interests.
      - ralmeida 9y ago
        This only holds if you believe that deliberately making decisions that will cost the company more than they should is in your career interests. As said in this topic - it's true that in some places, resume-driven development pays off for the developers. It's not the same everywhere and to me it looks more like a symptom of a dysfunctional structure than par for the course in business. This means it is in your career interests to increase company revenue and/or reduce costs, because this will make you more attractive to many companies (the ones we would likely want to work on) when you move on.
mattjaynes 9y ago
Great point. Ultimately, you need to create a situation where they will also be winning career-wise. I recently wrote about this. The TLDR is that 20% time is a great investment and can ultimately save the company a ton of time and money. It gives the engineers some playtime in order to build their CV's and "get their wiggles out". Ultimately, if done right it can protect your production systems from a lot of madness. https://hvops.com/articles/how-to-keep-passion-from-hugging-your-systems-to-death/ https://hvops.com/articles/how-to-keep-passion-from-hugging-...
- bsg75 9y ago
  > Ultimately, you need to create a situation where they will also be winning career-wise. The message could be that the goal is about the destination not the journey.
komali2 9y ago
Sounds like a good time to go over their head. Depending on the size of the company, straight to someone that deals with money - CFO? Dick move but so is rejecting an objectively better solution to save one's own middle management ass.
- cookiecaper 9y ago
  This is terrible advice. A C-level will see you as a rogue engineer. He will trust his managerial layers over a peon. I know this as I spent far too long being that rogue, peon engineer, screaming into the void. I just didn't get it. It's not about the tech used, and it never has been. It's about being enterprisey. The grandparent is correct. This is 100% a political problem. We have a bad habit of discounting such problems in tech. We shouldn't do that anymore. Life is much better when you cooperate, instead of fighting an uphill battle. Compact, reasonable solutions are the domain of startups. Bloating the engineering layer beyond any reasonable limit is an inherent cost of growing the company, and we shouldn't try to counteract that. We must operate within the framework we're given. The political concerns extend beyond the company's own internals. They must appear enterprisey if they expect to be treated enterprisey. Today, enterprises have "data science" departments and blow $100k/mo on useless crap. If they're not doing that, it's a liability for the whole company, not just from the petty territorial perspective of one individual. It doesn't matter that it's possible to accomplish the same thing with one one-hundredth of the monthly expense. The enterprise signaling is worth the cost.
  - jimbokun 9y ago
    What is the concrete ROI on being "enterprisey"? Google, Facebook, Amazon, Apple are definitely not enterprisey, and I don't think it has hurt their profits. I think they tend to be pretty frugal, overall, when spending their own money on hardware and infrastructure. And I bet if the "enterprisey" company ever tries to compete with a company like Google, Facebook, Amazon, or Apple, they will be destroyed in that market.
    - cookiecaper 9y ago
osrec 9y ago
Reminds me of almost every technology project I've seen in finance! It's all about building complex empires, rather than simple, functional solutions. Sometimes, it's also just about using up the assigned technology budget so that it's not scaled down the following year!
- ythn 9y ago
  Seems like you could spin up a side project business that has a simple solution to their problem but charges enough to soak up their whole budget. Everybody is happy!
  - cookiecaper 9y ago
    Have to provide enough bits and twiddles to allow the dept head to hire his favorite people and you're right. This is basically what Tableau et al have done. There are enough buttons that you can have a few "Tableau guys", and you don't have to hire custom grapher guys.
- rhizome 9y ago
  Funny, I had a phone screen today and in the midst of asking about how I saw my future and what I wanted to work in, the recruiter was tiptoeing around a situation where someone at the company had suggested React or something and the devs had pushed back (successfully) on the basis that the site didn't need it. I got the feeling he was trying not to step on my dreams of being a cutting-edge JS frontend trailblazer, but it is really a point in the company's favor (to me) that they were able to resist the urge. Basically, they are looking for web developers but it seems like they have to filter out all the frontend ninja rockstars.
  - Rapzid 9y ago
    Why didn't they "need" it? How do they decide what they "need"? This sounds like typical anti-change pushback, which I have learned can actually be a good thing. However, this anecdote is severely lacking in insight; much like most people's support of, or opposition to, change. Further, like the widespread belief that sentences shouldn't start with conjunctions; much less conjunctive adverbs.
vlunkr 9y ago
I'm sad about how right you probably are.
treebeard901 9y ago
> You were saving the company money but hurting their resumes. To take it a step further, Management has to own up to their original failure and try to explain to their bosses how they could spend so much time and money unnecessarily. Another psychological problem here is the perception by people who do not understand the technology that the higher priced solution is better because it costs more.
- hinkley 9y ago
  You have the machine that goes 'ping'! You're very lucky. This is the most expensive piece of equipment in the hospital. - Monty Python, Meaning of Life
- mowenz 9y ago
  And a step further: a 100k project had commissions, meetings, expenses all associated with a 100k project. If people are cheating enough to waste 100k on useless crap expect them to cheat some of that for themselves. That means hotel rooms, airline tickets, meals, etc and all, and any commission the sales guy shares with you for making it happen. And it does happen...
conjectures 9y ago
I would love to agree with you but there is another interpretation: they are clueless. Suppose management in a firm uncritically contracted the big data revolution meme. Then they believe they are in the position of a city mayor trying to build a bridge and some wiseacre comes along and says they can do it in an afternoon with two pieces of string and a stick of gum. The problem is that the analogy doesn't hold, but they don't know that.
vivekd 9y ago
Add to that, if anything goes wrong, they would have a much easier time cya-ing and convincing their superiors they did all they can do to prevent it if they build a large, complex solution than if they just went to the quickest, simplest one. In fact justifying anything going wrong at all is much easier with large, complex undertakings than small, simple ones. Plus people jump on fads, big data is trendy right now and when you read and hear about something a lot your mind tends to go to it first.
1290cc 9y ago
This. If you've ever heard the story about the 1 man engineer with the better, faster cheaper solution that competed with IBM (slower, thousands of $$$ more) for business? IBM won every time because the buyers wanted to be associated with big impressive projects(and have IBM on their resume). This is actually a sales pricing tactic that every engineer should learn so that they dont price themselves out of the market - by being too cheap.

phd514 9y ago

I've worked in a similar situation -- a transaction processing system with ~1M tx/day and ~2TB total data. They used 15 Cassandra nodes and a monster Redshift cluster (>64 CPU cores, ~512GB RAM, 30+TB disk) for the OLTP and OLAP queries. I almost can't put Cassandra and Redshift on my resume because when more knowledgeable people ask about my experience with them, that pathetic use case makes me look bad by association.

scapecast 9y ago
good lord, why did they need a cluster with 30TB when all they had was 2TB? you could downsize that cluster by a factor of 10x.

pcarolan 9y ago

'they had a 25 Kafka instances' < that is absolutely insane

aub3bhat 9y ago

I fail to understand the hype behind Kafka, my guess is that 99% of Kafka use cases can be handled by a simple RabbitMQ/Celery etc. But maybe its not considered cool enough?

monstrado 9y ago
Perhaps you haven't spent enough time reading into Kafka? It solves very different problems than the RabbitMQ/Clery "worker queue" pattern you're referring to. That doesn't necessarily mean people don't use Kafka for the wrong reasons.
- aub3bhat 9y ago
  I don't deny that Kafka solves different problems and has excellent performance at high loads (100K msg/second). >> That doesn't necessarily mean people don't use Kafka for the wrong reason. Exactly what I meant.
- andy_ppp 9y ago
  This is the best explanation I've found about Kafka and Samza: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://www.confluent.io/blog/turning-the-database-inside-ou...
bsg75 9y ago
I too would be very interested in understanding more about the Kafka hype. Its everywhere lately.
- monstrado 9y ago
  Why not just go to their documentation? http://kafka.apache.org/documentation/#introduction http://kafka.apache.org/documentation/#introduction
  - bsg75 9y ago
    I can understand the product from the docs, but not necessarily why Kafka-everything seems to be approaching Hadoop-everything.

rcurry 9y ago

I should know better by now, but I'm still surprised by the relentless pursuit of the shiny that some teams get mired in and even more so that their management allows them to do it. There are an awful lot of problems that can be solved with a simple Postgres instance running on RDS.

mattjaynes 9y ago

Yes! Had a startup client with a 4 person full-time "Big Data" team. They brought me into help them automate their server infrastructure. As part of the process I had them show me the Hadoop jobs they were running. 100% of the jobs could have been done as rudimentary SQL queries to their DB. Welcome to the land of engineering, where we will fine-tune an algorithm to save milliseconds, yet waste man-years of engineering time to look like the big kids. So many engineering choices are based on fashion rather than need. Gotta get that hot new bag from Prada to show off to your engineering friends! :D

arenaninja 9y ago
Yep. This is why I'm not sure that I want to do a deeper dive into algorithms. It seems like I could learn to implement all this awesome stuff with the least number of CPU cycles, but it's probably best to learn to communicate these things extremely well just to keep my sanity
- ktRolster 9y ago
  It's worth it, don't let that stop you.
ves 9y ago
They're an RDD shop (Resume Driven Development).
- humanrebar 9y ago
  To be fair, your employer is not concerned with your resume. They certainly don't pay you a premium for working on things that look boring or harmful on a resume. If companies were serious about providing realistic advancement opportunities (in salary, in tech, in responsibility) there wouldn't be as much of an impulse to work on "cool" (i.e., marketable) projects.
- scarmig 9y ago
  What's the alternative? Many dev jobs can be accomplished by pretty tried and true technology. For a startup and even most mid sized companies, MySQL and a web framework will get you pretty much everything you need. But any kid a year out of school can do that. So those positions are not really well paid, certainly not enough for a family in the Bay Area. To become a mid-level engineer and make more money, you've got to implement some kind of complicated distributed system. Incorrectly. And then you can "fix" it (or someone else's) to get the next promotion! This is stylized and not 100% true to reality, but I think it gets at a core truth.

badthingfactory 9y ago

Do you have any tips on how to convince management they don't have big data? My company's largest database has been in production for 10 years and could fit into the RAM on my dev machine, yet I'm constantly pushing back against "big data" stacks. This post made me laugh because "big data" and Hadoop were mentioned in our standup meeting yesterday morning.

pps43 9y ago
A good definition of "big data" is "data that won't fit on one machine". Corresponding rule of thumb is that you don't need big data tools unless you have big data.
- cookiecaper 9y ago
  I don't think that's a good definition. "One machine" of data is highly variable; everyone has a different impression of the size of "one machine". Does "fit" mean fit in memory or on disk? Why is "Big Data" automatically superior to sharding with a traditional RDBMS, or a clustered document database?
  - aurelianito 9y ago
    I usually change "one machine" with "my notebook" and "fit" with "can analyze". It is big data if I cannot analyze it using my notebook. So it depends both on the size of the data (a petabyte is big data), the performance requirements (10GB/s is big data even if I keep 1 minute of data in the system) and also depends on the kind of analysis (doing TSP on a 1000000-node graph is big data, even if it fits my notebook memory). I also define "small data" as anything that can be analyzed using Excel. It is usually dramatic enough to get buy-in :).
  - marcosdumay 9y ago
    It's the best definition. It makes "big data" the name of the problem you have when your data can not be worked in a coherent way, what must be solved by distributed tools. If you can buy a bigger machine, you can make "big data" bigger, and maybe evade this problem; if you must access it a lot of times, fitting on disk is useless and "big data" just got smaller; etc.

Spooky23 9y ago

One bunch of yahoos I ran into was building a Hadoop cluster to do some bullshit to user and PC configuration data before loading it into a big helpdesk system like Remedy or ServiceNow. They were 6 months in and made no progress other than building 30 servers. I wrote an awk script to process the CSVs and did the rest in Excel in about 30 minutes. I had an intern automate the process with a perl script, which took 3-4 days! :) The program management was very upset, mostly because they looked like a pack of clowns.

unixhero 9y ago
As a client I would have been comically outraged
ktRolster 9y ago
The program management was very upset, mostly because they looked like a pack of clowns. How did they respond to that? Any kind of retaliation?
- Spooky23 9y ago
  Not happy! Pulling a move like that requires extensive, pre-positioned aircover.
  - lucidguppy 9y ago
    Why not just output "HADOOP CLUSTER OUTPUT" at the top of the awk script...

ende 9y ago

Their solution is too hot, but yours is maybe too cold. Somewhere is an approach that is just right. While I too am weary of those needing to use the latest craze on any simple problem, I've become equally weary of the "tried and true" orthodox. The Python/Flask-Postgres stack can be great for rapidly prototyping a functioning application, but sometimes this solution is unable to evolve or scale to address irregular needs. It's almost always fine for your very typical web app. It can struggle with more complicated data processing, especially where irregular resource utilization and complex workflow orchestration are concerned. Celery workers can only address those problems to a degree. Home grown ETL is easy at version 1 and usually a nightmare by version 2. It's a hard problem with lots of wheel reinventing, so it's good that there are some emerging common platforms (particularly things like Airflow). A full on hadoop stack is rarely warranted, but I can understand the reasoning behind wanting a flexible enough processing capacity to accommodate any anticipated load regardless of frequency.

jimbokun 9y ago
Yes, but if you try to design for what you anticipate to be the bottlenecks as you scale, you are almost guaranteed to discover the real bottlenecks are in a entirely different part of the architecture than you anticipated. So there is still a good argument to be made for developing the Minimum Viable Product in whatever technologies are most productive for your developers, and figure out how to scale as you grow. It's a difficult balancing act.

ChuckMcM 9y ago

Some of this however is 'cookbook coding'. Too many people learn recipes rather than ingredients. Consider your programmer that goes to a 'big data' class and is taught how to use the stack. They are taught this generally on a 'toy' application because it would take to long to set up a real application. They are there to learn the stack, so they either ignore or get only lip service to the 'when would this be appropriate' slide. Now they come back to work and they know the recipe for using this big data stack. The boss gives them their task, they apply their recipe, and voila, they have a solution. All win right? If it is any consolation the type of engineering situation you are experiencing does (in my experience at least) eventually correct with the manager being moved out.

joyofdata 9y ago

okay - but what if frequency increases to three files and beyond within a year, the file size doubles and the expected latency of processing needs to be reduced? Also compressing doesn't go well with ad hoc analysis. Sounds like maybe a Hadoop setup is not the worst idea to be ready for the future.

jononor 9y ago
If you cannot engineer to accommodate changing requirements, you are not ready for the future.

z3t4 9y ago

The requirements, hopes and dreams was probably bigger then one document per second.

ajhurliman 9y ago

I was interviewing for a company that had "big data" once; they were looking for someone with Hadoop experience. It turns out 8GB is big data, and I told them they might want to explore other options, since they could do that in-memory with Spark or Pandas. Never got an email back.

_wmd 9y ago
Similar story. It became clear over the course of a (contracting) interview that the cloth-eared PM who according to LinkedIn had just completed some big data course was using every imaginable buzzword he could fit on the whiteboard, and seemed less than impressed at my modelling attempts interfering with his drawing the most ridiculous architectural diagram to power.. an executive management dashboard. I just flat out told him I didn't want anything to do with that kind of project and asked to be shown the exit.

maxxxxx 9y ago

I am going through this right now. We are producing 1 TB per year and have to run a few SQL queries per week. Some people in corporate IT have sold our management on Hadoop and now the whole project has turned from super simple storing into a SQL database into this crazy cycle of writing requirements for IT for every little thing. We are not even allowed to write directly into Hadoop but have to go through their badly defined API. I have run a SQL database on a USB disk with the same data without problems but some people are just attached to the idea of "big data" so Hadoop it.

pavanred 9y ago

This seems to be a pretty common sentiment here. But, here's my question, if you start up doesn't have big data now, then should you assume it will not have big data tomorrow. I work with a start up which currently doesn't have "big data", but perhaps "medium data". And, I can perhaps manage without a big data stack in production. But, if I look at the company/sales targets then in the next 6-12 months we will be working with clients that assures "big data". Now, here are my choices - 1. Stick to python scripts and/or large aws instances because it works for now. If the sales team closes deals in the next few months after working tirelessly for months, then though the sales team sold the client a great solution, but in reality we can't scale and we fail. 2. Plan for your start up to succeed, plan according to the company targets. Try and strike a balance for your production stack that isn't an huge overkill now but isn't under planned/worked either. It's easy to say we shouldn't use big data stack till we have big data, but its too late (specially for a startup) to start building a big data stack after you have big data.

amenod 9y ago
Been there. Believe me: stick to Python scripts! Always. And when you finally land that first customer and you have trouble scaling, first scale vertically (buy better machine) and work days and nights to build a scalable solution. But no sooner. Why? Because your problem is not technical, it is business related. You have no idea why your startup will fail or why you will need to pivot. Because if you did, it wouldn't be a startup. Or you would have had that client already. You might need to throw away your solution because it is not solving the right problem. Actually, it is almost certain that it is solving a problem nobody is prepared to pay for. So stick to Python until people start throwing money at you - because you don't have a product-market fit yet. And your fancy Big Data solution will be worth nothing, because it will be so damn impossible to adapt it to new requirements. I wish I could send this comment back in time to myself... :-/ But since I can't, how about at least you learn from my mistakes and not yours? EDIT: good luck!
- knz 9y ago
  > first scale vertically (buy better machine) With tools like message queues and Docker making it so easy to scale horizontally you don't even have to go vertically. We just won an industry award at work for a multi billion data point spatial analysis project that was all done with Python scripts + Docker on EC2 and PostgreSQL/PostGIS on RDS. A consultant was working in parallel with Hadoop etc and we kept up just fine. Use what works not what is "best". Edit: A dumbed down version of the Python/Docker piece for anyone interested (https://medium.com/@mbaker/horizontally-scaling-gis-python-analysis-with-docker-65eedf3d75ef https://medium.com/@mbaker/horizontally-scaling-gis-python-a...). It's really easy to scale horizontally with Docker...

marcosdumay 9y ago

Your manager is certainly certain that "you get what you pay for", and you are paying much more for the other solution.

dwhitena 9y ago

Thanks for sharing this story. My experience has been that data scientists and analysts aren't able to efficiently use Hadoop/Spark even in cases where it is warranted. These individuals don't generally like working with Java/Scala and/or haven't spend time understanding the underlying structures used (e.g., RDDs, caching, etc.). As a result, they either don't put their sophisticated modeling or analyses into production, or they hand off their application to other engineers to implement for production size "big data." This produces all sorts of problems and inefficiencies, not the least of which is the fact that the engineers don't understand the analyses and the data scientists don't understand the implementation. My (biased, as I work for them) opinion is that something like Pachyderm (http://pachyderm.io/ http://pachyderm.io/) will ease some of these struggles. The philosophy of those who work on this open source project is that data people should be able to use the tooling and frameworks they like/need and be able to push there analyses to production pipelines without re-writing, lots of frictions, or worrying about things like data sharding and parallelism. For example, in Pachyderm you can create a nice, simple Python/R script that is single threaded and runs nicely on your laptop. You can then put the exact same script into pachyderm and run it in a distributed way across many workers on a cluster. Thus, keeping your code simple and approachable, while still allowing people to push things into infrastructure and create value.

SatvikBeri 9y ago

I work in data science. Before interviewing at a company, I try to find an engineer and ask them two things: 1. What's your tech stack? 2. Why? This blatantly disqualifies ~90% of startups which are doing crazy things like using Hadoop for 10gb of data. OTOH, I get really impressed when someone describes effectively using "old" technologies for large amounts of data, or can pinpoint precisely why they use something with reasons other than data size. One good example: "We use Kafka because having a centralized difference log is the best way we've found for several different data stores to read from one source of truth, and we started doing this years ago. If we were starting today, we might use Kinesis on AWS, but the benefits are small compared to the amount of specific infrastructure we've built at this point."

mrweasel 9y ago

Not quit the same, but years back I was to take on the title of "BizTalk consultant". My company and another consulting company had decided that what our customer needed was a BizTalk server. When the BizTalk implementation broke down and I first needed to look into how it worked and what it did, I found that it just moved a small XML file to an SFTP server and wrote a log entry. So I replaced the entire setup with 50 lines of C#. Luckily my boss was onboard, arguing that we didn't really have the qualifications to do BizTalk. The idea had originally been that the customer, a harbour, would in the long run need to do a ton of data exchanges with the government organisations and shipping companies. The thing is that they where planing for a future that never really happened.

RandyRanderson 9y ago

I'm no fan of big data but this is completely unbelievable. And by that i mean i don't believe you. The industry standard is ~100-200 debugged lines of code a day. (If your team tracks hours, look this up on a big project; taking into account all the hours one spends, not just coding.) So your claim, being generous, is this team spent 6 months to not produce the equivalent of 100 lines of code? Even if completely true, this comes off as a humblebrag.

gregw134 9y ago
Sounds believable to me. If you don't know what you're doing you can easily waste six months setting up a hadoop cluster with ingestion.

gopalv 9y ago

> I just pulled out a python script, and combine that with Postgres and within an afternoon I had completed the project (albeit not production ready) There are two problems here - one is that people prototype their architecture using massively over-engineered systems and the second is that a rough prototype makes it way into production. So, as a Hadoop perf engineer, I deal with both issues - "We have a Kafka stream hooked up to a Storm pipeline and it is always breaking and we can't debug ... what is this shit?" or "Postgres is stuck on BYTEA vaccuum loops and you can fix it with Hadoop, right?". There are significant advantages in prototyping with an easy to debug architecture, until all the business requirements stabilize. Sometimes the right answer is to use Postgres better (specifically, table inheritance for deletion of old data + indexes instead of delete from with <=), sometimes the right answer is a big data system designed for cold data storage & deep scans. This all comes back to the "We made a plan, now it is a death march" problem - the iterative failure process of building a prototype and having it actually fail is very important, but most people feel like they'll get fired if their work fails in the real world. Over-engineering is usually career insulation and somewhat defense against an architecture council.

victor106 9y ago

It is difficult to get a man to understand something, when his salary depends on his not understanding it. -Upton Sinclair

BurningFrog 9y ago

I've tried to argue at two different jobs that if the data fits on my phone, it is not Big Data.

FridgeSeal 9y ago

I had a job interview for a Data Science position, in which I was talking about data science stuff I'd done in my current position. I mentioned that I started with R, but it would keep choking and running out of memory, so I moved to Python and largely fixed the issue. To which one of the interviewers asked why I didn't use a spark cluster. I was almost too stunned to answer. Their solution, was not to just use something more efficient, or even to just rent a single, more powerful EC2 instance or something, but to go to the effort of setting up and running a cluster. The dataset wasn't even that big: a few gb on disk.

lmm 9y ago
There's no a priori reason setting up a cluster has to be more effort than setting up a non-cluster. I mean, a computer is more complicated than a breadboard of custom-wired components, but most people find it easier to solve their problems with a computer. And Spark is usually much more efficient than R or Python (and gives you a nicer language to work with IMO, though that's very subjective). It's entirely possible a 1-node cluster would have outperformed your non-cluster approach, and while a 1-node cluster seems stupid it's useful if you know you're eventually going to have to scale, because it ensures that you test your scalability assumptions and don't embed any non-distributable steps into your logic.

__coaxialcabal 9y ago

Hadoop doesn't take six months for the project you described. It sounds like your team was learning Hadoop, not executing on Hadoop. Are you concerned that you fought a utility in favor of a tactical solution that now requires its own independent maintenance maintenancelifecycle?

est 9y ago

I recently upgraded ElasticSearch to 5.x I cant use a static HTML plugin called head, because port 9200 was already taken and site plugins were disabled. Wanna use head plugin? Install 300 megabytes docker image and setup CORS.

lloydjatkinson 9y ago

I have no experience in this area but $100k a month seems like an insane amount of money.