7 ms·
As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head. Most, heck something like 99.99% of all so-ca
by kamaal 14y ago
As some one who is currently dealing with these sort of things I can tell this article hits the nail on its head.
Most, heck something like 99.99% of all so-called big data I've dealt is something I wouldn't even classify as small data. I've seen data feeds in KB's sent over to be handled in as big data. It happens all the time. A simple data problem sufficient enough to be easily solved on something like a small db solution like sqlite is generally taken to 'grid' these days. It reminds me of the XML days when everything had to be XML. I mean every damn thing, these days its NoSQL and Big data.
People wrongly do their schema design just so that it can get into a NoSQL, then use something like Pig to generate data for it. The net result is they end up badly reinventing parts of SQL all over the place. If only they understand a little SQL and why its exists they can save themselves all that pointless complexity they get into. Besides avoiding to use SQL where its appropriate creates all sorts of data problems in your system. You will go endlessly reinventing ways doing things similar to what SQL offers while bloating your code. You will go reading a big part of the code, only to figure out the person actually intended to something like a nested select query albeit here very badly.
Besides I find much of this big data thing a total sham. Back in the yester years we would write Perl scripts to do all sorts complex data processing(With SQL of course). Heck I've run some very big analytic systems, and automation set ups in Perl to do far difficult things people do using 'Big data tools' today.
In larger corporation this has become fashion now. If you want to be known as a great 'architect' all you need to do is bring in these pointless complexities. Ensure the set up becomes so complicated it can't explained without the help of a hundred jargons totally incomprehensible to anybody beyond your cubicle. That is how you get promoted to become a architect these days.
- dinkumthinkum 14y agoIn some ways, it kind of is a sham. I think it is perpetuated by the blog/youtube style programming knowledge transference paradigm. Those mediums are fine but there seems to be a rallying cry against actually learning anything about computing in anything other than bite size pieces and thus get a lot of fad driven movements and an over population of redundant frameworks and libraries.
- jiggy2011 14y agoYes, and then because companies start using NoSQL or whatever for problems they could have done fine in mysql they start asking for NoSQL experts when they are hiring. This makes devs think that they need NoSQL experience and therefor they will find ways to shoehorn NoSQL into whatever problems they are currently solving.
- petercooper 14y agoThe modern equivalent of object oriented programming, perhaps. I remember similar comments about it in the early/mid 90s.
- rmrfrmrf 14y agoI think it's more perpetuated by the fact that executives and other corporate-folk are easily hypnotized by numbers, however irrelevant they may be. "Interestingly, peak purchases for our product in March occurred Monday through Friday between 7am and 4:30pm, with 807 people looked at page x before purchasing and 806 people looking at page y. Through running our data against national averages, we've found that our peak purchasing times align and are roughly proportionate to population density across the country." You can watch in amazement as every MBA in the room's eyes gloss over; a dopey smile overcomes them slowly as the data intoxicates them. Throw in a line graph and a heat map and you're basically on a fast track to a promotion without even saying a single useful thing.
- hso9791 14y agoAgreed. It is quite amazing what modern ANSI SQL has built into it. If one picks the right storage engine, an appropriate schema design and is rigorous about data quality, the flexibility and performance is incredible.
- rmrfrmrf 14y agoI really don't understand why anyone has a problem with relational databases. Once you take the time to understand how they work (by taking a class or reading a book), it's really straightforward and makes a lot of sense. Not to mention it's really fast and quite reliable. I get that a NoSQL-ish alternative makes sense for companies that have tons of shards spanning the globe, but for the vast majority of people, a relational database serves just fine.
- matwood 14y agoI grew up on rdbmses and think they are great. I've created numerous pieces of software backed by them. I have long preached the mantra that data is more important than the application. The data will almost always outlive the original application and become shared to others. With that said, the above comes at a time/flexibility cost (not as bad as it used to be but still there) when building a product that isn't quite sure what it will be yet. In these cases a different data store can be beneficial since the app itself is key until traction is gained, if ever.
- zapov 14y agoTime/flexibility problem can be solved (but it's hard). Take a look at DSL Platform (shameless plug) if you are interested in building on top of database, while having very flexible model.
- maggit 14y agoI don't have a problem with relational databases. Or rather, I don't have a problem with relational data modelling. I do have a problem with Oracle. Even the Oracle experts at my former job could barely get Oracle to do something sensible, and running it on my own computer was basically a death sentence for getting anything done. I have a problem with MySQL, from a sysadmin perspective. When I had it installed, MySQL was the package that would always break on every update. No upgrade was small enough that the data files would continue working. (I don't have experience with Postgres, but SQLite seems more comfortable than any of the mentioned alternatives) I have a problem with schemas in my database. It requires upfront work with modelling my data. I'd rather iterate. Also, nobody I've worked with seems to put the schemas in automatically; you need to run the special "initdb" script that isn't maintained to make it start working. I have a problem with SQL. It would be awesome if we had a standard query language, but we don't. You can't apply the same SQL to different database engines mostly because it won't even compile, secondarily because it will give different results and finally because it will have a completely different performance profile. All of this can be fixed by learning stuff, so I know better what I am doing. But I already know CouchDB [1]. It took me little effort to learn, and it makes a lot of sense to my mind. I can solve problems with it, and it has neat properties with regards to master-master replication. So for me, CouchDB works just fine, just like a relational database works just fine for you :) So, from my perspective, it seems that using some SQL solution would be the time consuming option. [1]: CouchDB can't be considered a "Big data" database for many cases. It is slow. But it scales neatly :)
- mathnode 14y agoHanging off this great comment, please anyone wanting to learn about relational databases I strongly suggest reading Database in Depth by C J Date. It's around 200 pages, you will learn a lot about the relational model, database/query/index design in a short amount of time, with a little bit of logic thrown in there too. Just need a good SQL reference? 'The Art of SQL' or 'SQL and Relational Theory' Are you not entertained? Fine, choke on this : http://en.wikipedia.org/wiki/The_Third_Manifesto http://en.wikipedia.org/wiki/The_Third_Manifesto
- Djehngo 14y agoHa, I knew that looked familiar, Hugh Darwin is a lecturer where I was at university so half of the coursework for a databases module was in Tutorial D.
- lifeisstillgood 14y agoSorry I missed the reference - mathnode == Hugh Darwin ?
- mathnode 14y agomathnode != Hugh Darwen They are some big boots to fill. I'm just some DBA in London.
- IsaacL 14y agoFunny to see another ex-Warwickian here. Yes, I remember that coursework - I also found in interesting that Hugh Darwin thought SQL was too lax and flexible. I wish I'd taken the chance to ask his opinions on nosql.
- dspillett 14y agoAnd once you know the basics, I strongly recommend "SQL Anti-Patterns" for a good guide for what not to do and why not (and what mitigating circumstances might make an otherwise bad choice OK (or simply the only available option)). I read it while considering myself experienced and found it to be a useful refresher. The style/tone is light and well organised by task/objective, so I suspect everyone down to a beginner will find it well worth their time perusing.
- giulianob 14y agoI've also seen cases where people who run on EC2 end up having to do a lot of extra work because of the bad IO/CPU. They end up with these solutions that are way overkill for such small websites.