6 ms·
Most people who think they have big data don't.
by jandrese 8y ago
Most people who think they have big data don't.
- paulmd 8y agoBut we have millions of rows! -- my actual employer (the actual problem is our SAN is getting us a grand total of 10 iops. No, that's not wrong, 10. The "what the christ, just buy us a system that actually works" momentum has been building for 2 years now but hasn't managed to take over. Hardware comes out of a different budget than wasted time, yo.)
- mirceal 8y agoSo you have big data. Now you just need the big hardware /s
- candiodari 8y agoA cynical person might point out that for empire-building managers the goals are : 1) minimize hardware spend at all costs 2) maximize lost time (if spent under the manager's control)
- cortesoft 8y agoI don’t know why these ‘you don’t have big data’ type articles bother me so much, but they really do. I know it isn’t saying NO ONE has big data, but I feel defensive anyway. Some of us DO work for companies that need to process millions of log lines a second. I know the article is not for me, but it still feels like a dismissal of our real, actual, big data problem.
- realusername 8y agoThese articles are not targeted at you indeed but to the thousands of companies trying to setup a massive architecture to process 3GB of data per year. All these big data solutions are still necessary of course, it's just not for everyone.
- rectangletangle 8y agoAgreed they are tools designed to fill a niche. This doesn't make them bad, or uneccessary, just specialized. A spanner drive screw isn't worse than a hex drive, it's just designed for a different more specific use case. Really this just goes to show how impressive RDMSs like Postgres are. There's nothing out there that's drastically better in the general case. So alternative database systems tend to just nibble around the edges. My rule of thumb is always try to implement it using a relational model first, and only when that proves itself untenable, look into other more specialized tools.
- candiodari 8y agoYeah but the whole cloud paradigm is predicated on big data, so large actors are pushing it where it makes no sense, and it makes everyone less effective. Not more. Ever notice how with cloud it's actually in the interest of cloud providers to have the worst programmers, worst possible solutions for their clients ? Those will maximize spend on cloud, which is what these companies are going for. Of course, they hold up the massive carrot of "you can get a job here if you ...".
- mhotchen 8y agoFor every programmer like you there's ten more at places where I've been employed trying to claim they need big data tools for a few million (or in some cases a few hundred thousand) rows in mysql. I get why you could feel attacked when this message is repeated so often, but apparently it still isn't repeated anywhere near enough.
- collyw 8y ago"Relational databases don't scale". I hope the NoSQL hype is over by now and people are back to choosing relational as the default choice. (The same people will probably chasing block-chain solutions to everything by now...)
- bshacklett 8y agoWhy should relational be the default choice? There are many cases where people are storing non-relational data and a nosql database can be the right solution regardless of whether scaling is the constraint. Most nosql SaaS platforms are significantly easier to consume than the average service oriented RDBMS platform. If all the DBMS is doing is handling simple CRUD transactions, there's a good chance that relational databases are overkill for the workload and could even be harmful to the delivery process. The key is to take the time to truly understand each workload before just assuming that one's preferred data storage solution is the right way to go.
- Retric 8y agoWhat's non relational data? You can have minimally relational data such as URL : website, but that's still improved by going URL : ID, ID : website because you can insert those ID's into the website data. Now plenty of DB's have terrible designs, but there I have yet to year of actually non relational data.
- cortesoft 8y agoWait, why is that an improvement? If there is a 1 to 1 mapping of url to website, splitting it into two tables is just bad database design.
- therealdrag0 8y agoJust curious what do you need to process so many log lines for? Is this an ELK type setup, and/or do you use the log processing for business logic?
- Groxx 8y agoAt an absolute minimum, I'd say "big data" begins when you can't buy hardware with that much memory. Apparently, in 2017, that was somewhere around 16 terabytes. https://www.theregister.co.uk/2017/05/16/aws_ram_cram/ https://www.theregister.co.uk/2017/05/16/aws_ram_cram/ Heck, you can trivially get a 4TB instance from Amazon nowadays: https://aws.amazon.com/ec2/instance-types/ https://aws.amazon.com/ec2/instance-types/ The biggest DBs I've worked on have been a few tens of billions of rows, and several hundreds of gigabytes. That's like... nothing. A laughable start. You can make trivially inefficient mistakes and for the most part e.g. MySQL will still work fine. Absolutely nowhere near "big data". And odds are pretty good you could just toss existing more-than-4TB-data through grep and then into your DB and still ignore "big data" problems.
- gaius 8y ago10 years ago I was working on a 50Tb datawarehouse. Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.
- Groxx 8y agoYeah, or R. When I need to do stuff bigger than RAM on a laptop I just reach for sqlite (assuming it can even fit on disk).
- gaius 8y agoThe funny thing is we didn’t even consider our 50Tb to be particularly big, we knew guys running 200Tb... BTW in R data.table is better for larger datasets than the default dataframe... or use the Revolution Analytics stuff on a server.
- Groxx 8y agoOh yeah, the default dataframe is practically only for toy problems / very-already-filtered data (which accounts for a LOT of useful scenarios!). It's more that I've run into "big data" people who (╯°□°)╯︵ ┻━┻ when the default settings + dataframe choke, which is around 200MB of heap.
- bryanrasmussen 8y agoI was working in a company where they brought in some consultants to do the analytics and they were going to use Hadoop, and they said straight up "We don't need it for the amount of data, but we prefer working in Hadoop"
- onion2k 8y agoIt's far more common for your assumptions/thinking/design to be the root cause of any real problem in things you build than your choice of tech. It doesn't matter if you use the 'wrong' tech to solve a problem if the solution works effectively, and very often using something you know well means you'll solve the problem faster and better than you would with the 'right' tool. Those consultants were probably doing the right thing.
- bryanrasmussen 8y agoI actually don't think they had great expertise in Hadoop though, since it was quite new at the time. but maybe
- cm2187 8y agoI remember a Microsoft talk about running big data in Excel...