9 ms·
Ask HN: To everybody who uses MapReduce: what problems do you solve?
- roschdal 13y ago"How can I make Google shareholders richer?"
- PaulHoule 13y agoMapReduce is great for ETL problems where there is a large mass of data and you want to filter and summarize it.
- darkxanthos 13y agoYup. This. Once I filter down to the data I actually care about, I typically find I'm no longer anywhere near "Big Data" size.
- ahulak 13y agoI came here to say the same thing. We use to rip through anywhere from ~100M-3B rows of data every day.. updates to this dataset are time sensitive so we use it to finish the job as quickly as possible.
- clubhi 13y ago"How do I make all my projects take 10x longer"
- deleted 13y ago[deleted]
- alexrson 13y agoHow strongly does a RNA binding protein bind to each possible sequence of RNA?
- nextos 13y agoInteresting, I'm using it for protein localization prediction, PTMs, and NGS data.
- rch 13y agoNGS phage display data here.
- collyw 13y agoHow big is your data? Yours is the first example where I a decent knowledge of the field, so can understand the needs accurately. In most cases I see people using NoSQL in places where MySQL could handle it easily. Maybe I am just too set in my relational database ways of thinking (having used them for 13 years), but there are few cases where I see NoSQL solutions being beneficial, other than for ease of use(most people are not running Facebook or Google). I a bit sceptical in most cases (though I would certainly like to know where they are appropriate).
- rch 13y agoFirst, I should note that my needs are fairly specific, and not typical of the rest of the NGS world. The datasets are essentially the same though. The rate at which we are acquiring new data has been accelerating, but each of our Illumina datasets is only 30GB or so. The total accumulated data is still just a few TB. The real imperative for using MR is more about the processing of that data. Integrating HMMER, for instance, into Postgres wouldn't be impossible, but I don't know of anything that's available now. Edit: A FDW for PostgreSQL around HMMER just made my to do list.
- collyw 13y agoSo is it fair to say it is an "ease of use" use case?
- rch 13y agoIs that the same as an 'impossible to do otherwise' case? Edit: I should say 'currently impossible' since as I noted, I can imagine being able to build SQL queries around PSSM comparisons and the like. I just can't build a system to last 5+ years around something that might be available at some point. Since I can't reply directly- agreed :)
- collyw 13y agoThat comment was based on your "into Postgres wouldn't be impossible" phrase. No fair enough of its the only way you can get things to work. I still see lots of people jumping on the "big data" bandwagon with very moderate sized data.
- sidcool 13y agoA host of batch jobs that source data from up to 28 different systems and then apply business rules to extract a substrate of useful data.
- iurisilvio 13y agoHow much can I save in my mobile phone and plan?
- hurrycane 13y agoBillion of metric values.
- willvarfar 13y agoWe had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users. But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes. Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted. Add: down voters can explain their problem with this data point?
- srl 13y agoThis doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving. (Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)
- res0nat0r 13y ago> not what problem you're actually solving. > targetting expressions over billions of time series events from users.
- collyw 13y agoYou moved back to MySQL. What was the motivation to move to Map Reduce in the first place if a well understood technology, MySQL, works fine? (sorry if I am posting a lot on this topic. I am really interested in finding answers rather than trying to prove any point that relational databases are better in case anyone thinks otherwise).
- willvarfar 13y ago(Poster) we moved from mysql+innodb/myisam to hadoop because of performance problems. We did test and evaluate hadoop, and then jumped. Then tokudb comes along (technically we moved back to mariadb) and puts performance advantage firmly back on mysql's side. I imagine impala and presto and any other column based, non-map-reduce engines would give tokudb a fairer fight though.
- benihana 13y agoHow can I make myself feel better about what I do by trying to diminish the work other people do using the same technologies?
- bijanv 13y agoTaking dumps of analytics logs and pulling out relevant info for our customers on app usage
- sitkack 13y agoThis is the `grep/awk` use case. The nice thing about streaming mr interface to hadoop (calling external programs) is that you can literally take your grep/awk workflow and move it to the cluster. Retaining line oriented records is a huge step in having a portable data processing workflow.
- alecco 13y agoA large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before doing any Map/Reduce (or equivalent), please I beg you to check out Introduction to Data Science at Coursera https://www.coursera.org/course/datasci https://www.coursera.org/course/datasci
- thrownaway2424 13y agoI strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total. Right tool for the job and all that. If you need to process a 50PB input though, map/reduce is the way to go.
- collyw 13y agoI completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).
- alecco 13y ago> Does anyone have a use case where data is on a single machine and map reduce is still relevant? What matters is the running data structure. For example, you can have Petabytes of logs but you need a map/table of some kind to do aggregations/transformations. Or a sparse-matrix based model. There are types of problems that can partition the data structure and work in parallel in the RAM of many servers. Related: it's extremely common to confuse for CPU bottleneck what's actually a problem of TLB-miss, cache-miss, or bandwidth limits (RAM, disk, network). I'd rather invest time in improving the basic DS/Algorithm than porting to Map/Reduce.
- joeblau 13y agoI was using it on a D3.JS chart to aggregate data flow though our custom real-time analytic pipeline.
- shamsulbuddy 13y agoWill u mind sharing some more info on this real time analytics using d3 as i am also planning to do something like tht..
- jackgolding 13y agolook up crossfilter.js
- cstigler 13y agoMongoDB: where you need Map/Reduce to do any aggregation at all.
- tzury 13y agoYou may consider using Mongo's aggregation framework[1] instead which is zillion times faster[2]. [1] http://docs.mongodb.org/manual/aggregation/ http://docs.mongodb.org/manual/aggregation/ [2] http://stackoverflow.com/questions/13908438/is-mongodb-aggregation-framework-faster-than-map-reduce http://stackoverflow.com/questions/13908438/is-mongodb-aggre...
- redwood 13y agoObviously, in today's theme, Facebook is not using mapreduce effectively to figure out who of our "friends" we actually care about :)
- KyeRussell 13y agoCould you be any more edgy?
- jread 13y agoGenerating web traffic summaries from nginx logs for a CDN with 150 servers, 10-15 billion hits/day. Summaries then stored in MySQL/TokuDB.
- zengr 13y agoWe use MR using Pig (data in cassandra/CFS) with a 6 node hadoop cluster to process timeseries data. The events contain user metrics like which view was tapped, user behavior, search result, clicks etc. We process these events to use it downstream for our search relevancy, internal metrics, see top products. We did this on mysql for a long time but things went really slow. We could have optimized mysql for performance but cassandra was an easier way to go about it and it works for us for now.
- oacgnol 13y agoCan you estimate how much faster your processing is now vs. before on MySQL? I find it interesting that your cluster is only 6 nodes - relatively small compared to what I've seen and from what I've read about. It'd be interesting to know the benefits of small-scale usage of big data tech.
- alecco 13y agoDid you try a columnar database first? Vertica, InfinyDB, MonetDB... There are many.
- lclarkmichalek 13y agoThis system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium data at best. However, we are bound by CPU in a big way, so between throwing more cores at the problem and rewriting everything we can in C, we think we can reduce processing times to an acceptable point (currently about ~4 mins, hoping to hit <30s).
- baudehlo 13y agoThis sounds like something you could just do in SQL and have it all done in milliseconds.
- collyw 13y agoMost of the NoSQL cases I have heard seem to be that they could be done at least as well in SQL. I have asked a lot of questions on this topic, and no one has yet convinced me (please do if you have a legitimate NoSQL case).
- msellout 13y agoExecution time, after a point, is less important than development time. NoSQL is often faster for development and refactoring, because the schema is easier to change in code than in the database. Also there's the scale issue. Append-only is helpful. Why have SQL if you can't use all the features?
- lclarkmichalek 13y agoFor our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.
- batbomb 13y agoI'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world. If processing latencies don't matter much, it's an easier more flexible system to use.
- evanb 13y agoIt is built into QDP++, which is the low-level code underlying lattice QCD code bases like Chroma and MILC, though! Not for gauge configuration generation, which has to be done sequentially, but inversions and measurements can be mapped / reduced.
- virtuabhi 13y agoYes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc. I do hope physics people consider MR systems in addition to the existing ones they use.
- apw 13y agoMPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC. It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.
- seanmcdirmid 13y agoMPI can't schedule to optimize IO/data shuffling like MapReduce, this is the price for being more general. The biggest competition to MPI in the HPC space is actually CUDA and GPUs. In either case, the data pipeline is exposed to the compiler or run time for optimization, where as MPI everything is just messaging.
- 13y ago
- gregoryw 13y agoValidating performance testing simulations. Tie the inputs of the load generators to the outputs from the application server logs and verify the system works as designed at scale.
- rubyfan 13y agoWe are using Map/Reduce to analyze raw XML as well as event activity streams, for example analyzing a collection of events and meta data to understand how discreet events relate to each other as well as patterns leading to certain outcomes. I am primarily using Ruby+Wukong via the Hadoop-Streaming interface as well as Hive to analyze output and for more normalized data problems. The company is a large Fortune 500 P&C insurer and has a small (30 node) Cloudera 4 based cluster in heavy use by different R&D, analytic and technology groups within the company. Those other groups use a variety of toolsets in the environment, I know of Python, R, Java, Pig, Hive, Ruby in use as well as more traditional tools on the periphery in the BI and R&D spaces such as Microstrategy, Ab Initio, SAS, etc.
- clockwork_189 13y agoData aggregation to calculate analytics.
- Aqueous 13y agoTo most people who use MapReduce in a cluster: You probably don't need to use MapReduce. You are either vastly overstating the amount of data you are dealing with and the complexity of what you need to do with that data, or you are vastly understating the amount of computational power a single node actually has. Either way, see how fast you can do it on a single machine before trying to run it on a cluster.
- sitkack 13y agoImage restoration, OCR, face detection, full text indexing. Mostly just a parallel job scheduler.
- DenisM 13y agoIt's worth noting that CouchDB is using map-reduce to define materialized views. Whereas normally MR parallelization is used to scale out, in this case it's used instead to allow incremental updates for the materialized views, which is to say incremental updates for arbitrarily defined indexes! By contrast SQL databases allow incremental updates only for indexes whose definition is well understood by the database engine. I found this to be pretty clever.
- smoyer 13y agoI've been using CouchDB (and now BigCouch) for about four years and it's both clever and useful. We're storing engineering documents (as attachments) and using map/reduce (CouchDB views) to segment the documents by the metadata stored in the fields. The only downside is that adding a view with trillions of rows can take quite a while.
- DenisM 13y agoStrangely, there aren't many discussions of Couch* family on hacker news. Do you know why would that be? I'm thinking about basing a new product around couchbase lite, but lack of popular acceptance is one of the things holding me back.
- smoyer 13y agoI'd be happy to take the discussion off-line if you want to get into more depth but I have to wonder if the merger of Memcached and CouchDB happened right as CouchDB would have made a name for itself. Right around that time, I think CouchDB and Mongo were equally well-known.
- platz 13y agoBillions
- btown 13y agoThis is a system in early development, but my research group is planning on using MapReduce for each iteration of a MCMC algorithm to infer latent characteristics for 70TB of astronomical images. Far too much to store on one node. Planning on using something like PySpark as the MapReduce framework.
- bcoughlan 13y agoA lot of people in this thread are saying that most data is not big enough for MapReduce. I use Hadoop on a single node for ~20GB of data because it is an excellent utility for sorting and grouping data, not because of its size. What should I be using instead?
- grogenaut 13y agoObviously you should throw out the solution that worked for you and start over. 20GB just isn't cool enough to use M/R.
- alexatkeplar 13y agoWe use Elastic MapReduce at Snowplow to validate and enrich raw user events (collected in S3 from web, Lua, Arduino etc clients) into "full fat" schema'ed Snowplow events containing geo-IP information, referer attribution etc. We then load those events from S3 into Redshift and Postgres. So we are solving the problem of processing raw user behavioural data at scale using MapReduce. All of our MapReduce code is written in Scalding, which is a Scala DSL on top of Cascading, which is an ETL/query framework for Hadoop. You can check out our MapReduce code here: https://github.com/snowplow/snowplow/tree/master/3-enrich/hadoop-etl https://github.com/snowplow/snowplow/tree/master/3-enrich/ha...
- sandGorgon 13y agothanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce. have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.
- alexatkeplar 13y agoHi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful: - http://stackoverflow.com/questions/6206105/running-web-fetches-from-within-a-hadoop-cluster http://stackoverflow.com/questions/6206105/running-web-fetch... - http://petewarden.com/2011/05/02/using-hadoop-with-external-api-calls/ http://petewarden.com/2011/05/02/using-hadoop-with-external-... I think a gatekeeper service could make sense; or alternatively you could write something which runs prior to your MapReduce job and e.g. just loads your results into HDFS/HBase, for the MapReduce to then lookup into. Akka or maybe Storm could be choices here. We have done a small prototype project to pull data out of Twitter & Facebook - that was only a Python/RDS pilot, but it gave us some ideas for how a proper social feed into Snowplow could work.
- 13y ago
- pjbrunet 13y agoIt would be nice if someone collected/counted the actual answers. I read the whole thread and "analyzing server logs" was the only answer. If you don't have funding or have dreams/plans, you're not really _using_ the technology.
- mrgriscom 13y agoThings that could be trivially done in SQL :(
- Tossrock 13y agoWe use our own highly customized fork of Hadoop to generate traffic graphs [1] and demographic information for hundreds of thousands of sites from petabytes of data, as well as building predictive models that power targeted display advertising. [1]: https://www.quantcast.com/tumblr.com https://www.quantcast.com/tumblr.com
- espennilsen 13y agoSaw this tweet today https://twitter.com/Andst7/status/399333514803294208/photo/1 https://twitter.com/Andst7/status/399333514803294208/photo/1
- conradev 13y agoAirbnb uses distributed computing to analyze the huge amount of data that it generates every day[1]. The results are used for all sorts of things: assessing how Airbnb affects the local economy (for government relations), optimizing user growth, etc. [1] http://nerds.airbnb.com/distributed-computing-at-airbnb/ http://nerds.airbnb.com/distributed-computing-at-airbnb/
- nl 13y agoWhile the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context. In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data. That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines. HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.
- jfxberns 13y agoIt still is a valid pattern. As disks get bigger, the harder it is to move the data and the more efficient it is to move the compute to the data.
- kirang1989 13y agoJust FYI: Even small problems can be solved using the concept of MapReduce. It is a concept and not an implementation that is tied to BigData and NoSQL. A simple example would be MergeSort. It uses the concept of MR to sort data.
- estebanz01 13y agoProcess a huge amount of spanish text in paper surveys
- mfeldman 13y ago"Things that could be trivially done in SQL :(" We use HIVE over HDFS. Sure the type of things we are doing could have been done in SQL, if we had carefully curated our data set and chosen which parts to keep and which to throw away. Unfortunately, we are greedy in the storage phase. Hive allows us to save much more than we reasonably would need, which actually is great when we need to get it long after the insert was made.