3 ms·
Does anyone use Hadoop for job management? We have millions of XML documents in a document database. Many of the questions we want to ask about those document
by jimbokun 13y ago
Does anyone use Hadoop for job management?
We have millions of XML documents in a document database. Many of the questions we want to ask about those documents can be answered through the native database querying capabilities.
But there are always questions falling outside the scope of the query capabilities, that could be answered by a simple map function applied to each document, with a reduce to combine the results.
Seems like a pain to always query for the documents you want to process, find some place to store them on disk, then run a program locally to get the result, vs. writing a map and reduce job then pointing it at the documents in the database to run against (this document store has a Hadoop integration API). Hadoop also seems to have a lot of nice job frameworks and monitoring tools and APIs to track job progress.
Anyone have similar situation where you used Hadoop just to get job management and tracking and flexibility in performing data analysis tasks? Are there easier ways to accomplish this goal?
- Chico75 13y agoThe company I work for creates a product which may help you : http://www.syncsort.com/en/Data-Integration/Products/DMX-h/DMX-h-Overview http://www.syncsort.com/en/Data-Integration/Products/DMX-h/D...