5 ms·
Show HN: Hadoop in Excel
- karamazov 13y agoHi, I'm one of the developers. I'd be happy to answer any questions you have on this. If you're in New York, I'd love to meet you in person at our Big Data in Excel meetup this Monday: http://www.meetup.com/DataNitro/events/149402612/ http://www.meetup.com/DataNitro/events/149402612/ And, as the page says, we're looking for beta users! If you're interested in this, know someone who might be, or just have an opinion, I'd love to talk to you. You can comment here or reach me at ben at datanitro.com.
- cs702 13y agoLooks like a killer product, because there are a lot of business people who readily know how to write what is essentially functional code in Excel but for whatever reason cannot or will not write even simple map and reduce functions in a conventional programming language to extract information from a large Hadoop data set. Are formulas or spreadsheet browsing limited in any way?
- karamazov 13y agoFormulas should be used as columnar operations: you can apply one formula to every element of a column (map) or aggregate an entire column (reduce). (This isn't that much of a restriction - you can use If-Statements to make complex expressions here.) Spreadsheet browsing is limited to a sample of the data (head + tail); you can set the size of the sample before pulling.
- ajasmin 13y agoI don't think "conventional programming languages" (and in particular Java) are well suited to the task of analyzing and querying data. But Excel is also an odd beast. It's used for laying out reports, pivot tables, plotting graphs, working with database, calculations and VBA macros. Not all features play well together.
- deleted 13y ago[deleted]
- pvnick 13y agoDid you write your mappers and reducers in java using the hadoop api or does this translate into hiveql or some other higher-level language? Great job btw, this looks super helpful for the business types to get useful reports on their own rather than interrupt the workflow of someone with more formal training (huge issue typically).
- karamazov 13y agoThanks! We're working directly in Java right now, but might explore alternatives later. We're also planning to add support for Impala/Presto/etc.
- staunch 13y agoFunny as this sounds it may be in fact exactly perfect for a large subset of Hadoop use-cases. If it works well.
- eigenvalue 13y agoI think this would really benefit from a dead simple tool that would allow users to import from csv files into a local Hadoop instance, without having to do anything besides install Hadoop. But this seems like something that could really democratize data analysis on large data sets considering the number of people who are pretty good with Excel.
- monstrado 13y agoWhat are you using on the back-end to perform the queries? Are you using MapReduce? What is the average latency expectations when using the application?
- karamazov 13y agoWe are using MapReduce. Latency will depend on your cluster and the query; it's just a regular MapReduce operation from Hadoop's point of view.
- jackmaney 13y agoUmmmm...doesn't Excel have a row limit of somewhere around 1 million?
- karamazov 13y agoYes, it does. This doesn't involve pulling all of your data into a spreadsheet.
- prawks 13y agoBeing pretty naive to the space, I'm assuming the killer differentiator from Microsoft's own Power Query (which looks like it can pull from Hadoop) is that this pulls a subset of data as an initial workspace, while Power Query pulls all of the data? Any other key differences? Really cool tool! Wish I had some large real-world Hadoop cluster to try it out on...
- karamazov 13y agoThe major difference is the ability to run queries on Hadoop in addition to the being able to pull data.
- wbsun 13y agoCan Excel open a 1-billion-row data file?
- retr0rocket 13y agothere is a limit and its no where near 1 billion.
- karamazov 13y agoNo, it can't - the limit is just over one million rows. This doesn't involve pulling anywhere near that many rows into your spreadsheet.
- Fomite 13y agoWhile impressive in terms of a technical achievement, Excel is a pretty appalling analysis tool generally. I fear for what it will turn into when you throw this much at it. Big Data doesn't let you power through being wrong.
- karamazov 13y agoThis is aimed at people doing simple analyses on massive sets of data, which can work extremely well. [1] We're not advocating that people without a data science background start doing ML or something. [1] See "The Unreasonable Effectiveness of Data", by Peter Norvig, Alon Halevy, and Fernando Pereira at Google.
- Fomite 13y agoWhich is why I'm not besmirching your technical achievement as much as...Excel is widely abused by the ignorant, Big Data is widely abused by the ignorant...Hadoop in Excel...
- RobGoretsky 13y agoI've seen demos of a tool called Datameer which seems to offer very similar functionality (an Excel-like interface for configuring a job on a small set of data, followed by submission of that job to a Hadoop cluster as a MapReduce job). How does DataNitro compare to that?