6 ms·
This talk was great, especially its focus on non-parametric techniques. A lot of people are mislead by the traditional statistics approach, where it looks like
by pavpanchekha 11y ago
This talk was great, especially its focus on non-parametric techniques. A lot of people are mislead by the traditional statistics approach, where it looks like stats is about memorizing a relatively short list of (really scary) formulas that let you compute one or another thing. People do not think about how these formulas come about, and the simple ideas behind them.
On the other hand, I think there is not enough written about how to do statistics with real, large, complex data sets. I tried to write something like this (https://pavpanchekha.com/blog/stats1.html https://pavpanchekha.com/blog/stats1.html) but of course the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post. In a large dataset, the difficulties are the how and when you make decisions about what to analyze.
- minimaxir 11y ago> the difficulty is that investigating a complex dataset is by definition too complex to really fit into a blog post It's not complex, but it requires answering scaleability questions. How do you fit the data into memory? How do you even get big data? I managed to analyze 5.6M Hacker News comments (http://minimaxir.com/2014/10/hn-comments-about-comments/ http://minimaxir.com/2014/10/hn-comments-about-comments/ ) by writing a scraper which dumps everything into a PostgreSQL database (https://github.com/minimaxir/get-all-hacker-news-submissions-comments https://github.com/minimaxir/get-all-hacker-news-submissions... ) because it's way too big to fit in memory. Likewise, to visualize the recent NYC taxi data, (http://minimaxir.com/2015/08/nyc-map/ http://minimaxir.com/2015/08/nyc-map/ ), I just used BigQuery and saved myself a load of time downloading the data and setting things up. But in many cases, data is not easily accessible, and not cheap to store and process. I want to switch to an actual big data workflow with Spark, but running the numbers with EC2 storage and processing, it won't be cheap and I may have to set up a Patreon to subsidize things.
- MengerSponge 11y agoThis might be useful the next time you need to scope your project's memory requirements. http://yourdatafitsinram.com/ http://yourdatafitsinram.com/
- minimaxir 11y agoHence the "it won't be cheap" qualifier. :P
- deleted 11y ago[deleted]
- eximius 11y agoRiiiiight.... Not many places can afford a server with 6TB of RAM. Dear Lord, I didn't even know that was possible with commodity hardware. This would be hard to justify at a data-driven business even.
- kragen 11y agoStoring your data on disk costs less money for the hardware per byte stored; storing your data in RAM costs less money per (transaction * byte stored). On EC2, 6TB of RAM would be 25 r3.8xlarge instances, which is US$70 per hour. If your six-terabyte task is batch processing rather than interactive, that might be cheaper than spending a lot of time optimizing your software for spinning-rust relics. Contrariwise, if it's an interactive computing task that you can do with AWS Lambda and which you can split into small pieces, 4000 Lambda functions each with 1.5 GiB of RAM, for a total of 6TB, will cost you US$0.01 per 100 milliseconds, or US$0.10 per second. Supposing your overall in-RAM computing task can be done with 10 seconds per Lambda function (a total of 40 000 CPU seconds (11 CPU hours)), can actually be parallelized to that extent, and that 10 seconds is an acceptable response time, then you can do it for US$1. I feel that this must be a mistake, since this is 700 times cheaper than EC2. (Lambda surely has some total capacity limitations, but I imagine they're quite a bit more than 6TB.) I am deeply unhappy about the move to such a centralized infrastructure for the internet, but at least this version of centralized infrastructure allows you to run your own code on the centralized servers, unlike the API vision promoted by Facebook and (mostly) Google.
- calpaterson 11y agoI think he was talking about the complexity of the analysis and not about the technical complexity of getting and storing the data.
- minimaxir 11y agoStorage/scalability must be solved before doing the complex analysis, through, and is the primary bottleneck.
- deleted 11y ago[deleted]
- rsy96 11y agoEvery textbook on math I have ever read always start with intuition behind the idea, and then present definitions and proofs. Only some reference books, intended for look ups only, only enumerate formulas. I never understand why anyone claims that they only learn math by memorizing formulas.
- eachro 11y agoPeople claim they only learn math by memorizing formulas because for most, that's all the math they see before getting irreversibly turned off by the subject. Most people learn math by memorizing plug and chug formulas in high school. Aside from 9th grade euclidean geometry, I can't recall an instance where I had to derive anything in my hs math courses. College math courses, of course, are usually much more rigorous.
- wodenokoto 11y agoBecause they learn math in junior high where they really don't care about the subject, so it gets boiled down to "remember this to pass the exam", regardless of what the book says.
- pavpanchekha 11y agoTake a look at what AP Stats teaches.