3 ms·
Making big data manageable
- fnord123 10y agoThis is neat. I bet if you use a hypercircle technique then you could project wikipedia articles into a GiST or BallTree or similar n dimensional tree structure. Then search can be done by projecting a query into the hypercircle and returning results close by.
- stdbrouw 10y agoYou would have a representative dataset but you'd still be stuck with the curse of dimensionality that plagues nearest neighbors: in high dimensions, everything is far away from each other and nothing is close to the center.
- unix-junkie 10y agoHere is the paper: https://papers.nips.cc/paper/6596-dimensionality-reduction-of-massive-sparse-datasets-using-coresets.pdf https://papers.nips.cc/paper/6596-dimensionality-reduction-o...
- collyw 10y agoThis sounds like something that could be done with a materialized view, or am I missing something?
- zbjornson 10y agoThe point of this algorithm is sort of to construct a materialized view, but how it's done is the cool part. It's not a simple database query -- it reduces the number of dimensions into a manageable number.
- jortiz81 10y agoIndeed, the core contribution (section 1.2) is the cool part.
- afian 10y agoI know these people! They are amazing :)
- MeteorMarc 10y agoBut then it is not big data anymore!