4 ms·
> D3 is probably good for its domain, but intuition tells me you'll have a problem if you mainly use Javascript to handle a 20GB dataset. (I'm not dismissing th
by anko 12y ago
> D3 is probably good for its domain, but intuition tells me you'll have a problem if you mainly use Javascript to handle a 20GB dataset. (I'm not dismissing this categorically; I am not very familiar with these tools).
I use d3 on the client with a 80GB (currently) dataset, by putting the dataset in elasticsearch. It's a pretty fantastic combination. You can do multi-value aggregation from unstructured data, or geo-spacial searches, or lightning-quick full text search.
The server has 8GB of ram and 2 cores, and with about 1.2 million new documents every hour, barely breaks a sweat.
- capkutay 12y agoWhat type of queries do you run on elasticsearch to pull into D3? I'm doing a very similar project (elasticsearch + web data vis) so I'm legitimately curious.
- anko 12y agoBasically I'm importing logs and system events. I run queries like show me the top 10 events over the last 24 hours from this source that were marked critical. Or for each farm shipping web logs, aggregate on the hosts, and then aggregate on the status code, and then give me the number of documents in each summed for each hour of the day. At the moment I roll up daily stats and store them in a separate database for longitudinal analysis, but eventually I'd like to ship data that is more than a couple of weeks old to hadoop.