3 ms·
Re: theory #1- there are commodity solutions to the queriable data warehouse problem, e.g. Vertica or Greenplum. Looking at these can give you an idea of what
by nohuck13 13y ago
Re: theory #1- there are commodity solutions to the queriable data warehouse problem, e.g. Vertica or Greenplum. Looking at these can give you an idea of what might be hard. You've just described Vertica pretty accurately (column-oriented storage, materialised on disk for particular query patterns, ad-hoc deletion comparatively costly). And ageing out in Vertica is easy- you just drop a partition. I can't imagine a custom system would be built without so e provision for this.
- bigiain 13y agoSure, if you were building it from scratch today, but Vertical was founded in 2005 - the article talks about data going back to 1987. Cynical-me suspects the NSA has been "requiring" them to retain all that data since before it became easy/practical to expire it properly.