6 ms·
> It genuinely makes sense for incumbent database players to offer vector search, because that eliminates unnecessary data movement to separate vector databases
by esafak 3y ago
> It genuinely makes sense for incumbent database players to offer vector search, because that eliminates unnecessary data movement to separate vector databases. Co-locating vectors and original documents also reduces latency.
Yet OLAP databases continue to thrive alongside OLTP databases, the nascence of NewSQL hybrid (HTAP) databases notwithstanding. Different needs dictate different design choices for optimality.
- appplication 3y agoIt’s interesting that I never considered why OLTP and OLAP are basically orthogonal technologies. Are there any major players that have an integrated solution for both? I guess it makes sense because the infra is so different, but I’m not sure whether it need be.
- barrkel 3y agoEffecient OLAP queries need a different shape of data - some combination of columnar storage for efficient scanning, and roll-up tables with pre-aggregated measures. Even in an integrated scenario, behind the scenes there will need to be a bunch of copying to transpose and / or refresh roll-ups.
- taeric 3y agoThe pre-aggregation is one too many try and skip. People seem to think they can build a single schema to rule all things, and then assume they can quickly calculate any aggregation on demand.
- esafak 3y agoAs far as the big players are concerned, Google offers AlloyDB (https://cloud.google.com/alloydb https://cloud.google.com/alloydb) while Amazon offers Aurora (https://aws.amazon.com/rds/aurora/ https://aws.amazon.com/rds/aurora/)
- anentropic 3y agoI had understood Aurora as just cloud-native MySQL / PostgreSQL How does that relate to the OLTP vs OLAP dimension? Are they not both primarily OLTP dbs still?
- xvinci 3y agoI think they have, they are just not that well known. E.g. SQL Server - https://learn.microsoft.com/en-us/sql/relational-databases/indexes/columnstore-indexes-overview?view=sql-server-ver16 https://learn.microsoft.com/en-us/sql/relational-databases/i... - you can also find quite a lot of papers by microsoft employees on the designs and capabilities (starting around 2016 I believe, so "pretty new"). I have used it with TPC-H and it worked wonders, never got around to using it in a production workload though.
- mr_toad 3y agoIs anyone considering a new OLAP system these days? If “NewSQL” (which seems to be a fancy buzzword for running analytics in your transactional database) takes off wont it be the final nail in the coffin for OLAP?
- panda888888 3y agoI think the opposite concept, "reverse ETL," is actually more popular. You put everything into your data warehouse and then pump whatever you need out from there.
- redwood 3y agoNot if you expect to do a bunch of data transformation before you materialize it into one of these engines.. if the way you want a materialize it is materially different than the way you handle it transactionally you're going to be moving it anyway and if you're moving it why not put it into an optimized context
- d_watt 3y ago"NewSQL" is not about analytical performance, it's about multimaster distributed writes and reads. EG, you don't want to run count(*) on spanner. "HTAP" is the buzzword you're looking for. It's promising, but also complex and nascent. It'll be interesting to see how much traction it gets over time, but things like TiDB and Unistore are pretty early on to call a nail in the coffin for redshift/bigquery/clickhouse etc.
- nhourcard 3y agoDifferent needs dictate different design choices for optimality. Could not agree more. Even for time series, which could be seen as a subset of OLAP, trade-offs and design choices inherent to time-series data are necessary. As an example of a TSDB that I know well, QuestDB: Data is always ordered by time once it lands on the disk, the data is partitioned by time, and the ingestion protocol is conceived to stream large volumes of data, which can be either continuous or in bursts.