3 ms·

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if yo

by brancz 4y ago

Hey all, one of the creators of ArcticDB here. We're going to be around for a while and answer any questions you might have about it! It's open source so if you just want to check out the repo: https://github.com/polarsignals/arcticdb https://github.com/polarsignals/arcticdb

eatonphil 4y ago
It looks like there's not a textual query interface to it at the moment. Do I have that wrong or are you interested in adding a high level query interface in the future?
- brancz 4y ago
  Very good question! While we may add a textual query interface (maybe even SQL) in the future, we very intentionally started with this abstraction, since most Observability projects out there (eg. Prometheus, Grafana Loki, Parca) have specialised query languages that we wanted to have a lower level abstractions that those languages could be implemented on top of, instead of having to transpile to SQL (or whatever proprietary language we might have come up with).
bfm 4y ago
What was the motivation for building your own instead of using something like DuckDB which support parquet out of the box? What are the differences?
- eatonphil 4y ago
  In addition to other reasons they may have, an embedded database for Go being built in Go means they don't need to require CGO, which DuckDB in Go would require.
  - brancz 4y ago
    This was definitely part of the motivation, but even more importantly (and it's possible that we missed it in the duckdb documentation when we explored doing exactly this) we needed the ability to add columns dynamically when we see a new label-name. This is sort of an analogy of wide-columns in Cassandra, but forcing it into the columnar layout to allow it to be searched and aggregated by efficiently. From our research all the open source column databases at best support a map type, through which we loose the columnar layout since the values of the map are all stored together giving us row-based database characteristics. (all databases except InfluxDB IOx, whose developers we talked to extensively and who highly inspired this design)
    - bfm 4y ago
      It makes sense, DuckDB's documentation has significantly improved, but it is still lacking when it comes to the limitations of using parquet. We have also hit some roadblocks when updating schemas backed by parquet files, so we now only use DuckDB for querying parquet via SQL.
jtwebman 4y ago
You really should change the front page to why I should use it vs why we built it. I don't care why you built it that it is in Go. I do care why I should use it which must be hidden somewhere in why you built it but I don't have that kind of time.
- brancz 4y ago
  Thank you so much for the feedback! By front page I assume you mean the project readme? I agree, I think we leaned a bit too far into design documentation in the readme rather than explaining why and how to use it. We'll get that fixed!
caust1c 4y ago
How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like storage engines, except using something like a K-Way Merge algorithm and a heap structure to manage that somehow? I'm more of an operator and user of these systems, so as an operator I care more about the usability than what's underneath, but also am reasonably skeptical of new databases since theres literally hundreds being written every year. So what benefits does this structure and data format provide over classical LSM-like databases which are currently dominating the high-write-throughput embedded DB space?
- brancz 4y ago
  It's closer to DuckDB rather than Badger/RocksDB/LevelDB, but similar in the sense that it is an embeddable database, not one that is operated standalone. It's not unlike an LSM tree, but the difference is that the leafs in the tree are not individual keys, but rather describe a range of values that are all read at once if read. So this allows high write throughput _and_ high read throughput, trading off mutability (which is ok for the Observability use cases we aim for it to fulfil). I think it's reasonable to be skeptical about new databases, if it helps, we worked on the Prometheus and Thanos project storage layers before we started the work on this, which now powers hundreds of thousands of monitoring stacks out there.
  - caust1c 4y ago
    It was my understanding that those databases do have sparse indexes, but admittedly I may have applied that assumption based on my majority experience with Clickhouse where it uses LSM Tree engines and also has a sparse index. Would it be correct to say this is like an embeddable clickhouse engine, minus the SQL interface and using Arrow and Parquet as the storage format?
    - brancz 4y ago
      Yes, that's correct! Plus the dynamic column feature, which we think is crucial for Observability-type workloads (from what we know only InfluxDB IOx supports a similar feature).
bigcat12345678 4y ago
Does ArcticDB support store data in memory, and flush overflown & old data onto disk (or other persistent storage sink)?
- brancz 4y ago
  Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parquet format into object storage from where it can be consumed from any parquet compatible tools. That plus additional metadata is what we’ll use for long term storage for profiling data ourselves.
  - bigcat12345678 4y ago
    Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use? We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually. Read the other comment: https://news.ycombinator.com/item?id=31263825 https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.
    - brancz 4y ago
      ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames. Hope that explains it, but happy to elaborate more!
      - bigcat12345678 4y ago
        IIUC: The query written by users of arcticdb is in a dataframe-like python-ish language. The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames. Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames. Does the query engine produce the query plan before selecting the arrow frames, and uses the plan for the selecting process as well?

by brancz 4y ago

eatonphil 4y ago

It looks like there's not a textual query interface to it at the moment. Do I have that wrong or are you interested in adding a high level query interface in the future?

brancz 4y ago
Very good question! While we may add a textual query interface (maybe even SQL) in the future, we very intentionally started with this abstraction, since most Observability projects out there (eg. Prometheus, Grafana Loki, Parca) have specialised query languages that we wanted to have a lower level abstractions that those languages could be implemented on top of, instead of having to transpile to SQL (or whatever proprietary language we might have come up with).

bfm 4y ago

What was the motivation for building your own instead of using something like DuckDB which support parquet out of the box? What are the differences?

eatonphil 4y ago
In addition to other reasons they may have, an embedded database for Go being built in Go means they don't need to require CGO, which DuckDB in Go would require.
- brancz 4y ago
  This was definitely part of the motivation, but even more importantly (and it's possible that we missed it in the duckdb documentation when we explored doing exactly this) we needed the ability to add columns dynamically when we see a new label-name. This is sort of an analogy of wide-columns in Cassandra, but forcing it into the columnar layout to allow it to be searched and aggregated by efficiently. From our research all the open source column databases at best support a map type, through which we loose the columnar layout since the values of the map are all stored together giving us row-based database characteristics. (all databases except InfluxDB IOx, whose developers we talked to extensively and who highly inspired this design)
  - bfm 4y ago
    It makes sense, DuckDB's documentation has significantly improved, but it is still lacking when it comes to the limitations of using parquet. We have also hit some roadblocks when updating schemas backed by parquet files, so we now only use DuckDB for querying parquet via SQL.

jtwebman 4y ago

You really should change the front page to why I should use it vs why we built it. I don't care why you built it that it is in Go. I do care why I should use it which must be hidden somewhere in why you built it but I don't have that kind of time.

brancz 4y ago
Thank you so much for the feedback! By front page I assume you mean the project readme? I agree, I think we leaned a bit too far into design documentation in the readme rather than explaining why and how to use it. We'll get that fixed!

caust1c 4y ago

How does it compare to BadgerDB/RocksDB/LevelDB? I see that it's using Arrow and Parquet of course, but the Sparse Index sounds very similar to LSM Tree like storage engines, except using something like a K-Way Merge algorithm and a heap structure to manage that somehow? I'm more of an operator and user of these systems, so as an operator I care more about the usability than what's underneath, but also am reasonably skeptical of new databases since theres literally hundreds being written every year. So what benefits does this structure and data format provide over classical LSM-like databases which are currently dominating the high-write-throughput embedded DB space?

brancz 4y ago
It's closer to DuckDB rather than Badger/RocksDB/LevelDB, but similar in the sense that it is an embeddable database, not one that is operated standalone. It's not unlike an LSM tree, but the difference is that the leafs in the tree are not individual keys, but rather describe a range of values that are all read at once if read. So this allows high write throughput _and_ high read throughput, trading off mutability (which is ok for the Observability use cases we aim for it to fulfil). I think it's reasonable to be skeptical about new databases, if it helps, we worked on the Prometheus and Thanos project storage layers before we started the work on this, which now powers hundreds of thousands of monitoring stacks out there.
- caust1c 4y ago
  It was my understanding that those databases do have sparse indexes, but admittedly I may have applied that assumption based on my majority experience with Clickhouse where it uses LSM Tree engines and also has a sparse index. Would it be correct to say this is like an embeddable clickhouse engine, minus the SQL interface and using Arrow and Parquet as the storage format?
  - brancz 4y ago
    Yes, that's correct! Plus the dynamic column feature, which we think is crucial for Observability-type workloads (from what we know only InfluxDB IOx supports a similar feature).

bigcat12345678 4y ago

Does ArcticDB support store data in memory, and flush overflown & old data onto disk (or other persistent storage sink)?

brancz 4y ago
Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parquet format into object storage from where it can be consumed from any parquet compatible tools. That plus additional metadata is what we’ll use for long term storage for profiling data ourselves.
- bigcat12345678 4y ago
  Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use? We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually. Read the other comment: https://news.ycombinator.com/item?id=31263825 https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.
  - brancz 4y ago
    ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames. Hope that explains it, but happy to elaborate more!
    - bigcat12345678 4y ago
      IIUC: The query written by users of arcticdb is in a dataframe-like python-ish language. The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames. Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames. Does the query engine produce the query plan before selecting the arrow frames, and uses the plan for the selecting process as well?