4 ms·

Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parque

by brancz 4y ago

Not yet, but we will soon, though we already rotate the in memory state when a certain (configurable) size is reached. The idea is that we’ll put data in parquet format into object storage from where it can be consumed from any parquet compatible tools. That plus additional metadata is what we’ll use for long term storage for profiling data ourselves.

bigcat12345678 4y ago
Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use? We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually. Read the other comment: https://news.ycombinator.com/item?id=31263825 https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.
- brancz 4y ago
  ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames. Hope that explains it, but happy to elaborate more!
  - bigcat12345678 4y ago
    IIUC: The query written by users of arcticdb is in a dataframe-like python-ish language. The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames. Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames. Does the query engine produce the query plan before selecting the arrow frames, and uses the plan for the selecting process as well?
    - brancz 4y ago
      As it stands, right now the query planner doesn’t look at any of the data (yet) when putting together the plan, but this will inevitably happen.

by brancz 4y ago

bigcat12345678 4y ago

Just to clarify more on the in-memory part, it offers the same Arrow APIs for querying, right? Or only specialized query API tailored to Parca's use? We'd like it to be Arrow APIs, such that we can use it for other purposes, but still in the Observability space actually. Read the other comment: https://news.ycombinator.com/item?id=31263825 https://news.ycombinator.com/item?id=31263825 Already explained that the in-memory part, but missing the serialization part.

brancz 4y ago
ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames. Hope that explains it, but happy to elaborate more!
- bigcat12345678 4y ago
  IIUC: The query written by users of arcticdb is in a dataframe-like python-ish language. The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames. Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames. Does the query engine produce the query plan before selecting the arrow frames, and uses the plan for the selecting process as well?
  - brancz 4y ago
    As it stands, right now the query planner doesn’t look at any of the data (yet) when putting together the plan, but this will inevitably happen.