3 ms·

It's great to see more solutions in this space, and congratulations on launching and open sourcing it. Is it possible to explain why it matters to an end users?

It's great to see more solutions in this space, and congratulations on launching and open sourcing it. Is it possible to explain why it matters to an end users? The Why We Built It section seems solely focus on implementation and what matters to the team. > First, we needed something embeddable for Go I'm not sure why it matters if I just need to store and analyze large quantity of profiling data, but let me assume this matters to the creator of the db. > The second and more pressing argument was: in order for us to be able to translate the label-based data-model to a table-based layout, we needed the ability to create columns whenever we see a label-name for the first time. Why does this matter to me, an end user? Does it make ingestion faster? Does it make query faster? Does it support higher throughput compared with M3DB or Netflix Atlas or FB Gorilla? Does it make distributed query more scalable? Does it enable the support of higher cardinality and more dimensions? Does it enable more expressive aggregations or query semantics in general? Does it enable the db to model data beyond multi-dimensional timer series?

brancz 5y ago
Great point! It’s important to have the data that we want to separately search and aggregate by laid out in their own column and sorted in order to allow fast processing of that data (query latency) but also achieve better compression since repetitive values can be efficiently encoded (saving both memory and disk once we persist data). We can keep cost of ingestion low because we make trade offs about the mutability of data, as sorting will never change if data is immutable. We globally maintain sorting by requiring writes to be sorted, that way, worst case we look at each in-memory block but in practice at far less when inserting. One thing to clarify, for now arcticDB is just an embeddable database similar to badger, but it's possible we might make it a distributed database in the future (for now this suffices for what we need it to do). Let me know if that clarifies it, happy to elaborate further!

brancz 5y ago

Great point! It’s important to have the data that we want to separately search and aggregate by laid out in their own column and sorted in order to allow fast processing of that data (query latency) but also achieve better compression since repetitive values can be efficiently encoded (saving both memory and disk once we persist data). We can keep cost of ingestion low because we make trade offs about the mutability of data, as sorting will never change if data is immutable. We globally maintain sorting by requiring writes to be sorted, that way, worst case we look at each in-memory block but in practice at far less when inserting. One thing to clarify, for now arcticDB is just an embeddable database similar to badger, but it's possible we might make it a distributed database in the future (for now this suffices for what we need it to do). Let me know if that clarifies it, happy to elaborate further!