Great point! It’s important to have the data that we want to separately search and aggregate by laid out in their own column and sorted in order to allow fast p

Great point! It’s important to have the data that we want to separately search and aggregate by laid out in their own column and sorted in order to allow fast processing of that data (query latency) but also achieve better compression since repetitive values can be efficiently encoded (saving both memory and disk once we persist data). We can keep cost of ingestion low because we make trade offs about the mutability of data, as sorting will never change if data is immutable. We globally maintain sorting by requiring writes to be sorted, that way, worst case we look at each in-memory block but in practice at far less when inserting. One thing to clarify, for now arcticDB is just an embeddable database similar to badger, but it's possible we might make it a distributed database in the future (for now this suffices for what we need it to do). Let me know if that clarifies it, happy to elaborate further!