3 ms·
> The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you Absolutely. The challeng
by bob1029 5y ago
> The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you
Absolutely. The challenge is having enough faith that it will take long enough to catch up to you.
Statistically speaking, it won't catch up to you and if it does, it will take so long you should have seen it coming from miles away and had time to prepare.
In my systems that use an in-memory/append-only technique, I try to keep only the pointers and basic indexes in memory. With modern PCIe flash storage, there is no good justification for keeping big fat blobs around in memory anymore.
- aprdm 5y agoCould you expand what you mean by keeping pointers and basic index in memory?
- bob1029 5y agoPointers are tuples of (Id, LogOffset) and are used to map logical identities to positions of those objects in the append-only log. Indexes are usually a tuple of (Some64BitKey, Id) and are used to map physical business keys to logical object identities. These entries are only candidates in the case where the key material needs to be hashed and inspected for actual equivalence. One big advantage with this approach is that you can stream big blobs directly out of the log to a caller-supplied buffer or stream. No intermediate allocations required aside from some small buffers.
- aprdm 5y agoAwesome, thank you for sharing!