9 ms·
Time Series and FoundationDB (2019)
- Xeoncross 4y agoThis is a really neat experiment. I really appreciate all the time that went into documenting and training the reader on why and how. Efficient compression of time series data is something that is fun to think about just like probabilistic structures or tries which provide space/time saves orders of magnitude more than naive approaches.
- richieartoul 4y agoKind of surprised how fast this took off, I wrote this 2 years ago and was showing it to a friend today who was asking how they might build an inverted index using FoundationDB and decided to repost it. If you’re looking for more examples of using FoundationDB to store timeseries data, we describe a much more scaleable approach here: https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky...
- rektide 4y agoRecently submitted: https://news.ycombinator.com/item?id=31416843 https://news.ycombinator.com/item?id=31416843 (185 points, 11d ago, 15 comments)
- deleted 4y ago[deleted]
- yed 4y agoIn Husky, how do you maintain very short cache times in your Writers without blowing up your S3 write costs?
- richieartoul 4y agoIt’s (very lightly) hinted at in the blog post (the section where we discuss “shard router”), but basically data locality in our pipelines by tenant + a little bit of buffering goes a long way. I’m hoping I can coax one of my colleagues who works on the routing side of things to write a detailed blog post on just that topic though :)
- yed 4y agoThat would be great! Really appreciate your write up, this space is really fascinating. The short cache / no stateful Writer / no searches on Writers thing stood out to me as a key differentiator to something like Grafana Loki.
- javajosh 4y ago"imagine you have a fleet of 50,000 servers..." It's interesting to consider when you would ever need so many servers, from first principles, considering that a single server can easily handle a million simultaneous connections, and very few applications are used by every man woman and child on Earth and even fewer are used by multiples of that.
- tilolebo 4y ago> considering that a single server can easily handle a million simultaneous connections Can you elaborate on that part?
- joisig 4y agoNot the GP but they might be referring to [0] or one of several other articles you will find if you Google "handle a million connections in *". Realistically you also usually need to perform some non-trivial work from time to time for some non-trivial portion of those connections, which will further load your server, but still. [0] https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://phoenixframework.org/blog/the-road-to-2-million-webs...
- crgwbr 4y agoGenuinely curious on this—with port numbers only being 16 bits, how is it possible for one machine to ever handle more than 65k concurrent connections?
- dxhdr 4y agoA connection is the tuple {source_ip, source_port, dest_ip, dest_port}
- pipe_connector 4y agoConnections must have unique IP:Port pairs between client and server. You're limited to 65K concurrent connections for the same client. In practice, no one is opening that many connections from a single client.
- endisneigh 4y agoI thought about making something using SQLite virtual tables to back all of the data on FDB in order to basically provide a SQL interface with FDB without writing a new layer, but some others tried and apparently didn't go well since you can't have indexes using virtual tables. Truly a shame. Any other storage engines that support using another interface as the underlying store?
- alexchamberlain 4y agoPostgres Foreign Data Wrappers?
- snissn 4y agoYeah was gonna suggest this! I’m not the biggest expert but you can put Postgres in front of all your data sources and have pg route the queries through a heterogeneous infrastructure and get a nice interface
- richieartoul 4y agoYeah that would be cool. You could try forking an existing “SQL over key value” system like Cockroach or TiDB and replacing the KV store with FoundationDB. That is probably what I would explore first. Some of my old colleagues/friends are working on building a transactional document store (among other things) on top of FoundationDB, so if you can forgo SQL that’s another project you could explore working on: https://github.com/tigrisdata/tigris https://github.com/tigrisdata/tigris
- zitterbewegung 4y agoFoundationDB uses SQLite as the storage layer unless you mean to access the storage layer in FoundationDB to perform that task?
- endisneigh 4y agoyes, access through storage layer through FDB using a SQL interface. aka SQL interface with FDB guarantees.
- SnowHill9902 4y agoWhy does he define the sample schema with “series_id TEXT, timestamp integer”? Isn’t it more reasonable to use “series_id bigint, timestamp timestamptz”?
- Mo3 4y agoseries_id - definitely looks like a mistake - timestamp could just be Unix time? I always use ints for unix times too, I have no hard data on that decision, but it feels like that might avoid a little bit of overhead.
- SnowHill9902 4y agoSomeone could chime in but timestamptz is just an 8 byte integer like a bigint.
- alberth 4y agoWhen should you use FDB over Redis?
- deleted 4y ago[deleted]
- endisneigh 4y agoNot super familiar with Redis performance - is it much better than FDB for single machine?
- datalopers 4y agoFor single-machine key-value needs (and lists, sets, sorted sets, hash tables, and various full-text operations) it’s by far fastest in my experience. But it’s not going to have the level of transactional guarantees in a clustered setup or full ACID (writes aren’t immediately durable) that some use-cases will require. Treat Redis more like a cache or job queue or pub-sub or event streaming platform or full-text search engine than a primary datastore of mission-critical data.
- endisneigh 4y agoIs Redis really more performant than FDB on a single instance when Redis persists to disk? It wouldn't surprise me that Redis is faster in-memory, tho.
- datalopers 4y agoNo idea, I haven't benchmarked a 100% durable redis setup (aof, fsync every query) against FDB but that will definitely have a significant hit to write performance. I generally stick to the model of not using Redis for mission-critical durability use cases. I don't really view them as competing technologies in the first place though.
- WJW 4y agoOne underappreciated part about Redis is that its singlethreaded in-memory approach makes every thing extremely atomic. There is literally no way for simultaneous operations to conflict if you only ever run one operation at a time. That said, you indeed need to be very careful around durability since the dataset is kept in memory. Most persistence options are a bit meh compared to "real" databases IMO. Redis is a very fast ACI-compliant datastore if you will.
- beebmam 4y agoIt seems like it is a suboptimal choice to use a garbage collected language when trying to optimize for throughput. What made you decide to use Go instead of a language like Rust or C++?
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- richieartoul 4y agoThis is a controversial opinion, but I’ve been generally unimpressed with the performance of real world systems written in Rust or C++. While the theoretical performance ceiling is higher, I’ve personally observed that it usually takes several times longer to write the systems in Rust/C++ and often the “naive” Rust implementation is less performant than the “naive” go implementation. There are obviously cases where using Rust or C++ makes a lot of sense, but I prefer Go. I can get a performant prototype up and running quickly, the “critical path” can usually be micro optimized to within +/- 10% of Rust if you know what you’re doing, and by the time the Rust implementation matures the Go implementation has already had time to iterate repeatedly on key architectural, data structure, and algorithmic choices based on real world usage. That’s just my 2cents though and doesn’t reflect the views of all the people I work with.
- purplerabbit 4y agoVery astute observation. Although it's a bit painful to admit, as Rust scratches a perfectionist itch which Go absolutely does not. I've read a bit about algorithm-driven performance gains vs. hardware-driven performance gains... this seems like an argument for a third category: DX-driven performance gains.
- the_duke 4y agoThese systems usually spend a lot of time waiting for network/disk IO, which means there is plenty of CPU time available to do the computations or GC runs, and the added memory pressure isn't so significant. The Go runtime is also finely tuned for exactly these kinds of workloads, with language level insight into blocking operations. If we take the Tokio async runtime for Rust: it's not exactly the fastest runtime out there, comparatively new, you need to carefully avoid introducing blocking calls that stall the worker threads, there is no language level integration, and Rusts trait system currently often requires boxing of futures. Combine all of that and you often end up with code that isn't noticeably faster than Go in practice. One counterpoint though: I've found that Rust code takes longer to write, but it's significantly easier to refactor, thanks to the much stronger type system.
- avinassh 4y ago> Time series means different things to different people. In this case, I want to focus on the type of time series storage engine that could efficiently power an OLTP system (strong consistency and immediately read your writes) or a monitoring / observability workload as opposed to a time series database designed for OLAP workloads. I am more interested in the latter part: > as opposed to a time series database designed for OLAP workloads. how it would be any different? Can anyone explain how/why time series for OLTP would be different than OLAP? (I always thought time series is stored in OLAP databases)
- richieartoul 4y agoI described a modern take on an OLAP timeseries system recently here: https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky... I don’t know if that helps, but contrasting those two systems might give you some idea of the difference.
- pradeepchhetri 4y agoI see you mentioned M3DB throughout the README. Have you looked at VictoriaMetrics ?
- richieartoul 4y agoI only mentioned M3DB because I was a maintainer of it at the time, so it’s what I was familiar with. I’m a fan of VictoriaMetrics; it’s performant and well designed. The main dev Aliaksandr is prolific, and a really nice guy to boot. I’ve messaged him a few times to ask for advice / help and he’s always sent me very detailed and thoughtful responses. These days I’m working on “non metrics” timeseries storage (I.E full fidelity events) at Datadog so my focus has shifted to the “real time OLAP data warehouse” side of things which is quite a bit different from traditional metrics stores.
- pradeepchhetri 4y agoThank you!
- andrew-ld 4y agogeneral comment and not just about this post: on hackernews we keep seeing relatively dead projects made in go in a few lines of code that promise to handle a really large amount of data in a short time, every time we end up with fake benchmarks or partial implementations of how it should really be, please before criticising the real projects (which unlike them are larger than tens of thousands of lines) check the claims carefully.
- richieartoul 4y agoI was pretty clear in the first paragraph that the code is a half-baked research PoC that should not be used for anything serious.