3 ms·
Thanks for trying it out! I haven't had time to run any benchmarks, so I really appreciate you taking the time to do this (especially since it took so long!).
by misframer 12y ago
Thanks for trying it out! I haven't had time to run any benchmarks, so I really appreciate you taking the time to do this (especially since it took so long!).
I'm not sure what the best batch size is at the moment. Timestamps are int64s, and it's up to the user to interpret them as they wish. Partition sizes are in terms of the number of timestamps. If you had timestamps which correspond to seconds, and you wanted each partition to be 1 day, you'd choose 86400. This isn't configurable yet unless you modify the source.
I'm not surprised it's that slow. I'm not a storage engine expert, still a college student, and this has my first lock-free list implementation :). There is a lot of silliness going on with inserts, like launching a goroutine for each metric on inserts[0], using lock-free O(n) lists for sources and metrics[1] when I could have used a map, there's still a lock left that should be removed[2], and a bunch of others.
On an unrelated note, I see that someone from Prometheus will be at Monitorama. I'll be there too, so I'd love to you guys some more.
Thanks again!
[0] https://github.com/PreetamJinka/catena/blob/8e068b1b95ce1a107cfed75e2f9e33693ce4bff5/mem_partition.go#L163-L169 https://github.com/PreetamJinka/catena/blob/8e068b1b95ce1a10...
[1] https://github.com/PreetamJinka/catena/blob/8e068b1b95ce1a107cfed75e2f9e33693ce4bff5/mem_partition.go#L191 https://github.com/PreetamJinka/catena/blob/8e068b1b95ce1a10...
[2] https://github.com/PreetamJinka/catena/blob/8e068b1b95ce1a107cfed75e2f9e33693ce4bff5/mem_partition.go#L273-L274 https://github.com/PreetamJinka/catena/blob/8e068b1b95ce1a10...
- jrv 12y agoAh ok, so some obvious things left to optimize then :) Thanks for the explanations! As far as I know, none of the Prometheus core developers will be at Monitorama this year. But we'll be at SRECon Dublin, DevOpsCon Berlin, GopherCon Denver, and hopefully also at one (or both) of the VelocityConfs later this year. Indeed, it would be great to catch up in person if you're at any of these.
- misframer 12y agoI see. I should be going to GopherCon too, so I hope to see you there.