5 ms·

Vector – Open-source Rust utility for logs, metrics and events

145 points by weitzj 7y ago

cthalupa 7y ago
Slightly unfortunate naming, as there is already a Vector project in the metrics space. http://getvector.io/ http://getvector.io/
- hnarn 7y ago
  It seems to be somewhat of a trend to prepend or somehow integrate the language used in naming schemes to ensure you have a somewhat unique name, something like "rvector" would at least make this a lot more searchable, because I suspect most people would search for something like "vector rust" with some additions anyway when looking for this.
  - CDSlice 7y ago
    The problem with searching "vector rust" is that vectors are one of the most commonly used data structures in Rust and right now searching that will pull up lots of info on Rust vectors (the data structure), not this project
dreyfiz 7y ago
Impressive! I've been using Fluentd a lot, will look at this. p.s. Search for "complaince" and fix the typo :-)
lemmox 7y ago
That's a nice list of sinks and I see the v0.5 Kafka source is already done a few days ago. 1) is there any talk of breaking changes on the road to 1.0? 2) are there any articles about lessons learned from the Timber folks about running Vector at scale?
- binarylogic 7y ago
  Hey Lemmox, good questions: 1) Not currently, but it's certainly possible. We don't foresee any major fundamental changes. The most likely change is around transforming. Specifically, how you shape both log and metrics data structures. Right now it's a little too piecemeal and we hope to provide ways to consolidate that more. For example, we have a transform for adding, removing, renaming, and coercing fields. We'll probably introduce a single transform to support that or think about ways to support schemas. 2) Yep! We're organizing our content and education strategy to start sharing this information. Expect to see high-quality guides, blog posts, and comparisons in the next 6-12 months. Hope that helps!
atonse 7y ago
Does timber use Elixir for their ingestion pipeline?
swsieber 7y ago
A nice discussion here 2 months ago: https://news.ycombinator.com/item?id=20334779 https://news.ycombinator.com/item?id=20334779
BubRoss 7y ago
Of all the things to name a project these people named it after the most commonly used data structure.
- TeMPOraL 7y ago
  Or one of the most common mathematical terms in programming.
jedisct1 7y ago
No comparison with Flowgger? https://github.com/awslabs/flowgger https://github.com/awslabs/flowgger
jeroenhd 7y ago
Perhaps unrelated, but I appreciate the honest performance comparison graphs on this website. Instead of just showcasing how much better this piece of software is, it also clearly shows that there are cases where its competitors are performing better. Moet product pages I've seen just want to sell you their product, telling you all about how it's better than everyone else. With graphs like these, I find the product page to be a lot more representative and reliable than other sales pages. I wish more software developers would do this.
- giancarlostoro 7y ago
  Seems to be really sweet in regards to memory which is one thing that kills me about Logstash / ELK in general. I may have to further evaluate this one as a sane alternative. The one nice thing that ELK has is Kibana though. Wonder how I can generate decent dashboards for this. Would also love to see a database option that is a little smarter. If ElasticSearch crashes or anything it becomes inaccessible for a minute. We lost log data due to instability in ElasticSearch.
  - zX41ZdbW 7y ago
    https://github.com/timberio/vector/pull/693 https://github.com/timberio/vector/pull/693
    - giancarlostoro 7y ago
      Nice, although ClickHouse seems to be a Yandex project. Not sure my boss would appreciate ClickHouse, wonder how does this compare to ElasticSearch though? I'm not looking for 8GB of RAM is not something I can sell my manager.
      - PeterZaitsev 7y ago
        Check out Altinity - this is US based Clickhouse vendor if your boss needs commercial support and less ties to Russia
rsrsrs86 7y ago
Terrible name choice...

145 points by weitzj 7y ago

Slightly unfortunate naming, as there is already a Vector project in the metrics space. http://getvector.io/ http://getvector.io/

hnarn 7y ago
It seems to be somewhat of a trend to prepend or somehow integrate the language used in naming schemes to ensure you have a somewhat unique name, something like "rvector" would at least make this a lot more searchable, because I suspect most people would search for something like "vector rust" with some additions anyway when looking for this.
- CDSlice 7y ago
  The problem with searching "vector rust" is that vectors are one of the most commonly used data structures in Rust and right now searching that will pull up lots of info on Rust vectors (the data structure), not this project

dreyfiz 7y ago

Impressive! I've been using Fluentd a lot, will look at this. p.s. Search for "complaince" and fix the typo :-)

lemmox 7y ago

That's a nice list of sinks and I see the v0.5 Kafka source is already done a few days ago. 1) is there any talk of breaking changes on the road to 1.0? 2) are there any articles about lessons learned from the Timber folks about running Vector at scale?

binarylogic 7y ago
Hey Lemmox, good questions: 1) Not currently, but it's certainly possible. We don't foresee any major fundamental changes. The most likely change is around transforming. Specifically, how you shape both log and metrics data structures. Right now it's a little too piecemeal and we hope to provide ways to consolidate that more. For example, we have a transform for adding, removing, renaming, and coercing fields. We'll probably introduce a single transform to support that or think about ways to support schemas. 2) Yep! We're organizing our content and education strategy to start sharing this information. Expect to see high-quality guides, blog posts, and comparisons in the next 6-12 months. Hope that helps!

atonse 7y ago

Does timber use Elixir for their ingestion pipeline?

swsieber 7y ago

A nice discussion here 2 months ago: https://news.ycombinator.com/item?id=20334779 https://news.ycombinator.com/item?id=20334779

BubRoss 7y ago

Of all the things to name a project these people named it after the most commonly used data structure.

TeMPOraL 7y ago
Or one of the most common mathematical terms in programming.

jedisct1 7y ago

No comparison with Flowgger? https://github.com/awslabs/flowgger https://github.com/awslabs/flowgger

jeroenhd 7y ago

Perhaps unrelated, but I appreciate the honest performance comparison graphs on this website. Instead of just showcasing how much better this piece of software is, it also clearly shows that there are cases where its competitors are performing better. Moet product pages I've seen just want to sell you their product, telling you all about how it's better than everyone else. With graphs like these, I find the product page to be a lot more representative and reliable than other sales pages. I wish more software developers would do this.

giancarlostoro 7y ago
Seems to be really sweet in regards to memory which is one thing that kills me about Logstash / ELK in general. I may have to further evaluate this one as a sane alternative. The one nice thing that ELK has is Kibana though. Wonder how I can generate decent dashboards for this. Would also love to see a database option that is a little smarter. If ElasticSearch crashes or anything it becomes inaccessible for a minute. We lost log data due to instability in ElasticSearch.
- zX41ZdbW 7y ago
  https://github.com/timberio/vector/pull/693 https://github.com/timberio/vector/pull/693
  - giancarlostoro 7y ago
    Nice, although ClickHouse seems to be a Yandex project. Not sure my boss would appreciate ClickHouse, wonder how does this compare to ElasticSearch though? I'm not looking for 8GB of RAM is not something I can sell my manager.
    - PeterZaitsev 7y ago
      Check out Altinity - this is US based Clickhouse vendor if your boss needs commercial support and less ties to Russia

rsrsrs86 7y ago

Terrible name choice...