4 ms·

For the last year we have been working in a lightweight log shipper product called Fluent Bit[0]. Originally made for Embedded Linux now is taking it place in c

For the last year we have been working in a lightweight log shipper product called Fluent Bit[0]. Originally made for Embedded Linux now is taking it place in common environments. It's pretty similar to Fluentd in architecture, some features are: - Event-Driven (async network I/O). - Input / Output plugins. - Data routing based on Tags. - Optional SSL/TLS for networking operations when required. Next major version 0.9 will come with buffering support (memory/file system). Ah, it's fully made in C. [0] http://fluentbit.io http://fluentbit.io http://fluentbit.io/documentation/0.8 http://fluentbit.io/documentation/0.8 http://github.com/fluent/fluent-bit http://github.com/fluent/fluent-bit

forgotpwtomain 10y ago
I couldn't find a link to your git repo on the website btw (repo (https://github.com/fluent/fluent-bit https://github.com/fluent/fluent-bit) ). I see that your input/output plugins are written in C[0]. I'm guessing this is because of the constraints of the embedded environment, but it really doesn't seem like it would be worth it in a normal one. The LUA sandbox model (e.g. Heka) just seems highly preferable. My main problem with Logstash/Fluentd is precisely the fragility and non-robustness of the plugin system. [0] https://github.com/fluent/fluent-bit/blob/master/plugins/out_es/es.c https://github.com/fluent/fluent-bit/blob/master/plugins/out...
- edsiper2 10y ago
  The whole project is in C, there is a planed Lua support for the future versions that can help to filter/modify records, more news about it in the incoming weeks ;) The decision about "why C" is: flexibility, performance and adaptability (note that it was originally designed for Embedded Linux targets, but now going everywhere). In order to make things easier for output plugins, every time a set of records needs to be flushed through some output plugin, a co-routine is created so any plugin can yield/resume at any time. For example out_http, out_es and out_forward relies on network I/O, having an event loop and a coroutine associated allows to simplify the plugin development and state management: connect, write, read, etc. This model is the foundation and allow the next step to integrate scripting more smoothly. For environments without co-routines support (old compilers), a POSIX thread model exists. What are the specific "fragility"/concerns you see in Fluentd plugin model?

forgotpwtomain 10y ago

I couldn't find a link to your git repo on the website btw (repo (https://github.com/fluent/fluent-bit https://github.com/fluent/fluent-bit) ). I see that your input/output plugins are written in C[0]. I'm guessing this is because of the constraints of the embedded environment, but it really doesn't seem like it would be worth it in a normal one. The LUA sandbox model (e.g. Heka) just seems highly preferable. My main problem with Logstash/Fluentd is precisely the fragility and non-robustness of the plugin system. [0] https://github.com/fluent/fluent-bit/blob/master/plugins/out_es/es.c https://github.com/fluent/fluent-bit/blob/master/plugins/out...

edsiper2 10y ago
The whole project is in C, there is a planed Lua support for the future versions that can help to filter/modify records, more news about it in the incoming weeks ;) The decision about "why C" is: flexibility, performance and adaptability (note that it was originally designed for Embedded Linux targets, but now going everywhere). In order to make things easier for output plugins, every time a set of records needs to be flushed through some output plugin, a co-routine is created so any plugin can yield/resume at any time. For example out_http, out_es and out_forward relies on network I/O, having an event loop and a coroutine associated allows to simplify the plugin development and state management: connect, write, read, etc. This model is the foundation and allow the next step to integrate scripting more smoothly. For environments without co-routines support (old compilers), a POSIX thread model exists. What are the specific "fragility"/concerns you see in Fluentd plugin model?