4 ms·

Does Fluentd handle storage and query?

• dozzie 10y ago
  Of course not, it's a message router, not a storage engine. ... Actually, it has some basic capability to store everything in a set of files stamped with time (and optionally gzipped), but this hardly counts.
  • sagichmal 10y ago
    It seems to me that you should be able to use your Fluentd pipeline in place of the forwarder component, I don't think there's any magic going on there. The engineering here seems to be in the ingest, replication, storage, and query side of things, none of which Fluentd addresses, really.
    • dozzie 10y ago
      The problem is that instead of submitting logs using a plugin (written in Ruby) I need to (a) spawn a separate process using (b) unrelated binary. I should have the possibility to not jump out to a separate process and I shouldn't need to ship binary to my log routers (a plugin in Ruby or Python is just a text, so it's easier to distribute with configuration management). The same thing stands for querying. Now the only possible (documented) way is to call the binary. As I said, this is a problem with lack of protocol specifications, not with the architecture.