3 ms·

The absolute bare minimum I'd say is 2GB RAM, though in the README we do say 4GB and 2 cores for testing, obviously more if you're at scale and need performance

The absolute bare minimum I'd say is 2GB RAM, though in the README we do say 4GB and 2 cores for testing, obviously more if you're at scale and need performance. For Syslog - it's something we're actually pretty close to because we already support Heroku's syslog based messages (though it's over HTTP), but largely need to test the otel Syslog receiver + parsing pipeline will translate as well as it should (PRs always welcome of course but it shouldn't be too far out from now ourselves :)). I'm curious are you using TLS/TCP syslog or plain TCP or UDP? Here's my docker stats on a x64 linux VM where it's doing some minimal self-logging, I suspect the otel collector memory can be tuned down to bring the memory usage closer to 1GB, but this is the default out-of-the-box stats, and the miner can be turned off if log patterns isn't needed: CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 439e3f426ca6 hdx-oss-miner 0.89% 167.2MiB / 7.771GiB 2.10% 3.25MB / 6.06MB 8.85MB / 0B 21 7dae9d72913d hdx-oss-task-check-alerts 0.03% 83.65MiB / 7.771GiB 1.05% 6.79MB / 9.54MB 147kB / 0B 11 5abd59211cd7 hdx-oss-app 0.00% 56.32MiB / 7.771GiB 0.71% 467kB / 551kB 6.23MB / 0B 11 90c0ef1634c7 hdx-oss-api 0.02% 93.71MiB / 7.771GiB 1.18% 13.2MB / 7.87MB 57.3kB / 0B 11 39737209c58f hdx-oss-hostmetrics 0.03% 72.27MiB / 7.771GiB 0.91% 3.83GB / 173MB 3.84MB / 0B 11 e13c9416c06e hdx-oss-ingestor 0.04% 23.11MiB / 7.771GiB 0.29% 73.2MB / 89.4MB 77.8kB / 0B 5 36d57eaac8b2 hdx-oss-otel-collector 0.33% 880MiB / 7.771GiB 11.06% 104MB / 68.9MB 1.24MB / 0B 11 78ac89d8e28d hdx-oss-aggregator 0.07% 88.08MiB / 7.771GiB 1.11% 141MB / 223MB 147kB / 0B 11 8a2de809efed hdx-oss-redis 0.19% 3.738MiB / 7.771GiB 0.05% 4.36MB / 76.5MB 8.19kB / 4.1kB 5 2f2eac07bedf hdx-oss-db 1.34% 75.62MiB / 7.771GiB 0.95% 105MB / 3.79GB 1.32MB / 246MB 56 032ae2b50b2f hdx-oss-ch-server 0.54% 128.7MiB / 7.771GiB 1.62% 194MB / 45MB 88.4MB / 65.5kB 316

nodesocket 3y ago
Thanks for the reply and providing detailed system requirements and docker stats. Seems I missed the note in the README. :-) Actually I am not really using syslog per say, but systemd journalctl which default behaviour on Debian (rsyslog) also duplicates to /var/log/syslog. StandardOutput=journal StandardError=journal Is there a better integration to pull logs from my systemd services and journalctl up to HyperDX?
- mikeshi42 3y ago
  Ah yeah the easiest way is probably using the OpenTelemetry collector to set up a process to pull your logs out of jounrnald and send them via otel logs to HyperDX (or anywhere else that speaks otel) - the docs might be a bit tricky to go around depending on your familiarity with OpenTelemetry but this is what you'd be looking for: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver#example-configurations https://github.com/open-telemetry/opentelemetry-collector-co... Happy to dive more into the discord too if you'd like!

nodesocket 3y ago

Thanks for the reply and providing detailed system requirements and docker stats. Seems I missed the note in the README. :-) Actually I am not really using syslog per say, but systemd journalctl which default behaviour on Debian (rsyslog) also duplicates to /var/log/syslog. StandardOutput=journal StandardError=journal Is there a better integration to pull logs from my systemd services and journalctl up to HyperDX?

mikeshi42 3y ago
Ah yeah the easiest way is probably using the OpenTelemetry collector to set up a process to pull your logs out of jounrnald and send them via otel logs to HyperDX (or anywhere else that speaks otel) - the docs might be a bit tricky to go around depending on your familiarity with OpenTelemetry but this is what you'd be looking for: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/journaldreceiver#example-configurations https://github.com/open-telemetry/opentelemetry-collector-co... Happy to dive more into the discord too if you'd like!