4 ms·

This looks very interesting! My suggestion for the self-hosting is to create docker images and use docker-compose. The self-hosting currently is a bit of effor

This looks very interesting! My suggestion for the self-hosting is to create docker images and use docker-compose. The self-hosting currently is a bit of effort to setup. I also wonder if PHP is a good language for this. For the UI, yea that's fine and makes sense. But for the log processor that's going to need to handle a high throughput which PHP just isn't good at. For the same resources, you can have Go doing thousands of requests per second vs PHP doing hundreds of requests per second.

herbst 2y ago
On the other side some people (me) are happy to have an actual self hosting setup and not being forced to use a docker setup with unknown overhead.
- xinu2020 2y ago
  Why not both? It's not much trouble to publish a Dockerfile while still documenting a normal installation.
  - herbst 2y ago
    It's not, but more often than not it's just a dockerfile
majkinetor 2y ago
No benefit using go over C#, IMO, and I am also baffled by the switch
- that_guy_iain 2y ago
  I just used go as an example, any compiled language would be better.
  - withinboredom 2y ago
    I highly suspect it wouldn't be better in brainfuck...
majkinetor 2y ago
It uses Clickhouse, though, which should be xtremelly fast for this.
- that_guy_iain 2y ago
  Yes. But PHP still needs to process it before it goes to Clickhouse. PHP is the bottleneck.
  - wiseowise 2y ago
    [flagged]
  - axelthegerman 2y ago
    If that "bottleneck" is thousands of requests per second then it doesn't really matter for smaller deployments does it? (Which seems to be the target audience and not FAANG) I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language.
    - that_guy_iain 2y ago
      To tweak a PHP deployment to handle hundreds of requests per second which is very very realistic for a basic logging for a mid-sized application you're looking at having a very beefy server setup. Most PHP deployments barely reach a hundred per server. And this is an open source project is should be designed to handle basic production workloads which it could but it'll cost you a bunch more than if you used the correct languages. > I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language. Honestly, I thought it was so obvious that an interpreted language is not good for high throughput endpoints that it didn't need to be proven. I also thought it was obvious that a logging system is going to handle lots and lots of data. It could be easily proven by doing a bunch of work but obviously there is no point in me proving it.
      - 2y ago
withinboredom 2y ago
PHP is arguably the best solution here. If a log ingestion process breaks everything, no other logs are harmed (a default shared-nothing architecture). Using something like Go, C#, etc, it might be "faster" but less resilient -- or more complex to handle the resiliency. > But for the log processor that's going to need to handle a high throughput which PHP just isn't good at. I'm sorry, but wut? PHP is probably one of the fastest languages out there if you can ignore frameworks. It's backed by some of the most tuned C code out there and should be just about as fast as C for most tasks. The only reason it is not is due to the function call overhead -- which is by-far the slowest aspect of PHP. > you can have Go doing thousands of requests per second vs PHP doing hundreds of requests per second. This is mostly due to nginx and friends ... There is frankenphp (a frontend for php running in caddy which is written in go) which can easily handle 80k+ requests per second.
- that_guy_iain 2y ago
  I'm going to have to also reply with, sorry but what?! PHP is one of the fastest-interpreted languages. But compiled are going to be faster than interpreted pretty much everytime. It loses benchmarks against every language. That's not to mention it's slowed down by the fact it have to rebuild everything per request. As a PHP developer for 15+ years, I can tell you what PHP is good at and what PHP is not good at. High throughput API endpoints such as log ingestion are not a good fit for PHP. Your argument that if it breaks it's fine. Yea, who wants a log system that will only log some of your logs? No one. It's not mission critical but it's pretty important to keep working if you want to keep your system working. And in fact, some places it is a legal requirement.
  - withinboredom 2y ago
    > It loses benchmarks against every language. Every language loses benchmarks against every other language. That's not surprising. Since you didn't provide a specific benchmark, it's hard to say why it lost. > High throughput API endpoints such as log ingestion are not a good fit for PHP. I disagree; but ultimately, it depends on how you're doing it. You can beat or exceed compiled languages in some cases. PHP allows some low-level stuff directly implemented in C and also the high-level stuff you're used to in interpreted languages.
williebeek 2y ago
Thanks for the tip, I will check if inserting rows with Go is any faster. For reference, inserting a log takes three steps, first the log data is stored in a Redis Stream (memory), a number of logs are taken from the stream and saved to disk and finally inserted in batches in ClickHouse. I've created it so you can take the ClickHouse server offline without losing any data (it will be inserted later). For reference, moving about 4k logs from memory to disk takes less than 0.1 second. This is a real log from one of the webservers: Start new cron loop: 2024-12-18 08:11:16.397...stored 3818 rows in /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc (0.0652 seconds). Storing this data in ClickHouse takes a bit more than 0.1 second: Start new cron loop: 2024-12-18 08:11:17.124...parsing file /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc * Inserting 3818 row(s) on database server 1...0.137 seconds (approx. 3021.15 KB). * Removed /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc As for Docker, I'm too much of a Docker noob but I appreciate the suggestion.
ryanianian 2y ago
PHP trivially scales up to multiple nodes behind an LB. You're really only limited by your backend storage connection count and throughput. Go and friends may make for more efficient resource utilization, but it will be marginal in the grand scheme of things unless there are plans to do massively different things. As it is this code is very simple. I haven't used PHP in 15 years and I was able to trace through this from front-end to back-end in less than 3 minutes. To me it look like a really great level of complexity for the problem it solves. Keep it up, OP.
- that_guy_iain 2y ago
  You can but that costs more money... > Keep it up, OP. Live in the real world. No one wants to have a fleet of servers for their logging infra when there are options to run it on a single server.
hipadev23 2y ago
> PHP doing hundreds of requests per second. You may want to update your understanding of PHP and Go's speed . Both of your estimates are off by a couple orders of magnitude on commodity hardware. There are also numerous ways to make PHP extremely fast today (e.g. swoole, ngx_php, or frankenphp) instead of the 1999 best practice of apache with mod_php. Go is absolutely an excellent choice, but your opinion on PHP is quite dated. Here are benchmarks for numerous Go (green) and PHP (blue) web frameworks: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22&l=zijnjz-cn3 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...
- p_ing 2y ago
  As soon as you add C#, ASP.NET Core shoots to the top of the Fortune stack.
  - deleted 2y ago
    [deleted]
- that_guy_iain 2y ago
  What you're talking about is generally not considered production-ready. While you can use these tools you will almost certainly run into problems. I know this because as an active PHP developer for over a decade I'm very much paying attention to that field of PHP. What we see here is a classic case of benchmarks saying one thing when the reality of production code says something else. Also, I used go as a generic example of compiled languages. But what we see is production-grade Go languages outperforming non-production-ready experimental PHP tooling. And if we go to look at all of them https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s... We'll see that even the experimental PHP solution is 43 and being beat out by compiled languages.
  - hipadev23 2y ago
    Nobody is suggesting PHP beats compiled. We’re arguing with you about your utter lack of expertise in the language, knowledge of the ecosystem and “production-ready” status of the many options, and your overall coding ability when it comes to PHP.

herbst 2y ago

On the other side some people (me) are happy to have an actual self hosting setup and not being forced to use a docker setup with unknown overhead.

xinu2020 2y ago
Why not both? It's not much trouble to publish a Dockerfile while still documenting a normal installation.
- herbst 2y ago
  It's not, but more often than not it's just a dockerfile

majkinetor 2y ago

No benefit using go over C#, IMO, and I am also baffled by the switch

that_guy_iain 2y ago
I just used go as an example, any compiled language would be better.
- withinboredom 2y ago
  I highly suspect it wouldn't be better in brainfuck...

majkinetor 2y ago

It uses Clickhouse, though, which should be xtremelly fast for this.

that_guy_iain 2y ago
Yes. But PHP still needs to process it before it goes to Clickhouse. PHP is the bottleneck.
- wiseowise 2y ago
  [flagged]
- axelthegerman 2y ago
  If that "bottleneck" is thousands of requests per second then it doesn't really matter for smaller deployments does it? (Which seems to be the target audience and not FAANG) I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language.
  - that_guy_iain 2y ago
    To tweak a PHP deployment to handle hundreds of requests per second which is very very realistic for a basic logging for a mid-sized application you're looking at having a very beefy server setup. Most PHP deployments barely reach a hundred per server. And this is an open source project is should be designed to handle basic production workloads which it could but it'll cost you a bunch more than if you used the correct languages. > I'm not a big fan when folks call out languages as bottlenecks when they have no proof on the actual overhead and how much faster it would be in another language. Honestly, I thought it was so obvious that an interpreted language is not good for high throughput endpoints that it didn't need to be proven. I also thought it was obvious that a logging system is going to handle lots and lots of data. It could be easily proven by doing a bunch of work but obviously there is no point in me proving it.
    - 2y ago

withinboredom 2y ago

PHP is arguably the best solution here. If a log ingestion process breaks everything, no other logs are harmed (a default shared-nothing architecture). Using something like Go, C#, etc, it might be "faster" but less resilient -- or more complex to handle the resiliency. > But for the log processor that's going to need to handle a high throughput which PHP just isn't good at. I'm sorry, but wut? PHP is probably one of the fastest languages out there if you can ignore frameworks. It's backed by some of the most tuned C code out there and should be just about as fast as C for most tasks. The only reason it is not is due to the function call overhead -- which is by-far the slowest aspect of PHP. > you can have Go doing thousands of requests per second vs PHP doing hundreds of requests per second. This is mostly due to nginx and friends ... There is frankenphp (a frontend for php running in caddy which is written in go) which can easily handle 80k+ requests per second.

that_guy_iain 2y ago
I'm going to have to also reply with, sorry but what?! PHP is one of the fastest-interpreted languages. But compiled are going to be faster than interpreted pretty much everytime. It loses benchmarks against every language. That's not to mention it's slowed down by the fact it have to rebuild everything per request. As a PHP developer for 15+ years, I can tell you what PHP is good at and what PHP is not good at. High throughput API endpoints such as log ingestion are not a good fit for PHP. Your argument that if it breaks it's fine. Yea, who wants a log system that will only log some of your logs? No one. It's not mission critical but it's pretty important to keep working if you want to keep your system working. And in fact, some places it is a legal requirement.
- withinboredom 2y ago
  > It loses benchmarks against every language. Every language loses benchmarks against every other language. That's not surprising. Since you didn't provide a specific benchmark, it's hard to say why it lost. > High throughput API endpoints such as log ingestion are not a good fit for PHP. I disagree; but ultimately, it depends on how you're doing it. You can beat or exceed compiled languages in some cases. PHP allows some low-level stuff directly implemented in C and also the high-level stuff you're used to in interpreted languages.

williebeek 2y ago

Thanks for the tip, I will check if inserting rows with Go is any faster. For reference, inserting a log takes three steps, first the log data is stored in a Redis Stream (memory), a number of logs are taken from the stream and saved to disk and finally inserted in batches in ClickHouse. I've created it so you can take the ClickHouse server offline without losing any data (it will be inserted later). For reference, moving about 4k logs from memory to disk takes less than 0.1 second. This is a real log from one of the webservers: Start new cron loop: 2024-12-18 08:11:16.397...stored 3818 rows in /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc (0.0652 seconds). Storing this data in ClickHouse takes a bit more than 0.1 second: Start new cron loop: 2024-12-18 08:11:17.124...parsing file /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc * Inserting 3818 row(s) on database server 1...0.137 seconds (approx. 3021.15 KB). * Removed /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc As for Docker, I'm too much of a Docker noob but I appreciate the suggestion.

ryanianian 2y ago

PHP trivially scales up to multiple nodes behind an LB. You're really only limited by your backend storage connection count and throughput. Go and friends may make for more efficient resource utilization, but it will be marginal in the grand scheme of things unless there are plans to do massively different things. As it is this code is very simple. I haven't used PHP in 15 years and I was able to trace through this from front-end to back-end in less than 3 minutes. To me it look like a really great level of complexity for the problem it solves. Keep it up, OP.

that_guy_iain 2y ago
You can but that costs more money... > Keep it up, OP. Live in the real world. No one wants to have a fleet of servers for their logging infra when there are options to run it on a single server.

hipadev23 2y ago

> PHP doing hundreds of requests per second. You may want to update your understanding of PHP and Go's speed . Both of your estimates are off by a couple orders of magnitude on commodity hardware. There are also numerous ways to make PHP extremely fast today (e.g. swoole, ngx_php, or frankenphp) instead of the 1999 best practice of apache with mod_php. Go is absolutely an excellent choice, but your opinion on PHP is quite dated. Here are benchmarks for numerous Go (green) and PHP (blue) web frameworks: https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22&l=zijnjz-cn3 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...

p_ing 2y ago
As soon as you add C#, ASP.NET Core shoots to the top of the Fortune stack.
- deleted 2y ago
  [deleted]
that_guy_iain 2y ago
What you're talking about is generally not considered production-ready. While you can use these tools you will almost certainly run into problems. I know this because as an active PHP developer for over a decade I'm very much paying attention to that field of PHP. What we see here is a classic case of benchmarks saying one thing when the reality of production code says something else. Also, I used go as a generic example of compiled languages. But what we see is production-grade Go languages outperforming non-production-ready experimental PHP tooling. And if we go to look at all of them https://www.techempower.com/benchmarks/#hw=ph&test=fortune&section=data-r22 https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s... We'll see that even the experimental PHP solution is 43 and being beat out by compiled languages.
- hipadev23 2y ago
  Nobody is suggesting PHP beats compiled. We’re arguing with you about your utter lack of expertise in the language, knowledge of the ecosystem and “production-ready” status of the many options, and your overall coding ability when it comes to PHP.