26 ms·

How to run Valgrind on a CGI program in C

32 points by ingve 2y ago

amelius 2y ago
I want to know how to run Valgrind on a Python program that uses lots of different modules. The amount of noise I usually get is so large that Valgrind becomes useless.
- carlmr 2y ago
  https://stackoverflow.com/a/45248015 https://stackoverflow.com/a/45248015
  - amelius 2y ago
    Thanks, but I think I looked into that when I ran into the problem. Iirc, the main problem was that most Python modules are not designed with Valgrind in mind (so they generate a lot of noisy/irrelevant warnings and errors) I wish module maintainers would make running Valgrind a part of their QC process.
    - carlmr 2y ago
      Understandable. I don't think there's a good way around this though. Even if you run valgrind on a C++ repo that hasn't seen much valgrind use, you'll get a lot of output. But that output is because the memory management is usually broken in every larger C++ repo which isn't developed by a small team of experts.
- wallstprog 2y ago
  You might want to take a look at https://github.com/nyfix/memleaktest https://github.com/nyfix/memleaktest We've open-sourced the tools we use to run valgrind (and ASAN) on large mixed C++/Java code bases. The JVM in particular triggers a slew of errors which can make filtering valgrind output impractical, but the scripts we developed can handle that. FWIW, we use these tools every day on the code that goes into NYFIX Marketplace (https://www.broadridge.com/financial-services/capital-markets/trading-and-connectivity/order-routing-network https://www.broadridge.com/financial-services/capital-market...).
  - amelius 2y ago
    That's very interesting, thanks!
jijji 2y ago
why not run valgrind on the web server and wait until it crashes on your POST request and then inspect the logs
- guappa 2y ago
  A decent server wouldn't crash if a cgi binary crashes.
- spc476 2y ago
  One, CGI scripts are run as a separate process, so running valgrind on the web server won't get status about the CGI program. Two, it would probably generate a ton of output not related to the CGI program even if it did work. Three, the CGI program wasn't crashing, it was to check for memory errors.
D4ckard 2y ago
I don’t get why people use (or used) CGI programs for web servers. This seems crazy inefficient and prone to break
- F3nd0 2y ago
  What should they have been using instead?
- jagged-chisel 2y ago
  You don’t get, when web servers were a new thing, why anyone continued using the same kind of executables they always had to handle workloads that required executable code. Probably because there was not yet any data AT ALL about demand from the network, the patterns that random visitors could cause, etc. “I don’t get why the Romans didn’t build suspension bridges with concrete, steel, and cables.”
- marcus0x62 2y ago
  I have no idea why someone would use one now. In the mid-90’s it was pretty much the only practical cross-language way to create dynamic web content. Someone certainly could have written their CGI program as a separate web server, but: 1. proxy support was limited or non-existent in the main webservers that existed at the time, so routing the request over would have been a problem. 2. there were no frameworks to speak of to make any of this easier. CGI.pm would help parse and decode GET and POST data. That was just about all there was. There was no PHP (as it exists now,) no Rails, no http frameworks like actix-web. Nothing. “Inefficient and prone to break” is an excellent description of CGI. Most practical people moved away from as quickly as possible.
- PhilipRoman 2y ago
  It's actually pretty fast if written in a lean language like C/rust. See "cgit" for a good example. And if you want to distribute a program which talks HTTP, it's kind of the obvious choice - the alternative is each program packing it's own HTTP server, which needs a reverse proxy/SSL in front of it anyway.
  - 2y ago
zokier 2y ago
to me it would have been simpler to wrap the program in a small shellscript or something, e.g. #!/bin/sh exec valgrind --log-file=/tmp/valgrind.%p.log /path/to/my/app and have the web server execute that instead of the original executable
- mananaysiempre 2y ago
  Being able to repeat the experiment at your leisure without digging through web server logs is valuable. Still, a shell script does indeed sound like the way to go: `set` should already dump the environment as required here, and then you can exec your CGI program. (Unfortunately, it doesn’t in bash; `set -o posix` first to fix that.)
deleted 2y ago
[deleted]

32 points by ingve 2y ago

I want to know how to run Valgrind on a Python program that uses lots of different modules. The amount of noise I usually get is so large that Valgrind becomes useless.

carlmr 2y ago
https://stackoverflow.com/a/45248015 https://stackoverflow.com/a/45248015
- amelius 2y ago
  Thanks, but I think I looked into that when I ran into the problem. Iirc, the main problem was that most Python modules are not designed with Valgrind in mind (so they generate a lot of noisy/irrelevant warnings and errors) I wish module maintainers would make running Valgrind a part of their QC process.
  - carlmr 2y ago
    Understandable. I don't think there's a good way around this though. Even if you run valgrind on a C++ repo that hasn't seen much valgrind use, you'll get a lot of output. But that output is because the memory management is usually broken in every larger C++ repo which isn't developed by a small team of experts.
wallstprog 2y ago
You might want to take a look at https://github.com/nyfix/memleaktest https://github.com/nyfix/memleaktest We've open-sourced the tools we use to run valgrind (and ASAN) on large mixed C++/Java code bases. The JVM in particular triggers a slew of errors which can make filtering valgrind output impractical, but the scripts we developed can handle that. FWIW, we use these tools every day on the code that goes into NYFIX Marketplace (https://www.broadridge.com/financial-services/capital-markets/trading-and-connectivity/order-routing-network https://www.broadridge.com/financial-services/capital-market...).
- amelius 2y ago
  That's very interesting, thanks!

jijji 2y ago

why not run valgrind on the web server and wait until it crashes on your POST request and then inspect the logs

guappa 2y ago
A decent server wouldn't crash if a cgi binary crashes.
spc476 2y ago
One, CGI scripts are run as a separate process, so running valgrind on the web server won't get status about the CGI program. Two, it would probably generate a ton of output not related to the CGI program even if it did work. Three, the CGI program wasn't crashing, it was to check for memory errors.

D4ckard 2y ago

I don’t get why people use (or used) CGI programs for web servers. This seems crazy inefficient and prone to break

F3nd0 2y ago
What should they have been using instead?
jagged-chisel 2y ago
You don’t get, when web servers were a new thing, why anyone continued using the same kind of executables they always had to handle workloads that required executable code. Probably because there was not yet any data AT ALL about demand from the network, the patterns that random visitors could cause, etc. “I don’t get why the Romans didn’t build suspension bridges with concrete, steel, and cables.”
marcus0x62 2y ago
I have no idea why someone would use one now. In the mid-90’s it was pretty much the only practical cross-language way to create dynamic web content. Someone certainly could have written their CGI program as a separate web server, but: 1. proxy support was limited or non-existent in the main webservers that existed at the time, so routing the request over would have been a problem. 2. there were no frameworks to speak of to make any of this easier. CGI.pm would help parse and decode GET and POST data. That was just about all there was. There was no PHP (as it exists now,) no Rails, no http frameworks like actix-web. Nothing. “Inefficient and prone to break” is an excellent description of CGI. Most practical people moved away from as quickly as possible.
PhilipRoman 2y ago
It's actually pretty fast if written in a lean language like C/rust. See "cgit" for a good example. And if you want to distribute a program which talks HTTP, it's kind of the obvious choice - the alternative is each program packing it's own HTTP server, which needs a reverse proxy/SSL in front of it anyway.
- 2y ago

zokier 2y ago

to me it would have been simpler to wrap the program in a small shellscript or something, e.g. #!/bin/sh exec valgrind --log-file=/tmp/valgrind.%p.log /path/to/my/app and have the web server execute that instead of the original executable

mananaysiempre 2y ago
Being able to repeat the experiment at your leisure without digging through web server logs is valuable. Still, a shell script does indeed sound like the way to go: `set` should already dump the environment as required here, and then you can exec your CGI program. (Unfortunately, it doesn’t in bash; `set -o posix` first to fix that.)

deleted 2y ago

[deleted]