6 ms·

With this tool I am wary that I'll encounter system issues that are dramatically more difficult to diagnose and troubleshoot because I'll have drifted from a st

With this tool I am wary that I'll encounter system issues that are dramatically more difficult to diagnose and troubleshoot because I'll have drifted from a standard distro configuration. And in ways I'm unaware of. Is this a reasonable hesitation?

pbhjpbhj 2y ago
>"bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out." (from OP GitHub readme) The rmem example seems to allay fears that it will make changes one can't reverse.
- admax88qqq 2y ago
  It’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.
  - pbhjpbhj 2y ago
    I focused primarily on guesswho's "in ways I am unaware of". Your issue appears to be true for any system change. Although, risk will of course vary.
  - nehal3m 2y ago
    If they can be reversed individually you can simply deduce by rolling back changes one by one, no?
    - jstanley 2y ago
      Only if you already suspect that this tool caused the problem.
    - spenczar5 2y ago
      Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between too high and too low. Can you see how this is not a matter of simple deduction and rollbacks? This scenario is plausible. Autotuning generally has issues with feedback, since the overall system lacks control theoretic structure. And the premise here is that you use this to tune a large number of machines where individual admin is infeasible.
      - pbhjpbhj 2y ago
        >not only can we observe the system and tune appropriately, we can also observe the effect of that tuning and re-tune if necessary. // Does sound like a potential way to implement literal chaos. Surely it's like anything else, you do pre-release testing and balance the benefits for you against the risks?
      - Modified3019 2y ago
        Sounds like you have your answer of “don’t use it” then.
      - pstuart 2y ago
        In that scenario you could run it on a couple servers, compare and contrast, and then apply globally via whatever management tool you use.
      - KennyBlanken 2y ago
        Presumably one would use autotune to find optimized parameters, and then roll those out via change control, either one parameter at a time, or a mix of parameters across the systems. Alternatively: if you have a fleet of thousands of machines you can very easily do a binary search with them to a)establish the problem with the auto-tuner and then b)which of the changes it settled on are causing your problems. I get the impression you've never actually managed a "fleet" of systems, because these techniques would have immediately occurred to you.
        spenczar5 2y ago
        Certainly when we managed Twitch’s ~10,000 boxes of video servers, neither of the tasks you describe would have been simple. We underinvested in tools, for sure. Even so, I don’t think you can really argue that dynamically changing configs like this are going to make life easier!
      - toast0 2y ago
        When you have a thousand machines, you can usually get feedback pretty quick, in my experience. Run the tune on one machine. Looks good? Put it on ten. Looks good? Put it on one hundred. Looks good? Put it on everyone. Find an issue a week later, and want to dig into it? Run 100 machines back on the old tune, and 100 machines with half the difference. See what happens.
  - yourapostasy 2y ago
    Record changes in git and then git bisect issues, maybe? Without change capture, solid regression testing, or observability, it seems difficult to manage these changes. I’d like to how others are managing these kinds of changes to readily troubleshoot them, without lots of regression testing or observability, if anyone has successes to share.
sgarland 2y ago
Yes, it is. IMO, except for learning (which should not be done in prod), you shouldn’t make changes that you don’t understand. The tools seems to mostly tweak various networking settings. You could set up a test instance with monitoring, throw load at it, and change the parameters the tool modifies (one at a time!) to see how it reacts.
- nine_k 2y ago
  I'd run such a tool on prod in "advice mode". It should suggest the tweaks, explaining the reasoning behind them, and listing the actions necessary to implement them. Then humans would decide if they want to implement that as is, partly, modified, or not at all.
  - sgarland 2y ago
    Fair point, though I didn’t see any such option with this tool.
    - nine_k 2y ago
      It's developed in the open; we can create Github issue. Actually https://github.com/oracle/bpftune/issues/99 https://github.com/oracle/bpftune/issues/99
      - westurner 2y ago
        In the existing issue, we can link to the code and docs that would need to be understood and changed: usage, main() https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d889e3a60de6c12feb76/src/bpftune.c#L280 https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d... - [ ] CLI opts: --pretend-allow <tuner> or --log-only-allow <tuner> or [...] Probably relevant function headers in libbpftune.c: bpftune_sysctl_write( bpftuner_tunable_sysctl_write( bpftune_module_load( static void bpftuner_scenario_log(struct bpftuner *tuner, unsigned int tunable, ; https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d889e3a60de6c12feb76/src/libbpftune.c#L716 https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d... https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d889e3a60de6c12feb76/src/libbpftune.c#L1047 https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d...
deleted 2y ago
[deleted]
trelliscoded 2y ago
If your staging doesn’t do capacity checks in excess of what production sees, yes.
Twirrim 2y ago
Disclaimer: I work for Oracle, who publish this tool, though I have nothing to do with the org or engineers that created it I've been running this for a while on my laptop. So far yet to see any particular weirdness, but also I don't know that I can state with any confidence it has a positive impact either. I've not carried out any benchmarks in either direction. It logs all changes that it's going to make including what they were on before. Here's an example from my logs: bpftune[1852994]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput bpftune[1852994]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 7864320) -> (4096 131072 9830400)
gerdesj 2y ago
"because I'll have drifted from a standard distro configuration" You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. This tool logs all the changes it makes via the standard logging system, which can be easily captured, shipped and aggregated and then queried and reported on. This is not a tool from Clown Cars R US, it's from a reasonably reputable source - Oracle (lol etc). Even better, you can read the code and learn or critique. Not being funny but I'd rather this sort of thing by far than any amount of wooo handwavy wankery. Would you prefer openly described and documented or "take our word for it"?
- cortesoft 2y ago
  > You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. Which is now a list you will have to check for every issue. I don't think they are complaining they don't trust the writers of the code, just that it adds confounding variables to your system
  - gerdesj 2y ago
    We (in IT security) are expected to abrogate responsibility to funky AI or whatevs anti virus and other stuff. Buy and install a security package from ... whoever ... and all will be well. This is an expert system/advice run by real people (at a reasonably well respected firm) not an AI wankery thingie. It is literally expert advice and it is being given away and in code form which you can read. What on earth is wrong with that?
    - cortesoft 2y ago
      If the alternative is those proprietary anti virus products, sure this is better. The original comment was comparing to doing nothing and just using the standard distro, I believe.
      - 2y ago

pbhjpbhj 2y ago

>"bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out." (from OP GitHub readme) The rmem example seems to allay fears that it will make changes one can't reverse.

admax88qqq 2y ago
It’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.
- pbhjpbhj 2y ago
  I focused primarily on guesswho's "in ways I am unaware of". Your issue appears to be true for any system change. Although, risk will of course vary.
- nehal3m 2y ago
  If they can be reversed individually you can simply deduce by rolling back changes one by one, no?
  - jstanley 2y ago
    Only if you already suspect that this tool caused the problem.
  - spenczar5 2y ago
    Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between too high and too low. Can you see how this is not a matter of simple deduction and rollbacks? This scenario is plausible. Autotuning generally has issues with feedback, since the overall system lacks control theoretic structure. And the premise here is that you use this to tune a large number of machines where individual admin is infeasible.
    - pbhjpbhj 2y ago
      >not only can we observe the system and tune appropriately, we can also observe the effect of that tuning and re-tune if necessary. // Does sound like a potential way to implement literal chaos. Surely it's like anything else, you do pre-release testing and balance the benefits for you against the risks?
    - Modified3019 2y ago
      Sounds like you have your answer of “don’t use it” then.
    - pstuart 2y ago
      In that scenario you could run it on a couple servers, compare and contrast, and then apply globally via whatever management tool you use.
    - KennyBlanken 2y ago
      Presumably one would use autotune to find optimized parameters, and then roll those out via change control, either one parameter at a time, or a mix of parameters across the systems. Alternatively: if you have a fleet of thousands of machines you can very easily do a binary search with them to a)establish the problem with the auto-tuner and then b)which of the changes it settled on are causing your problems. I get the impression you've never actually managed a "fleet" of systems, because these techniques would have immediately occurred to you.
      - spenczar5 2y ago
        Certainly when we managed Twitch’s ~10,000 boxes of video servers, neither of the tasks you describe would have been simple. We underinvested in tools, for sure. Even so, I don’t think you can really argue that dynamically changing configs like this are going to make life easier!
    - toast0 2y ago
      When you have a thousand machines, you can usually get feedback pretty quick, in my experience. Run the tune on one machine. Looks good? Put it on ten. Looks good? Put it on one hundred. Looks good? Put it on everyone. Find an issue a week later, and want to dig into it? Run 100 machines back on the old tune, and 100 machines with half the difference. See what happens.
- yourapostasy 2y ago
  Record changes in git and then git bisect issues, maybe? Without change capture, solid regression testing, or observability, it seems difficult to manage these changes. I’d like to how others are managing these kinds of changes to readily troubleshoot them, without lots of regression testing or observability, if anyone has successes to share.

sgarland 2y ago

Yes, it is. IMO, except for learning (which should not be done in prod), you shouldn’t make changes that you don’t understand. The tools seems to mostly tweak various networking settings. You could set up a test instance with monitoring, throw load at it, and change the parameters the tool modifies (one at a time!) to see how it reacts.

nine_k 2y ago
I'd run such a tool on prod in "advice mode". It should suggest the tweaks, explaining the reasoning behind them, and listing the actions necessary to implement them. Then humans would decide if they want to implement that as is, partly, modified, or not at all.
- sgarland 2y ago
  Fair point, though I didn’t see any such option with this tool.
  - nine_k 2y ago
    It's developed in the open; we can create Github issue. Actually https://github.com/oracle/bpftune/issues/99 https://github.com/oracle/bpftune/issues/99
    - westurner 2y ago
      In the existing issue, we can link to the code and docs that would need to be understood and changed: usage, main() https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d889e3a60de6c12feb76/src/bpftune.c#L280 https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d... - [ ] CLI opts: --pretend-allow <tuner> or --log-only-allow <tuner> or [...] Probably relevant function headers in libbpftune.c: bpftune_sysctl_write( bpftuner_tunable_sysctl_write( bpftune_module_load( static void bpftuner_scenario_log(struct bpftuner *tuner, unsigned int tunable, ; https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d889e3a60de6c12feb76/src/libbpftune.c#L716 https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d... https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d889e3a60de6c12feb76/src/libbpftune.c#L1047 https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d...

deleted 2y ago

[deleted]

trelliscoded 2y ago

If your staging doesn’t do capacity checks in excess of what production sees, yes.

Twirrim 2y ago

Disclaimer: I work for Oracle, who publish this tool, though I have nothing to do with the org or engineers that created it I've been running this for a while on my laptop. So far yet to see any particular weirdness, but also I don't know that I can state with any confidence it has a positive impact either. I've not carried out any benchmarks in either direction. It logs all changes that it's going to make including what they were on before. Here's an example from my logs: bpftune[1852994]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput bpftune[1852994]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 7864320) -> (4096 131072 9830400)

gerdesj 2y ago

"because I'll have drifted from a standard distro configuration" You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. This tool logs all the changes it makes via the standard logging system, which can be easily captured, shipped and aggregated and then queried and reported on. This is not a tool from Clown Cars R US, it's from a reasonably reputable source - Oracle (lol etc). Even better, you can read the code and learn or critique. Not being funny but I'd rather this sort of thing by far than any amount of wooo handwavy wankery. Would you prefer openly described and documented or "take our word for it"?

cortesoft 2y ago
> You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. Which is now a list you will have to check for every issue. I don't think they are complaining they don't trust the writers of the code, just that it adds confounding variables to your system
- gerdesj 2y ago
  We (in IT security) are expected to abrogate responsibility to funky AI or whatevs anti virus and other stuff. Buy and install a security package from ... whoever ... and all will be well. This is an expert system/advice run by real people (at a reasonably well respected firm) not an AI wankery thingie. It is literally expert advice and it is being given away and in code form which you can read. What on earth is wrong with that?
  - cortesoft 2y ago
    If the alternative is those proprietary anti virus products, sure this is better. The original comment was comparing to doing nothing and just using the standard distro, I believe.
    - 2y ago