3 ms·

Maybe this will sound dumb, but I had to deal with medium-large (not "very large") logs in the past, and I found that spending some time actually trimming down

Maybe this will sound dumb, but I had to deal with medium-large (not "very large") logs in the past, and I found that spending some time actually trimming down the messages was really helpful. Especially if you're using it for CI, some commands (e.g., wget, installers like apt-get, etc.) tend to produce what is essentially useless garbage (like XXXX.... progress indicators, "preparing to unpack", "unpacking", etc.). It's not something everyone bothers to do, but I spent some time making commands more quiet and actually keeping only the important parts, and I felt it paid off. All of which is to say—have you tried going this route? I'm not familiar with Yocto, but I feel like it's worth a shot regardless of the UI. There's only so much useful information to log, and it makes it easier to read the logs too, even with a perfect UI.

jordemort 6y ago
I haven't looked at what knobs are available to turn it down yet, but most of what it's logging is actually of interest to me - if it was any less verbose, it would make answering "why did this build take so much longer than the others?" much more difficult, and that's a question I have to answer with some regularity.
- dataflow 6y ago
  For diagnosing build speed, one thing I find helpful is trimming N messages down to 1-2 per command. e.g., if you spawn a process, don't emit "created process", "started process", "shutting down process", "initializing process", "terminated process"... while they all seem useful, just including one of them (like "started process") can trim things down significantly with negligible effect on the usability. I can't speak the same for your use case necessarily, but it might be worth spending a couple hours at some point looking into what you can trim. It's easy to feel the upfront time investment isn't worth it, but it really compounds on a daily basis for every single person, so it's nice if you can address it.
saagarjha 6y ago
I like verbose logs, because when things fail then you don't have the run the build again to get those log results. It's much easier to trim than to generate debug information from thin air.
- dataflow 6y ago
  ...that's exactly why I spent some time writing concrete examples of precisely where that's not the case. How exactly is seeing all this progress mess in your log helpful? If your network connection suddenly breaks at 158K, are you going to do anything different than if it broke at 60K? Do you really need every single line here when all you're doing is just an external download? 0K .......... .......... .......... .......... .......... 0% 153K 67s 50K .......... .......... .......... .......... .......... 0% 310K 49s 100K .......... .......... .......... .......... .......... 1% 300K 44s 150K .......... .......... .......... .......... .......... 1% 10.5M 33s 200K .......... .......... .......... .......... .......... 2% 305K 33s 250K .......... .......... .......... .......... .......... 2% 152K 38s 300K .......... .......... .......... .......... .......... 3% 51.1M 33s 350K .......... .......... .......... .......... .......... 3% 305K 32s 400K .......... .......... .......... .......... .......... 4% 302K 32s 450K .......... .......... .......... .......... .......... 4% 306K 32s 500K .......... .......... .......... .......... .......... 5% 158K 35s What I think you're not realizing is you waste so much time normally filtering through all this cruft every time you scroll through and filter logs; it's not a zero-cost addition. You're paying for it every single time, and the noise actually makes it harder to find stuff you're looking for. At best it wastes seconds of your (and your teammates') time every time; at worst it makes you (and your teammates) actually miss real problems. It actually helps productivity to filter things out, and often (like here) it doesn't even have an ongoing downside to begin with.
  - saagarjha 6y ago
    I use these, though. I can check what time the network connection died and correlate it against any container timeouts I have, or maybe networking troubles. I can run the job again and see if it fails at the same spot it not. It’s useful to have! (Although, I wouldn’t object to it being “collapsed” as many logs will show it.)
    - dataflow 6y ago
      You're making a... very tenuous argument, to say the least. For starters, your entire premise was "you don't have the run the build again". But now that I ask what you're going to do, you tell me you're... going to run the build again. So you've basically proven my point... you're going to waste that time sitting around anyway, and you'll need to reproduce it anyway, so just kick off an extra build or two in parallel if you need to correlate offsets. It's not like the 2nd one was going to be your last build anyway. Now re: diagnosing networks by looking at the byte offsets... so you correlate it (2 builds sufficed apparently?) and then... what exactly are you going to do now? This is GitHub Actions we're talking about, with dependencies downloaded from third-party servers. You're not managing either party's network or storage infrastructure to try to diagnose intermittent "network troubles". All you see is, for whatever reason, ubuntu/pypi/nuget/whatever isn't returning the full file. Maybe it fails in the same spot every time, maybe it doesn't. What exactly are you going to do differently based on what # byte it's failing to return? Either you can increase some timeout or switch to another server or whatever, or you can't. It's not like you're going to pull half the file and just use that. All the progress information does is satisfy idle curiosity and make you go "oh, that's funny...", and then you're left with the same options you had anyway. And honestly, how often does this even happen that you can't afford to tweak the build and increase the verbosity or whatever? These messages are just as useful as garbage >99.9% for the time, for the entire team... you really encounter these issues often enough (and find 2 builds sufficient for fixing them) to warrant that? To say I have a hard time swallowing this is quite the understatement.

jordemort 6y ago

I haven't looked at what knobs are available to turn it down yet, but most of what it's logging is actually of interest to me - if it was any less verbose, it would make answering "why did this build take so much longer than the others?" much more difficult, and that's a question I have to answer with some regularity.

dataflow 6y ago
For diagnosing build speed, one thing I find helpful is trimming N messages down to 1-2 per command. e.g., if you spawn a process, don't emit "created process", "started process", "shutting down process", "initializing process", "terminated process"... while they all seem useful, just including one of them (like "started process") can trim things down significantly with negligible effect on the usability. I can't speak the same for your use case necessarily, but it might be worth spending a couple hours at some point looking into what you can trim. It's easy to feel the upfront time investment isn't worth it, but it really compounds on a daily basis for every single person, so it's nice if you can address it.

saagarjha 6y ago

I like verbose logs, because when things fail then you don't have the run the build again to get those log results. It's much easier to trim than to generate debug information from thin air.

dataflow 6y ago
...that's exactly why I spent some time writing concrete examples of precisely where that's not the case. How exactly is seeing all this progress mess in your log helpful? If your network connection suddenly breaks at 158K, are you going to do anything different than if it broke at 60K? Do you really need every single line here when all you're doing is just an external download? 0K .......... .......... .......... .......... .......... 0% 153K 67s 50K .......... .......... .......... .......... .......... 0% 310K 49s 100K .......... .......... .......... .......... .......... 1% 300K 44s 150K .......... .......... .......... .......... .......... 1% 10.5M 33s 200K .......... .......... .......... .......... .......... 2% 305K 33s 250K .......... .......... .......... .......... .......... 2% 152K 38s 300K .......... .......... .......... .......... .......... 3% 51.1M 33s 350K .......... .......... .......... .......... .......... 3% 305K 32s 400K .......... .......... .......... .......... .......... 4% 302K 32s 450K .......... .......... .......... .......... .......... 4% 306K 32s 500K .......... .......... .......... .......... .......... 5% 158K 35s What I think you're not realizing is you waste so much time normally filtering through all this cruft every time you scroll through and filter logs; it's not a zero-cost addition. You're paying for it every single time, and the noise actually makes it harder to find stuff you're looking for. At best it wastes seconds of your (and your teammates') time every time; at worst it makes you (and your teammates) actually miss real problems. It actually helps productivity to filter things out, and often (like here) it doesn't even have an ongoing downside to begin with.
- saagarjha 6y ago
  I use these, though. I can check what time the network connection died and correlate it against any container timeouts I have, or maybe networking troubles. I can run the job again and see if it fails at the same spot it not. It’s useful to have! (Although, I wouldn’t object to it being “collapsed” as many logs will show it.)
  - dataflow 6y ago
    You're making a... very tenuous argument, to say the least. For starters, your entire premise was "you don't have the run the build again". But now that I ask what you're going to do, you tell me you're... going to run the build again. So you've basically proven my point... you're going to waste that time sitting around anyway, and you'll need to reproduce it anyway, so just kick off an extra build or two in parallel if you need to correlate offsets. It's not like the 2nd one was going to be your last build anyway. Now re: diagnosing networks by looking at the byte offsets... so you correlate it (2 builds sufficed apparently?) and then... what exactly are you going to do now? This is GitHub Actions we're talking about, with dependencies downloaded from third-party servers. You're not managing either party's network or storage infrastructure to try to diagnose intermittent "network troubles". All you see is, for whatever reason, ubuntu/pypi/nuget/whatever isn't returning the full file. Maybe it fails in the same spot every time, maybe it doesn't. What exactly are you going to do differently based on what # byte it's failing to return? Either you can increase some timeout or switch to another server or whatever, or you can't. It's not like you're going to pull half the file and just use that. All the progress information does is satisfy idle curiosity and make you go "oh, that's funny...", and then you're left with the same options you had anyway. And honestly, how often does this even happen that you can't afford to tweak the build and increase the verbosity or whatever? These messages are just as useful as garbage >99.9% for the time, for the entire team... you really encounter these issues often enough (and find 2 builds sufficient for fixing them) to warrant that? To say I have a hard time swallowing this is quite the understatement.