5 ms·

I regularly work with Actions that produce very large logs (building Yocto) and I've found trying to diagnose build failures to be extremely frustrating since t

I regularly work with Actions that produce very large logs (building Yocto) and I've found trying to diagnose build failures to be extremely frustrating since this change rolled out. I want to jump to the end of the log immediately, since that's presumably where I will find my error, but instead of being able to do that in one step, I get stuck in a treadmill of endlessly lazy-loaded log lines; every time I get to the bottom of the window, it's no longer the bottom of the window anymore, regardless of if I used the mouse wheel or tried to yank the scrollbar down. Now when I have build failures, I find myself immediately just going for the raw logs, because I can jump to the end of them without putting in unreasonable effort.

dataflow 6y ago
Maybe this will sound dumb, but I had to deal with medium-large (not "very large") logs in the past, and I found that spending some time actually trimming down the messages was really helpful. Especially if you're using it for CI, some commands (e.g., wget, installers like apt-get, etc.) tend to produce what is essentially useless garbage (like XXXX.... progress indicators, "preparing to unpack", "unpacking", etc.). It's not something everyone bothers to do, but I spent some time making commands more quiet and actually keeping only the important parts, and I felt it paid off. All of which is to say—have you tried going this route? I'm not familiar with Yocto, but I feel like it's worth a shot regardless of the UI. There's only so much useful information to log, and it makes it easier to read the logs too, even with a perfect UI.
- jordemort 6y ago
  I haven't looked at what knobs are available to turn it down yet, but most of what it's logging is actually of interest to me - if it was any less verbose, it would make answering "why did this build take so much longer than the others?" much more difficult, and that's a question I have to answer with some regularity.
  - dataflow 6y ago
    For diagnosing build speed, one thing I find helpful is trimming N messages down to 1-2 per command. e.g., if you spawn a process, don't emit "created process", "started process", "shutting down process", "initializing process", "terminated process"... while they all seem useful, just including one of them (like "started process") can trim things down significantly with negligible effect on the usability. I can't speak the same for your use case necessarily, but it might be worth spending a couple hours at some point looking into what you can trim. It's easy to feel the upfront time investment isn't worth it, but it really compounds on a daily basis for every single person, so it's nice if you can address it.
- saagarjha 6y ago
  I like verbose logs, because when things fail then you don't have the run the build again to get those log results. It's much easier to trim than to generate debug information from thin air.
  - dataflow 6y ago
    ...that's exactly why I spent some time writing concrete examples of precisely where that's not the case. How exactly is seeing all this progress mess in your log helpful? If your network connection suddenly breaks at 158K, are you going to do anything different than if it broke at 60K? Do you really need every single line here when all you're doing is just an external download? 0K .......... .......... .......... .......... .......... 0% 153K 67s 50K .......... .......... .......... .......... .......... 0% 310K 49s 100K .......... .......... .......... .......... .......... 1% 300K 44s 150K .......... .......... .......... .......... .......... 1% 10.5M 33s 200K .......... .......... .......... .......... .......... 2% 305K 33s 250K .......... .......... .......... .......... .......... 2% 152K 38s 300K .......... .......... .......... .......... .......... 3% 51.1M 33s 350K .......... .......... .......... .......... .......... 3% 305K 32s 400K .......... .......... .......... .......... .......... 4% 302K 32s 450K .......... .......... .......... .......... .......... 4% 306K 32s 500K .......... .......... .......... .......... .......... 5% 158K 35s What I think you're not realizing is you waste so much time normally filtering through all this cruft every time you scroll through and filter logs; it's not a zero-cost addition. You're paying for it every single time, and the noise actually makes it harder to find stuff you're looking for. At best it wastes seconds of your (and your teammates') time every time; at worst it makes you (and your teammates) actually miss real problems. It actually helps productivity to filter things out, and often (like here) it doesn't even have an ongoing downside to begin with.
    - saagarjha 6y ago
      I use these, though. I can check what time the network connection died and correlate it against any container timeouts I have, or maybe networking troubles. I can run the job again and see if it fails at the same spot it not. It’s useful to have! (Although, I wouldn’t object to it being “collapsed” as many logs will show it.)

dataflow 6y ago

Maybe this will sound dumb, but I had to deal with medium-large (not "very large") logs in the past, and I found that spending some time actually trimming down the messages was really helpful. Especially if you're using it for CI, some commands (e.g., wget, installers like apt-get, etc.) tend to produce what is essentially useless garbage (like XXXX.... progress indicators, "preparing to unpack", "unpacking", etc.). It's not something everyone bothers to do, but I spent some time making commands more quiet and actually keeping only the important parts, and I felt it paid off. All of which is to say—have you tried going this route? I'm not familiar with Yocto, but I feel like it's worth a shot regardless of the UI. There's only so much useful information to log, and it makes it easier to read the logs too, even with a perfect UI.

jordemort 6y ago
I haven't looked at what knobs are available to turn it down yet, but most of what it's logging is actually of interest to me - if it was any less verbose, it would make answering "why did this build take so much longer than the others?" much more difficult, and that's a question I have to answer with some regularity.
- dataflow 6y ago
  For diagnosing build speed, one thing I find helpful is trimming N messages down to 1-2 per command. e.g., if you spawn a process, don't emit "created process", "started process", "shutting down process", "initializing process", "terminated process"... while they all seem useful, just including one of them (like "started process") can trim things down significantly with negligible effect on the usability. I can't speak the same for your use case necessarily, but it might be worth spending a couple hours at some point looking into what you can trim. It's easy to feel the upfront time investment isn't worth it, but it really compounds on a daily basis for every single person, so it's nice if you can address it.
saagarjha 6y ago
I like verbose logs, because when things fail then you don't have the run the build again to get those log results. It's much easier to trim than to generate debug information from thin air.
- dataflow 6y ago
  ...that's exactly why I spent some time writing concrete examples of precisely where that's not the case. How exactly is seeing all this progress mess in your log helpful? If your network connection suddenly breaks at 158K, are you going to do anything different than if it broke at 60K? Do you really need every single line here when all you're doing is just an external download? 0K .......... .......... .......... .......... .......... 0% 153K 67s 50K .......... .......... .......... .......... .......... 0% 310K 49s 100K .......... .......... .......... .......... .......... 1% 300K 44s 150K .......... .......... .......... .......... .......... 1% 10.5M 33s 200K .......... .......... .......... .......... .......... 2% 305K 33s 250K .......... .......... .......... .......... .......... 2% 152K 38s 300K .......... .......... .......... .......... .......... 3% 51.1M 33s 350K .......... .......... .......... .......... .......... 3% 305K 32s 400K .......... .......... .......... .......... .......... 4% 302K 32s 450K .......... .......... .......... .......... .......... 4% 306K 32s 500K .......... .......... .......... .......... .......... 5% 158K 35s What I think you're not realizing is you waste so much time normally filtering through all this cruft every time you scroll through and filter logs; it's not a zero-cost addition. You're paying for it every single time, and the noise actually makes it harder to find stuff you're looking for. At best it wastes seconds of your (and your teammates') time every time; at worst it makes you (and your teammates) actually miss real problems. It actually helps productivity to filter things out, and often (like here) it doesn't even have an ongoing downside to begin with.
  - saagarjha 6y ago
    I use these, though. I can check what time the network connection died and correlate it against any container timeouts I have, or maybe networking troubles. I can run the job again and see if it fails at the same spot it not. It’s useful to have! (Although, I wouldn’t object to it being “collapsed” as many logs will show it.)