3 ms·

...that's exactly why I spent some time writing concrete examples of precisely where that's not the case. How exactly is seeing all this progress mess in your

...that's exactly why I spent some time writing concrete examples of precisely where that's not the case. How exactly is seeing all this progress mess in your log helpful? If your network connection suddenly breaks at 158K, are you going to do anything different than if it broke at 60K? Do you really need every single line here when all you're doing is just an external download? 0K .......... .......... .......... .......... .......... 0% 153K 67s 50K .......... .......... .......... .......... .......... 0% 310K 49s 100K .......... .......... .......... .......... .......... 1% 300K 44s 150K .......... .......... .......... .......... .......... 1% 10.5M 33s 200K .......... .......... .......... .......... .......... 2% 305K 33s 250K .......... .......... .......... .......... .......... 2% 152K 38s 300K .......... .......... .......... .......... .......... 3% 51.1M 33s 350K .......... .......... .......... .......... .......... 3% 305K 32s 400K .......... .......... .......... .......... .......... 4% 302K 32s 450K .......... .......... .......... .......... .......... 4% 306K 32s 500K .......... .......... .......... .......... .......... 5% 158K 35s What I think you're not realizing is you waste so much time normally filtering through all this cruft every time you scroll through and filter logs; it's not a zero-cost addition. You're paying for it every single time, and the noise actually makes it harder to find stuff you're looking for. At best it wastes seconds of your (and your teammates') time every time; at worst it makes you (and your teammates) actually miss real problems. It actually helps productivity to filter things out, and often (like here) it doesn't even have an ongoing downside to begin with.

saagarjha 6y ago
I use these, though. I can check what time the network connection died and correlate it against any container timeouts I have, or maybe networking troubles. I can run the job again and see if it fails at the same spot it not. It’s useful to have! (Although, I wouldn’t object to it being “collapsed” as many logs will show it.)
- dataflow 6y ago
  You're making a... very tenuous argument, to say the least. For starters, your entire premise was "you don't have the run the build again". But now that I ask what you're going to do, you tell me you're... going to run the build again. So you've basically proven my point... you're going to waste that time sitting around anyway, and you'll need to reproduce it anyway, so just kick off an extra build or two in parallel if you need to correlate offsets. It's not like the 2nd one was going to be your last build anyway. Now re: diagnosing networks by looking at the byte offsets... so you correlate it (2 builds sufficed apparently?) and then... what exactly are you going to do now? This is GitHub Actions we're talking about, with dependencies downloaded from third-party servers. You're not managing either party's network or storage infrastructure to try to diagnose intermittent "network troubles". All you see is, for whatever reason, ubuntu/pypi/nuget/whatever isn't returning the full file. Maybe it fails in the same spot every time, maybe it doesn't. What exactly are you going to do differently based on what # byte it's failing to return? Either you can increase some timeout or switch to another server or whatever, or you can't. It's not like you're going to pull half the file and just use that. All the progress information does is satisfy idle curiosity and make you go "oh, that's funny...", and then you're left with the same options you had anyway. And honestly, how often does this even happen that you can't afford to tweak the build and increase the verbosity or whatever? These messages are just as useful as garbage >99.9% for the time, for the entire team... you really encounter these issues often enough (and find 2 builds sufficient for fixing them) to warrant that? To say I have a hard time swallowing this is quite the understatement.
  - saagarjha 6y ago
    You still save a build, because (if needed) you can get results the next run rather than running two builds to figure out what went wrong. It’s not that I only have a chance to run two builds, but fewer builds saves time, especially in situations where running one takes a long time or things are falling over in such a way that it’s even hard to get a build scheduled. As for what I would do once I have that information…for GitHub Actions not much other than report it to someone who might actually have use for that information. But for internal CI it can be useful because it can tell you things like “oh the build always fails 30 minutes in, perhaps something is timing out” or “it always fails at 68%…hmm, I can download this file when not on the corporate network. Ah, the firewall doesn’t like that packet”. Again, I’m not saying I don’t agree with you that this is often not useful, but if you have a way to hide it away until it is necessary I would generally lean towards keeping it.
mikepurvis 6y ago
Note that apt in particular has one progress output which can be difficult to silence since it's actually coming from the underlying dpkg tool. Rather than passing a ton of flags into every apt invocation to deal with this, it can be helpful to drop a file like this into `etc/apt/apt.conf.d/`, especially if you know that you're in a throwaway CI environment (or a base container only used for such things) where no one will be jumping into an interactive session afterward and be confused by differing defaults: APT::Install-Recommends "0"; APT::Install-Suggests "0"; APT::Acquire::Retries "20"; Dpkg::Use-Pty "0"; Dpkg::Options "--force-confnew";
- dataflow 6y ago
  Yup, thanks for sharing this! That's why I mentioned apt as an example. It's an upfront time investment (I think I spent hours trying to figure out how to suppress apt messages) so I can understand why it's not appealing, but you do it once and then it's so much cleaner forever after.
- djsavvy 6y ago
  This doesn't seem to be working for me on Ubuntu 20.04 (and according to a comment on https://askubuntu.com/questions/258219/how-do-i-make-apt-get-install-less-noisy https://askubuntu.com/questions/258219/how-do-i-make-apt-get... it seems like I'm not the only one). If, by chance, you're on Ubuntu 20.04, did you happen to figure out a way to get rid of the rest of `apt-get`'s output?

saagarjha 6y ago

I use these, though. I can check what time the network connection died and correlate it against any container timeouts I have, or maybe networking troubles. I can run the job again and see if it fails at the same spot it not. It’s useful to have! (Although, I wouldn’t object to it being “collapsed” as many logs will show it.)

dataflow 6y ago
You're making a... very tenuous argument, to say the least. For starters, your entire premise was "you don't have the run the build again". But now that I ask what you're going to do, you tell me you're... going to run the build again. So you've basically proven my point... you're going to waste that time sitting around anyway, and you'll need to reproduce it anyway, so just kick off an extra build or two in parallel if you need to correlate offsets. It's not like the 2nd one was going to be your last build anyway. Now re: diagnosing networks by looking at the byte offsets... so you correlate it (2 builds sufficed apparently?) and then... what exactly are you going to do now? This is GitHub Actions we're talking about, with dependencies downloaded from third-party servers. You're not managing either party's network or storage infrastructure to try to diagnose intermittent "network troubles". All you see is, for whatever reason, ubuntu/pypi/nuget/whatever isn't returning the full file. Maybe it fails in the same spot every time, maybe it doesn't. What exactly are you going to do differently based on what # byte it's failing to return? Either you can increase some timeout or switch to another server or whatever, or you can't. It's not like you're going to pull half the file and just use that. All the progress information does is satisfy idle curiosity and make you go "oh, that's funny...", and then you're left with the same options you had anyway. And honestly, how often does this even happen that you can't afford to tweak the build and increase the verbosity or whatever? These messages are just as useful as garbage >99.9% for the time, for the entire team... you really encounter these issues often enough (and find 2 builds sufficient for fixing them) to warrant that? To say I have a hard time swallowing this is quite the understatement.
- saagarjha 6y ago
  You still save a build, because (if needed) you can get results the next run rather than running two builds to figure out what went wrong. It’s not that I only have a chance to run two builds, but fewer builds saves time, especially in situations where running one takes a long time or things are falling over in such a way that it’s even hard to get a build scheduled. As for what I would do once I have that information…for GitHub Actions not much other than report it to someone who might actually have use for that information. But for internal CI it can be useful because it can tell you things like “oh the build always fails 30 minutes in, perhaps something is timing out” or “it always fails at 68%…hmm, I can download this file when not on the corporate network. Ah, the firewall doesn’t like that packet”. Again, I’m not saying I don’t agree with you that this is often not useful, but if you have a way to hide it away until it is necessary I would generally lean towards keeping it.

mikepurvis 6y ago

Note that apt in particular has one progress output which can be difficult to silence since it's actually coming from the underlying dpkg tool. Rather than passing a ton of flags into every apt invocation to deal with this, it can be helpful to drop a file like this into `etc/apt/apt.conf.d/`, especially if you know that you're in a throwaway CI environment (or a base container only used for such things) where no one will be jumping into an interactive session afterward and be confused by differing defaults: APT::Install-Recommends "0"; APT::Install-Suggests "0"; APT::Acquire::Retries "20"; Dpkg::Use-Pty "0"; Dpkg::Options "--force-confnew";

dataflow 6y ago
Yup, thanks for sharing this! That's why I mentioned apt as an example. It's an upfront time investment (I think I spent hours trying to figure out how to suppress apt messages) so I can understand why it's not appealing, but you do it once and then it's so much cleaner forever after.
djsavvy 6y ago
This doesn't seem to be working for me on Ubuntu 20.04 (and according to a comment on https://askubuntu.com/questions/258219/how-do-i-make-apt-get-install-less-noisy https://askubuntu.com/questions/258219/how-do-i-make-apt-get... it seems like I'm not the only one). If, by chance, you're on Ubuntu 20.04, did you happen to figure out a way to get rid of the rest of `apt-get`'s output?