7 ms·

Microsoft Ubuntu repositories are broken because of space issues

191 points by NiekvdMaas 5y ago

phab 5y ago
The packages.microsoft.com repositories are regularly broken, which results in perennial build failures if you use GitHub Actions which includes it by default. GitHub's response to "your default configuration is broken" was basically "welp, NOMFP": https://github.com/actions/virtual-environments/issues/3410 https://github.com/actions/virtual-environments/issues/3410
- ilaksh 5y ago
  Is there a way to easily change the default and if so what is a good mirror?
  - phab 5y ago
    I have no need for MS packages, and so I simply for broken_apt_repository in `grep -lr microsoft /etc/apt/sources.list.d/`; do sudo rm $broken_apt_repository; done before I do anything else. Others are not so lucky.
- unwind 5y ago
  Uh somewhat off topic but is NOMFP a typo of NMFP (="not my fine problem")? Hadn't seen it before and big G suggests NMFP is way more common, with no hit that I saw explaining the 'O'.
  - Aeolun 5y ago
    Is this another typo? NFMP = Not Fine My Problem? That said, I’d also like to know.
    - CallMeMarc 5y ago
      I’d have guessed „none of my fucking problems“
    - unwind 5y ago
      Typo yes, that's what I get from HN:ing on my phone. Thanks, and fixed.
  - phab 5y ago
    https://www.google.com/search?client=firefox-b-d&q=NOMFP https://www.google.com/search?client=firefox-b-d&q=NOMFP (It's a quote from the UK political satire series "The Thick Of It", in which the character Malcolm Tucker remarks, "NOMFP. N-O-M-F-P. Not My Fucking Problem. I quite like that. Did you like that? I'll use that quite a lot today.")
    - unwind 5y ago
      Aha, that was a couple of fathoms below my radar. Thanks!
    - scrapcode 5y ago
      At first glance I thought "Not Our MFing Problem." Thanks.
      - dhimes 5y ago
        This seems quite plausible- moreso than using the first two letters of "Not."
  - bitwize 5y ago
    I thought it was "not our motherfucking problem".
    - cghendrix 5y ago
      Not sure why but thought it was: not on my f*cking premises
      - Cthulhu_ 5y ago
        You don't need to self-censor, we're all adults, we know what you mean, the US government won't deplatform you for typing an u.
  - comradesmith 5y ago
    Maybe “Not on my fucking plate”?
- paulcarroty 5y ago
  Oh, had this several times and removed MS repos in some projects. Maybe it worth to create the extension.
- shrikant 5y ago
  The packages.microsoft.com repositories are by far THE SLOWEST for `apt` updates. I wonder if there's a way to pick a faster mirror.
  - G3rn0ti 5y ago
    If you are on Ubuntu you could install from Canonical‘s snap repo hosted by Canonical and not MS.
    - shrikant 5y ago
      I don't use snaps though...
G3rn0ti 5y ago
Wanted to update my Teams on Ubuntu 20 in the morning (after coming to office after months working from home) and had the same issue. Removed the package and installed it from snap. Works like charm so far.
merb 5y ago
it's not just ubuntu, it's basically all of them. basically my debian docker build failed because of that.
- Scandiravian 5y ago
  Same. More or less all of my company's pipelines are currently broken
throwawaybutwhy 5y ago
Not only Ubuntu. And not for the first time [0], [1], [3] (Stack Overflow). Would be surprised, however, if the servers are in space instead of under water [2]. [0] https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22-4e8d-a534-3e79b3cb3113/failed-to-fetch https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22... [1] https://github.com/MicrosoftDocs/windowsserverdocs/issues/5051 https://github.com/MicrosoftDocs/windowsserverdocs/issues/50... [2] https://news.microsoft.com/innovation-stories/project-natick-underwater-datacenter/ https://news.microsoft.com/innovation-stories/project-natick... [3] https://stackoverflow.com/questions/59624773/failed-to-fetch-https-packages-microsoft-com-repos-azure-cli-dists-bionic-main https://stackoverflow.com/questions/59624773/failed-to-fetch...
azalemeth 5y ago
What SLA are Ubuntu mirrors held to? This doesn't seem great, but won't a quick run of mirrorselect remove the offending repos and let you get on with life?
- NiekvdMaas 5y ago
  For your home setup: sure, it should be resolved soon, not too big of an issue. What's much more annoying is that this breaks CI/Docker builds. Right now you can't deploy new container images that depend on a MS repository. I hope it's resolved soon.
  - dvfjsdhgfv 5y ago
    We had the same problems already in the 90s and resolved them by caching, it's unthinkable that the whole build is broken by a problem with an external dependence - if you assume everything will be 100% available you basically make sure a problem will happen.
    - creshal 5y ago
      https://wiki.debian.org/AptCacherNg https://wiki.debian.org/AptCacherNg still exists and still works fine.
      - tambre 5y ago
        Aptly [0] is also really nice for maintaining mirrors and one's own Apt repositories though it is unmaintained. [0]: https://github.com/aptly-dev/aptly https://github.com/aptly-dev/aptly
    - silon42 5y ago
      Not caching... mirroring or vendoring.
    - rkeene2 5y ago
      I wrote "hashcache" [0] [1] for this for my Linux distribution (which is the build from source variety) to avoid issues with upstream going away, or worse changing the contents of a file. Many different caches can be used, in case one is down as well. Simple example download script at [2]. [0] http://hashcache.rkeene.org/ http://hashcache.rkeene.org/ [1] https://chiselapp.com/user/rkeene/repository/hashcache/ https://chiselapp.com/user/rkeene/repository/hashcache/ [2] https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce807 https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce8...
mixxit 5y ago
had this last night :( thanks for posting
vmception 5y ago
Of all the social issues in the world, apt-get and wget 404ing for my version of linux are the only things that trigger me
deleted 5y ago
[deleted]
alfiedotwtf 5y ago
I've also found the ':' character in filenames also breaks Linux under Windows :(
fabbari 5y ago
None commented yet on the suggested issue? They ran out of space? Really? Azure Monitoring, Azure Sentinel - surely they have the simplest metric: used vs remaining space. Or did they hit a limit in the blob/file storage. Still: no monitoring? You may have noticed I have so many questions on Microsoft Ops.
- throwawaybutwhy 5y ago
  rm -rf --no-preserve-root is a space issue, too.
  - rvanlaar 5y ago
    Reminds me of an F1 story. A car was able to finish the race but stopped before reaching the pit lane. The team communicated it as being a fuel pick up problem. The underlying cause was that there was no fuel left to be picked up.
yrro 5y ago
Microsoft's dpkg packages themselves do some fairly weird/annoying stuff in their maintainer scripts. e.g., azuredatastudio contains code that manages a /usr/local/bin/code symlink... ... and it contains code that converts a PGP key in armored form to binary form and dumps it in /etc/apt/trusted.gpg.d, even if the sysadmin already took the time to verify the PGP key and put it into their own .asc file in the same directory (ok, I guess at least they aren't force-appending their key to the old/deprecated/monolithic /etc/apt/trusted.gpg file like many others)... ... oh and in doing so they dump out microsoft.gpg to the current directory, whatever that may be... they should at least be using mktemp! ... and they are doing other things that they should be relying on debhelper to do for them, e.g. installing shared-mime-info-spec files manually rather than with dh_installmime; installing desktop-entry-spec files manually rather than relying on the triggers that already handle installation of such files... As for teams, it has its own oddball way of monkeying with /etc/apt, and one other weirdness: it explicitly changes /usr/share/teams/chrome-sandbox to be setuid. If that file is supposed to be setuid then ship it as such... shipping it non-setuid and then modifing it in a maintainer script sets off alarm bells and breaks dpkg-statoverride...
- Nux 5y ago
  I have zero trust on how they handle this, judging by your post it is justified. This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.
  - mcny 5y ago
    > This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software. I always recommend this now if it is an option. Even if Microsoft does nothing wrong, it is way too easy for me to make an error and somehow enable Microsoft antivirus stuff on my Fedora machine which I still don't know how I managed to do.
  - yrro 5y ago
    Flatpak is best iff the vendor is producing the packages. I have found all too often than I'm stuck on an outdated version of a orphaned, unofficial pacakge...
enriquto 5y ago
> because of space issues yeah, spaces in filenames. A well-known yet insidious problem. Is there any mount option to change raw spaces in the exposed filesystem with an unicode non-breaking space, or something?
- craftinator 5y ago
  > yeah, spaces in filenames. A well-known yet insidious problem. It'll be fixed as soon as they finish their Backslash Conversion Project!
jijji 5y ago
its probably a silly question, but why are people using microsoft to fetch debian packages?
- deleted 5y ago
  [deleted]
- krautsourced 5y ago
  I'd assume they host some of their own packages plus it's probably the default for the Azure space. (just an assumption)
  - fapjacks 5y ago
    Microsoft repos were silently added to Raspberry Pi OS earlier this year, ostensibly "in case anyone wants to use VSCode" and definitely not just as a super shitty way of adding telemetry to every Raspberry Pi OS install without asking the user if they were cool with this.
- userbinator 5y ago
  Microsoft's packages of its own software for Debian.
- minedwiz 5y ago
  For example, my team's services run on Debian, but are .NET Core-based, so our dev boxes & CI machines rely on Microsoft deb repos.
deleted 5y ago
[deleted]
TimSchumann 5y ago
Just to clarify the title, Space in this context is the 'Biggest Key on your Keyboard' kind not the 'On Disk' kind. It looks like a parsing issue where something in MS's automated process is replacing various parts of filenames with spaces. Looks like / and _ at a glance, but there could be more. Just trying to prevent this type of comment, which I can only assume came from HN as nowhere in the Github issue does anyone mention disk space. -- https://github.com/dotnet/core/issues/6381#issuecomment-863157425 https://github.com/dotnet/core/issues/6381#issuecomment-8631...
- tyingq 5y ago
  "nowhere in the Github issue does anyone mention disk space" What is mentioned is ambiguous. So while disk space isn't specifically mentioned, neither is spaces in the context of filenames. Here's what people are reading in that issue: "Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet." There isn't enough context to know what it means.
  - jbverschoor 5y ago
    "escaping space characters" would be clear. I thought there was some diskspace issue
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
sproketboy 5y ago
.NET Core is a waste of time.

191 points by NiekvdMaas 5y ago

phab 5y ago

The packages.microsoft.com repositories are regularly broken, which results in perennial build failures if you use GitHub Actions which includes it by default. GitHub's response to "your default configuration is broken" was basically "welp, NOMFP": https://github.com/actions/virtual-environments/issues/3410 https://github.com/actions/virtual-environments/issues/3410

ilaksh 5y ago
Is there a way to easily change the default and if so what is a good mirror?
- phab 5y ago
  I have no need for MS packages, and so I simply for broken_apt_repository in `grep -lr microsoft /etc/apt/sources.list.d/`; do sudo rm $broken_apt_repository; done before I do anything else. Others are not so lucky.
unwind 5y ago
Uh somewhat off topic but is NOMFP a typo of NMFP (="not my fine problem")? Hadn't seen it before and big G suggests NMFP is way more common, with no hit that I saw explaining the 'O'.
- Aeolun 5y ago
  Is this another typo? NFMP = Not Fine My Problem? That said, I’d also like to know.
  - CallMeMarc 5y ago
    I’d have guessed „none of my fucking problems“
  - unwind 5y ago
    Typo yes, that's what I get from HN:ing on my phone. Thanks, and fixed.
- phab 5y ago
  https://www.google.com/search?client=firefox-b-d&q=NOMFP https://www.google.com/search?client=firefox-b-d&q=NOMFP (It's a quote from the UK political satire series "The Thick Of It", in which the character Malcolm Tucker remarks, "NOMFP. N-O-M-F-P. Not My Fucking Problem. I quite like that. Did you like that? I'll use that quite a lot today.")
  - unwind 5y ago
    Aha, that was a couple of fathoms below my radar. Thanks!
  - scrapcode 5y ago
    At first glance I thought "Not Our MFing Problem." Thanks.
    - dhimes 5y ago
      This seems quite plausible- moreso than using the first two letters of "Not."
- bitwize 5y ago
  I thought it was "not our motherfucking problem".
  - cghendrix 5y ago
    Not sure why but thought it was: not on my f*cking premises
    - Cthulhu_ 5y ago
      You don't need to self-censor, we're all adults, we know what you mean, the US government won't deplatform you for typing an u.
- comradesmith 5y ago
  Maybe “Not on my fucking plate”?
paulcarroty 5y ago
Oh, had this several times and removed MS repos in some projects. Maybe it worth to create the extension.
shrikant 5y ago
The packages.microsoft.com repositories are by far THE SLOWEST for `apt` updates. I wonder if there's a way to pick a faster mirror.
- G3rn0ti 5y ago
  If you are on Ubuntu you could install from Canonical‘s snap repo hosted by Canonical and not MS.
  - shrikant 5y ago
    I don't use snaps though...

G3rn0ti 5y ago

Wanted to update my Teams on Ubuntu 20 in the morning (after coming to office after months working from home) and had the same issue. Removed the package and installed it from snap. Works like charm so far.

merb 5y ago

it's not just ubuntu, it's basically all of them. basically my debian docker build failed because of that.

Scandiravian 5y ago
Same. More or less all of my company's pipelines are currently broken

throwawaybutwhy 5y ago

Not only Ubuntu. And not for the first time [0], [1], [3] (Stack Overflow). Would be surprised, however, if the servers are in space instead of under water [2]. [0] https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22-4e8d-a534-3e79b3cb3113/failed-to-fetch https://social.msdn.microsoft.com/Forums/en-US/5d090bb8-6c22... [1] https://github.com/MicrosoftDocs/windowsserverdocs/issues/5051 https://github.com/MicrosoftDocs/windowsserverdocs/issues/50... [2] https://news.microsoft.com/innovation-stories/project-natick-underwater-datacenter/ https://news.microsoft.com/innovation-stories/project-natick... [3] https://stackoverflow.com/questions/59624773/failed-to-fetch-https-packages-microsoft-com-repos-azure-cli-dists-bionic-main https://stackoverflow.com/questions/59624773/failed-to-fetch...

azalemeth 5y ago

What SLA are Ubuntu mirrors held to? This doesn't seem great, but won't a quick run of mirrorselect remove the offending repos and let you get on with life?

NiekvdMaas 5y ago
For your home setup: sure, it should be resolved soon, not too big of an issue. What's much more annoying is that this breaks CI/Docker builds. Right now you can't deploy new container images that depend on a MS repository. I hope it's resolved soon.
- dvfjsdhgfv 5y ago
  We had the same problems already in the 90s and resolved them by caching, it's unthinkable that the whole build is broken by a problem with an external dependence - if you assume everything will be 100% available you basically make sure a problem will happen.
  - creshal 5y ago
    https://wiki.debian.org/AptCacherNg https://wiki.debian.org/AptCacherNg still exists and still works fine.
    - tambre 5y ago
      Aptly [0] is also really nice for maintaining mirrors and one's own Apt repositories though it is unmaintained. [0]: https://github.com/aptly-dev/aptly https://github.com/aptly-dev/aptly
  - silon42 5y ago
    Not caching... mirroring or vendoring.
  - rkeene2 5y ago
    I wrote "hashcache" [0] [1] for this for my Linux distribution (which is the build from source variety) to avoid issues with upstream going away, or worse changing the contents of a file. Many different caches can be used, in case one is down as well. Simple example download script at [2]. [0] http://hashcache.rkeene.org/ http://hashcache.rkeene.org/ [1] https://chiselapp.com/user/rkeene/repository/hashcache/ https://chiselapp.com/user/rkeene/repository/hashcache/ [2] https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce807 https://kitcreator.rkeene.org/fossil/artifact/d546c4b48dfce8...

mixxit 5y ago

had this last night :( thanks for posting

vmception 5y ago

Of all the social issues in the world, apt-get and wget 404ing for my version of linux are the only things that trigger me

deleted 5y ago

[deleted]

alfiedotwtf 5y ago

I've also found the ':' character in filenames also breaks Linux under Windows :(

fabbari 5y ago

None commented yet on the suggested issue? They ran out of space? Really? Azure Monitoring, Azure Sentinel - surely they have the simplest metric: used vs remaining space. Or did they hit a limit in the blob/file storage. Still: no monitoring? You may have noticed I have so many questions on Microsoft Ops.

throwawaybutwhy 5y ago
rm -rf --no-preserve-root is a space issue, too.
- rvanlaar 5y ago
  Reminds me of an F1 story. A car was able to finish the race but stopped before reaching the pit lane. The team communicated it as being a fuel pick up problem. The underlying cause was that there was no fuel left to be picked up.

yrro 5y ago

Microsoft's dpkg packages themselves do some fairly weird/annoying stuff in their maintainer scripts. e.g., azuredatastudio contains code that manages a /usr/local/bin/code symlink... ... and it contains code that converts a PGP key in armored form to binary form and dumps it in /etc/apt/trusted.gpg.d, even if the sysadmin already took the time to verify the PGP key and put it into their own .asc file in the same directory (ok, I guess at least they aren't force-appending their key to the old/deprecated/monolithic /etc/apt/trusted.gpg file like many others)... ... oh and in doing so they dump out microsoft.gpg to the current directory, whatever that may be... they should at least be using mktemp! ... and they are doing other things that they should be relying on debhelper to do for them, e.g. installing shared-mime-info-spec files manually rather than with dh_installmime; installing desktop-entry-spec files manually rather than relying on the triggers that already handle installation of such files... As for teams, it has its own oddball way of monkeying with /etc/apt, and one other weirdness: it explicitly changes /usr/share/teams/chrome-sandbox to be setuid. If that file is supposed to be setuid then ship it as such... shipping it non-setuid and then modifing it in a maintainer script sets off alarm bells and breaks dpkg-statoverride...

Nux 5y ago
I have zero trust on how they handle this, judging by your post it is justified. This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software.
- mcny 5y ago
  > This is why I decided to bite the bullet and go flatpak for the likes of teams, slack and other proprietary software. I always recommend this now if it is an option. Even if Microsoft does nothing wrong, it is way too easy for me to make an error and somehow enable Microsoft antivirus stuff on my Fedora machine which I still don't know how I managed to do.
- yrro 5y ago
  Flatpak is best iff the vendor is producing the packages. I have found all too often than I'm stuck on an outdated version of a orphaned, unofficial pacakge...

enriquto 5y ago

> because of space issues yeah, spaces in filenames. A well-known yet insidious problem. Is there any mount option to change raw spaces in the exposed filesystem with an unicode non-breaking space, or something?

craftinator 5y ago
> yeah, spaces in filenames. A well-known yet insidious problem. It'll be fixed as soon as they finish their Backslash Conversion Project!

jijji 5y ago

its probably a silly question, but why are people using microsoft to fetch debian packages?

deleted 5y ago
[deleted]
krautsourced 5y ago
I'd assume they host some of their own packages plus it's probably the default for the Azure space. (just an assumption)
- fapjacks 5y ago
  Microsoft repos were silently added to Raspberry Pi OS earlier this year, ostensibly "in case anyone wants to use VSCode" and definitely not just as a super shitty way of adding telemetry to every Raspberry Pi OS install without asking the user if they were cool with this.
userbinator 5y ago
Microsoft's packages of its own software for Debian.
minedwiz 5y ago
For example, my team's services run on Debian, but are .NET Core-based, so our dev boxes & CI machines rely on Microsoft deb repos.

deleted 5y ago

[deleted]

TimSchumann 5y ago

Just to clarify the title, Space in this context is the 'Biggest Key on your Keyboard' kind not the 'On Disk' kind. It looks like a parsing issue where something in MS's automated process is replacing various parts of filenames with spaces. Looks like / and _ at a glance, but there could be more. Just trying to prevent this type of comment, which I can only assume came from HN as nowhere in the Github issue does anyone mention disk space. -- https://github.com/dotnet/core/issues/6381#issuecomment-863157425 https://github.com/dotnet/core/issues/6381#issuecomment-8631...

tyingq 5y ago
"nowhere in the Github issue does anyone mention disk space" What is mentioned is ambiguous. So while disk space isn't specifically mentioned, neither is spaces in the context of filenames. Here's what people are reading in that issue: "Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet." There isn't enough context to know what it means.
- jbverschoor 5y ago
  "escaping space characters" would be clear. I thought there was some diskspace issue

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

sproketboy 5y ago

.NET Core is a waste of time.