6 ms·

This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days? https://developers.redhat.com/blog/2021/01/05/building-red-hat

This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days? https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level https://developers.redhat.com/blog/2021/01/05/building-red-h... Think of how much faster their servers would be with one of those Epyc consumer cpus. I was about to ask people to donate, but they have $80k in their coffers. I realize their budget is only $17,000 a year, but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers as they are around under $2k under budget. If they have a fleet of these old servers I imagine a Zen5 one can replace at least a few of them and consume far less power and space. https://opencollective.com/f-droid#category-BUDGET https://opencollective.com/f-droid#category-BUDGET Not sure if this includes their Librapay donations either: https://liberapay.com/F-Droid-Data/donate https://liberapay.com/F-Droid-Data/donate

FirmwareBurner 1y ago
>they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers I would also like to know this.
- lupusreal 1y ago
  Probably a case of "don't fix it if it ain't broke" keeping old machines in service too long, so now they broke.
  - FirmwareBurner 1y ago
    That's like ignoring your 'Check Engine' light because the engine still runs.
- pastage 1y ago
  I would much rather they spent that on having the devs network and travel, the servers work.
  - melodyogonna 1y ago
    Why are the builds failing then?
    - tcfhgj 1y ago
      planned obsolescence by Google
      - shadowgovt 1y ago
        Beginning to use a CPU opcode that is 19 years old doesn't feel like planned obsolescence. if anything, it feels like unplanned obsolescence... "Oh hell what do you mean your CPU doesn't have that opcode no we've just been running the compiler with the default flags and that opcode got added to the default two months ago after a 10-year fight about the possible consequences of changing defaults!" Although I'm a little surprised to learn that the binary itself doesn't have enough information in its header to be able to declare that it needs SSSE3 to be executed; that feels like something that should be statically-analyzed-and-cached to avoid a lot of debugging headaches.
        tcfhgj 1y ago
        > "Oh hell what do you mean your CPU doesn't have that opcode [...]" hobbyst dev? sure Google? nope
        shadowgovt 1y ago
        Did they make any explicit guarantees that their newly-cut binaries would continue to support 20-year-old architectures? Googlers aren't gods. It's a 100,000-person company; they're as vulnerable to "We didn't really think of that one way or the other" as anyone else. ETA: It's actually not even Google code that changed (directly); Gradle apparently began requiring SSSE3 (https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153 https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153) and Google's toolchain just consumed the new constraint from its upstream. Here, I'm not surprised at all; Google is not the kind of firm that keeps a test-lab of older hardware for every application they ship, so (particularly for their dev tooling) "It worked on my machine" is probably ship-worthy. I bet they don't even have an explicit architecture target for the Android build toolchain beyond the company's default (which is generally "The two most recent versions" of whatever we're talking about).
- Perz1val 1y ago
  Yeah and everybody was complaining how slow the builds are for years. I really want to know too
Timshel 1y ago
$2-3k ? That’s barely the price of a lower end Threadripper bare cpu not a full Epyc server ???
- wongarsu 1y ago
  At our supplier $2k would pay for a 1U server with a 16 core 3GHz Epyc 7313P with 32GB RAM, a tiny SSD and non-redundant power. $3k pays for a 1U server with a 32 core 2.6GHz Epyc 7513 with 128GB RAM and 960GB of non-redundant SSD storage (probably fine for build servers). All using server CPUs, since that was easier to find. If you want more cores or more than 3GHz things get considerably more expensive.
  - Timshel 1y ago
    Yes but thoose are Zen 3 Milan cpu released in 2021 I believe. Not that they are bad and would not be way better than what they have, just that I though the parent was quite the optimist with his Zen4/Zen5 pricing.
    - wtallis 1y ago
      OP did say "consumer Epyc", so presumably referring to the parts using the AM5 socket. From a quick check on Newegg, it looks like barebones servers for that platform start at under $1000, to which you need to add CPU, RAM, and storage. So a $3000 budget to assemble a low-end Zen4/5 EPYC server is realistic: $570 for the 16-core EPYC 4565P, a few hundred for DDR5 ECC unbuffered modules, a few hundred for an enterprise SSD, and you have a credible current-gen server from readily available parts at retail prices, without any of the enterprise pricing and procurement hassle.
      - csdreamer7 1y ago
        That was my intention; mATX AM5 parts.
      - BizarroLand 1y ago
        I imagine they would need quite a few servers to replace their current setup. Then there's also the overhead of setting up and maintaining the hardware in their location. It's not just a "solve this problem for ~$2,000 and be done with it". I don't know the actual specs or requirements. Maybe 1 build server is sufficient, but from what I know there's nearly 4,000 apps on FDroid. 1 server might be swamped handling that much overhead in a timely manner.
doublepg23 1y ago
Perhaps the servers run Coreboot / Libreboot?
bayindirh 1y ago
> This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days? This is not always a given. In our virtualization platform, we have upgraded a vendor supplied VM recently, and while it booted, some of the services on it failed to start despite exposing a x86_64v2 + AES CPU to the said VM. Minimum requirements cited "Pentium and Celeron", so it was more than enough. It turned out that one of the services used a single instruction added in a v3 or v4 CPU, and failed to start. We changed the exposed CPU and things have returned to normal. So, their servers might be capable and misconfigured, or the binary might require more that what it states, or something else.
- lucb1e 1y ago
  A developer on the ticket writes: "Our machines run older server grade CPUs, that indeed do not support the newer SSE4_1 and SSSE3"
  - bayindirh 1y ago
    Ooh. They are at least ~15 years old, then. Maybe they have scored on some old, 4 socket Dell R815s. 48 cores ain't that bad for a build server.
    - lucb1e 1y ago
      It's kinda good they use such old systems, as the vast majority of pollution occurs during manufacturing of devices since we usually use them only a handful of years. Iirc the break-even point was somewhere around 25 years, as in, upgrading for energy efficiency then becomes worth it (source: https://wimvanderbauwhede.codeberg.page/articles/frugal-computing/ https://wimvanderbauwhede.codeberg.page/articles/frugal-comp...). 15 goes a long way towards that! On the other hand, I didn't dig very deep into the ticket history now but it sounds like this could have been expected: it broke once already 4 years ago (2021), so maybe planning an upgrade for when this happens again would be good foresight. Then again, volunteers... It's not like I picked up the work as an f-droid user either
      - 1y ago
pclmulqdq 1y ago
I'm not even sure mainline Linux supports machines this old at this point. The cmpxchg16b instruction isn't that old, and I believe it's required now.
- csdreamer7 1y ago
  32 bit Linux is still supported by the kernel and Debian, Arch, and Fedora still supports baseline x86_64. RHEL 8 is still supported and Ubuntu is still baseline x86_64 I believe for commercial distros. Not sure about SuSE.
  - squiffsquiff 1y ago
    > 32 bit Linux is still supported by the kernel and Debian Deprecated for Debian https://www.debian.org/releases/stable/release-notes/issues.en.html#i386-reduced-support https://www.debian.org/releases/stable/release-notes/issues....
    - csdreamer7 1y ago
      >> 32 bit Linux is still supported by the kernel and Debian > Deprecated for Debian > https://www.debian.org/releases/stable/release-notes/issues https://www.debian.org/releases/stable/release-notes/issues.... 32 bit Linux is still supported by the kernel... and... 'Debian, Arch, and Fedora still supports baseline x86_64'. Please do not take things out of context.
- cwillu 1y ago
  CMPXCHG8B is required as of a month or two ago, not 16B (i.e., the version from the 90's is now required) See https://lkml.org/lkml/2025/4/25/409 https://lkml.org/lkml/2025/4/25/409
ignoramous 1y ago
> about to ask people to donate, but they have $80k in their coffers I'd still ask folks to donate. £80k isn't much at all given the time and effort I've seen their volunteers spend on keeping the lights on. From what I recall, they do want to modernize their build infrastructure, but it is as big as an investment they can make. If they had enough in their "coffers", I'm sure they'd feel more confident about it. It isn't like they don't have any other things to fix or address.
- csdreamer7 1y ago
  I would too but do you have a link to them talking about it?
  - deleted 1y ago
    [deleted]
  - ignoramous 1y ago
    They publish weekly reports now on the work they're doing: https://f-droid.org/en/news https://f-droid.org/en/news For example, here's a recent NLNet sponsorship that helped reproducible builds ship (an effort that began in 2023): https://f-droid.org/en/2025/05/21/making-reproducible-builds-visible.html https://f-droid.org/en/2025/05/21/making-reproducible-builds... Their 2025 year in review: https://f-droid.org/en/2025/01/21/a-look-back-at-2024-f-droids-progress-and-whats-coming-in-2025.html https://f-droid.org/en/2025/01/21/a-look-back-at-2024-f-droi...
    - deleted 1y ago
      [deleted]

FirmwareBurner 1y ago

>they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers I would also like to know this.

lupusreal 1y ago
Probably a case of "don't fix it if it ain't broke" keeping old machines in service too long, so now they broke.
- FirmwareBurner 1y ago
  That's like ignoring your 'Check Engine' light because the engine still runs.
pastage 1y ago
I would much rather they spent that on having the devs network and travel, the servers work.
- melodyogonna 1y ago
  Why are the builds failing then?
  - tcfhgj 1y ago
    planned obsolescence by Google
    - shadowgovt 1y ago
      Beginning to use a CPU opcode that is 19 years old doesn't feel like planned obsolescence. if anything, it feels like unplanned obsolescence... "Oh hell what do you mean your CPU doesn't have that opcode no we've just been running the compiler with the default flags and that opcode got added to the default two months ago after a 10-year fight about the possible consequences of changing defaults!" Although I'm a little surprised to learn that the binary itself doesn't have enough information in its header to be able to declare that it needs SSSE3 to be executed; that feels like something that should be statically-analyzed-and-cached to avoid a lot of debugging headaches.
      - tcfhgj 1y ago
        > "Oh hell what do you mean your CPU doesn't have that opcode [...]" hobbyst dev? sure Google? nope
        shadowgovt 1y ago
        Did they make any explicit guarantees that their newly-cut binaries would continue to support 20-year-old architectures? Googlers aren't gods. It's a 100,000-person company; they're as vulnerable to "We didn't really think of that one way or the other" as anyone else. ETA: It's actually not even Google code that changed (directly); Gradle apparently began requiring SSSE3 (https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153 https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153) and Google's toolchain just consumed the new constraint from its upstream. Here, I'm not surprised at all; Google is not the kind of firm that keeps a test-lab of older hardware for every application they ship, so (particularly for their dev tooling) "It worked on my machine" is probably ship-worthy. I bet they don't even have an explicit architecture target for the Android build toolchain beyond the company's default (which is generally "The two most recent versions" of whatever we're talking about).
Perz1val 1y ago
Yeah and everybody was complaining how slow the builds are for years. I really want to know too

Timshel 1y ago

$2-3k ? That’s barely the price of a lower end Threadripper bare cpu not a full Epyc server ???

wongarsu 1y ago
At our supplier $2k would pay for a 1U server with a 16 core 3GHz Epyc 7313P with 32GB RAM, a tiny SSD and non-redundant power. $3k pays for a 1U server with a 32 core 2.6GHz Epyc 7513 with 128GB RAM and 960GB of non-redundant SSD storage (probably fine for build servers). All using server CPUs, since that was easier to find. If you want more cores or more than 3GHz things get considerably more expensive.
- Timshel 1y ago
  Yes but thoose are Zen 3 Milan cpu released in 2021 I believe. Not that they are bad and would not be way better than what they have, just that I though the parent was quite the optimist with his Zen4/Zen5 pricing.
  - wtallis 1y ago
    OP did say "consumer Epyc", so presumably referring to the parts using the AM5 socket. From a quick check on Newegg, it looks like barebones servers for that platform start at under $1000, to which you need to add CPU, RAM, and storage. So a $3000 budget to assemble a low-end Zen4/5 EPYC server is realistic: $570 for the 16-core EPYC 4565P, a few hundred for DDR5 ECC unbuffered modules, a few hundred for an enterprise SSD, and you have a credible current-gen server from readily available parts at retail prices, without any of the enterprise pricing and procurement hassle.
    - csdreamer7 1y ago
      That was my intention; mATX AM5 parts.
    - BizarroLand 1y ago
      I imagine they would need quite a few servers to replace their current setup. Then there's also the overhead of setting up and maintaining the hardware in their location. It's not just a "solve this problem for ~$2,000 and be done with it". I don't know the actual specs or requirements. Maybe 1 build server is sufficient, but from what I know there's nearly 4,000 apps on FDroid. 1 server might be swamped handling that much overhead in a timely manner.

doublepg23 1y ago

Perhaps the servers run Coreboot / Libreboot?

bayindirh 1y ago

> This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days? This is not always a given. In our virtualization platform, we have upgraded a vendor supplied VM recently, and while it booted, some of the services on it failed to start despite exposing a x86_64v2 + AES CPU to the said VM. Minimum requirements cited "Pentium and Celeron", so it was more than enough. It turned out that one of the services used a single instruction added in a v3 or v4 CPU, and failed to start. We changed the exposed CPU and things have returned to normal. So, their servers might be capable and misconfigured, or the binary might require more that what it states, or something else.

lucb1e 1y ago
A developer on the ticket writes: "Our machines run older server grade CPUs, that indeed do not support the newer SSE4_1 and SSSE3"
- bayindirh 1y ago
  Ooh. They are at least ~15 years old, then. Maybe they have scored on some old, 4 socket Dell R815s. 48 cores ain't that bad for a build server.
  - lucb1e 1y ago
    It's kinda good they use such old systems, as the vast majority of pollution occurs during manufacturing of devices since we usually use them only a handful of years. Iirc the break-even point was somewhere around 25 years, as in, upgrading for energy efficiency then becomes worth it (source: https://wimvanderbauwhede.codeberg.page/articles/frugal-computing/ https://wimvanderbauwhede.codeberg.page/articles/frugal-comp...). 15 goes a long way towards that! On the other hand, I didn't dig very deep into the ticket history now but it sounds like this could have been expected: it broke once already 4 years ago (2021), so maybe planning an upgrade for when this happens again would be good foresight. Then again, volunteers... It's not like I picked up the work as an f-droid user either
    - 1y ago

pclmulqdq 1y ago

I'm not even sure mainline Linux supports machines this old at this point. The cmpxchg16b instruction isn't that old, and I believe it's required now.

csdreamer7 1y ago
32 bit Linux is still supported by the kernel and Debian, Arch, and Fedora still supports baseline x86_64. RHEL 8 is still supported and Ubuntu is still baseline x86_64 I believe for commercial distros. Not sure about SuSE.
- squiffsquiff 1y ago
  > 32 bit Linux is still supported by the kernel and Debian Deprecated for Debian https://www.debian.org/releases/stable/release-notes/issues.en.html#i386-reduced-support https://www.debian.org/releases/stable/release-notes/issues....
  - csdreamer7 1y ago
    >> 32 bit Linux is still supported by the kernel and Debian > Deprecated for Debian > https://www.debian.org/releases/stable/release-notes/issues https://www.debian.org/releases/stable/release-notes/issues.... 32 bit Linux is still supported by the kernel... and... 'Debian, Arch, and Fedora still supports baseline x86_64'. Please do not take things out of context.
cwillu 1y ago
CMPXCHG8B is required as of a month or two ago, not 16B (i.e., the version from the 90's is now required) See https://lkml.org/lkml/2025/4/25/409 https://lkml.org/lkml/2025/4/25/409

ignoramous 1y ago

> about to ask people to donate, but they have $80k in their coffers I'd still ask folks to donate. £80k isn't much at all given the time and effort I've seen their volunteers spend on keeping the lights on. From what I recall, they do want to modernize their build infrastructure, but it is as big as an investment they can make. If they had enough in their "coffers", I'm sure they'd feel more confident about it. It isn't like they don't have any other things to fix or address.

csdreamer7 1y ago
I would too but do you have a link to them talking about it?
- deleted 1y ago
  [deleted]
- ignoramous 1y ago
  They publish weekly reports now on the work they're doing: https://f-droid.org/en/news https://f-droid.org/en/news For example, here's a recent NLNet sponsorship that helped reproducible builds ship (an effort that began in 2023): https://f-droid.org/en/2025/05/21/making-reproducible-builds-visible.html https://f-droid.org/en/2025/05/21/making-reproducible-builds... Their 2025 year in review: https://f-droid.org/en/2025/01/21/a-look-back-at-2024-f-droids-progress-and-whats-coming-in-2025.html https://f-droid.org/en/2025/01/21/a-look-back-at-2024-f-droi...
  - deleted 1y ago
    [deleted]