17 ms·

F-Droid build servers can't build modern Android apps due to outdated CPUs

442 points by nativeforks 1y ago

On August 7, 2025, a new build problem started hitting many Android apps on F-Droid. Many Android apps on F-Droid have been unable to publish updates if they use Android Gradle Plugin (AGP) 8.12.0 or Gradle 9.0. The root cause: Google’s new aapt2 binary in AGP 8.12.0 started requiring CPU instructions (SSE4.1, SSSE3) that F-Droid’s build farm hardware doesn’t support. This is similar to a 2021 AGP 4.1.0 issue, but it has returned, and now affects hundreds of apps. As an example, my open-source app MBCompass hit this issue. I downgraded to AGP 8.11.1 with Gradle 8.13 to make it build, but even then, F-Droid failed due to a baseline profile reproducibility bug in AGP. The only workaround was disabling baseline profiles and pushing yet another release. This has led to multiple “maintenance” versions in a short time, confusing users and wasting developer time, just to work around infrastructure issues outside the developer’s control. References: - F-Droid admin issue: https://gitlab.com/fdroid/admin/-/issues/593 https://gitlab.com/fdroid/admin/-/issues/593 - Catima example: https://github.com/CatimaLoyalty/Android/issues/2608 https://github.com/CatimaLoyalty/Android/issues/2608 - MBCompass case: https://github.com/CompassMB/MBCompass/issues/88 https://github.com/CompassMB/MBCompass/issues/88

nativeforks 1y ago
References: F-Droid admin issue: https://gitlab.com/fdroid/admin/-/issues/593 https://gitlab.com/fdroid/admin/-/issues/593 Catima example: https://github.com/CatimaLoyalty/Android/issues/2608 https://github.com/CatimaLoyalty/Android/issues/2608 MBCompass case: https://github.com/CompassMB/MBCompass/issues/88 https://github.com/CompassMB/MBCompass/issues/88
- benrutter 1y ago
  The Catima thread makes FDroid sound like a really difficult commmunity to work with. Although I'm basing this on one person's comment and other people agreeing, not on any knowledge or experience. > But this is like everything with F-Droid: everything always falls on a deaf man's ears. So I would rather not waste more time talking to a brick wall. If I had the feeling it was possible to improve F-Droid by raising issues and trying to discuss how to solve them I wouldn't have left the project out of frustration after years of putting so much time and energy into it.
  - typpilol 1y ago
    I saw that too and was wondering what kind of drama happened in the past
    - noirscape 1y ago
      Very unexciting stuff; it's just your typical long-running FOSS project issues as I understand it. Lead maintainer of F-Droid is entrenched in his ways "cuz it works for me", which leads to stonewalling any attempts to change or improve the F-Droid workflow[0], but since he holds the keys to the kingdom (and the name recognition prevents forks), they keep him around. Everyone else then tries to work around him and through a mixture of emotional appealing, downplaying the importance of certain patches and doing everything in very tiny steps then try to improve things. It's an extremely mentally draining process that's prone to burnout on the part of the contributors, which eventually boils over and then some people quit... which might start a conversation on why nobody wants to contribute to the FOSS project. That conversation inevitably goes nowhere because the people you'd want to hold that conversation with are so fed up with how bad things have gotten that they'd rather just see the person causing trouble removed entirely. (Which may be the correct course of action, but this is an argument often given without putting forward a proper replacement/considering how the project might move forward without them. Some larger organizations can handle the removal of a core maintainer, most can't.) Rinse and repeat that cycle every five years or so. F-Droid isn't at all unique in this regard, and most people are willing to ignore it "because it's free, you shouldn't have any expectations". Any long running FOSS project that has significant infrastructure behind it will at some point have this issue and most haven't had a great history at handling it, since the bus factor of a lot of major FOSS projects is still pretty much one point five people. (As in, one actual maintainer and one guy that knows what levers to pull to seize control if the maintainer actually gets hit by a bus, with the warning that they stop being 0.5 of a bus factor and become 0 if they do that while the maintainer is still around.) [0]: Basically the inverse of https://xkcd.com/1172/ https://xkcd.com/1172/
      - Tade0 1y ago
        This is the sort of stuff that makes me want to pursue FIRE. There's so much good that could be done, but isn't because people need to be making money for someone else. Then again who is to say that I would be a better custodian than this guy?
trenchpilgrim 1y ago
I thought SSE 4.1 dates back to 2008 or so?
- nativeforks 1y ago
  Yes, SSE4.1 and SSSE3 have been introduced in ~2006. The F-Droid build server still uses that to build modern and some of the most popular FOSS apps.
- starkparker 1y ago
  The build servers appear to be AMD Opteron G3s, which only support part of SSE4 (SSE4a). Full SSE4 support didn't land until Bulldozer (late 2011).
  - karlgkk 1y ago
    I appreciate that this is a volunteer project, but my back of the hand math suggests that if they upgraded to a $300 laptop using a 10nm intel chip, it would pay for itself in power usage within a few years. Actually, probably less, considering an i3-N305 has more cores and substantially faster single thread. And yes, you could get that cost down easily.
    - wtallis 1y ago
      Yes, a used laptop would be an upgrade from server hardware of that vintage, in performance and probably in reliability. If they're really using hardware that old, that is itself a big red flag that F-Droid's infrastructure is fragile and unmaintained. (A server that old might not have any SSDs, which would be insane for a software build server unless it was doing everything in RAM.)
      - eimrine 1y ago
        There are some more possible virtues except of performance and probably-reliability.
      - johnklos 1y ago
benrutter 1y ago
This is pretty concerning, especially as FDroid is by far the largest non-google android store at the moment, something that I feel is really needed, regardless of your feelings about google. Does anyone know of plans to resolve this? Will FDroid update their servers? Are google looking into rolling back the requirement? (this last one sounds unlikely)
- dannyw 1y ago
  I agree it’s a bit concerning but please keep in mind F-Droid is a volunteer-run community project. Especially with some EU countries moving to open source software, it would be nice to see some public funding for projects like F-Droid.
  - nativeforks 1y ago
    > Nice to see some public funding for projects like F-Droid Definitely, SSE4.1 instruction set based CPU, for building apps in 2025, No way!!
  - benrutter 1y ago
    Hope I didn't come across as criticising FDroid here- It seems sucky to have build requirements change under your feet. It's just I think that FDroid is an important project, and hope this doesn't block their progress.
  - berkes 1y ago
    > please keep in mind F-Droid is a volunteer-run community project. To, me, that's the worrying part. Not that it's ran by volunteers. But that all there's left between a full-on "tech monopoly" or hegemony, and a free internet, is small bands of underfunded volunteers. Opposition to market dominance and monopolies by multibillion multinationals shouldn't just come from a few volunteers. If that's the case, just roll over and give up; the cause is lost. (As I've done, hence my defaitism) Aside from that: it being "a volunteer ran community" shouldn't be put as an excuse for why it's in trouble/has poor UX/is hard to use/is behind/etc. It should be a killer feature. Something that makes it more resilient/better attuned/easier/earlier adopting/etc.
wtallis 1y ago
It seems quite implausible that F-Droid is actually running on hardware that predates those instruction set extensions. They're seeing wider adoption by default these days precisely because hardware which doesn't support them is getting very rare, especially in servers still in production use. Are you sure this isn't simply a matter of F-Droid using VMs that are configured to not expose those instructions as supported?
- deleted 1y ago
  [deleted]
do_not_redeem 1y ago
[flagged]
- qart 1y ago
  I have it installed. But the only thing I get updates for is Obtainium itself. There's no catalogue of apps, so I haven't installed anything via Obtainium.
  - UberFly 1y ago
    I would uninstall. Author and app seem sketchy.
    - RedComet 1y ago
      Will you elaborate?
  - user070223 1y ago
    Try Discoverium
  - wishfish 1y ago
    Here's a catalog of apps from the Obtainium wiki. https://apps.obtainium.imranr.dev/ https://apps.obtainium.imranr.dev/ They put the disclaimer on top that this list is not meant as an app store or catalog. It's meant for apps with somewhat complex requirements for adding to Obtainium. But it serves well as a catalog since most of the major open source apps are listed.
- oguz-ismail 1y ago
  How is this not another middleman (with a political banner in its README no less)?
  - spacemule 1y ago
    [flagged]
mjevans 1y ago
https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions#Later_versions https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions#Late... Even my last, crazy long in the tooth, desktop supported this and it lived to almost 10 years old before being replaced. However at the same time, not even offering a fallback path in non-assembly?
- wtallis 1y ago
  > However at the same time, not even offering a fallback path in non-assembly? There's probably not any hand-written assembly at issue here, just a compiler told to target x86_64-v2. Among others, RHEL 9 and derivatives were built with such options. (RHEL 10 bumped up the minimum spec again to x86_64-v3, allowing use of AVX.)
  - shadowgovt 1y ago
    Or even, a compiler told to target nothing in particular, and a default finally toggled over from "Oh, we're 'targeting x86'? So CPUs from the early 2000s then" to "Oh, we're 'targeting x86'? So CPUs from the mid-2010s then."
- vocx2tx 1y ago
  Looking at the issue their builders seem to be Opterons G3 (K10?)[0] [0] https://en.wikipedia.org/wiki/AMD_10h https://en.wikipedia.org/wiki/AMD_10h
  - ozim 1y ago
    I can imagine this has to be like that as they usually get $1500 per month in donations. You could buy a newer one but I guess they have other stuff they have to pay for.
    - a012 1y ago
      For $500 you can get a decent refurbished server on ebay that supports those “new” extensions
exabrial 1y ago
Man, Android could have been way cooler if it actually used real virtual machines, or at least the JVMs.
- pjmlp 1y ago
  I stood by Oracle, because in the long term as it has been proven, Android is Google's J++, and Kotlin became Google's C#. Hardly any different from what was in the genesis of .NET. Nowadays they support up to Java 17 LTS, a subset only as usual, mostly because Android was being left behind accessing the Java ecosystem on Maven central. And even though now ART is updatable via PlayStore, all the way down to Android 12, they see no need to move beyond Java 17 subset, until most likely they start again missing on key libraries that decided to adopt newer features. Also stuff like Panama, Loom, Vector, Valhala (if ever), don't count them ever being supported on ART. At least, they managed to push into mainstream the closest idea of OSes like Oberon, Inferno, Java OS and co, where regardless of what think about the superiotity of UNIX clones, here they have to contend themselves with a managed userspace, something that Microsoft failed at with Longhorn, Singularity and Midori due to their internal politics.
  - aembleton 1y ago
    > Kotlin became Google's C# Are Google buying Jetbrains?
    - pjmlp 1y ago
      They almost could, after all they have outsourced most of the Android tooling efforts to JetBrains, given that Android Studio is mostly InteliJ + Clion, and Kotlin is the main Android language nowadays. Also Kotlin Foundation is mostly JetBrains and Google employees.
  - exabrial 1y ago
    >Panama, Loom, Vector, Valhala (if ever), don't count them ever being supported on ART This is pretty sad IMHO, as Java17 was a true turning point. Java21 is icing and Java25 is an incredible refinement with some fascinating new features that are really well thought out.
o11c 1y ago
Note: the underlying blame here fundamentally belongs to whoever built AGP / Gradle with non-universal flags, then distributed it. It's fine to ship binaries with hard-coded cpu flag requirements if you control the universe, but otherwise not, especially if you are in an ecosystem where you make it hard for users to rebuild everything from source.
- IshKebab 1y ago
  Exactly. Everything should be compiled to target i386. /s (should be obvious but probably not for this audience)
  - pabs3 1y ago
    They should be compiled for the CPU baseline of the ABI they are using, and check if newer instructions are available before using them. This is what Debian does, so they can have maximum hardware support. https://wiki.debian.org/InstructionSelection https://wiki.debian.org/InstructionSelection
    - IshKebab 1y ago
      Why? There's nothing wrong with having minimum requirements beyond that. They don't have to use Debian's policy, and multiversioning adds enough complexity that basically nobody does it (I've only ever seen it used in video codecs).
- userbinator 1y ago
  control the universe Guess what the company behind Android wants to do...
kijin 1y ago
Requiring (supposedly) universally available CPU instructions is one thing. Starting to require it in a minor version update (8.11.1 -> 8.12.0) is a whole different thing. What the heck happened to semantic versioning? We can't even trust patch updates anymore these days. The version numbers might as well be git commit IDs.
nicman23 1y ago
wtf they cannot be still running opterons. it was to be that they are using qemu with g3 as a cpu profile.. right?
shmerl 1y ago
Can't cross compilation help for that? The CPU compiling doesn't need to match the target.
- a99c43f2d565504 1y ago
  It's not the target that is now requiring new instructions, but one of the components in the build tools.
  - shmerl 1y ago
    I see.
jchw 1y ago
Apparently it was fixed upstream by Google? https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153 https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153 Not sure how long it will take to get resolved but that thread seems reassuring even if there isn't a direct source that it was fixed.
- nativeforks 1y ago
  Still haven't. Currently, most of the devs aren't aware of this underlying issue!
- AnssiH 1y ago
  It is not fixed. In the thread you linked to people are confusing a typo correction ("mas fixed" => "was fixed") as a claim about this new issue being fixed. The one that was fixed is this similar old issue from years ago: https://issuetracker.google.com/issues/172048751 https://issuetracker.google.com/issues/172048751
  - jchw 1y ago
    Oh, that's unfortunate, very confusing thread.
roywashere 1y ago
This is sort of like a bug I hit last year when the mysql docker container suddenly started requiring x86-64-v2 after a patch level upgrade and failed to start: https://github.com/docker-library/mysql/issues/1055 https://github.com/docker-library/mysql/issues/1055
userbinator 1y ago
Fortunately the source code is available: https://android.googlesource.com/platform/frameworks/base/+/refs/heads/master/tools/aapt2/ https://android.googlesource.com/platform/frameworks/base/+/... If I had the time, I'd try to compile a binary of it that will run on Win95 just to give my fuckings to the planned obsolescence crowd.
- maxloh 1y ago
  There is no point for Google to push planned obsolescence on the PC or server space. They don't have a market there.
  - userbinator 1y ago
    It does benefit them to make it harder for competitors.
    - maxloh 1y ago
      When you mention "competitors," what industries or markets are you referring to? No one would write Android apps on a Chromebook, and making it harder to do so would only reduce the incentive for companies to develop Android apps. How could Google benefit from pushing a newer instruction set standard on Windows and macOS?
      - heavyset_go 1y ago
        The one moderately popular competitor is the project in the OP that is suffering directly from this upstream change.
        maxloh 1y ago
        While your perspective makes some sense, it's highly improbable. It's unlikely that Google was aware of F-Droid's infrastructure specs, or its inability to fix the issue in advance. It seems you're suggesting a very specific, targeted attack.
wpm 1y ago
I’ve got an old Ivy Bridge-EP Dell workstation they can borrow goddamn SSE4.1 is nearly old enough to drink.
- rpcope1 1y ago
  Yeah I was kind of shocked too. Core 2 could do both of those instruction sets. A used Dell Precision can be had for very little and probably would be grossly more efficient than whatever they're using.
- jeroenhd 1y ago
  SSE4.1 can legally buy lightly alcoholic beverages in various European countries already. Next year, it can buy strong spirits. Using AMD hardware that's "only" 13 years old can also cause this problem, though.
yjftsjthsd-h 1y ago
> Google’s new aapt2 binary in AGP 8.12.0 Given F-Droid's emphasis on isolating and protecting their build environment, I'm kind of surprised that they're just using upstream binaries and not building from source.
- jraph 1y ago
  Relatedly, we don't really have any up to date free software build of the Android SDK AFAIK. To build Android apps, we all rely on the Google binaries, which are non-free. https://forum.f-droid.org/t/call-for-help-making-free-software-builds-of-the-android-sdk/4685 https://forum.f-droid.org/t/call-for-help-making-free-softwa...
micw 1y ago
As far as I can see, sse4.1 has been introduced in CPUs in 2011. That's more than 10 years ago. I wonder why such old servers are still in use. I'd assume that a modern CPU would do the same amount of work with a fraction of energy so that it does not even make economical sense to run such outdated hardware. Does anyone know the numbers of build servers and the specs?
- heavyset_go 1y ago
  Hardware after the first couple of generations of x86_64 muliticore processors are perfectly capable machines to use as servers, even for tasks you want to put off to a build farm.
- LukeShu 1y ago
  I was going to say that I assume that the reason for such old CPUs is the ability to use Canoeboot/GNU Boot. But you absolutely can put an SSE4.2 CPU in a KGPE-D16 motherboard. So IDK.
- cjaackie 1y ago
  I haven’t seen the real answer that I suspect here - the build servers are that one dual socket AMD board which runs open firmware and has no ME/PSP .
- adrian_b 1y ago
  It has been introduced in Intel Penryn, in November 2007. However the AMD CPUs did not implement it until Bulldozer, in mid 2011. While they lacked the many additional instructions provided by Bulldozer, also including AVX and FMA, for many applications the older Opteron CPUs were significantly faster than the Bulldozer-based CPUs, so there were few incentives for upgrading them, before the launch of AMD Epyc in mid 2017. SSE 4.1 is a cut point in supporting old CPUs for many software packages, because older CPUs have a very high overhead for divergent computations (e.g. with if ... else ...) inside loops that are parallelized with SIMD instructions.
- ffaser5gxlsll 1y ago
  On the server side, probably not, but I'd like to point out that old hardware is not uncommon, and it's going to be more and more likely as time passes especially in the desktop space. I was hit by this scenario in the 2000s with an old desktop pc I had, also in the 10ys range, I was using just for boring stuff and random browsing, which was old, but perfectly adequate for the purpose. With time programs got rebuilt with some version of SSE it didn't support. When even firefox switched to the new instruction set, I had to essentially trash a perfectly working desktop pc as it became useless for the purpose.
hulitu 1y ago
> The root cause: Google’s new aapt2 binary in AGP 8.12.0 started requiring CPU instructions (SSE4.1, SSSE3) that F-Droid’s build farm hardware doesn’t support. Very intelligent move from Google. Now you can't compile "Hello World" without SSE4.1, SSSE3. /s Are there any X86 tablets with Android ?
- vardump 1y ago
  There are very few 17+ years old build servers at this point. Or laptops and desktops for that matter.
karteum 1y ago
I don't fully understand: aren't gradle and aapt2 open-source ? If you want to build buildroot or openwrt, the first thing it will do is compiling your own toolchain (rather than reusing the one from your distro) so that it can lead to predictable results. I would have the same rationale for f-droid : why not compile the whole toolchain from source rather than using a binary gradle/aapt2 that uses unsupported instructions?
- a2128 1y ago
  SDK binaries provided by Google are still used, see https://forum.f-droid.org/t/call-for-help-making-free-software-builds-of-the-android-sdk/4685 https://forum.f-droid.org/t/call-for-help-making-free-softwa...
- mid-kid 1y ago
  I agree, this should be the case, but Gradle specifically relies on downloading prebuilt java libraries and such to build itself and anything you build with it, and sometimes these have prebuilt native code inside. Unlike buildroot and any linux distribution, there's no metadata to figure out how to build each library, and the process for them is different between each library (no standards like make, autotools and cmake), so building the gradle ecosystem from source is very tedious and difficult.
  - 1oooqooq 1y ago
    having worked with both mvn and gradle, i always have a good chuckle when i hear about npm "supply chain" hacks.
BoredPositron 1y ago
>> This has led to multiple “maintenance” versions in a short time, confusing users and wasting developer time, just to work around infrastructure issues outside the developer’s control. What an entitled conclusion.
deleted 1y ago
[deleted]
edgan 1y ago
That F-Droid even requires to do the build is one of the reasons I created Discoverium. https://github.com/cygnusx-1-org/Discoverium/ https://github.com/cygnusx-1-org/Discoverium/
- devrandoom 1y ago
  So I should take a binary from a random stranger because trust me bro?
  - edgan 1y ago
    It is a modified version of Obtainium. You get it from the author via GitHub.
    - devrandoom 1y ago
      It's still a binary from a stranger. You don't know from which source it was built.
- MYEUHD 1y ago
  That F-Droid requires to do the build ensures all apps provided by F-Droid are free software (as in freedom) and proven to be buildable by someone other than the app developer
  - edgan 1y ago
    The issue is more complicated than that.
    - yjftsjthsd-h 1y ago
      How so?
    - twodave 1y ago
      Do you mean the overall issue or that F-Droid’s guarantees are arguable? The guarantees may not be the whole discussion, but for many they are the most relevant piece. Edit: or perhaps you mean that isn’t the only way to provide such guarantees, which is the implication I got reading your other replies.
ivanjermakov 1y ago
Why not recompile aapt2 to correct target? It seems to be source available. https://android.googlesource.com/platform/frameworks/base/+/refs/heads/master/tools/aapt2/ https://android.googlesource.com/platform/frameworks/base/+/...
- munchlax 1y ago
  Have you tried building AOSP from available sources? Binaries everywhere. Tried to rebuild some of them with the available sources and noped the f out because that breaks the build so bad it's ridiculous.
  - zoobab 1y ago
    "Binaries everywhere" So much for "Open Source"
    - gbin 1y ago
      Everything is open source, if you can read assembly ;)
      - bluGill 1y ago
        Machine code. Assembly is higher level. since data and instructions can be mixed machine code is harder to decode - that might be a byte of data or an instruction. Mel would have [ab]used this fact to make his programs work. It is worse on x86 where instructions are not fixed length but even on arm you can run into problems at times
        snake42 1y ago
        You can always lift machine code to assembly. Its a 1 to 1 process.
        bluGill 1y ago
        No you cannot. While it is 1 to 1, you still need to know where to start as if you start at the wrong place data will be interrupted as an asm instruction and things will decode legally - but invalidly. It is worse on CISC (like x86) where instructions are different length and so you can jump to the middle byte of a long instruction and decode a shorter instruction. (RISC sometimes starts to get CISC features as they add more instructions as well). If the code was written reasonably you can usually find enough clues to figure out where to start decoding and thus get a reasonable assembly output, but even then you often need to restart the decoding several times because the decoder can get confused at function boundaries depending on what other data gets embedded and where it is embedded. Be glad self modifying code was going out of style in the 1980's and is mostly a memory today as that will kill any disassembly attempts. All the other tricks that Mel used (https://en.wikipedia.org/wiki/The_Story_of_Mel https://en.wikipedia.org/wiki/The_Story_of_Mel) also make your attempts at lifting machine code to assembly impossible.
fancythat 1y ago
I don't know how much servers are they using or server specs besides ancient Opterons, but how is this even an issue in 2025? On Hetnzer (not affiliated), at this moment, i7-8700 (AVX2 supported) with 128 GB RAM, 2x1 TB SSD and 1 Gbit uplink costs 42.48 eur per month, VAT included, in their server auction section. What are we missing here, besides that build farm was left to decay?
- WesolyKubeczek 1y ago
  Either they want to run on ideologically pure hardware too, without pesky management bits in it (or even indeed UEFI), or they are just "it used to work perfectly" guys. In the former case, I fail to see how ME or its absence is relevant to building Android apps, which they do using Google-provided binaries that have even more opportunity to inject naughty bits into the software. In the latter case, I better forget they exist.
  - fancythat 1y ago
    I agree with you. Unfortunately usually, the simplest explanation is often the truth, so they just probably ignored this issue, until it surfaced up.
    - WesolyKubeczek 1y ago
      In other words, > they are just "it used to work perfectly" guys.
  - bill_mcgonigle 1y ago
    Well if you wanted to compromise F-Droid you could target their build server's ME or a cloud vm's hypervisor. To do a supply-chain attack on Google's SDK would be much more expensive and less likely to succeed. Google isn't going to be the attacker. The recent attack on AMI/Gigabyte's ME shows how a zero-day can bootkit a UEFI server quite easily. There are newer Coreboot boards than Opteron, though. Some embedded-oriented BIOS'es let you fuse out the ME. You are warned this is permanent and irreversible. F-Droid likely has upgrade options even in the all-open scenario.
yyyk 1y ago
Their servers are so old, even an entirely different architecture emulating x86_64 would still see a performance increase... So there's no OSS argument here - they could even buy a Talos, have no closed firmware, and still see a performance increase with emulation. If they don't care about the firmware, there are plenty of very cheap x86 options which are still more modern.
- DrewADesign 1y ago
  > Their servers are so old When I read this, pop culture has trained me to expect an insult, like: “Their servers are so old, they sat next to Ben Franklin in kindergarten.”
  - queenkjuul 1y ago
    My home server is so old, it gets its driver's license next year
- nalinidash 1y ago
  https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 > For all those following this, we have the budget to buy new hardware, what we lack is a skilled hoster who has committed to physically hosting a new bare metal box for us.
Arech 1y ago
This is super annoying how SW vendors forcefully deprecate good enough hardware. Genuinely hate that, as Mozilla has deprived me from Firefox's translation feature because of that.
- sparkie 1y ago
  OTOH, if software wants to take advantage of modern features, it becomes hell to maintain if you have to have flags for every possible feature supported by CPUID. It's also unreasonable to expect maintainers to package dozens of builds for software that is unlikely to be used. There's some guidelines[1][2] for developers to follow for a reasonable set of features, where they only need to manage ~4 variants. In this proposal the lowest set of features include SSE4.1, which is basically includes nearly any x86_64 CPU from the past 15 years. In theory we could use a modern CPU to compile the 4 variants and ship them all in a FatELF, so we only need to distribute one set of binaries. This of course would be completely impractical if we had to support every possible CPU's distinct features, and the binaries would be huge. [1]:https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.html https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.h... [2]:https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels https://en.wikipedia.org/wiki/X86-64#Microarchitecture_level...
  - Arech 1y ago
    In most cases (and this was the case of Mozilla I referred to) it's only a matter of compiling code that already have all support necessary. They are using some upstream component that works perfectly fine on my architecture. They just decided to drop it, because they could.
    - sparkie 1y ago
      It's not only your own software, but also its dependencies. The link above is for glibc, and is specifically addressing incompatibliy issues between different software. Unless you are going to compile your own glibc (for example, doing Linux From Scratch), you're going to depend on features shipped by someone else. In this case that means either baseline, with no SIMD support at all, or level A, which includes SSE4.1. It makes no sense for developers to keep maintaining software for 20 year old CPUs when they can't test it.
tetris11 1y ago
I'm a bit lost in this thread, but I've written up what I know for other dummies like me Aapt2 is an x86_64 standalone binary used to build android APKs for various CPU targets Previous versions of it used a simpler instruction set, but the new version requires an extra SIMD instruction SSE4. A lot of CPUs after 2008 support this, but not F-droid's current server farm?
- its-summertime 1y ago
  > Our machines run older server grade CPUs So a bit of both of older hardware, and not-matched-with-consumer-featureset hardware. I'd imagine some server hardware vendors supported SSE4 way earlier than most, and some probably supported it way later than most too.
deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
csdreamer7 1y ago
This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days? https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level https://developers.redhat.com/blog/2021/01/05/building-red-h... Think of how much faster their servers would be with one of those Epyc consumer cpus. I was about to ask people to donate, but they have $80k in their coffers. I realize their budget is only $17,000 a year, but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers as they are around under $2k under budget. If they have a fleet of these old servers I imagine a Zen5 one can replace at least a few of them and consume far less power and space. https://opencollective.com/f-droid#category-BUDGET https://opencollective.com/f-droid#category-BUDGET Not sure if this includes their Librapay donations either: https://liberapay.com/F-Droid-Data/donate https://liberapay.com/F-Droid-Data/donate
- FirmwareBurner 1y ago
  >they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers I would also like to know this.
  - lupusreal 1y ago
    Probably a case of "don't fix it if it ain't broke" keeping old machines in service too long, so now they broke.
    - FirmwareBurner 1y ago
      That's like ignoring your 'Check Engine' light because the engine still runs.
  - pastage 1y ago
    I would much rather they spent that on having the devs network and travel, the servers work.
    - 1y ago
CommenterPerson 1y ago
Non-hacker here. The title says "modern". I don't need modern, have a 10 year old phone, can I still get the occasional simple app from F-Droid? I upped my (small) monthly contribution. Hope more people contribute, and also work to build public support. Also, for developers .. please include old fashioned credit cards as a payment method. I'd like to contribute but don't want to sign up for yet another payment method.
andix 1y ago
Do I get it correctly, that they run their build infrastructure on at least 15 year old hardware?
nativeforks 1y ago
There are even some "Unknown problem" on IzzyOnDroid repo for app publishing, even ensuring reproducible build, izzy says >>Not necessarily "your fault" – baseline often has such issues: https://github.com/CompassMB/MBCompass/issues/90 https://github.com/CompassMB/MBCompass/issues/90 Seems like he is talking about the developer being responsible for that also!
- SylvieLorxu 1y ago
  IzzyOnDroid can publish updates even if it's not reproducible, this is not an "app publishing" issue at all. IzzyOnDroid can deal with AGP 8.12 fine. Also "not necessarily your fault" means "probably not your fault", the opposite of "your fault"
bluGill 1y ago
QEMU static on linux supports automatic emulating of missing instructions. Depending on details that I haven't figured out it can be a lot slower running this way or close enough to native. I have got that working, but it was a pain and I don't remember what was needed (most of the work was done by someone else, but I helped)
solodolo 1y ago
[dead]
solodolo 1y ago
[dead]
Sent1n3l 1y ago
[flagged]
- guappa 1y ago
  As a user, i'm glad when devs use old tools so that my battery has a chance of lasting the whole day and my apps don't take 10 seconds just to open.
  - cnst 1y ago
    > As a user, i'm glad when devs use old tools so that my battery has a chance of lasting the whole day and my apps don't take 10 seconds just to open. Yup, same here! The story is as old as time, and the examples are plentiful. First Slashdot, then Reddit, then now GitHub, all became far-far-far slower and less usable, once they've been "improved" by the folk engaging in the resume-driven development: Why is GitHub UI getting slower? - https://news.ycombinator.com/item?id=44799861 https://news.ycombinator.com/item?id=44799861 - Aug 2025 (115 comments) I am, too, as a user, quite pleased that F-Droid is keeping it cool and reliable for the actual users.
    - guappa 1y ago
      On github besides the slowness the number of clicks is increasing! I now have to click a "..." thing that opens a menu that only has 1 item in it to see the test build. And of course that (proprietary) tool follows the trend so I need another number of clicks to finally get to the logs and see what failed.
- AshamedCaptain 1y ago
  A shitton of people, not to mention including all F-Droid users, would take FOSS ideology over new fangled bloated "non-decrepit" development tools _any day_. But in any case, this is false dichotomy, and likely exaggerated one to begin with.
- mid-kid 1y ago
  I think it's extremely useful to have more strict requirements on how programs are built, to make sure that developers don't do stupid things that makes code harder for others to compile. The tools in question in OP should be easy to build from source and not rely on the host's architecture, to be usable on platforms like ARM and RISCV. It's clear that in the android ecosystem, people don't care, so F-Droid can't do miracles (the java/gradle ecosystem is just really bad at this), but this would not happen if the build tools had proper build recipes themselves.
mandown2308 1y ago
On the other hand, we have "personal" data centers for AI and mining farms for crypto.
flykespice 1y ago
> The root cause: Google’s new aapt2 binary in AGP 8.12.0 started requiring CPU instructions (SSE4.1, SSSE3) that F-Droid’s build farm hardware doesn’t support. This is similar to a 2021 AGP 4.1.0 issue, but it has returned, and now affects hundreds of apps. I don't know why they have enabled modern CPU flags for a simple intermediary tool that compiles the apk resources files, it was so unneccesary Welp there goes my plans on savaging an old laptop to build my android apps.
deleted 1y ago
[deleted]
deleted 1y ago
[deleted]
jdbdnxjdhe 1y ago
I don't get the issue, binary target is completely independent from host target on all but the most basic setups
deleted 1y ago
[deleted]
skyzouwdev 1y ago
That’s a tough one. It’s ironic that the very platform meant to keep apps open and accessible is now bottlenecked by outdated hardware. Upgrading the build farm CPUs seems like the obvious fix, but I’m guessing funding and coordination make it less straightforward. In the meantime, forcing devs to downgrade AGP or strip baseline profiles just to ship feels like a pretty big friction point. Long term, I wonder if F-Droid could offer an optional “modern build lane” with newer hardware, even if it means fewer guarantees of full reproducibility at first. That might at least keep apps from stalling out entirely.
- 1970-01-01 1y ago
  I've said this before, but I'll say it again. Running on donations is not a viable strategy for any long-term goal. FOSS needs to passively invest the donations. That is a viable long-term strategy. Now when things like this happen, it becomes a major line item moment, and not a limp-along situation, with yet another WE NEED YOUR HELP banner blocking off 1/2 their website.
1970-01-01 1y ago
Put another way, Google is requiring you to have 65nm Intel chips. 2009-ish.
OldfieldFund 1y ago
I think this might give Google some ideas...
shrubble 1y ago
Is it the CPUs or the compilers? Or possibly a CI/CD runner that has to run something that can’t run on these CPUs?
1vuio0pswjnm7 1y ago
Perhaps there should be more than one F-Droid For example, if they published their exact setup for building Android apps so others could replicate it How many Android users compile the own apps they use Perhaps increasing that number would be a goal worth pursuing
SylvieLorxu 1y ago
Might be worth noting that several devs have suggested users use IzzyOnDroid instead. Due to IzzyOnDroid distributing official upstream builds (after scanning), they're not dependent on any build server. Although they do have build servers for the purpose of confirming upstream APKs match the source code using reproducible builds, but those are separate processes that don't block each other (unlike F-Droid's rather monolithic structure). IzzyOnDroid has been faster with updates than F-Droid for years, releasing app updates within 24 hours for most cases.
rasz 1y ago
> (SSE4.1, SSSE3) This means their build infrastructure burns excessive amounts of power being run by volunteers in basements/homelabs on vintage museum grade (15 year old Opterons/Phenoms) hardware. Gamers have been there 14 years ago with 'No Man's Sky' being the first big game requiring SSE 4.1 for no particular reason.
pabs3 1y ago
Google should be compiling for the CPU baseline of the ABI their binaries are for, and then check if newer instructions are available before using them. Just like glibc and other projects do. The Debian documentation for this mentions tools to do this, like SIMDe and GCC/clang FMV. https://wiki.debian.org/InstructionSelection https://wiki.debian.org/InstructionSelection
- wtallis 1y ago
  Am I missing something, or does SIMDe only help for cases where a program is using instruction intrinsics, and it doesn't do anything to address cases where the compiler decides to use SIMD as a result of auto-vectorization?
  - pabs3 1y ago
    Thats correct, but usually compilers don't do that if you use the CPU baseline.
    - wtallis 1y ago
      > but usually compilers don't do that if you use the CPU baseline That's a problem that people are trying to solve by not using an ancient CPU baseline. Do you have a reasonable proposal for how else we should enable widespread use of hardware functionality that's less than two decades old?
      - pabs3 1y ago
        The compiler could auto-enable function multi-versioning (FMV) for functions where auto-vectorisation gets triggered. At program start, FMV checks which instructions are available and updates function pointers to the right functions. Things like glibc use FMV to switch things like memcpy to SIMD-optimised versions.
nicman23 1y ago
now that i think of it, is this because they want to run without blobs and without ME/ PSP ?
eighthave 1y ago
The limiting factor for upgrading our buildserver is finding a trusted, skilled sysadmin to physically install, setup and maintain new hardware at the high level of security that is needed for a release buildserver for a project like F-Droid. It also needs to be in a trusted physical location. Hetzner is definitely not that.
doppelgunner 1y ago
Imagine explaining to your app that it cannot compile because the server thinks Snapdragon 800 is still the future. F-Droid is basically the grandparent who insists their flip phone works just fine.
nalinidash 1y ago
https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 > For all those following this, we have the budget to buy new hardware, what we lack is a skilled hoster who has committed to physically hosting a new bare metal box for us.

442 points by nativeforks 1y ago

nativeforks 1y ago

References: F-Droid admin issue: https://gitlab.com/fdroid/admin/-/issues/593 https://gitlab.com/fdroid/admin/-/issues/593 Catima example: https://github.com/CatimaLoyalty/Android/issues/2608 https://github.com/CatimaLoyalty/Android/issues/2608 MBCompass case: https://github.com/CompassMB/MBCompass/issues/88 https://github.com/CompassMB/MBCompass/issues/88

benrutter 1y ago
The Catima thread makes FDroid sound like a really difficult commmunity to work with. Although I'm basing this on one person's comment and other people agreeing, not on any knowledge or experience. > But this is like everything with F-Droid: everything always falls on a deaf man's ears. So I would rather not waste more time talking to a brick wall. If I had the feeling it was possible to improve F-Droid by raising issues and trying to discuss how to solve them I wouldn't have left the project out of frustration after years of putting so much time and energy into it.
- typpilol 1y ago
  I saw that too and was wondering what kind of drama happened in the past
  - noirscape 1y ago
    Very unexciting stuff; it's just your typical long-running FOSS project issues as I understand it. Lead maintainer of F-Droid is entrenched in his ways "cuz it works for me", which leads to stonewalling any attempts to change or improve the F-Droid workflow[0], but since he holds the keys to the kingdom (and the name recognition prevents forks), they keep him around. Everyone else then tries to work around him and through a mixture of emotional appealing, downplaying the importance of certain patches and doing everything in very tiny steps then try to improve things. It's an extremely mentally draining process that's prone to burnout on the part of the contributors, which eventually boils over and then some people quit... which might start a conversation on why nobody wants to contribute to the FOSS project. That conversation inevitably goes nowhere because the people you'd want to hold that conversation with are so fed up with how bad things have gotten that they'd rather just see the person causing trouble removed entirely. (Which may be the correct course of action, but this is an argument often given without putting forward a proper replacement/considering how the project might move forward without them. Some larger organizations can handle the removal of a core maintainer, most can't.) Rinse and repeat that cycle every five years or so. F-Droid isn't at all unique in this regard, and most people are willing to ignore it "because it's free, you shouldn't have any expectations". Any long running FOSS project that has significant infrastructure behind it will at some point have this issue and most haven't had a great history at handling it, since the bus factor of a lot of major FOSS projects is still pretty much one point five people. (As in, one actual maintainer and one guy that knows what levers to pull to seize control if the maintainer actually gets hit by a bus, with the warning that they stop being 0.5 of a bus factor and become 0 if they do that while the maintainer is still around.) [0]: Basically the inverse of https://xkcd.com/1172/ https://xkcd.com/1172/
    - Tade0 1y ago
      This is the sort of stuff that makes me want to pursue FIRE. There's so much good that could be done, but isn't because people need to be making money for someone else. Then again who is to say that I would be a better custodian than this guy?

trenchpilgrim 1y ago

I thought SSE 4.1 dates back to 2008 or so?

nativeforks 1y ago
Yes, SSE4.1 and SSSE3 have been introduced in ~2006. The F-Droid build server still uses that to build modern and some of the most popular FOSS apps.
starkparker 1y ago
The build servers appear to be AMD Opteron G3s, which only support part of SSE4 (SSE4a). Full SSE4 support didn't land until Bulldozer (late 2011).
- karlgkk 1y ago
  I appreciate that this is a volunteer project, but my back of the hand math suggests that if they upgraded to a $300 laptop using a 10nm intel chip, it would pay for itself in power usage within a few years. Actually, probably less, considering an i3-N305 has more cores and substantially faster single thread. And yes, you could get that cost down easily.
  - wtallis 1y ago
    Yes, a used laptop would be an upgrade from server hardware of that vintage, in performance and probably in reliability. If they're really using hardware that old, that is itself a big red flag that F-Droid's infrastructure is fragile and unmaintained. (A server that old might not have any SSDs, which would be insane for a software build server unless it was doing everything in RAM.)
    - eimrine 1y ago
      There are some more possible virtues except of performance and probably-reliability.
    - johnklos 1y ago

benrutter 1y ago

This is pretty concerning, especially as FDroid is by far the largest non-google android store at the moment, something that I feel is really needed, regardless of your feelings about google. Does anyone know of plans to resolve this? Will FDroid update their servers? Are google looking into rolling back the requirement? (this last one sounds unlikely)

dannyw 1y ago
I agree it’s a bit concerning but please keep in mind F-Droid is a volunteer-run community project. Especially with some EU countries moving to open source software, it would be nice to see some public funding for projects like F-Droid.
- nativeforks 1y ago
  > Nice to see some public funding for projects like F-Droid Definitely, SSE4.1 instruction set based CPU, for building apps in 2025, No way!!
- benrutter 1y ago
  Hope I didn't come across as criticising FDroid here- It seems sucky to have build requirements change under your feet. It's just I think that FDroid is an important project, and hope this doesn't block their progress.
- berkes 1y ago
  > please keep in mind F-Droid is a volunteer-run community project. To, me, that's the worrying part. Not that it's ran by volunteers. But that all there's left between a full-on "tech monopoly" or hegemony, and a free internet, is small bands of underfunded volunteers. Opposition to market dominance and monopolies by multibillion multinationals shouldn't just come from a few volunteers. If that's the case, just roll over and give up; the cause is lost. (As I've done, hence my defaitism) Aside from that: it being "a volunteer ran community" shouldn't be put as an excuse for why it's in trouble/has poor UX/is hard to use/is behind/etc. It should be a killer feature. Something that makes it more resilient/better attuned/easier/earlier adopting/etc.

wtallis 1y ago

It seems quite implausible that F-Droid is actually running on hardware that predates those instruction set extensions. They're seeing wider adoption by default these days precisely because hardware which doesn't support them is getting very rare, especially in servers still in production use. Are you sure this isn't simply a matter of F-Droid using VMs that are configured to not expose those instructions as supported?

deleted 1y ago
[deleted]

do_not_redeem 1y ago

[flagged]

qart 1y ago
I have it installed. But the only thing I get updates for is Obtainium itself. There's no catalogue of apps, so I haven't installed anything via Obtainium.
- UberFly 1y ago
  I would uninstall. Author and app seem sketchy.
  - RedComet 1y ago
    Will you elaborate?
- user070223 1y ago
  Try Discoverium
- wishfish 1y ago
  Here's a catalog of apps from the Obtainium wiki. https://apps.obtainium.imranr.dev/ https://apps.obtainium.imranr.dev/ They put the disclaimer on top that this list is not meant as an app store or catalog. It's meant for apps with somewhat complex requirements for adding to Obtainium. But it serves well as a catalog since most of the major open source apps are listed.
oguz-ismail 1y ago
How is this not another middleman (with a political banner in its README no less)?
- spacemule 1y ago
  [flagged]

mjevans 1y ago

https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions#Later_versions https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions#Late... Even my last, crazy long in the tooth, desktop supported this and it lived to almost 10 years old before being replaced. However at the same time, not even offering a fallback path in non-assembly?

wtallis 1y ago
> However at the same time, not even offering a fallback path in non-assembly? There's probably not any hand-written assembly at issue here, just a compiler told to target x86_64-v2. Among others, RHEL 9 and derivatives were built with such options. (RHEL 10 bumped up the minimum spec again to x86_64-v3, allowing use of AVX.)
- shadowgovt 1y ago
  Or even, a compiler told to target nothing in particular, and a default finally toggled over from "Oh, we're 'targeting x86'? So CPUs from the early 2000s then" to "Oh, we're 'targeting x86'? So CPUs from the mid-2010s then."
vocx2tx 1y ago
Looking at the issue their builders seem to be Opterons G3 (K10?)[0] [0] https://en.wikipedia.org/wiki/AMD_10h https://en.wikipedia.org/wiki/AMD_10h
- ozim 1y ago
  I can imagine this has to be like that as they usually get $1500 per month in donations. You could buy a newer one but I guess they have other stuff they have to pay for.
  - a012 1y ago
    For $500 you can get a decent refurbished server on ebay that supports those “new” extensions

exabrial 1y ago

Man, Android could have been way cooler if it actually used real virtual machines, or at least the JVMs.

pjmlp 1y ago
I stood by Oracle, because in the long term as it has been proven, Android is Google's J++, and Kotlin became Google's C#. Hardly any different from what was in the genesis of .NET. Nowadays they support up to Java 17 LTS, a subset only as usual, mostly because Android was being left behind accessing the Java ecosystem on Maven central. And even though now ART is updatable via PlayStore, all the way down to Android 12, they see no need to move beyond Java 17 subset, until most likely they start again missing on key libraries that decided to adopt newer features. Also stuff like Panama, Loom, Vector, Valhala (if ever), don't count them ever being supported on ART. At least, they managed to push into mainstream the closest idea of OSes like Oberon, Inferno, Java OS and co, where regardless of what think about the superiotity of UNIX clones, here they have to contend themselves with a managed userspace, something that Microsoft failed at with Longhorn, Singularity and Midori due to their internal politics.
- aembleton 1y ago
  > Kotlin became Google's C# Are Google buying Jetbrains?
  - pjmlp 1y ago
    They almost could, after all they have outsourced most of the Android tooling efforts to JetBrains, given that Android Studio is mostly InteliJ + Clion, and Kotlin is the main Android language nowadays. Also Kotlin Foundation is mostly JetBrains and Google employees.
- exabrial 1y ago
  >Panama, Loom, Vector, Valhala (if ever), don't count them ever being supported on ART This is pretty sad IMHO, as Java17 was a true turning point. Java21 is icing and Java25 is an incredible refinement with some fascinating new features that are really well thought out.

o11c 1y ago

Note: the underlying blame here fundamentally belongs to whoever built AGP / Gradle with non-universal flags, then distributed it. It's fine to ship binaries with hard-coded cpu flag requirements if you control the universe, but otherwise not, especially if you are in an ecosystem where you make it hard for users to rebuild everything from source.

IshKebab 1y ago
Exactly. Everything should be compiled to target i386. /s (should be obvious but probably not for this audience)
- pabs3 1y ago
  They should be compiled for the CPU baseline of the ABI they are using, and check if newer instructions are available before using them. This is what Debian does, so they can have maximum hardware support. https://wiki.debian.org/InstructionSelection https://wiki.debian.org/InstructionSelection
  - IshKebab 1y ago
    Why? There's nothing wrong with having minimum requirements beyond that. They don't have to use Debian's policy, and multiversioning adds enough complexity that basically nobody does it (I've only ever seen it used in video codecs).
userbinator 1y ago
control the universe Guess what the company behind Android wants to do...

kijin 1y ago

Requiring (supposedly) universally available CPU instructions is one thing. Starting to require it in a minor version update (8.11.1 -> 8.12.0) is a whole different thing. What the heck happened to semantic versioning? We can't even trust patch updates anymore these days. The version numbers might as well be git commit IDs.

nicman23 1y ago

wtf they cannot be still running opterons. it was to be that they are using qemu with g3 as a cpu profile.. right?

shmerl 1y ago

Can't cross compilation help for that? The CPU compiling doesn't need to match the target.

a99c43f2d565504 1y ago
It's not the target that is now requiring new instructions, but one of the components in the build tools.
- shmerl 1y ago
  I see.

jchw 1y ago

Apparently it was fixed upstream by Google? https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153 https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153 Not sure how long it will take to get resolved but that thread seems reassuring even if there isn't a direct source that it was fixed.

nativeforks 1y ago
Still haven't. Currently, most of the devs aren't aware of this underlying issue!
AnssiH 1y ago
It is not fixed. In the thread you linked to people are confusing a typo correction ("mas fixed" => "was fixed") as a claim about this new issue being fixed. The one that was fixed is this similar old issue from years ago: https://issuetracker.google.com/issues/172048751 https://issuetracker.google.com/issues/172048751
- jchw 1y ago
  Oh, that's unfortunate, very confusing thread.

roywashere 1y ago

This is sort of like a bug I hit last year when the mysql docker container suddenly started requiring x86-64-v2 after a patch level upgrade and failed to start: https://github.com/docker-library/mysql/issues/1055 https://github.com/docker-library/mysql/issues/1055

userbinator 1y ago

Fortunately the source code is available: https://android.googlesource.com/platform/frameworks/base/+/refs/heads/master/tools/aapt2/ https://android.googlesource.com/platform/frameworks/base/+/... If I had the time, I'd try to compile a binary of it that will run on Win95 just to give my fuckings to the planned obsolescence crowd.

maxloh 1y ago
There is no point for Google to push planned obsolescence on the PC or server space. They don't have a market there.
- userbinator 1y ago
  It does benefit them to make it harder for competitors.
  - maxloh 1y ago
    When you mention "competitors," what industries or markets are you referring to? No one would write Android apps on a Chromebook, and making it harder to do so would only reduce the incentive for companies to develop Android apps. How could Google benefit from pushing a newer instruction set standard on Windows and macOS?
    - heavyset_go 1y ago
      The one moderately popular competitor is the project in the OP that is suffering directly from this upstream change.
      - maxloh 1y ago
        While your perspective makes some sense, it's highly improbable. It's unlikely that Google was aware of F-Droid's infrastructure specs, or its inability to fix the issue in advance. It seems you're suggesting a very specific, targeted attack.

wpm 1y ago

I’ve got an old Ivy Bridge-EP Dell workstation they can borrow goddamn SSE4.1 is nearly old enough to drink.

rpcope1 1y ago
Yeah I was kind of shocked too. Core 2 could do both of those instruction sets. A used Dell Precision can be had for very little and probably would be grossly more efficient than whatever they're using.
jeroenhd 1y ago
SSE4.1 can legally buy lightly alcoholic beverages in various European countries already. Next year, it can buy strong spirits. Using AMD hardware that's "only" 13 years old can also cause this problem, though.

yjftsjthsd-h 1y ago

> Google’s new aapt2 binary in AGP 8.12.0 Given F-Droid's emphasis on isolating and protecting their build environment, I'm kind of surprised that they're just using upstream binaries and not building from source.

jraph 1y ago
Relatedly, we don't really have any up to date free software build of the Android SDK AFAIK. To build Android apps, we all rely on the Google binaries, which are non-free. https://forum.f-droid.org/t/call-for-help-making-free-software-builds-of-the-android-sdk/4685 https://forum.f-droid.org/t/call-for-help-making-free-softwa...

micw 1y ago

As far as I can see, sse4.1 has been introduced in CPUs in 2011. That's more than 10 years ago. I wonder why such old servers are still in use. I'd assume that a modern CPU would do the same amount of work with a fraction of energy so that it does not even make economical sense to run such outdated hardware. Does anyone know the numbers of build servers and the specs?

heavyset_go 1y ago
Hardware after the first couple of generations of x86_64 muliticore processors are perfectly capable machines to use as servers, even for tasks you want to put off to a build farm.
LukeShu 1y ago
I was going to say that I assume that the reason for such old CPUs is the ability to use Canoeboot/GNU Boot. But you absolutely can put an SSE4.2 CPU in a KGPE-D16 motherboard. So IDK.
cjaackie 1y ago
I haven’t seen the real answer that I suspect here - the build servers are that one dual socket AMD board which runs open firmware and has no ME/PSP .
adrian_b 1y ago
It has been introduced in Intel Penryn, in November 2007. However the AMD CPUs did not implement it until Bulldozer, in mid 2011. While they lacked the many additional instructions provided by Bulldozer, also including AVX and FMA, for many applications the older Opteron CPUs were significantly faster than the Bulldozer-based CPUs, so there were few incentives for upgrading them, before the launch of AMD Epyc in mid 2017. SSE 4.1 is a cut point in supporting old CPUs for many software packages, because older CPUs have a very high overhead for divergent computations (e.g. with if ... else ...) inside loops that are parallelized with SIMD instructions.
ffaser5gxlsll 1y ago
On the server side, probably not, but I'd like to point out that old hardware is not uncommon, and it's going to be more and more likely as time passes especially in the desktop space. I was hit by this scenario in the 2000s with an old desktop pc I had, also in the 10ys range, I was using just for boring stuff and random browsing, which was old, but perfectly adequate for the purpose. With time programs got rebuilt with some version of SSE it didn't support. When even firefox switched to the new instruction set, I had to essentially trash a perfectly working desktop pc as it became useless for the purpose.

hulitu 1y ago

> The root cause: Google’s new aapt2 binary in AGP 8.12.0 started requiring CPU instructions (SSE4.1, SSSE3) that F-Droid’s build farm hardware doesn’t support. Very intelligent move from Google. Now you can't compile "Hello World" without SSE4.1, SSSE3. /s Are there any X86 tablets with Android ?

vardump 1y ago
There are very few 17+ years old build servers at this point. Or laptops and desktops for that matter.

karteum 1y ago

I don't fully understand: aren't gradle and aapt2 open-source ? If you want to build buildroot or openwrt, the first thing it will do is compiling your own toolchain (rather than reusing the one from your distro) so that it can lead to predictable results. I would have the same rationale for f-droid : why not compile the whole toolchain from source rather than using a binary gradle/aapt2 that uses unsupported instructions?

a2128 1y ago
SDK binaries provided by Google are still used, see https://forum.f-droid.org/t/call-for-help-making-free-software-builds-of-the-android-sdk/4685 https://forum.f-droid.org/t/call-for-help-making-free-softwa...
mid-kid 1y ago
I agree, this should be the case, but Gradle specifically relies on downloading prebuilt java libraries and such to build itself and anything you build with it, and sometimes these have prebuilt native code inside. Unlike buildroot and any linux distribution, there's no metadata to figure out how to build each library, and the process for them is different between each library (no standards like make, autotools and cmake), so building the gradle ecosystem from source is very tedious and difficult.
- 1oooqooq 1y ago
  having worked with both mvn and gradle, i always have a good chuckle when i hear about npm "supply chain" hacks.

BoredPositron 1y ago

>> This has led to multiple “maintenance” versions in a short time, confusing users and wasting developer time, just to work around infrastructure issues outside the developer’s control. What an entitled conclusion.

deleted 1y ago

[deleted]

edgan 1y ago

That F-Droid even requires to do the build is one of the reasons I created Discoverium. https://github.com/cygnusx-1-org/Discoverium/ https://github.com/cygnusx-1-org/Discoverium/

devrandoom 1y ago
So I should take a binary from a random stranger because trust me bro?
- edgan 1y ago
  It is a modified version of Obtainium. You get it from the author via GitHub.
  - devrandoom 1y ago
    It's still a binary from a stranger. You don't know from which source it was built.
MYEUHD 1y ago
That F-Droid requires to do the build ensures all apps provided by F-Droid are free software (as in freedom) and proven to be buildable by someone other than the app developer
- edgan 1y ago
  The issue is more complicated than that.
  - yjftsjthsd-h 1y ago
    How so?
  - twodave 1y ago
    Do you mean the overall issue or that F-Droid’s guarantees are arguable? The guarantees may not be the whole discussion, but for many they are the most relevant piece. Edit: or perhaps you mean that isn’t the only way to provide such guarantees, which is the implication I got reading your other replies.

ivanjermakov 1y ago

Why not recompile aapt2 to correct target? It seems to be source available. https://android.googlesource.com/platform/frameworks/base/+/refs/heads/master/tools/aapt2/ https://android.googlesource.com/platform/frameworks/base/+/...

munchlax 1y ago
Have you tried building AOSP from available sources? Binaries everywhere. Tried to rebuild some of them with the available sources and noped the f out because that breaks the build so bad it's ridiculous.
- zoobab 1y ago
  "Binaries everywhere" So much for "Open Source"
  - gbin 1y ago
    Everything is open source, if you can read assembly ;)
    - bluGill 1y ago
      Machine code. Assembly is higher level. since data and instructions can be mixed machine code is harder to decode - that might be a byte of data or an instruction. Mel would have [ab]used this fact to make his programs work. It is worse on x86 where instructions are not fixed length but even on arm you can run into problems at times
      - snake42 1y ago
        You can always lift machine code to assembly. Its a 1 to 1 process.
        bluGill 1y ago
        No you cannot. While it is 1 to 1, you still need to know where to start as if you start at the wrong place data will be interrupted as an asm instruction and things will decode legally - but invalidly. It is worse on CISC (like x86) where instructions are different length and so you can jump to the middle byte of a long instruction and decode a shorter instruction. (RISC sometimes starts to get CISC features as they add more instructions as well). If the code was written reasonably you can usually find enough clues to figure out where to start decoding and thus get a reasonable assembly output, but even then you often need to restart the decoding several times because the decoder can get confused at function boundaries depending on what other data gets embedded and where it is embedded. Be glad self modifying code was going out of style in the 1980's and is mostly a memory today as that will kill any disassembly attempts. All the other tricks that Mel used (https://en.wikipedia.org/wiki/The_Story_of_Mel https://en.wikipedia.org/wiki/The_Story_of_Mel) also make your attempts at lifting machine code to assembly impossible.

fancythat 1y ago

I don't know how much servers are they using or server specs besides ancient Opterons, but how is this even an issue in 2025? On Hetnzer (not affiliated), at this moment, i7-8700 (AVX2 supported) with 128 GB RAM, 2x1 TB SSD and 1 Gbit uplink costs 42.48 eur per month, VAT included, in their server auction section. What are we missing here, besides that build farm was left to decay?

WesolyKubeczek 1y ago
Either they want to run on ideologically pure hardware too, without pesky management bits in it (or even indeed UEFI), or they are just "it used to work perfectly" guys. In the former case, I fail to see how ME or its absence is relevant to building Android apps, which they do using Google-provided binaries that have even more opportunity to inject naughty bits into the software. In the latter case, I better forget they exist.
- fancythat 1y ago
  I agree with you. Unfortunately usually, the simplest explanation is often the truth, so they just probably ignored this issue, until it surfaced up.
  - WesolyKubeczek 1y ago
    In other words, > they are just "it used to work perfectly" guys.
- bill_mcgonigle 1y ago
  Well if you wanted to compromise F-Droid you could target their build server's ME or a cloud vm's hypervisor. To do a supply-chain attack on Google's SDK would be much more expensive and less likely to succeed. Google isn't going to be the attacker. The recent attack on AMI/Gigabyte's ME shows how a zero-day can bootkit a UEFI server quite easily. There are newer Coreboot boards than Opteron, though. Some embedded-oriented BIOS'es let you fuse out the ME. You are warned this is permanent and irreversible. F-Droid likely has upgrade options even in the all-open scenario.

yyyk 1y ago

Their servers are so old, even an entirely different architecture emulating x86_64 would still see a performance increase... So there's no OSS argument here - they could even buy a Talos, have no closed firmware, and still see a performance increase with emulation. If they don't care about the firmware, there are plenty of very cheap x86 options which are still more modern.

DrewADesign 1y ago
> Their servers are so old When I read this, pop culture has trained me to expect an insult, like: “Their servers are so old, they sat next to Ben Franklin in kindergarten.”
- queenkjuul 1y ago
  My home server is so old, it gets its driver's license next year
nalinidash 1y ago
https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 > For all those following this, we have the budget to buy new hardware, what we lack is a skilled hoster who has committed to physically hosting a new bare metal box for us.

Arech 1y ago

This is super annoying how SW vendors forcefully deprecate good enough hardware. Genuinely hate that, as Mozilla has deprived me from Firefox's translation feature because of that.

sparkie 1y ago
OTOH, if software wants to take advantage of modern features, it becomes hell to maintain if you have to have flags for every possible feature supported by CPUID. It's also unreasonable to expect maintainers to package dozens of builds for software that is unlikely to be used. There's some guidelines[1][2] for developers to follow for a reasonable set of features, where they only need to manage ~4 variants. In this proposal the lowest set of features include SSE4.1, which is basically includes nearly any x86_64 CPU from the past 15 years. In theory we could use a modern CPU to compile the 4 variants and ship them all in a FatELF, so we only need to distribute one set of binaries. This of course would be completely impractical if we had to support every possible CPU's distinct features, and the binaries would be huge. [1]:https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.html https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.h... [2]:https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels https://en.wikipedia.org/wiki/X86-64#Microarchitecture_level...
- Arech 1y ago
  In most cases (and this was the case of Mozilla I referred to) it's only a matter of compiling code that already have all support necessary. They are using some upstream component that works perfectly fine on my architecture. They just decided to drop it, because they could.
  - sparkie 1y ago
    It's not only your own software, but also its dependencies. The link above is for glibc, and is specifically addressing incompatibliy issues between different software. Unless you are going to compile your own glibc (for example, doing Linux From Scratch), you're going to depend on features shipped by someone else. In this case that means either baseline, with no SIMD support at all, or level A, which includes SSE4.1. It makes no sense for developers to keep maintaining software for 20 year old CPUs when they can't test it.

tetris11 1y ago

I'm a bit lost in this thread, but I've written up what I know for other dummies like me Aapt2 is an x86_64 standalone binary used to build android APKs for various CPU targets Previous versions of it used a simpler instruction set, but the new version requires an extra SIMD instruction SSE4. A lot of CPUs after 2008 support this, but not F-droid's current server farm?

its-summertime 1y ago
> Our machines run older server grade CPUs So a bit of both of older hardware, and not-matched-with-consumer-featureset hardware. I'd imagine some server hardware vendors supported SSE4 way earlier than most, and some probably supported it way later than most too.

deleted 1y ago

[deleted]

deleted 1y ago

[deleted]

csdreamer7 1y ago

This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days? https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level https://developers.redhat.com/blog/2021/01/05/building-red-h... Think of how much faster their servers would be with one of those Epyc consumer cpus. I was about to ask people to donate, but they have $80k in their coffers. I realize their budget is only $17,000 a year, but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers as they are around under $2k under budget. If they have a fleet of these old servers I imagine a Zen5 one can replace at least a few of them and consume far less power and space. https://opencollective.com/f-droid#category-BUDGET https://opencollective.com/f-droid#category-BUDGET Not sure if this includes their Librapay donations either: https://liberapay.com/F-Droid-Data/donate https://liberapay.com/F-Droid-Data/donate

FirmwareBurner 1y ago
>they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers I would also like to know this.
- lupusreal 1y ago
  Probably a case of "don't fix it if it ain't broke" keeping old machines in service too long, so now they broke.
  - FirmwareBurner 1y ago
    That's like ignoring your 'Check Engine' light because the engine still runs.
- pastage 1y ago
  I would much rather they spent that on having the devs network and travel, the servers work.
  - 1y ago

CommenterPerson 1y ago

Non-hacker here. The title says "modern". I don't need modern, have a 10 year old phone, can I still get the occasional simple app from F-Droid? I upped my (small) monthly contribution. Hope more people contribute, and also work to build public support. Also, for developers .. please include old fashioned credit cards as a payment method. I'd like to contribute but don't want to sign up for yet another payment method.

andix 1y ago

Do I get it correctly, that they run their build infrastructure on at least 15 year old hardware?

nativeforks 1y ago

There are even some "Unknown problem" on IzzyOnDroid repo for app publishing, even ensuring reproducible build, izzy says >>Not necessarily "your fault" – baseline often has such issues: https://github.com/CompassMB/MBCompass/issues/90 https://github.com/CompassMB/MBCompass/issues/90 Seems like he is talking about the developer being responsible for that also!

SylvieLorxu 1y ago
IzzyOnDroid can publish updates even if it's not reproducible, this is not an "app publishing" issue at all. IzzyOnDroid can deal with AGP 8.12 fine. Also "not necessarily your fault" means "probably not your fault", the opposite of "your fault"

bluGill 1y ago

QEMU static on linux supports automatic emulating of missing instructions. Depending on details that I haven't figured out it can be a lot slower running this way or close enough to native. I have got that working, but it was a pain and I don't remember what was needed (most of the work was done by someone else, but I helped)

solodolo 1y ago

[dead]

solodolo 1y ago

[dead]

Sent1n3l 1y ago

[flagged]

guappa 1y ago
As a user, i'm glad when devs use old tools so that my battery has a chance of lasting the whole day and my apps don't take 10 seconds just to open.
- cnst 1y ago
  > As a user, i'm glad when devs use old tools so that my battery has a chance of lasting the whole day and my apps don't take 10 seconds just to open. Yup, same here! The story is as old as time, and the examples are plentiful. First Slashdot, then Reddit, then now GitHub, all became far-far-far slower and less usable, once they've been "improved" by the folk engaging in the resume-driven development: Why is GitHub UI getting slower? - https://news.ycombinator.com/item?id=44799861 https://news.ycombinator.com/item?id=44799861 - Aug 2025 (115 comments) I am, too, as a user, quite pleased that F-Droid is keeping it cool and reliable for the actual users.
  - guappa 1y ago
    On github besides the slowness the number of clicks is increasing! I now have to click a "..." thing that opens a menu that only has 1 item in it to see the test build. And of course that (proprietary) tool follows the trend so I need another number of clicks to finally get to the logs and see what failed.
AshamedCaptain 1y ago
A shitton of people, not to mention including all F-Droid users, would take FOSS ideology over new fangled bloated "non-decrepit" development tools _any day_. But in any case, this is false dichotomy, and likely exaggerated one to begin with.
mid-kid 1y ago
I think it's extremely useful to have more strict requirements on how programs are built, to make sure that developers don't do stupid things that makes code harder for others to compile. The tools in question in OP should be easy to build from source and not rely on the host's architecture, to be usable on platforms like ARM and RISCV. It's clear that in the android ecosystem, people don't care, so F-Droid can't do miracles (the java/gradle ecosystem is just really bad at this), but this would not happen if the build tools had proper build recipes themselves.

mandown2308 1y ago

On the other hand, we have "personal" data centers for AI and mining farms for crypto.

flykespice 1y ago

> The root cause: Google’s new aapt2 binary in AGP 8.12.0 started requiring CPU instructions (SSE4.1, SSSE3) that F-Droid’s build farm hardware doesn’t support. This is similar to a 2021 AGP 4.1.0 issue, but it has returned, and now affects hundreds of apps. I don't know why they have enabled modern CPU flags for a simple intermediary tool that compiles the apk resources files, it was so unneccesary Welp there goes my plans on savaging an old laptop to build my android apps.

deleted 1y ago

[deleted]

deleted 1y ago

[deleted]

jdbdnxjdhe 1y ago

I don't get the issue, binary target is completely independent from host target on all but the most basic setups

deleted 1y ago

[deleted]

skyzouwdev 1y ago

That’s a tough one. It’s ironic that the very platform meant to keep apps open and accessible is now bottlenecked by outdated hardware. Upgrading the build farm CPUs seems like the obvious fix, but I’m guessing funding and coordination make it less straightforward. In the meantime, forcing devs to downgrade AGP or strip baseline profiles just to ship feels like a pretty big friction point. Long term, I wonder if F-Droid could offer an optional “modern build lane” with newer hardware, even if it means fewer guarantees of full reproducibility at first. That might at least keep apps from stalling out entirely.

1970-01-01 1y ago
I've said this before, but I'll say it again. Running on donations is not a viable strategy for any long-term goal. FOSS needs to passively invest the donations. That is a viable long-term strategy. Now when things like this happen, it becomes a major line item moment, and not a limp-along situation, with yet another WE NEED YOUR HELP banner blocking off 1/2 their website.

1970-01-01 1y ago

Put another way, Google is requiring you to have 65nm Intel chips. 2009-ish.

OldfieldFund 1y ago

I think this might give Google some ideas...

shrubble 1y ago

Is it the CPUs or the compilers? Or possibly a CI/CD runner that has to run something that can’t run on these CPUs?

1vuio0pswjnm7 1y ago

Perhaps there should be more than one F-Droid For example, if they published their exact setup for building Android apps so others could replicate it How many Android users compile the own apps they use Perhaps increasing that number would be a goal worth pursuing

SylvieLorxu 1y ago

Might be worth noting that several devs have suggested users use IzzyOnDroid instead. Due to IzzyOnDroid distributing official upstream builds (after scanning), they're not dependent on any build server. Although they do have build servers for the purpose of confirming upstream APKs match the source code using reproducible builds, but those are separate processes that don't block each other (unlike F-Droid's rather monolithic structure). IzzyOnDroid has been faster with updates than F-Droid for years, releasing app updates within 24 hours for most cases.

rasz 1y ago

> (SSE4.1, SSSE3) This means their build infrastructure burns excessive amounts of power being run by volunteers in basements/homelabs on vintage museum grade (15 year old Opterons/Phenoms) hardware. Gamers have been there 14 years ago with 'No Man's Sky' being the first big game requiring SSE 4.1 for no particular reason.

pabs3 1y ago

Google should be compiling for the CPU baseline of the ABI their binaries are for, and then check if newer instructions are available before using them. Just like glibc and other projects do. The Debian documentation for this mentions tools to do this, like SIMDe and GCC/clang FMV. https://wiki.debian.org/InstructionSelection https://wiki.debian.org/InstructionSelection

wtallis 1y ago
Am I missing something, or does SIMDe only help for cases where a program is using instruction intrinsics, and it doesn't do anything to address cases where the compiler decides to use SIMD as a result of auto-vectorization?
- pabs3 1y ago
  Thats correct, but usually compilers don't do that if you use the CPU baseline.
  - wtallis 1y ago
    > but usually compilers don't do that if you use the CPU baseline That's a problem that people are trying to solve by not using an ancient CPU baseline. Do you have a reasonable proposal for how else we should enable widespread use of hardware functionality that's less than two decades old?
    - pabs3 1y ago
      The compiler could auto-enable function multi-versioning (FMV) for functions where auto-vectorisation gets triggered. At program start, FMV checks which instructions are available and updates function pointers to the right functions. Things like glibc use FMV to switch things like memcpy to SIMD-optimised versions.

nicman23 1y ago

now that i think of it, is this because they want to run without blobs and without ME/ PSP ?

eighthave 1y ago

The limiting factor for upgrading our buildserver is finding a trusted, skilled sysadmin to physically install, setup and maintain new hardware at the high level of security that is needed for a release buildserver for a project like F-Droid. It also needs to be in a trusted physical location. Hetzner is definitely not that.

doppelgunner 1y ago

Imagine explaining to your app that it cannot compile because the server thinks Snapdragon 800 is still the future. F-Droid is basically the grandparent who insists their flip phone works just fine.

nalinidash 1y ago

https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 https://gitlab.com/fdroid/admin/-/issues/593#note_2690000843 > For all those following this, we have the budget to buy new hardware, what we lack is a skilled hoster who has committed to physically hosting a new bare metal box for us.