8 ms·
Several years ago I did want to fix something in Firefox, and it was far easier to just patch a few bytes in the binary (I think it might've been a string const
by userbinator 7y ago
Several years ago I did want to fix something in Firefox, and it was far easier to just patch a few bytes in the binary (I think it might've been a string constant) than go through the whole build process with all the resources that takes, including trying to find where in the original source to actually make the change, and then still have a totally different binary than the one you wanted to fix, with other (bug-causing) changes you didn't want due to the different environment you built it in.
Of course, if you try to fix/revert something that was purposely broken or removed for some Mozilla reason, it is not going to accept your changes either.
One of the ideals of open-source was to make it easy to modify and share software. For a lot of the larger projects, I feel like that has missed the mark. I write (mostly small and always portable) software, yet the "build bloat" is a huge turn-off. On the other hand, I've had good experience with the smaller open-source projects, those that need a minimal amount of tooling and resources to build. Some examples I can think of include libpng, zlib, and even OpenSSL.
- geofft 7y agoThis is IMO one of the under-celebrated benefits of Linux distros. You should be able to (e.g.) sudo apt-get build-dep firefox; apt-get source firefox; cd firefox[TAB]; debuild and have yourself a working package. Whatever dependencies it needs are all installed in a standardized way.
- panpanna 7y agoYou should not forgive dependency bloat just because Linux makes it a bit easier to handle.
- jerf 7y agoI'm not sure that's an applicable term here. I see "dependency bloat" as an application having a lot of unnecessary dependencies, like when I start a basic project skeleton in Hot Javascript Framework of the Week and get 1,000 dependencies loaded in (which, in npm land, is a conservative number; I felt bad about saying 10,000 but from what I've seen it isn't necessarily far off the mark). But when you have a project that really does touch every XWindows library there is, and really does need to know about every major image format, and really does speak every major network protocol, and runs a full JIT interpreter and compiler, and integrates with every OS security mechanism there is, and uses every build protection library it can find, and on and on... well, better those dependencies than re-implementation of all that from scratch. Fortunately, there aren't very many things nowadays other than a browser that has that characteristic. (Office suites, maybe. QT and other similar things considered as a whole, but not necessarily what any given user of QT will need.)
- panpanna 7y agoI was indeed thinking of npm when I wrote that comment but we are starting to see similar problems in some large rust projects. It's simply too easy to add a new dependency. Anyway, some poster child FOSS projects in C++ take hours to compile due to dependency bloat.
- EpicEng 7y agoWhich FF dependencies do you think are unnecessary?
- spookthesunset 7y agoAnd the alternative to depending on those codebases is.....? Write your own? Don't build out the feature? What, exactly do you propose?
- est31 7y agoSometimes you depend on stuff you don't even use. Or your dependency depends on it. Or the stuff only gives you little advantages. E.g. in Rust the failure crate saves you maybe 10 lines of code or such. But it has tons of dependencies. Or the lalrpop library crate depends on things only the binary crate needs because both are named the same way. You end up downloading and building things you don't even need. This stuff is fixable, the problem is that people don't care much about it.
- NilsIRL 7y agoNix aims to fix this. https://nixos.org/ https://nixos.org/
- secure 7y agoOn Debian (and derivatives), check out https://github.com/Debian/pk4 https://github.com/Debian/pk4 for this purpose!
- geofft 7y agoWow, hadn't seen that before, thanks!
- BeetleB 7y agoNot 100% sure I follow you, but isn't that what Gentoo provides? I can easily tell Portage to fetch the sources. If I want to make a change, I create a patch with my changes, and put it in the patches directory where the Firefox ebuilds live. I don't even think you have to edit the ebuild to add a patch - just name the patch properly and drop it in the right place.
- geofft 7y agoYes, basically any Linux distro or equivalent product (like Homebrew) provides this. I'm drawing a distinction between having a distro and putting the pieces together yourself, not between any two distros. (I'm mostly familiar with Debian but I think it is just about as easy everywhere.)
- fluffything 7y agoI don't think that whether a program is small or large is the issue. The problem here is that most build systems suck. I work on large-ish scale Rust software (~250kLOC), we ship to all major OSes and many minor ones: android, macos, ios, windows, linux, freebsd, openbsd, ... we build binaries for x86 (32 and 64 bit), arm6, arm7, arm8, arm64, ppc, ppc64le, sparc64, s390x, wasm, ... and many platforms (e.g. glibc and musl on linux). We did pretty much all of this with a wrapper over Rust's cargo build system called "cross", and to build for a target, with just had to write `cross build --target=...` instead of `cargo build`. If you want to run your tests for a target, you can actually write `cross test --target` and your tests will run. For a long time, that was it. Pretty much zero setup, cross compile ready-to-publish binaries to any target, run the tests of your program on that target, all of it by just writing cross instead of cargo. It just worked. The way cross works is quite simple. It comes with a bunch of maintained Docker images, containing cross-compilation toolchains for C, C++, ..., right versions of kernel headers, glibc/musl, some utilities, etc. These also come with qemu-user or qemu-system, depending on the target. So cross would use the Rust compiler and the C toolchains and appropriate libraries to cross-compile on your HOST, which is quite fast, and when running the tests, it would often just run them under qemu-user, which is quite fast as well. For Windows targets it uses wine, for Android targets it install the SDK, NDK, emulators, for iOS you can use the iOS simulator that come with Xcode, etc. Cross is by far, the damn best build tool I've ever used. Zero configuration, and it not only worked, but it did a lot. Want to target sparc64? No problem: cross test --target=sparc64-unknown-linux-gnu compiles your project runs your test, and you are good to go. The reason we stopped using it wasn't that our project got big. It was that, at some point, the build tool wasn't enough anymore. We added more dependencies, and needed to modify the Docker container slightly. cross lets you reuse their containers to build your own adding whatever you need, so we did that for a while, and we still do. The MacOS support wasn't really there, so we were running macos and cross natively. There is another Rust build tool for macos, that is like cross, but supports iOS, so you could run `dinghy test --device=...` and it would cross-compile to the device, copy the tests to the device, and run your tests on an aarch64 device. That meant having to ssh into the right machine, install the tool in your account, and do all that, just to see if a test fails. At some point we wanted to start supporting other targets, like FreeBSD and OpenBSD. Both operating system are, in my experience, very hard to cross-compile to, providing no cross-compilation toolchains for Linux at least. So with cross we were compiling to FreeBSD under qemu-system which was super slow. When we started supporting OpenBSD, we ended up spawning native FreeBSD and OpenBSD VMs, and compiling directly from those hosts. So how are we doing all of this right now ? Well we initially forked cross, which is written in Rust, and hacked some specific modifications in, but at this point we just wrote our own crappy duct-tapish python script, that calls cross when it can, and does something else when it cannot. This is where the barrier of entry for new developers appears. We aren't using cargo, cross, or anything else that they are familiar with. We are using an undocumented, super hacked, unfamiliar solution, that calls other build systems. Most of the time, `./b.py build` or `./b.py test --target=xyz` is all they need to know, but if they need to modify our build system, the curve is steep, because they now need to learn about all the horrible problems it solves and the super hacky ways in which it solves them. So why do Firefox, Chrome and other big projects have their own build systems? Being big has nothing to do with this. These projects run on all versions of all OSes on all hardware that these OSes support with all possible user-system configurations running the tests that must be run for each of these configuration and generating and publishing documentation, packages, and artifacts for all sorts of documentation and package formats to all sort of domains while preventing hundreds of developers trying to break everything with their changes from doing so. Imagine if we were to solve this problem writing Makefiles, CMake, SCons, or any other crappy DSL of any build system in existence. I'd rather suffer for all eternity in hell.
- nullwasamistake 7y agoI always have a ton of problems getting C binaries building. Java and JS are generally super easy. Python and Ruby somewhere in the middle. Kinda a proxy for cross platform compatibility I guess. Some projects now just build in their own docker container. I hope this becomes more common, it's always a single click build no matter how many crazy dependencies you need installed.
- gwbas1c 7y agoHonestly, this shows why projects of major complexity need professional full-time developers. I think it also shows a very minor flaw in the open source / pull request process: Expecting casual and amateur contributors to work at the same level as a professional who works with a given codebase on a daily basis is unreasonable. In my day-to-day job, sometimes I can fix a bug in an hour, but then I need the rest of the day to update tests, avoid regressions, and think through my fix to make sure it's long-term maintainable / readable / performant. Even more important, if I'm working with code that I don't usually touch, I have to make sure I follow the expected conventions that everyone else expects; and I have to be much more careful about assumptions that I haven't learned yet. So, a 1-hour fix easily balloons into 5-15 hours! Perhaps the easiest way to encourage new contributors, who are assumed to be casual, is to encourage incomplete and partial fixes that the main team completes?
- cameronbrown 7y agoIsn't the point of a pull request to submit code that functionally fixes a bug/adds a small feature, and then work on that patch until it's of sufficient quality to merge in? Things like style formatting, running tests among other things could be totally automated away here.
- gwbas1c 7y agoBugfixes and small features need new tests. The tests are often harder, and more time consuming, to write than the bugfix or feature. Edit: And furthermore, when a casual / amateur creates such a pull request, it's not reasonable to expect that person to complete the pull request.
- Crinus 7y ago> Honestly, this shows why projects of major complexity need professional full-time developers. I think the opposite, it shows why projects that care about the practical side of open source and free software should always have in mind their "hackability" from regular developers. I remember a case where RMS didn't want to make some change in a project (i think it was Emacs) because it would make the project harder for people who didn't work on it to understand it. And it makes sense, if you believe that what makes FLOSS good is that anyone could change and/or improve it, then it follows that you should try to maximize the pool of that "anyone".
- snek 7y agoI think browsers are somewhat unique even among "large projects" for the sheer breadth of their codebases. Everything from networking to graphics to midi is somewhere in there, and they all have to provide a somewhat common interface. It just has to be complex, no matter how well the project is organized. On a sidenote, firefox has fantastic build times (like 20-45 minutes cold), and the codebase is fairly well organized, so if you want to try contributing to a browser, that's probably the one to start with. Chromium has a much nicer source IMO, but cold build times can be as high as seven or eight hours.
- erikpukinskis 7y ago> Everything from networking to graphics to midi Wow, I had to look it up: (midi) https://developer.mozilla.org/en-US/docs/Web/API/MIDIAccess https://developer.mozilla.org/en-US/docs/Web/API/MIDIAccess Chrome and Opera only it seems. So no excuse for Firefox having a complex codebase! But yes, browsers are operating systems in every sense except that they target another operating system and not bare metal.
- cakebrewery 7y agoI recently discovered Chrome had MIDI as well while looking for a music sight-reading app. Unfortunately, Chrome for iOS does not support it even though Android does.
- charrondev 7y agoThat’s because chrome isn’t allowed to ship it’s own engine on apple’s platform. Chrome for iOS is essentially safari with a wrapper for syncing some data.
- jes5199 7y agoBrowser MIDI is kinda half-baked- you can use it to do most MIDI things, but it’s not visible to the rest of your MIDI network. So, for example, you can make a web page that functions as a MIDI synth, but your recording studio app won’t be able to select it as output. Also, they expose MIDI messages as arrays of bytes rather than parsing them, which is a weird choice
- pbalau 7y agoI disagree with "One of the ideals of open-source was to make it easy to modify [...] software". I think the ideal is/should be "make it possible to modify software".
- hinkley 7y agoThat's the kind of bar you trip over. We can do better than that. There's a lot of cognitive dissonance around "why are you creating your own when X exists" and then saying, "Well I don't have to do a good job on X because it's free. If you don't like it make your own." We can't actually handle an infinite number of solutions in a space. If I enter that space and do not leave, I'm carving out territory whether I want to or not. There's an opportunity cost to me producing a library, the way there's an opportunity cost of throwing a party (don't think so? Throw a party the same time as your friend's birthday party and see how that goes over). And like a party, you owe it to your guests to put in a modicum of effort. They are perfectly entitled to badmouth you if you show up and do a lousy enough job. Showing up is step one. It's the most important step, but it's not the only important step. If I were going to hire full timers to maintain a piece of FOSS software, I think I might start with a designer. You need conceptual coherence and a designer is one of several ways to get that. Next I might hire a toolsmith. Someone to create a coherent system for contributing. If there's any money left over after that I'd start courting some of my most prolific contributors (although they may be prolific because of the project they work on, and taking them off that project may kill the oracle.)
- userbinator 7y agoIt was always "possible to modify software". PC magazines from the 80s and early 90s even published patches of the form "change bytes at offset X from Y to Z", almost always discovered by someone with a debugger and not the original author. There are quite a few unofficial Windows patches around too, and that was never open-source. Perhaps it's time we rid ourselves of the misconception that source code is necessary to understand and modify software. It may make it easier, but it may also make it harder (by, among other things, helping to propagate the misconception.) Imagine if RMS didn't base the GPL on the availability of source, but simply enforced the legality of understanding and distributing software. I suspect the "free software" movement could've become something far more interesting and powerful, and reverse-engineering techniques would also become far more advanced than they are today.
- jes5199 7y agothis problem goes back to the very beginning of open source browsers. The first source release of Mozilla couldn’t be compiled at all!