11 ms·

Make Ubuntu packages 90% faster by rebuilding them

542 points by jeffbee 2y ago

cyounkins 2y ago
Why is glibc malloc() not more performant? Are tcmalloc/mimalloc making tradeoffs that maintainers are unwilling to make in glibc?
- citrin_ru 2y ago
  AFAIK ptmalloc (on which glibc is based) was created decades ago and both multi-threaded application and multi CPU systems where rare back then (at least in the Linux world) so multi-threaded performance didn't matter. Some improvement in glibc were made since then but I don't think it's possible to significantly improve glibc malloc without rewriting it more or less fully. At that point it would make more sense to import some existing malloc implementation. And we are speaking about trafeoff the default number of arenas in glibc malloc is 8 times CPU which is a terrible tradeoff - on many workloads it cause heap fragmentation and memory usage (RSS) many times higher than allocated memory size, that's why it is common to find advice to set MALLOC_ARENA_MAX to 1 or 2. But probably such high number of areans allows glib to look less bad on synthetic benchmakrs. Jemalloc, tcmalloc, mimalloc all were created with focus on multi-threaded applications from the beginning and while they don't work better than glibc malloc for single threaded application they don't work worse for this use case either. Probably the main disadvantage of using je/tc/mi mallocs for a single threaded app is large code size.
mooktakim 2y ago
Was big fan of Gentoo. Would spend hours and hours optimizing the compile flags to maximise performance. Fun times.
yjftsjthsd-h 2y ago
> What happens if you grab the jq source code from Launchpad, then configure and rebuild it with no flags at all? Even that is about 2-4% faster than the Ubuntu binary package. Why wouldn't that be identical?
- jeffbee 2y ago
  Because instead of using the Debian build, I just used `./configure; make`
  - Aachen 2y ago
    How does that explain it?
- zetalyrae 2y ago
  -march=native maybe?
- cjbprime 2y ago
  Could be measurement noise..
  - motoboi 2y ago
    I tough about that. The output of hiperfine mention nothing chi-test or anything about testing distributions…
taeric 2y ago
Surprised it would be 90%. Are the packages all debug symbols? I recall trying with march=native and seeing some improvement, but not enough to care at a system level.
- jeffbee 2y ago
  > Are the packages all debug symbols? Upstream is stripped actually. My final build here is 43x larger than the distro binary, partly due to having plentiful debug info.
  - taeric 2y ago
    Sorry, I didn't mean symbols, I meant optimization level. Hilarious that they have symbols stripped, but still outclassed. Love to see why they wouldn't have more optimizations. I'm assuming this isn't the case for more packages? Feels too good.
    - dan-robertson 2y ago
      Debug symbols do not need to be paged in so shouldn’t make much/any difference. Compiling with -O3 can increase code size a lot due to inlining, which can be a little bad.
      - pests 2y ago
        Mario 64 gets flak for some files not being compiled with optimizations on. There is a YouTuber, whose name I forget at the moment, who has been doing optimization and improvements on it and constantly talks about how this is for the better due to the low instruction cache size and slow speed on the console. Compiling with optimizations explodes code size (unrolling links, inlining, etc) to the point it’s a net loss after factoring in the increased load on the IC.
        chasil 2y ago
        The -Os flag optimizes for size. What impact does that have?
kstrauser 2y ago
(Well, rebuilding them with a different allocator that benchmarks well on their specific workflow.)
- dan-robertson 2y ago
  Is it even known what workloads the glibc malloc is good for?
  - searealist 2y ago
    Using all your memory on multi-threaded workflows.
  - rurban 2y ago
    It's using the least additional memory, compared to all others. Whilst being the slowest by far of all modern allocators.
- loeg 2y ago
  Everything outperforms glibc malloc. It's essentially malpractice that distros continue to use it instead of mimalloc or jemalloc.
  - throw16180339 2y ago
    It's been awhile since I looked into this, but it's not necessarily an easy change. glibc malloc has debugging APIs; a distro can't easily replace it without either emulating the API or patching programs that use it.
    - loeg 2y ago
      No need to even patch it out. It's relatively easy to change the default, rebuild the world (distros have a flow for this), and restore glibc for the tiny, tiny handful of individual programs that actually rely on glibc debugging APIs. From a end developer perspective: I have no particular familiarity with mimalloc, but I know jemalloc has pretty extensive debugging functionality (not API compatible with glibc malloc of course).
code-faster 2y ago
Could you release a fork of jq whose make does all this fanciness? I use jq A LOT so would love a turnkey solution to make it faster.
- jeffbee 2y ago
  If I had the time to fork jq, I would convert the relevant part to C++ so it doesn't spend literally all of its time dynamically allocating strings it is just about to discard.
  - jeffparsons 2y ago
    Why does that require C++? A quick search showed up plenty of material about "pure C" arena allocators, etc.
- internetter 2y ago
  https://github.com/01mf02/jaq?tab=readme-ov-file#performance https://github.com/01mf02/jaq?tab=readme-ov-file#performance jaq runs 5x faster on my machine in some cases
  - jeffbee 2y ago
    jaq is nice. It just loses, performance-wise, to the final step in this article, and it can't do the second mentioned workload (yet) so I didn't include it.
    - electromech 2y ago
      Which workload can't it do? I've had good success with jaq performance.
      - jeffbee 2y ago
        It bombs out on the jq program I use for the 2nd corpus that I mentioned. On further investigation, the show-stopping filter is strftime. In the jaq readme this is the only not-yet-checked box in the compatibility list, so perhaps some day soon.
Thaxll 2y ago
Sounds like Gentoo...
- Maxious 2y ago
  --funroll-loops for lyfe
  - leoh 2y ago
    Personally I like to use -O8, it’s just even faster that way
    - 01HNNWZ0MV43FF 2y ago
      --vtech just kicked in
    - JoshTriplett 2y ago
      -Omg -Wtf -Bbq
  - genpfault 2y ago
    --funsafe-math-optimizations Enables fun & safe math optimizations!
taeric 2y ago
Looks like the main gem here is mimalloc? Love to know more about that. Good places to look?
- gz09 2y ago
  https://github.com/microsoft/mimalloc https://github.com/microsoft/mimalloc we're using it in feldera and it's been giving us (slightly) better performance than jemalloc which is what we've used before mimalloc
  - taeric 2y ago
    Apologies, I did my first post on my phone, so was rather quick. I had found the repository and it is a good read. What I'd love to see is a comprehensive dive on the different options and a good general discussion on when each is superior. I realize, of course, that this could just be overall optimal. Feels more likely that the allocation patterns of the application using it will impact which is the better allocator?
marcodiego 2y ago
Or, if you want to do it pain free, install guix on it.
- otsukare 2y ago
  How would you do the things mentioned in the article using Guix? You could write your own custom package definitions, extending the default to change up compile flags and allocators, but then you need to do this for every single package (and maintain them all). I'm not sure Guix gives you much here, though maybe that's fine for one or two packages. The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but packages have to be defined as tunable for it to work (and not many are). Is there another option?
  - dannymi 2y ago
    guix build --with-configure-flag="jq=CFLAGS=-O3" jq If you want it to be permanent, then you can use a guix home profile (that's a declarative configuration of your home directory) with a patch function in the package list there: (define llama-tune (options->transformation `((tune . "znver3")))) ; Zen 3 (home-environment (packages (list (llama-tune (specification->package "llama"))))) or: (define jq-patch (options->transformation `((with-configure-flag . "jq=CFLAGS=-O3")))) [...] (jq-patch (specification->package "jq")) [...] You can also write a 10 line guile script to automatically do it for all dependencies (I sometimes do--for example for emacs). That would cause a massive rebuild, though. >The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but > packages have to be defined as tunable for it to work (and not many are). We did it that way on purpose--from prior experience, otherwise, you would get a combinatorial explosion of different package combinations. If it does help for some package X, please email us a 2 line patch adding (tunable? . #t) to that one package. If you do use --tune, it will tune everything that is tuneable in the dependency graph. But at least all dependents (not dependencies) will be just grafted--not be rebuilt.
ergonaught 2y ago
O3 is a bad idea if the package is important. While that’s a dated opinion based on older experiences (Gentoo, various server software, etc), I suspect it’s still the easiest way to get exposed to compiler bugs.
- motoboi 2y ago
  Is a bad ideia if you shove that into people expecting stability. But for self compiling is ok because you'll have a clear culprit in case of failure.
- jeffbee 2y ago
  If you're going to offer unsupported FUD, I would prefer you direct the FUD at NDEBUG and the fear that it could have disabled a load-bearing assertion.
  - wombatpm 2y ago
    Load bearing assertions. Even Design by Contract in Eiffel turns off assertions in production.
    - groby_b 2y ago
      Just because it's turned off by default doesn't mean it isn't load bearing. If you didn't ever encounter an ASSERT with side effects that were crucial to proper functioning, you just haven't seen enough ASSERT statements. Rule #1 of programming: If it can go wrong, it will.
      - eru 2y ago
        Also, you might just run into a new bug, and the assert might caught it, and at least told you about the bug.
      - fc417fc802 2y ago
        That's before we even consider instances where assert was used to check a condition that should always be verified. In my personal projects I define a "require" macro that I use liberally.
atonse 2y ago
I’m almost more amazed that someone figured out jq’s syntax and got some use out of it. In all seriousness though, are you sure some of this isn’t those blocks being loaded into some kind of file system cache the second and third times? How about if you rebooted and then ran the mimalloc version?
- matt123456789 2y ago
  The benchmarking tool being used in this post accounts for that using multiple runs for each invocation, together with a warmup run that is not included in the result metric.
  - atonse 2y ago
    Ah missed that, sorry.
- behnamoh 2y ago
  jq has point-free programming. it's intuitive once you wrap your head around it. See this: https://en.wikipedia.org/wiki/Tacit_programming#jq https://en.wikipedia.org/wiki/Tacit_programming#jq
  - mmastrac 2y ago
    I wish I had seen this earlier. My mental model was close but not quite there to the point where I needed to think too hard about how to solve problems. This is much more intuitive now.
  - atonse 2y ago
    Yeah after this post I sought a couple youtube videos explaining it. It's starting to make a bit more sense now. But the lightbulb hasn't gone off just yet. Appreciate the link.
  - Flimm 2y ago
    Thank you! That's helpful. I got the examples in that Wikipedia section to run using the `--null-input` flag: $ jq --null-input '[1, 2] | add' 3
deleted 2y ago
[deleted]
ltbarcly3 2y ago
Misleading title, it's 90% of the faster time. It's about 45% faster. It's actually a little bit interesting, if you are interested in how we use language. You could argue that now you now get 90% more work done in the same amount of time, and that would align with other 'speed' units that we commonly use (miles per hour, words per minute, bits per second). However, the convention in computer performance is to measure time for a fixed amount of work. I would guess that this is because generally we have a fixed amount of work and what might vary is how long we wait for it (and that is absolutely true in the case of this blog post) so we put time in the numerator. It's a very interesting post and very well done, but it's not 90% faster.
- xmprt 2y ago
  I think you're right. I think you could say the package (as in the code) is 45% faster or that the package increases parsing rate by 90%. But mixing the two is confusing.
  - Daneel_ 2y ago
    I think it's an interpretation issue. When you say it's 45% faster I interpret that as "the new package handles work at a rate of 145% when compared to the original, that is, 1.45 times as fast". I would rephrase your comment as "the package takes 45% less time to process a given data set".
- odo1242 2y ago
  I feel like using units of rate (90% faster) and not units of time makes more sense here. Plus, if you were using units of time, you wouldn’t use the word “faster.” “Takes 45% less time” and “45% faster” are very different assertions, but they both have meaning, both in programming and outside it.
  - ltbarcly3 2y ago
    It comes down to convention. When talking about proportional differences, we can fix the unitary example to be the smaller, larger, earlier, or later, object or subject. I think, generally, we fix on the earlier when talking about the change over time of a characteristic. "This stock went up 100%, this stock went down 50%". In both cases it's the earlier measurement that is taken as the unit. That makes this a 45% reduction in time to do the work, and that's actually what they measured. When talking about comparisons between two things that aren't time dependent it depends on if we talk in multiples or percents I think. A twin bed is half as big as a king bed. A king bed is twice as big as a twin bed. Both are idiomatic. A king bed is 100% bigger than a twin bed. Yes, you could talk like this. A twin bed is 100% smaller than a king bed. Right away you say wait, a twin bed isn't 0 size! Because we don't talk in terms of the smaller thing when talking about decreasing percents, only increasing. A twin bed is 50% smaller than a king bed (iffy). A twin bed is 50% as big as a king bed. There, that's idiomatic again.
emmelaich 2y ago
I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial. https://simdjson.org/ https://simdjson.org/
- llimllib 2y ago
  For kicks, I downloaded the file and compared duckdb with the spatial extension against jq. jq was about 2x as fast for interactive use, but if you had multiple queries to run, paying a small cost to create a duckdb database and then querying it would be vastly faster
  - datacynic 2y ago
    I also was nerdsniped into trying this and found that after extracting the features array into a newline delimited json file, DuckDB finishes the example query in 500 ms (M1 Mac), querying the 1.3 GB json file directly with read_json!
- wiredfool 2y ago
  Yeah, any of GeoPackage (sqllite db with spatial extensions), FlatGeoBuf, or GeoArrow/GeoParquet would be far better for querying.
reramuyc 2y ago
[dead]
deleted 2y ago
[deleted]
rlpb 2y ago
Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corruption and stack buffer overflow. (that one was for CVE-2017-9224, CVE-2017-9226, CVE-2017-9227, CVE-2017-9228 and CVE-2017-9229)
- topspin 2y ago
  This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.
  - smarx007 2y ago
    There are also many flags that should be enabled by default for non-debug builds like ubsan, stack protection, see https://news.ycombinator.com/item?id=35758898 https://news.ycombinator.com/item?id=35758898
    - ryao 2y ago
      UBSAN is usually a debug build only thing. You can run it in production for some added safety, but it comes at a performance cost and theoretically, if you test all execution paths on a debug build and fix all complaints, there should be no benefit to running it in production.
      - smarx007 2y ago
        I think it's time for the C/C++ communities to consider a mindset shift and pivot to having almost all protectors, canaries, sanitizers, assertions (e.g. via _GLIBCXX_ASSERTIONS) on by default and recommended for use in release builds in production. The opposite (i.e, the current state of affairs) should be discouraged and begrudginly accepted in select few cases. https://www.youtube.com/watch?v=gG4BJ23BFBE https://www.youtube.com/watch?v=gG4BJ23BFBE is a presentation that best represents my view on the kind of mindset that's long overdue to become the new norm in our industry.
teitoklien 2y ago
Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Linux internally.
- shanemhansen 2y ago
  So ChromeOS and also the OS for GKE are still basically built this way.
- 6SixTy 2y ago
  Afaik the Gentoo based ChromeOS is being replaced by Android.
  - mycall 2y ago
    It will be interesting if Gentoo supported Fuchsia next.
    - yjftsjthsd-h 2y ago
      Gentoo already has prefix support for non-Linux systems and used to have at least some interest in a full Gentoo/kFreeBSD, so it's plausible.
    - surajrmal 2y ago
      What would that even mean?
      - sham1 2y ago
        Portage on top of Fuchsia, probably. There's not all that much more to Gentoo all things considered.
ac130kz 2y ago
-march=native + mimalloc (or jemalloc) should be sufficient without causing significant undefined behavior like -O3 or most extra optimization related compiler arguments.
- eru 2y ago
  Yes, -O3 is seldom an improvement for most C compilers.
- loeg 2y ago
  -O3 doesn't cause undefined behavior.
  - MyOutfitIsVague 2y ago
    No, but it can often expose undefined behavior that doesn't stop a program from "working" on lower optimization levels. Which is why I'm in support of always building your own software with -O3.
    - vlovich123 2y ago
      Are you sure about that? I’m not aware of any additional UB exploitation enabled by O3 vs O2 or Os.
      - MyOutfitIsVague 2y ago
        Nope, I'm not sure about it. I remember when I was using Gentoo about 10 years ago, this was the common reason given for using -O2 instead of -O3 in your build flags, and I'm just speaking from that memory.
        vlovich123 2y ago
        Yeah there’s lots of cargo culting FUD like that that happens (see other posters in this thread that claim that glibc is a more stable allocator )
linkerdoo 2y ago
[dead]
DrNosferatu 2y ago
Sometimes comes to mind: Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware? Plus, the configuration could be automated - just push a button to (lazy, in background) recompile. Priority for the ones profiled to be performance critical.
- eru 2y ago
  > Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware? Well, in principle yes. But you'd also need to figure out what you mean by 'optimal' flags? The flags might differ between different packages, and not all packages will be compatible with all the flags. Even worse, in the example in the article they got most of their speedup out of moving to a different allocator, and that's not necessarily compatible with all packages (nor an improvement for all packages). However, if you still want to do all these things, then Gentoo Linux is the distribution for you. It supports this tinkering directly and you'll have a much happier time than trying to bend Ubuntu to your will.
Uptrenda 2y ago
[flagged]
- msm_ 2y ago
  Waste minutes of your time just so anonymous people on the internet can read your comment :). Not everything in life is an optimization problem. Fun little projects like this is what makes us human (and, arguably, are both educational and entertaining).
  - Uptrenda 2y ago
    I see what you're saying but I am also talking about how computers originally saved humans time and that the OP seems to now be slaving away to save the CPU time. Doesn't that seem a little backwards? If we're talking about micro-seconds of difference, the trade-off doesn't seem worth it. Even on a mass scale where this is somehow adopted, nobody is going to notice the difference. Maybe if this were in something like eCommerce and web browing where the lag translates to profit lost? Or perhaps game engines? IDK, I just consider humans time more precious than a slow package (that already runs blazingly fast on a CPU that it barely matters.)
    - alpaca128 2y ago
      Longer waiting times can decrease productivity a lot more than just the runtime increase. If a program takes long enough that the user switches context mentally, or goes to get another coffee or looks at a few posts on social media that now means far more wasted time. Humans are not robots and won’t just keep staring at the screen. Also a lot of simple optimizations could save seconds or minutes in the lives of millions of people, and across multiple devices/programs that adds up. Microsoft once sped up the booting process of their consoles by 5 seconds, by simply making the boot animation shorter by 5 seconds. Those consoles were sold to millions of people and it took MS 8 years or so to make the fix. That’s many lifetimes wasted once you add it all up.
- Daneel_ 2y ago
  Today it's 2.2GB of JSON files, tomorrow it's tens of petabytes of archived data that you need to look through for work. A speedup of 2x is nothing to scoff at. As a broader comment though, this is how we learn and discover things. Just because the specific outcome here is "trivial" doesn't mean the approach or the lessons learnt aren't valuable. I found it an interesting read despite not have any stake in the outcome.
zX41ZdbW 2y ago
ClickHouse will be faster for processing large JSON data files. The example from the gist will look like follows: ch -q "WITH arrayJoin(features) AS f SELECT f.properties.SitusCity WHERE f.properties.TotalNetValue < 193000 FROM 'data.json'" Reference: https://jsonbench.com/ https://jsonbench.com/
- jeffbee 2y ago
  I'm willing to believe that will execute in less than 2 seconds, but it doesn't work as given.
  - rustc 2y ago
    Can you post a link to the JSON file? Try this (the placement of FROM was incorrect): ch "WITH arrayJoin(features) AS f SELECT f.properties.SitusCity FROM 'a.json' WHERE f.properties.TotalNetValue < 193000"
    - jeffbee 2y ago
      It's a public data file accessible at https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcdba81_0/explore https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcd...
77pt77 2y ago
If you decide to go down this road just use Gentoo.
- benatkin 2y ago
  Not a bad idea. Arch also comes to mind, and its support on https://copy.sh/v86/ https://copy.sh/v86/ is a testament to its flexibility.
- m000 2y ago
  I haven't kept up with Gentoo. How long does it take these days to bootstrap a functional desktop from source? I remember when it was my main driver back in early '00s, running on a horribly underpowered PII at 266MHz. It would often be compiling 24/7 to keep up with the updates.
  - 77pt77 2y ago
    Haven't used it in a looong time, but by 2010 it was no longer that bad. My first install was 2002 and it took my a good 24 hours to get X to boot. It did catch a faulty memory DIM later on because the linking kept failing intermittently.
    - hy4000days 2y ago
      [dead]
  - hy4000days 2y ago
    [dead]
- hy4000days 2y ago
  [dead]
electromech 2y ago
I'd be curious how the performance compares to this Rust jq clone: cargo install --locked jaq (you might also be able to add RUSTFLAGS="-C target-cpu=native" to enable optimizations for your specific CPU family) "cargo install" is an underrated feature of Rust for exactly the kind of use case described in the article. Because it builds the tools from source, you can opt into platform-specific features/instructions that often aren't included in binaries built for compatibility with older CPUs. And no need to clone the repo or figure out how to build it; you get that for free. jaq[1] and yq[2] are my go-to options anytime I'm using jq and need a quick and easy performance boost. [1] https://github.com/01mf02/jaq https://github.com/01mf02/jaq [2] https://github.com/mikefarah/yq https://github.com/mikefarah/yq
- saghm 2y ago
  As a bonus that people might not be aware of, in the cases where you do want to use the repo directly (either because there isn't a published package or maybe you want the latest commit that hasn't been released), `cargo install` also has a `--git` flag that lets you specify a URL to a repo. I've used this a number of times in the past, especially as an easy way for me to quickly install personal stuff that I throw together and push to a repo without needing to put together any sort of release process or manually copy around binaries to personal machines and keep track of the exact commits I've used to build them.
- oguz-ismail 2y ago
  > I'd be curious how the performance compares to this Rust jq clone Every once in a while I test jaq against jq and gojq with my jq solution to AoC 2022 day 13 https://gist.github.com/oguz-ismail/8d0957dfeecc4f816ffee79d650c9d38 https://gist.github.com/oguz-ismail/8d0957dfeecc4f816ffee79d... It's still behind both as of today
zadler 2y ago
[flagged]
alberth 2y ago
Isn’t this what ClearLinux essentially is (way over generalizing)
rgmerk 2y ago
It's a while since I had to deal with this kind of thing, but my memory was that as soon as you go beyond the flags that the upstream developers use (just to be clear, I mean the upstream developers, not the distro packagers) you're buying yourself weird bugs and a whole lot of indifference if they occur. I haven't used a non-libc malloc before but I suspect the same applies.
- taeric 2y ago
  I've been building my own emacs for a long time, and have yet to hit any weird bugs. I thought that as long as you avoid any unsafe optimizations, you should be fine? Granted, I also thought that -march=native was the main boost that I was seeing. This post indicates that is not necessarily the case. I also suspect that any application using floats is more likely to have rough edges?
  - pertymcpert 2y ago
    Complex software usually has some undefined behavior lurking that at higher or even just different optimization levels can trigger the compiler to do unexpected things to the code. It happens all the time in my line of work. If there's an extensive test suite you can run to verify that it still works mostly as expected then it's easier.
    - taeric 2y ago
      This is one where I suspect we don't disagree. But "all the time" can have a very different feel between people. It also used to happen that just changing processors was likely to find some problems in the code. I have no doubt that still happens, but I'd also expect it has reduced. Some of this has to be a convergence on far fewer compilers than we used to encounter. I know there are still many c compilers. Seems there are only two common ones, though. Embedded, of course, is a whole other bag of worms.
  - Y_Y 2y ago
    Did you try -ffast-math? IIRC that used to break emacs in some subtle way, while providing no extra speed.
john-tells-all 2y ago
If you actually do this, just get Ubuntu to download the exact source package you want. In this case: apt-get source jq Then go into the package and recompile to your heart's content. You can even repackage it for distribution or archiving. You'll get a result much closer to the upstream Ubuntu, as opposed to getting lots of weird errors and misalignments.
smallstepforman 2y ago
Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch malloc again, others constantly release and acquire. How well do they handle fragmentation? What is the uptime (10 seconds or 10 years)? Sometimes the choice of allocators is the difference between long term stability vs short term speed. I experimented with different allocators devoloping a video editor testing 4K videos that caches frames. 32Mb per frame, at 60fps, thats almost 2Gb per second per track. You quickly hit allocator limitations, and realise that at least vanilla glibc allocator offers the best long term stability. But for short running benchmarks its the slowest. As already pointed out, engineering is a compromise.
- vlovich123 2y ago
  Mimalloc is a general purpose allocator like JEMalloc / TCMalloc. Glibc is known to have a pretty bad allocator that modern allocators like MIMalloc & the latest TCMalloc (not the one available by default in Ubuntu) run laps around. While of course speedup may be variable, generally the benchmarks show an across the board speedup (whether that matters for any given application is something entirely different). As for crashes, these are all general purpose multi-thread allocators and behave no differently from glibc (modulo bugs that can exist equally in glibc).
  - deleted 2y ago
    [deleted]
  - smallstepforman 2y ago
    Agree for most short running apps. I updated my comment to reflect issues with apps that are constantly reallocating, and running for longer that 60 seconds. But you are absolutely correct for most short running apps, 99% recommended to replace glibc. However, there is an app or two where glibc stability doesnt trigger a pathological use cases, and you have no choice. Hence why its the default, since there are less crashes in pathaligical cases. And the devs are exhausted dealing with crashing bugs which can be eliminated by using the slower allocator.
brucehoult 2y ago
Since the test data is not available for us to experiment ourselves, I've asked the poster to try it with Boehm GC compiled as a transparent malloc() replacement.
coolgoose 2y ago
Reminds me how I learned Linux by bootstrapping Gentoo from Knoppix live cd's on my Athlon xp 1600+ and using God knows how many compilations flags
dlcarrier 2y ago
For a lot of packages, installing the .deb file instead of the Snap package can in and of itself result in several times faster performance.
sedatk 2y ago
When I was using Linux in 1995, installing a package meant downloading the sources and running “make install”. Have we come full circle?
sneak 2y ago
I love the part where he showed us the commands to rebuild the packages with different compilers and LDFLAGS.
deleted 2y ago
[deleted]
rubatuga 2y ago
Chimera Linux defaults to using clang + mimalloc Read more here: https://chimera-linux.org/about/ https://chimera-linux.org/about/
polskibus 2y ago
This is gentoo resurrected! Great memories building entire Linux from scratch on install ;)
dgan 2y ago
what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes. i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files
- Hackbraten 2y ago
  There might me some kind of malware scanner at play. Maybe it's time to rebuild those with mimalloc, too.
- nurettin 2y ago
  Does ${corporate} prevent you from merging your jsons to a single file with a decent format like parquet?
  - dgan 2y ago
    they don't but that defeats the purpose. i want to be able to inspect the insides, so binary formats are out, i have to easily open & read the content
    - nurettin 2y ago
      I don't get how you can't put the content within the merged file and query it but good luck.
      - dgan 2y ago
        because that d be a several hundreds megabytes json :) Anyway with suggestion in another comment, i moved the root directory into an antivirus-excluded location, which gave me almost a 10 times performance win
- HdS84 2y ago
  Try excluding that directory from the virus scanner. Also a dev drive might come in handy for this workload
globular-toast 2y ago
No GCC march=native?
- penguin_booze 2y ago
  Wouldn't that be the default?
- rurban 2y ago
  He added it of course
  - globular-toast 2y ago
    With clang. GCC is known to do better with march=native, or at least it was.
yxhuvud 2y ago
What i wonder is when any main stream distro will wake up and change their default allocator away from the very dated glibc.
teekert 2y ago
Ah yes, this is nice but it sounds like Gentoo with extra steps. Probably also some extra benefits. But there is a reason we don’t all run Gentoo. Although given a large and seamless enough cache, maybe we would.
rurban 2y ago
He is still missing profile guided optimizations (with his testcase), and bolt. And in some/most cases -Os is faster than -O3 Boehm GC is usually slower than glibc with lots of allocs. MPS would be better, but needs lots of rewriting. https://github.com/Ravenbrook/mps https://github.com/Ravenbrook/mps
- roeles 2y ago
  > And in some/most cases -Os is faster than -O3 This surprises me. Can you please elaborate?
  - withinboredom 2y ago
    If the whole program can fit in the L1/L2 cache, it can speed up things quite a lot!
  - rurban 2y ago
    A IC cache miss costs 50x more on average. If your insn and data are small, the overhead of doing more work is less than the overhead of cache misses
- jeffbee 2y ago
  BOLT is only good for ~2% in this scenario, which is about what I would have expected from glancing at the profiles. -Os is much slower, as I would have expected. Nothing in the perf data suggested high L1i miss rate. The .text of the "size optimized" program is, for some reason, 16x larger.
hyperman1 2y ago
Reading this, I wonder a few things that seem distro quick wins: 1) Why don't distros replace the glibc allocator with one of the better ones? 2) Why don't distros allow for making server-specific builds for only a few packages? You don't have to pay the compilation cost for everything, just a list of 2 or 3 packages.
- vkazanov 2y ago
  Glibc is a reliable generalist while most other ones specialise in something: single thread, multiple threads, slab, arena style, etc .. Some of the other things disabled might be useful for distros in general, like ndebug assertions or compiling with debig symbols.
  - hyperman1 2y ago
    Is glibc being generalist still true? I might be wrong here, but I had the impression it was open and free but behind most others,and e.g. tcmalloc is better on most criteria. Do we have benchmarks and comparisons for the generalist case?
- hinkley 2y ago
  Is the glibc allocator still that bad? I thought they replaced that ages ago.
nottorp 2y ago
Make a specific ubuntu package 90% faster by rebuilding it after changing the memory allocator, that is. Should probably work for Debian and RedHat too. For this particular package. Edit: based just on the title i initially thought this is an article about turning Ubuntu into Linux From Scratch.
Aachen 2y ago
Reading that such a simple change can get such a big speedup, my first thought is to let the authors of jq know. Maybe there's a caveat to be aware of, maybe they'll test it and end up making it faster for everyone. Useful to drop a quick note pretty much no matter the result, I think? The article doesn't seem to even consider that option and I don't see any comment here mentioning this either. Am I missing something?
- chasil 2y ago
  I would be curious if Intel Clear Linux had similar gains, by using newer opcodes in the instruction set. I wonder if the glibc allocator is standard there. https://en.m.wikipedia.org/wiki/Clear_Linux_OS https://en.m.wikipedia.org/wiki/Clear_Linux_OS
jbverschoor 2y ago
On FreeBSD doing a make world / make buildworld used to be so common
wengo314 2y ago
"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in stabilizing said application's memory usage. it actually fixed a problem that appeared to be a memory leak, but probably wasn't fault of the app itself (likely memory fragmentation with standard malloc)
- dietr1ch 2y ago
  To me it's obviously a scam because there's no way such an improvement can be achieved globally with a single post explanation. 90% faster is a micro-benchmark number.
  - looofooo0 2y ago
    True that, I mean it is still interesting, that if you have a narrow task, you might achieve some significant speed up from rebuilding them. But this is a very niche application.
    - wengo314 2y ago
      true, i saw a thread recently on reddit where guy hand-tuned compilation flags and did pgo profiling for a video encoder app that he uses on video encode farm. In his case, even a gain of ~20% was significant. It calculated into extra bandwidth to encode a few thousand more video files per year.
  - arghwhat 2y ago
    This is neither a micro-benchmark nor a scam, but it is click-bait by not mentioning jq specifically. Micro-benchmarks would be testing e.g. a single library function or syscall rather than the whole application. This is the whole application, just not one you might care that much for the performance of. Other applications will of course see different results, but stuff like enabling LTO, tuning THP and picking a suitable allocator are good, universal recommendations.
lifeinthevoid 2y ago
People will run into subtle bugs and weird behavior when starting to compile things themselves and replacing libraries I think. Not worth the effort for me, fun as an exercise, but would only do it if it's really (really) needed for performance.
p0w3n3d 2y ago
I wonder why my brain calculates `90% faster' as 10% of the time (i.e. t1=0.1*t0) it used to be working. However, this makes sense only if we calculate the recompiled application running time as t1=t0/1.90
- withinboredom 2y ago
  "by 190%" instead of "90% faster" is far clearer.
freep1zza 2y ago
Even Canonical themselves sobered up and abandoned their v3 recompilation effort as it showed no serious benefits. Distros who went this path is going the way of the dinousaurs, together with their confused user base.
sheepscreek 2y ago
In the late 1990s, the most common method for distributing a package was by providing its source code as a tarball. You essentially had to run ./configure and make on every package. I believe ./configure does some of what the OP did in a clever way, checking for locally installed alternatives, etc. It’s still pretty cool, though. I didn’t think a malloc replacement would significantly improve performance.
mbar84 2y ago
Why would you give your product a quite specific name that's exactly the same as another product? "Core 2 Duo" immediately brings an Intel CPU to mind and will probably make it harder to search for anything related to the watch.
tpoacher 2y ago
Obviously the title / article is clickbait. But in general I have often found great benefit in many cases in obtaining packages from source and compiling them, or downloading official binaries, even if the repository does have the latest version in theory. (shameless plug incoming): My only obstacle with such "Manually Installed [and/or] Source-Compiled" (MISC) packages was checking for updates. So I made this: https://sr.ht/~tpapastylianou/misc-updater/ https://sr.ht/~tpapastylianou/misc-updater/ Works great, and I've been slowly improving it to add 'upgrader' commands for some of the packages too. It's not a full replacement for a package manager, nor is it meant to be. But it's made my ability to use MISC packages instead of repository ones a lot more streamlined and easier.
kedmi 2y ago
Please rebuild Chrome
- jeffbee 2y ago
  Chrome is optimized right up to its eyebrows. If you are using Google distributions of Chrome binaries, I wouldn't expect improvements from build tweaks.
simlevesque 2y ago
Isn't GeoJSON a really bad tool for what you are trying to do ? Just use GeoParquet: Converter: https://geoparquet.org/convert/ https://geoparquet.org/convert/ It'll scan way less data and be 90% faster than your 90% faster jq.
- jeffbee 2y ago
  GeoJSON is an interchange format that a government agency is using to provide this data, and it updates continuously, so at some point I will be interested in the efficiency of the ETL even if I start doing all my queries against an optimized format.
mannyv 2y ago
What you should take away from this is that there are things you can do to speed up your app that are outside of normal code optimization.
rajaravivarma_r 2y ago
It is funny, how being a corporate-Rails programmer taught me this early in my career. Because building Ruby from source was the only way you could install latest Ruby versions (using ruby-build) and installing Ruby from source means its dependencies like openssl, zlib had to be installed from source too, to match the required versions. And for a long time using Jemalloc was the only was to keep the memory usage constant with multi-threaded ruby programs like Puma and Sidekiq. This was either achieved through compiling Ruby with jemalloc or modifying the LD_LIBRARY_PATH. Some developers also reported 5% or so reduction in response times with jemalloc, iirc. The problem with this approach is though, when a package has a lot of dependencies like ImageMagick which relies on jpeg, png, ghost and a lot of other libraries, you have to take a trial and error approach until it succeeds. Fortunately fixing the dependency errors are the easiest, sometimes building Python from source would throw errors from headers which are impossible to understand. If you find a Stackoverflow solution then you are good, or you have to go down the rabbit hole coming either successful or empty handed based on your level of expertise.
alexmyczko 2y ago
I speed up /sbin/reboot for x86-64 1000 times! might want to run sync before. https://github.com/alexmyczko/autoexec.bat/blob/master/kbd-reboot.c https://github.com/alexmyczko/autoexec.bat/blob/master/kbd-r...
hy4000days 2y ago
[dead]
kristianp 2y ago
I tried this with a 1.3GB file [1], and got Gemini to convert the jq query into a go program, that takes 6.6s the second time its run.(to cache the geojson file). My laptop isn't particularly fast (i7-1250p), and go's json handling isn't particularly fast either, so jq's time is not impressing me when run on a ryzen 9 desktop processor on a 500MB file. It's surprising how quick this kind of processing can be in go. [1] https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcdba81_0/about https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcd... The program: package main import ( "encoding/json" "fmt" "os" ) type FeatureCollection struct { Features []Feature `json:"features"` } type Feature struct { Properties Properties `json:"properties"` } type Properties struct { TotalNetValue int `json:"TotalNetValue"` SitusCity string `json:"SitusCity"` } func main() { filename := "Parcels.geojson" data, err := os.ReadFile(filename) if err != nil { fmt.Println("Error reading file:", err) return } var featureCollection FeatureCollection err = json.Unmarshal(data, &featureCollection) if err != nil { fmt.Println("Error unmarshaling JSON:", err) return } for _, feature := range featureCollection.Features { if feature.Properties.TotalNetValue < 193000 { fmt.Println(feature.Properties.SitusCity) } } }
trod1234 2y ago
Isn't it against the rules to post a link like this where the content is walled behind needing an account to access? I know most of us have accounts here, but when we're seeing something we want to read its not conducive to have to have to deal with roadblocks before we can see what is being talked about. Same goes for paywalls.
- jeffbee 2y ago
  In what way does this require an account? It loads fine for me in a Chrome Incognito window.
  - trod1234 2y ago
    The link goes straight to a login page, doesn't let you see the page. On Firefox here.
sswam 2y ago
[dead]

542 points by jeffbee 2y ago

Why is glibc malloc() not more performant? Are tcmalloc/mimalloc making tradeoffs that maintainers are unwilling to make in glibc?

citrin_ru 2y ago
AFAIK ptmalloc (on which glibc is based) was created decades ago and both multi-threaded application and multi CPU systems where rare back then (at least in the Linux world) so multi-threaded performance didn't matter. Some improvement in glibc were made since then but I don't think it's possible to significantly improve glibc malloc without rewriting it more or less fully. At that point it would make more sense to import some existing malloc implementation. And we are speaking about trafeoff the default number of arenas in glibc malloc is 8 times CPU which is a terrible tradeoff - on many workloads it cause heap fragmentation and memory usage (RSS) many times higher than allocated memory size, that's why it is common to find advice to set MALLOC_ARENA_MAX to 1 or 2. But probably such high number of areans allows glib to look less bad on synthetic benchmakrs. Jemalloc, tcmalloc, mimalloc all were created with focus on multi-threaded applications from the beginning and while they don't work better than glibc malloc for single threaded application they don't work worse for this use case either. Probably the main disadvantage of using je/tc/mi mallocs for a single threaded app is large code size.

mooktakim 2y ago

Was big fan of Gentoo. Would spend hours and hours optimizing the compile flags to maximise performance. Fun times.

yjftsjthsd-h 2y ago

> What happens if you grab the jq source code from Launchpad, then configure and rebuild it with no flags at all? Even that is about 2-4% faster than the Ubuntu binary package. Why wouldn't that be identical?

jeffbee 2y ago
Because instead of using the Debian build, I just used `./configure; make`
- Aachen 2y ago
  How does that explain it?
zetalyrae 2y ago
-march=native maybe?
cjbprime 2y ago
Could be measurement noise..
- motoboi 2y ago
  I tough about that. The output of hiperfine mention nothing chi-test or anything about testing distributions…

taeric 2y ago

Surprised it would be 90%. Are the packages all debug symbols? I recall trying with march=native and seeing some improvement, but not enough to care at a system level.

jeffbee 2y ago
> Are the packages all debug symbols? Upstream is stripped actually. My final build here is 43x larger than the distro binary, partly due to having plentiful debug info.
- taeric 2y ago
  Sorry, I didn't mean symbols, I meant optimization level. Hilarious that they have symbols stripped, but still outclassed. Love to see why they wouldn't have more optimizations. I'm assuming this isn't the case for more packages? Feels too good.
  - dan-robertson 2y ago
    Debug symbols do not need to be paged in so shouldn’t make much/any difference. Compiling with -O3 can increase code size a lot due to inlining, which can be a little bad.
    - pests 2y ago
      Mario 64 gets flak for some files not being compiled with optimizations on. There is a YouTuber, whose name I forget at the moment, who has been doing optimization and improvements on it and constantly talks about how this is for the better due to the low instruction cache size and slow speed on the console. Compiling with optimizations explodes code size (unrolling links, inlining, etc) to the point it’s a net loss after factoring in the increased load on the IC.
      - chasil 2y ago
        The -Os flag optimizes for size. What impact does that have?

kstrauser 2y ago

(Well, rebuilding them with a different allocator that benchmarks well on their specific workflow.)

dan-robertson 2y ago
Is it even known what workloads the glibc malloc is good for?
- searealist 2y ago
  Using all your memory on multi-threaded workflows.
- rurban 2y ago
  It's using the least additional memory, compared to all others. Whilst being the slowest by far of all modern allocators.
loeg 2y ago
Everything outperforms glibc malloc. It's essentially malpractice that distros continue to use it instead of mimalloc or jemalloc.
- throw16180339 2y ago
  It's been awhile since I looked into this, but it's not necessarily an easy change. glibc malloc has debugging APIs; a distro can't easily replace it without either emulating the API or patching programs that use it.
  - loeg 2y ago
    No need to even patch it out. It's relatively easy to change the default, rebuild the world (distros have a flow for this), and restore glibc for the tiny, tiny handful of individual programs that actually rely on glibc debugging APIs. From a end developer perspective: I have no particular familiarity with mimalloc, but I know jemalloc has pretty extensive debugging functionality (not API compatible with glibc malloc of course).

code-faster 2y ago

Could you release a fork of jq whose make does all this fanciness? I use jq A LOT so would love a turnkey solution to make it faster.

jeffbee 2y ago
If I had the time to fork jq, I would convert the relevant part to C++ so it doesn't spend literally all of its time dynamically allocating strings it is just about to discard.
- jeffparsons 2y ago
  Why does that require C++? A quick search showed up plenty of material about "pure C" arena allocators, etc.
internetter 2y ago
https://github.com/01mf02/jaq?tab=readme-ov-file#performance https://github.com/01mf02/jaq?tab=readme-ov-file#performance jaq runs 5x faster on my machine in some cases
- jeffbee 2y ago
  jaq is nice. It just loses, performance-wise, to the final step in this article, and it can't do the second mentioned workload (yet) so I didn't include it.
  - electromech 2y ago
    Which workload can't it do? I've had good success with jaq performance.
    - jeffbee 2y ago
      It bombs out on the jq program I use for the 2nd corpus that I mentioned. On further investigation, the show-stopping filter is strftime. In the jaq readme this is the only not-yet-checked box in the compatibility list, so perhaps some day soon.

Thaxll 2y ago

Sounds like Gentoo...

Maxious 2y ago
--funroll-loops for lyfe
- leoh 2y ago
  Personally I like to use -O8, it’s just even faster that way
  - 01HNNWZ0MV43FF 2y ago
    --vtech just kicked in
  - JoshTriplett 2y ago
    -Omg -Wtf -Bbq
- genpfault 2y ago
  --funsafe-math-optimizations Enables fun & safe math optimizations!

taeric 2y ago

Looks like the main gem here is mimalloc? Love to know more about that. Good places to look?

gz09 2y ago
https://github.com/microsoft/mimalloc https://github.com/microsoft/mimalloc we're using it in feldera and it's been giving us (slightly) better performance than jemalloc which is what we've used before mimalloc
- taeric 2y ago
  Apologies, I did my first post on my phone, so was rather quick. I had found the repository and it is a good read. What I'd love to see is a comprehensive dive on the different options and a good general discussion on when each is superior. I realize, of course, that this could just be overall optimal. Feels more likely that the allocation patterns of the application using it will impact which is the better allocator?

marcodiego 2y ago

Or, if you want to do it pain free, install guix on it.

otsukare 2y ago
How would you do the things mentioned in the article using Guix? You could write your own custom package definitions, extending the default to change up compile flags and allocators, but then you need to do this for every single package (and maintain them all). I'm not sure Guix gives you much here, though maybe that's fine for one or two packages. The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but packages have to be defined as tunable for it to work (and not many are). Is there another option?
- dannymi 2y ago
  guix build --with-configure-flag="jq=CFLAGS=-O3" jq If you want it to be permanent, then you can use a guix home profile (that's a declarative configuration of your home directory) with a patch function in the package list there: (define llama-tune (options->transformation `((tune . "znver3")))) ; Zen 3 (home-environment (packages (list (llama-tune (specification->package "llama"))))) or: (define jq-patch (options->transformation `((with-configure-flag . "jq=CFLAGS=-O3")))) [...] (jq-patch (specification->package "jq")) [...] You can also write a 10 line guile script to automatically do it for all dependencies (I sometimes do--for example for emacs). That would cause a massive rebuild, though. >The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but > packages have to be defined as tunable for it to work (and not many are). We did it that way on purpose--from prior experience, otherwise, you would get a combinatorial explosion of different package combinations. If it does help for some package X, please email us a 2 line patch adding (tunable? . #t) to that one package. If you do use --tune, it will tune everything that is tuneable in the dependency graph. But at least all dependents (not dependencies) will be just grafted--not be rebuilt.

ergonaught 2y ago

O3 is a bad idea if the package is important. While that’s a dated opinion based on older experiences (Gentoo, various server software, etc), I suspect it’s still the easiest way to get exposed to compiler bugs.

motoboi 2y ago
Is a bad ideia if you shove that into people expecting stability. But for self compiling is ok because you'll have a clear culprit in case of failure.
jeffbee 2y ago
If you're going to offer unsupported FUD, I would prefer you direct the FUD at NDEBUG and the fear that it could have disabled a load-bearing assertion.
- wombatpm 2y ago
  Load bearing assertions. Even Design by Contract in Eiffel turns off assertions in production.
  - groby_b 2y ago
    Just because it's turned off by default doesn't mean it isn't load bearing. If you didn't ever encounter an ASSERT with side effects that were crucial to proper functioning, you just haven't seen enough ASSERT statements. Rule #1 of programming: If it can go wrong, it will.
    - eru 2y ago
      Also, you might just run into a new bug, and the assert might caught it, and at least told you about the bug.
    - fc417fc802 2y ago
      That's before we even consider instances where assert was used to check a condition that should always be verified. In my personal projects I define a "require" macro that I use liberally.

atonse 2y ago

I’m almost more amazed that someone figured out jq’s syntax and got some use out of it. In all seriousness though, are you sure some of this isn’t those blocks being loaded into some kind of file system cache the second and third times? How about if you rebooted and then ran the mimalloc version?

matt123456789 2y ago
The benchmarking tool being used in this post accounts for that using multiple runs for each invocation, together with a warmup run that is not included in the result metric.
- atonse 2y ago
  Ah missed that, sorry.
behnamoh 2y ago
jq has point-free programming. it's intuitive once you wrap your head around it. See this: https://en.wikipedia.org/wiki/Tacit_programming#jq https://en.wikipedia.org/wiki/Tacit_programming#jq
- mmastrac 2y ago
  I wish I had seen this earlier. My mental model was close but not quite there to the point where I needed to think too hard about how to solve problems. This is much more intuitive now.
- atonse 2y ago
  Yeah after this post I sought a couple youtube videos explaining it. It's starting to make a bit more sense now. But the lightbulb hasn't gone off just yet. Appreciate the link.
- Flimm 2y ago
  Thank you! That's helpful. I got the examples in that Wikipedia section to run using the `--null-input` flag: $ jq --null-input '[1, 2] | add' 3

deleted 2y ago

[deleted]

ltbarcly3 2y ago

Misleading title, it's 90% of the faster time. It's about 45% faster. It's actually a little bit interesting, if you are interested in how we use language. You could argue that now you now get 90% more work done in the same amount of time, and that would align with other 'speed' units that we commonly use (miles per hour, words per minute, bits per second). However, the convention in computer performance is to measure time for a fixed amount of work. I would guess that this is because generally we have a fixed amount of work and what might vary is how long we wait for it (and that is absolutely true in the case of this blog post) so we put time in the numerator. It's a very interesting post and very well done, but it's not 90% faster.

xmprt 2y ago
I think you're right. I think you could say the package (as in the code) is 45% faster or that the package increases parsing rate by 90%. But mixing the two is confusing.
- Daneel_ 2y ago
  I think it's an interpretation issue. When you say it's 45% faster I interpret that as "the new package handles work at a rate of 145% when compared to the original, that is, 1.45 times as fast". I would rephrase your comment as "the package takes 45% less time to process a given data set".
odo1242 2y ago
I feel like using units of rate (90% faster) and not units of time makes more sense here. Plus, if you were using units of time, you wouldn’t use the word “faster.” “Takes 45% less time” and “45% faster” are very different assertions, but they both have meaning, both in programming and outside it.
- ltbarcly3 2y ago
  It comes down to convention. When talking about proportional differences, we can fix the unitary example to be the smaller, larger, earlier, or later, object or subject. I think, generally, we fix on the earlier when talking about the change over time of a characteristic. "This stock went up 100%, this stock went down 50%". In both cases it's the earlier measurement that is taken as the unit. That makes this a 45% reduction in time to do the work, and that's actually what they measured. When talking about comparisons between two things that aren't time dependent it depends on if we talk in multiples or percents I think. A twin bed is half as big as a king bed. A king bed is twice as big as a twin bed. Both are idiomatic. A king bed is 100% bigger than a twin bed. Yes, you could talk like this. A twin bed is 100% smaller than a king bed. Right away you say wait, a twin bed isn't 0 size! Because we don't talk in terms of the smaller thing when talking about decreasing percents, only increasing. A twin bed is 50% smaller than a king bed (iffy). A twin bed is 50% as big as a king bed. There, that's idiomatic again.

emmelaich 2y ago

I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial. https://simdjson.org/ https://simdjson.org/

llimllib 2y ago
For kicks, I downloaded the file and compared duckdb with the spatial extension against jq. jq was about 2x as fast for interactive use, but if you had multiple queries to run, paying a small cost to create a duckdb database and then querying it would be vastly faster
- datacynic 2y ago
  I also was nerdsniped into trying this and found that after extracting the features array into a newline delimited json file, DuckDB finishes the example query in 500 ms (M1 Mac), querying the 1.3 GB json file directly with read_json!
wiredfool 2y ago
Yeah, any of GeoPackage (sqllite db with spatial extensions), FlatGeoBuf, or GeoArrow/GeoParquet would be far better for querying.

reramuyc 2y ago

[dead]

deleted 2y ago

[deleted]

rlpb 2y ago

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corruption and stack buffer overflow. (that one was for CVE-2017-9224, CVE-2017-9226, CVE-2017-9227, CVE-2017-9228 and CVE-2017-9229)

topspin 2y ago
This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.
- smarx007 2y ago
  There are also many flags that should be enabled by default for non-debug builds like ubsan, stack protection, see https://news.ycombinator.com/item?id=35758898 https://news.ycombinator.com/item?id=35758898
  - ryao 2y ago
    UBSAN is usually a debug build only thing. You can run it in production for some added safety, but it comes at a performance cost and theoretically, if you test all execution paths on a debug build and fix all complaints, there should be no benefit to running it in production.
    - smarx007 2y ago
      I think it's time for the C/C++ communities to consider a mindset shift and pivot to having almost all protectors, canaries, sanitizers, assertions (e.g. via _GLIBCXX_ASSERTIONS) on by default and recommended for use in release builds in production. The opposite (i.e, the current state of affairs) should be discouraged and begrudginly accepted in select few cases. https://www.youtube.com/watch?v=gG4BJ23BFBE https://www.youtube.com/watch?v=gG4BJ23BFBE is a presentation that best represents my view on the kind of mindset that's long overdue to become the new norm in our industry.

teitoklien 2y ago

Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Linux internally.

shanemhansen 2y ago
So ChromeOS and also the OS for GKE are still basically built this way.
6SixTy 2y ago
Afaik the Gentoo based ChromeOS is being replaced by Android.
- mycall 2y ago
  It will be interesting if Gentoo supported Fuchsia next.
  - yjftsjthsd-h 2y ago
    Gentoo already has prefix support for non-Linux systems and used to have at least some interest in a full Gentoo/kFreeBSD, so it's plausible.
  - surajrmal 2y ago
    What would that even mean?
    - sham1 2y ago
      Portage on top of Fuchsia, probably. There's not all that much more to Gentoo all things considered.

ac130kz 2y ago

-march=native + mimalloc (or jemalloc) should be sufficient without causing significant undefined behavior like -O3 or most extra optimization related compiler arguments.

eru 2y ago
Yes, -O3 is seldom an improvement for most C compilers.
loeg 2y ago
-O3 doesn't cause undefined behavior.
- MyOutfitIsVague 2y ago
  No, but it can often expose undefined behavior that doesn't stop a program from "working" on lower optimization levels. Which is why I'm in support of always building your own software with -O3.
  - vlovich123 2y ago
    Are you sure about that? I’m not aware of any additional UB exploitation enabled by O3 vs O2 or Os.
    - MyOutfitIsVague 2y ago
      Nope, I'm not sure about it. I remember when I was using Gentoo about 10 years ago, this was the common reason given for using -O2 instead of -O3 in your build flags, and I'm just speaking from that memory.
      - vlovich123 2y ago
        Yeah there’s lots of cargo culting FUD like that that happens (see other posters in this thread that claim that glibc is a more stable allocator )

linkerdoo 2y ago

[dead]

DrNosferatu 2y ago

Sometimes comes to mind: Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware? Plus, the configuration could be automated - just push a button to (lazy, in background) recompile. Priority for the ones profiled to be performance critical.

eru 2y ago
> Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware? Well, in principle yes. But you'd also need to figure out what you mean by 'optimal' flags? The flags might differ between different packages, and not all packages will be compatible with all the flags. Even worse, in the example in the article they got most of their speedup out of moving to a different allocator, and that's not necessarily compatible with all packages (nor an improvement for all packages). However, if you still want to do all these things, then Gentoo Linux is the distribution for you. It supports this tinkering directly and you'll have a much happier time than trying to bend Ubuntu to your will.

Uptrenda 2y ago

[flagged]

msm_ 2y ago
Waste minutes of your time just so anonymous people on the internet can read your comment :). Not everything in life is an optimization problem. Fun little projects like this is what makes us human (and, arguably, are both educational and entertaining).
- Uptrenda 2y ago
  I see what you're saying but I am also talking about how computers originally saved humans time and that the OP seems to now be slaving away to save the CPU time. Doesn't that seem a little backwards? If we're talking about micro-seconds of difference, the trade-off doesn't seem worth it. Even on a mass scale where this is somehow adopted, nobody is going to notice the difference. Maybe if this were in something like eCommerce and web browing where the lag translates to profit lost? Or perhaps game engines? IDK, I just consider humans time more precious than a slow package (that already runs blazingly fast on a CPU that it barely matters.)
  - alpaca128 2y ago
    Longer waiting times can decrease productivity a lot more than just the runtime increase. If a program takes long enough that the user switches context mentally, or goes to get another coffee or looks at a few posts on social media that now means far more wasted time. Humans are not robots and won’t just keep staring at the screen. Also a lot of simple optimizations could save seconds or minutes in the lives of millions of people, and across multiple devices/programs that adds up. Microsoft once sped up the booting process of their consoles by 5 seconds, by simply making the boot animation shorter by 5 seconds. Those consoles were sold to millions of people and it took MS 8 years or so to make the fix. That’s many lifetimes wasted once you add it all up.
Daneel_ 2y ago
Today it's 2.2GB of JSON files, tomorrow it's tens of petabytes of archived data that you need to look through for work. A speedup of 2x is nothing to scoff at. As a broader comment though, this is how we learn and discover things. Just because the specific outcome here is "trivial" doesn't mean the approach or the lessons learnt aren't valuable. I found it an interesting read despite not have any stake in the outcome.

zX41ZdbW 2y ago

ClickHouse will be faster for processing large JSON data files. The example from the gist will look like follows: ch -q "WITH arrayJoin(features) AS f SELECT f.properties.SitusCity WHERE f.properties.TotalNetValue < 193000 FROM 'data.json'" Reference: https://jsonbench.com/ https://jsonbench.com/

jeffbee 2y ago
I'm willing to believe that will execute in less than 2 seconds, but it doesn't work as given.
- rustc 2y ago
  Can you post a link to the JSON file? Try this (the placement of FROM was incorrect): ch "WITH arrayJoin(features) AS f SELECT f.properties.SitusCity FROM 'a.json' WHERE f.properties.TotalNetValue < 193000"
  - jeffbee 2y ago
    It's a public data file accessible at https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcdba81_0/explore https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcd...

77pt77 2y ago

If you decide to go down this road just use Gentoo.

benatkin 2y ago
Not a bad idea. Arch also comes to mind, and its support on https://copy.sh/v86/ https://copy.sh/v86/ is a testament to its flexibility.
m000 2y ago
I haven't kept up with Gentoo. How long does it take these days to bootstrap a functional desktop from source? I remember when it was my main driver back in early '00s, running on a horribly underpowered PII at 266MHz. It would often be compiling 24/7 to keep up with the updates.
- 77pt77 2y ago
  Haven't used it in a looong time, but by 2010 it was no longer that bad. My first install was 2002 and it took my a good 24 hours to get X to boot. It did catch a faulty memory DIM later on because the linking kept failing intermittently.
  - hy4000days 2y ago
    [dead]
- hy4000days 2y ago
  [dead]
hy4000days 2y ago
[dead]

electromech 2y ago

I'd be curious how the performance compares to this Rust jq clone: cargo install --locked jaq (you might also be able to add RUSTFLAGS="-C target-cpu=native" to enable optimizations for your specific CPU family) "cargo install" is an underrated feature of Rust for exactly the kind of use case described in the article. Because it builds the tools from source, you can opt into platform-specific features/instructions that often aren't included in binaries built for compatibility with older CPUs. And no need to clone the repo or figure out how to build it; you get that for free. jaq[1] and yq[2] are my go-to options anytime I'm using jq and need a quick and easy performance boost. [1] https://github.com/01mf02/jaq https://github.com/01mf02/jaq [2] https://github.com/mikefarah/yq https://github.com/mikefarah/yq

saghm 2y ago
As a bonus that people might not be aware of, in the cases where you do want to use the repo directly (either because there isn't a published package or maybe you want the latest commit that hasn't been released), `cargo install` also has a `--git` flag that lets you specify a URL to a repo. I've used this a number of times in the past, especially as an easy way for me to quickly install personal stuff that I throw together and push to a repo without needing to put together any sort of release process or manually copy around binaries to personal machines and keep track of the exact commits I've used to build them.
oguz-ismail 2y ago
> I'd be curious how the performance compares to this Rust jq clone Every once in a while I test jaq against jq and gojq with my jq solution to AoC 2022 day 13 https://gist.github.com/oguz-ismail/8d0957dfeecc4f816ffee79d650c9d38 https://gist.github.com/oguz-ismail/8d0957dfeecc4f816ffee79d... It's still behind both as of today

zadler 2y ago

[flagged]

alberth 2y ago

Isn’t this what ClearLinux essentially is (way over generalizing)

rgmerk 2y ago

It's a while since I had to deal with this kind of thing, but my memory was that as soon as you go beyond the flags that the upstream developers use (just to be clear, I mean the upstream developers, not the distro packagers) you're buying yourself weird bugs and a whole lot of indifference if they occur. I haven't used a non-libc malloc before but I suspect the same applies.

taeric 2y ago
I've been building my own emacs for a long time, and have yet to hit any weird bugs. I thought that as long as you avoid any unsafe optimizations, you should be fine? Granted, I also thought that -march=native was the main boost that I was seeing. This post indicates that is not necessarily the case. I also suspect that any application using floats is more likely to have rough edges?
- pertymcpert 2y ago
  Complex software usually has some undefined behavior lurking that at higher or even just different optimization levels can trigger the compiler to do unexpected things to the code. It happens all the time in my line of work. If there's an extensive test suite you can run to verify that it still works mostly as expected then it's easier.
  - taeric 2y ago
    This is one where I suspect we don't disagree. But "all the time" can have a very different feel between people. It also used to happen that just changing processors was likely to find some problems in the code. I have no doubt that still happens, but I'd also expect it has reduced. Some of this has to be a convergence on far fewer compilers than we used to encounter. I know there are still many c compilers. Seems there are only two common ones, though. Embedded, of course, is a whole other bag of worms.
- Y_Y 2y ago
  Did you try -ffast-math? IIRC that used to break emacs in some subtle way, while providing no extra speed.

john-tells-all 2y ago

If you actually do this, just get Ubuntu to download the exact source package you want. In this case: apt-get source jq Then go into the package and recompile to your heart's content. You can even repackage it for distribution or archiving. You'll get a result much closer to the upstream Ubuntu, as opposed to getting lots of weird errors and misalignments.

smallstepforman 2y ago

Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch malloc again, others constantly release and acquire. How well do they handle fragmentation? What is the uptime (10 seconds or 10 years)? Sometimes the choice of allocators is the difference between long term stability vs short term speed. I experimented with different allocators devoloping a video editor testing 4K videos that caches frames. 32Mb per frame, at 60fps, thats almost 2Gb per second per track. You quickly hit allocator limitations, and realise that at least vanilla glibc allocator offers the best long term stability. But for short running benchmarks its the slowest. As already pointed out, engineering is a compromise.

vlovich123 2y ago
Mimalloc is a general purpose allocator like JEMalloc / TCMalloc. Glibc is known to have a pretty bad allocator that modern allocators like MIMalloc & the latest TCMalloc (not the one available by default in Ubuntu) run laps around. While of course speedup may be variable, generally the benchmarks show an across the board speedup (whether that matters for any given application is something entirely different). As for crashes, these are all general purpose multi-thread allocators and behave no differently from glibc (modulo bugs that can exist equally in glibc).
- deleted 2y ago
  [deleted]
- smallstepforman 2y ago
  Agree for most short running apps. I updated my comment to reflect issues with apps that are constantly reallocating, and running for longer that 60 seconds. But you are absolutely correct for most short running apps, 99% recommended to replace glibc. However, there is an app or two where glibc stability doesnt trigger a pathological use cases, and you have no choice. Hence why its the default, since there are less crashes in pathaligical cases. And the devs are exhausted dealing with crashing bugs which can be eliminated by using the slower allocator.

brucehoult 2y ago

Since the test data is not available for us to experiment ourselves, I've asked the poster to try it with Boehm GC compiled as a transparent malloc() replacement.

coolgoose 2y ago

Reminds me how I learned Linux by bootstrapping Gentoo from Knoppix live cd's on my Athlon xp 1600+ and using God knows how many compilations flags

dlcarrier 2y ago

For a lot of packages, installing the .deb file instead of the Snap package can in and of itself result in several times faster performance.

sedatk 2y ago

When I was using Linux in 1995, installing a package meant downloading the sources and running “make install”. Have we come full circle?

sneak 2y ago

I love the part where he showed us the commands to rebuild the packages with different compilers and LDFLAGS.

deleted 2y ago

[deleted]

rubatuga 2y ago

Chimera Linux defaults to using clang + mimalloc Read more here: https://chimera-linux.org/about/ https://chimera-linux.org/about/

polskibus 2y ago

This is gentoo resurrected! Great memories building entire Linux from scratch on install ;)

dgan 2y ago

what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes. i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files

Hackbraten 2y ago
There might me some kind of malware scanner at play. Maybe it's time to rebuild those with mimalloc, too.
nurettin 2y ago
Does ${corporate} prevent you from merging your jsons to a single file with a decent format like parquet?
- dgan 2y ago
  they don't but that defeats the purpose. i want to be able to inspect the insides, so binary formats are out, i have to easily open & read the content
  - nurettin 2y ago
    I don't get how you can't put the content within the merged file and query it but good luck.
    - dgan 2y ago
      because that d be a several hundreds megabytes json :) Anyway with suggestion in another comment, i moved the root directory into an antivirus-excluded location, which gave me almost a 10 times performance win
HdS84 2y ago
Try excluding that directory from the virus scanner. Also a dev drive might come in handy for this workload

globular-toast 2y ago

No GCC march=native?

penguin_booze 2y ago
Wouldn't that be the default?
rurban 2y ago
He added it of course
- globular-toast 2y ago
  With clang. GCC is known to do better with march=native, or at least it was.

yxhuvud 2y ago

What i wonder is when any main stream distro will wake up and change their default allocator away from the very dated glibc.

teekert 2y ago

Ah yes, this is nice but it sounds like Gentoo with extra steps. Probably also some extra benefits. But there is a reason we don’t all run Gentoo. Although given a large and seamless enough cache, maybe we would.

rurban 2y ago

He is still missing profile guided optimizations (with his testcase), and bolt. And in some/most cases -Os is faster than -O3 Boehm GC is usually slower than glibc with lots of allocs. MPS would be better, but needs lots of rewriting. https://github.com/Ravenbrook/mps https://github.com/Ravenbrook/mps

roeles 2y ago
> And in some/most cases -Os is faster than -O3 This surprises me. Can you please elaborate?
- withinboredom 2y ago
  If the whole program can fit in the L1/L2 cache, it can speed up things quite a lot!
- rurban 2y ago
  A IC cache miss costs 50x more on average. If your insn and data are small, the overhead of doing more work is less than the overhead of cache misses
jeffbee 2y ago
BOLT is only good for ~2% in this scenario, which is about what I would have expected from glancing at the profiles. -Os is much slower, as I would have expected. Nothing in the perf data suggested high L1i miss rate. The .text of the "size optimized" program is, for some reason, 16x larger.

hyperman1 2y ago

Reading this, I wonder a few things that seem distro quick wins: 1) Why don't distros replace the glibc allocator with one of the better ones? 2) Why don't distros allow for making server-specific builds for only a few packages? You don't have to pay the compilation cost for everything, just a list of 2 or 3 packages.

vkazanov 2y ago
Glibc is a reliable generalist while most other ones specialise in something: single thread, multiple threads, slab, arena style, etc .. Some of the other things disabled might be useful for distros in general, like ndebug assertions or compiling with debig symbols.
- hyperman1 2y ago
  Is glibc being generalist still true? I might be wrong here, but I had the impression it was open and free but behind most others,and e.g. tcmalloc is better on most criteria. Do we have benchmarks and comparisons for the generalist case?
hinkley 2y ago
Is the glibc allocator still that bad? I thought they replaced that ages ago.

nottorp 2y ago

Make a specific ubuntu package 90% faster by rebuilding it after changing the memory allocator, that is. Should probably work for Debian and RedHat too. For this particular package. Edit: based just on the title i initially thought this is an article about turning Ubuntu into Linux From Scratch.

Aachen 2y ago

Reading that such a simple change can get such a big speedup, my first thought is to let the authors of jq know. Maybe there's a caveat to be aware of, maybe they'll test it and end up making it faster for everyone. Useful to drop a quick note pretty much no matter the result, I think? The article doesn't seem to even consider that option and I don't see any comment here mentioning this either. Am I missing something?

chasil 2y ago
I would be curious if Intel Clear Linux had similar gains, by using newer opcodes in the instruction set. I wonder if the glibc allocator is standard there. https://en.m.wikipedia.org/wiki/Clear_Linux_OS https://en.m.wikipedia.org/wiki/Clear_Linux_OS

jbverschoor 2y ago

On FreeBSD doing a make world / make buildworld used to be so common

wengo314 2y ago

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in stabilizing said application's memory usage. it actually fixed a problem that appeared to be a memory leak, but probably wasn't fault of the app itself (likely memory fragmentation with standard malloc)

dietr1ch 2y ago
To me it's obviously a scam because there's no way such an improvement can be achieved globally with a single post explanation. 90% faster is a micro-benchmark number.
- looofooo0 2y ago
  True that, I mean it is still interesting, that if you have a narrow task, you might achieve some significant speed up from rebuilding them. But this is a very niche application.
  - wengo314 2y ago
    true, i saw a thread recently on reddit where guy hand-tuned compilation flags and did pgo profiling for a video encoder app that he uses on video encode farm. In his case, even a gain of ~20% was significant. It calculated into extra bandwidth to encode a few thousand more video files per year.
- arghwhat 2y ago
  This is neither a micro-benchmark nor a scam, but it is click-bait by not mentioning jq specifically. Micro-benchmarks would be testing e.g. a single library function or syscall rather than the whole application. This is the whole application, just not one you might care that much for the performance of. Other applications will of course see different results, but stuff like enabling LTO, tuning THP and picking a suitable allocator are good, universal recommendations.

lifeinthevoid 2y ago

People will run into subtle bugs and weird behavior when starting to compile things themselves and replacing libraries I think. Not worth the effort for me, fun as an exercise, but would only do it if it's really (really) needed for performance.

p0w3n3d 2y ago

I wonder why my brain calculates `90% faster' as 10% of the time (i.e. t1=0.1*t0) it used to be working. However, this makes sense only if we calculate the recompiled application running time as t1=t0/1.90

withinboredom 2y ago
"by 190%" instead of "90% faster" is far clearer.

freep1zza 2y ago

Even Canonical themselves sobered up and abandoned their v3 recompilation effort as it showed no serious benefits. Distros who went this path is going the way of the dinousaurs, together with their confused user base.

sheepscreek 2y ago

In the late 1990s, the most common method for distributing a package was by providing its source code as a tarball. You essentially had to run ./configure and make on every package. I believe ./configure does some of what the OP did in a clever way, checking for locally installed alternatives, etc. It’s still pretty cool, though. I didn’t think a malloc replacement would significantly improve performance.

mbar84 2y ago

Why would you give your product a quite specific name that's exactly the same as another product? "Core 2 Duo" immediately brings an Intel CPU to mind and will probably make it harder to search for anything related to the watch.

tpoacher 2y ago

Obviously the title / article is clickbait. But in general I have often found great benefit in many cases in obtaining packages from source and compiling them, or downloading official binaries, even if the repository does have the latest version in theory. (shameless plug incoming): My only obstacle with such "Manually Installed [and/or] Source-Compiled" (MISC) packages was checking for updates. So I made this: https://sr.ht/~tpapastylianou/misc-updater/ https://sr.ht/~tpapastylianou/misc-updater/ Works great, and I've been slowly improving it to add 'upgrader' commands for some of the packages too. It's not a full replacement for a package manager, nor is it meant to be. But it's made my ability to use MISC packages instead of repository ones a lot more streamlined and easier.

kedmi 2y ago

Please rebuild Chrome

jeffbee 2y ago
Chrome is optimized right up to its eyebrows. If you are using Google distributions of Chrome binaries, I wouldn't expect improvements from build tweaks.

simlevesque 2y ago

Isn't GeoJSON a really bad tool for what you are trying to do ? Just use GeoParquet: Converter: https://geoparquet.org/convert/ https://geoparquet.org/convert/ It'll scan way less data and be 90% faster than your 90% faster jq.

jeffbee 2y ago
GeoJSON is an interchange format that a government agency is using to provide this data, and it updates continuously, so at some point I will be interested in the efficiency of the ETL even if I start doing all my queries against an optimized format.

mannyv 2y ago

What you should take away from this is that there are things you can do to speed up your app that are outside of normal code optimization.

rajaravivarma_r 2y ago

It is funny, how being a corporate-Rails programmer taught me this early in my career. Because building Ruby from source was the only way you could install latest Ruby versions (using ruby-build) and installing Ruby from source means its dependencies like openssl, zlib had to be installed from source too, to match the required versions. And for a long time using Jemalloc was the only was to keep the memory usage constant with multi-threaded ruby programs like Puma and Sidekiq. This was either achieved through compiling Ruby with jemalloc or modifying the LD_LIBRARY_PATH. Some developers also reported 5% or so reduction in response times with jemalloc, iirc. The problem with this approach is though, when a package has a lot of dependencies like ImageMagick which relies on jpeg, png, ghost and a lot of other libraries, you have to take a trial and error approach until it succeeds. Fortunately fixing the dependency errors are the easiest, sometimes building Python from source would throw errors from headers which are impossible to understand. If you find a Stackoverflow solution then you are good, or you have to go down the rabbit hole coming either successful or empty handed based on your level of expertise.

alexmyczko 2y ago

I speed up /sbin/reboot for x86-64 1000 times! might want to run sync before. https://github.com/alexmyczko/autoexec.bat/blob/master/kbd-reboot.c https://github.com/alexmyczko/autoexec.bat/blob/master/kbd-r...

hy4000days 2y ago

[dead]

kristianp 2y ago

I tried this with a 1.3GB file [1], and got Gemini to convert the jq query into a go program, that takes 6.6s the second time its run.(to cache the geojson file). My laptop isn't particularly fast (i7-1250p), and go's json handling isn't particularly fast either, so jq's time is not impressing me when run on a ryzen 9 desktop processor on a 500MB file. It's surprising how quick this kind of processing can be in go. [1] https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcdba81_0/about https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcd... The program: package main import ( "encoding/json" "fmt" "os" ) type FeatureCollection struct { Features []Feature `json:"features"` } type Feature struct { Properties Properties `json:"properties"` } type Properties struct { TotalNetValue int `json:"TotalNetValue"` SitusCity string `json:"SitusCity"` } func main() { filename := "Parcels.geojson" data, err := os.ReadFile(filename) if err != nil { fmt.Println("Error reading file:", err) return } var featureCollection FeatureCollection err = json.Unmarshal(data, &featureCollection) if err != nil { fmt.Println("Error unmarshaling JSON:", err) return } for _, feature := range featureCollection.Features { if feature.Properties.TotalNetValue < 193000 { fmt.Println(feature.Properties.SitusCity) } } }

trod1234 2y ago

Isn't it against the rules to post a link like this where the content is walled behind needing an account to access? I know most of us have accounts here, but when we're seeing something we want to read its not conducive to have to have to deal with roadblocks before we can see what is being talked about. Same goes for paywalls.

jeffbee 2y ago
In what way does this require an account? It loads fine for me in a Chrome Incognito window.
- trod1234 2y ago
  The link goes straight to a login page, doesn't let you see the page. On Firefox here.

sswam 2y ago

[dead]