6 ms·

Do you cross-compile from aarch64 or to aarch64? I tend to treat the desire to cross-compile as a rather strong indicator that the architecture is just not via

Do you cross-compile from aarch64 or to aarch64? I tend to treat the desire to cross-compile as a rather strong indicator that the architecture is just not viable for general-purpose computing. Either because it's impossible to get actual hardware at all, or the hardware doesn't have decent specs. In both cases, it's going to be difficult find a business justification for building for this architecture (again for general-purpose computing). Obviously, architecture bringup is an exception, but we are way past that for aarch64.

rcme 4y ago
[dead]
mise_en_place 4y ago
Well, afaik aarch64 is not some exotic target architecture. It is well known, well specified, and while implementations may vary hardware wise, it is generally compatible between different implementations. It’s not completely alien and new like IA64 or RISC-V. Even then it seems like porting from Alpha to IA64 wasn’t super painful.
- nine_k 4y ago
  BTW x64 ans aarch64 have pretty different memory consistency models, so some performance-sensitive C++ code may either end up unnecessarily slow, or even surface a bug. But this only applies to a very tiny amount of in-house code which hasn't been tested on aarch64 before. I'd expect 99.9% of C++ code to just cross-compile and run without issues.
  - liuliu 4y ago
    The trick is on how to locate that 0.1% of C++ code ...
    - nine_k 4y ago
      Usually stuff that implements advanced multithreading, concurrent thread-safe data structures, etc is pretty well isolated in a codebase, with spiky fences around it, and signs saying "You are not expected to understand this" [1] on the fences. [1]: http://9p.io/who/dmr/odd.html http://9p.io/who/dmr/odd.html
  - isityouyesitsme 4y ago
    I'm not so sure. Maybe by literal line count. Maybe not by all targets in a large project. Literally every time in my life I've updated from version A of a compiler to version B of the same exact compiler, I had weeks of change sets required for getting the existing code to build under compiler B.
motiejus 4y ago
Currently x86_64 to aarch64. It's a bootstrapping problem. It takes a lot of infra to run the first build host. We can transition to aarch64 for build hosts if we see the need later. Having cross-compilation makes things much easier from get-go.
- whyohwhywhy 4y ago
  Can you explain your rationale for moving your infrastructure to an architecture that is both more difficult to procure and more difficult to build for? Surely there is some huge win I'm missing here, so if you have figures showing the math you've done, I'm definitely interested.
  - connicpu 4y ago
    I'm assuming power efficiency would be a big part of it, that tends to be the draw for ARM servers
  - rincebrain 4y ago
    I don't think it's that aarch64 specifically is difficult to build for, it's that adding _any_ additional architecture is a significant upfront amount of work, since you need to set up all the dependent platform-specific things for the first time.
  - bri3d 4y ago
    For anyone operating exclusively in "the cloud," AWS Graviton2 instances provide a 10-20+% cost reduction for similar performance to the next comparable instance (of course, highly workload dependent), without procurement issues beyond the usual AWS random regional shortages which are generally accounted for at scale already. ARM instances from cloud providers are very appealing to most large-scale enterprise cloud customers, because they represent a significant cost savings at only a moderate upfront investment.
sliken 4y ago
Agreed, I view cross-compiling as a hint that the platform isn't ready for general purpose use. I bought a rk3588 arm widget for $120 ($140 with a nice metal case) with 8GB ram and 8 cores. I installed Ubuntu 22.04 LTS and was able to compile Rust and Go from source. So far I'm impressed. It's pretty fast (about half as fast as my 2015 Xeon) at compiling Rust. It seems generally pretty usable aarch64 platform and bodes well for other aarch64 implementations. I believe the rk3588 requires a custom (not mainline) kernel, but I believe the important bits have been upstreamed and should appear soon ... hopefully. Neat widget, no fan, cheap, and reasonably fast. Was something like 6-7 times faster than an Rpi4 at compiling rust.
- Fnoord 4y ago
  120 USD? Seems like a bargain. Do you have a link? QuartzPro64 supposedly will go for 300 USD.
  - sliken 4y ago
    https://www.friendlyelec.com/index.php?route=product/product&product_id=289 https://www.friendlyelec.com/index.php?route=product/product... Pretty happy with it, was easy to get ubuntu 22.04 going, but with their image. From what I can tell they use the same not upstreamed kernel on all their images. I've heard progress, in particular with OpenWRT. It does make a decent router, or just a nice mini server. 8GB ram means it can do handle a fair bit. Compiling rust was 6-8 times faster than a Pi4. It's been stable so far, it does require a usb-c power supply, which I had lying around. It does have a microSD slot. If in the USA you can get it quicker from amazon, but around $200.
    - Fnoord 4y ago
      Neat! I'll wait till OPNsense is properly supported but otherwise a nice bang for the buck. Decent replacement for my EdgeRouter-Lite
zX41ZdbW 4y ago
Cross-compiling is more convenient. If you have an amd64 desktop or aarch64 server, you can compile to amd64 and aarch64, and powerpc64 with almost the same command. The binary without dependencies can be copied to the server with `scp`. This is the approach we use at ClickHouse[1]. Even more, we always do cross-compilation, even if the host and target architectures are identical. It allows making the build hermetic, not dependent on the environment of the host system. [1] https://github.com/ClickHouse/ClickHouse/blob/master/PreLoad.cmake#L76 https://github.com/ClickHouse/ClickHouse/blob/master/PreLoad...

rcme 4y ago

[dead]

mise_en_place 4y ago

Well, afaik aarch64 is not some exotic target architecture. It is well known, well specified, and while implementations may vary hardware wise, it is generally compatible between different implementations. It’s not completely alien and new like IA64 or RISC-V. Even then it seems like porting from Alpha to IA64 wasn’t super painful.

nine_k 4y ago
BTW x64 ans aarch64 have pretty different memory consistency models, so some performance-sensitive C++ code may either end up unnecessarily slow, or even surface a bug. But this only applies to a very tiny amount of in-house code which hasn't been tested on aarch64 before. I'd expect 99.9% of C++ code to just cross-compile and run without issues.
- liuliu 4y ago
  The trick is on how to locate that 0.1% of C++ code ...
  - nine_k 4y ago
    Usually stuff that implements advanced multithreading, concurrent thread-safe data structures, etc is pretty well isolated in a codebase, with spiky fences around it, and signs saying "You are not expected to understand this" [1] on the fences. [1]: http://9p.io/who/dmr/odd.html http://9p.io/who/dmr/odd.html
- isityouyesitsme 4y ago
  I'm not so sure. Maybe by literal line count. Maybe not by all targets in a large project. Literally every time in my life I've updated from version A of a compiler to version B of the same exact compiler, I had weeks of change sets required for getting the existing code to build under compiler B.

motiejus 4y ago

Currently x86_64 to aarch64. It's a bootstrapping problem. It takes a lot of infra to run the first build host. We can transition to aarch64 for build hosts if we see the need later. Having cross-compilation makes things much easier from get-go.

whyohwhywhy 4y ago
Can you explain your rationale for moving your infrastructure to an architecture that is both more difficult to procure and more difficult to build for? Surely there is some huge win I'm missing here, so if you have figures showing the math you've done, I'm definitely interested.
- connicpu 4y ago
  I'm assuming power efficiency would be a big part of it, that tends to be the draw for ARM servers
- rincebrain 4y ago
  I don't think it's that aarch64 specifically is difficult to build for, it's that adding _any_ additional architecture is a significant upfront amount of work, since you need to set up all the dependent platform-specific things for the first time.
- bri3d 4y ago
  For anyone operating exclusively in "the cloud," AWS Graviton2 instances provide a 10-20+% cost reduction for similar performance to the next comparable instance (of course, highly workload dependent), without procurement issues beyond the usual AWS random regional shortages which are generally accounted for at scale already. ARM instances from cloud providers are very appealing to most large-scale enterprise cloud customers, because they represent a significant cost savings at only a moderate upfront investment.

sliken 4y ago

Agreed, I view cross-compiling as a hint that the platform isn't ready for general purpose use. I bought a rk3588 arm widget for $120 ($140 with a nice metal case) with 8GB ram and 8 cores. I installed Ubuntu 22.04 LTS and was able to compile Rust and Go from source. So far I'm impressed. It's pretty fast (about half as fast as my 2015 Xeon) at compiling Rust. It seems generally pretty usable aarch64 platform and bodes well for other aarch64 implementations. I believe the rk3588 requires a custom (not mainline) kernel, but I believe the important bits have been upstreamed and should appear soon ... hopefully. Neat widget, no fan, cheap, and reasonably fast. Was something like 6-7 times faster than an Rpi4 at compiling rust.

Fnoord 4y ago
120 USD? Seems like a bargain. Do you have a link? QuartzPro64 supposedly will go for 300 USD.
- sliken 4y ago
  https://www.friendlyelec.com/index.php?route=product/product&product_id=289 https://www.friendlyelec.com/index.php?route=product/product... Pretty happy with it, was easy to get ubuntu 22.04 going, but with their image. From what I can tell they use the same not upstreamed kernel on all their images. I've heard progress, in particular with OpenWRT. It does make a decent router, or just a nice mini server. 8GB ram means it can do handle a fair bit. Compiling rust was 6-8 times faster than a Pi4. It's been stable so far, it does require a usb-c power supply, which I had lying around. It does have a microSD slot. If in the USA you can get it quicker from amazon, but around $200.
  - Fnoord 4y ago
    Neat! I'll wait till OPNsense is properly supported but otherwise a nice bang for the buck. Decent replacement for my EdgeRouter-Lite

zX41ZdbW 4y ago

Cross-compiling is more convenient. If you have an amd64 desktop or aarch64 server, you can compile to amd64 and aarch64, and powerpc64 with almost the same command. The binary without dependencies can be copied to the server with `scp`. This is the approach we use at ClickHouse[1]. Even more, we always do cross-compilation, even if the host and target architectures are identical. It allows making the build hermetic, not dependent on the environment of the host system. [1] https://github.com/ClickHouse/ClickHouse/blob/master/PreLoad.cmake#L76 https://github.com/ClickHouse/ClickHouse/blob/master/PreLoad...