4 ms·

Is this 1.0 ready indeed? a comment in blake3_neon.c: // TODO: This is probably incorrect for big-endian ARM. How should that work?

kbumsik 5y ago
I know ARM in principal supports BE, but never heard real BE ARM products yet. Is there one? I am just curious.
- tyingq 5y ago
  The hardware products support setting endianness at boot time. It's just a matter of OS's supporting BE mode. Netbsd releases big endian versions that work on various ARM boards. Here's an announcement showing it works on an Rpi3 and below, for example: https://mail-index.netbsd.org/port-arm/2020/12/03/msg007117.html https://mail-index.netbsd.org/port-arm/2020/12/03/msg007117.... They said at the time that it's not yet working on the Rpi4 because of issues getting BE mode and UEFI to work together. I don't know that it's used often, but it does exist.
  - howinteresting 5y ago
    Why would anyone want to use BE mode at all?
    - tyingq 5y ago
      Handy for testing that your code doesn't have any endian bugs that might crop up if someone tried compiling it on older RISC hardware? Or perhaps very slightly more efficient networking code, since big endian is the default order for many operations there?
- gsnedders 5y ago
  It used to be much more common than it is nowadays; commonality between desktop systems used for development and production has slowly got rid of it. I know certain TVs and set-top boxes at least used to use big-endian, but mostly them long-ago migrated to little-endian nowadays.
- brandmeyer 5y ago
  TMS570 dual-lockstep automotive safety controllers are hardwired to be big-endian. Their close cousins the RM57 line look like nearly identical parts that are little-endian. I believe, but cannot prove, that the only difference in silicon is a factory one-time-programmable setting (fuse). "Why would anyone want to build a BE machine?" In this case, they are trying to gain market-share from big-endian POWER microcontrollers.
pornel 5y ago
1.0 is not the last and final version of a product. There can always be 1.0.1 or 1.1 or 2.0. Numbers are plentiful. Rust ecosystem is already overthinking 1.0 releases, which results in tons of crates having 0.x versions while being depended on as de-facto stable.
- st_goliath 5y ago
  > 1.0 is not the last and final version of a product. Of course, but if you bother to do semantic versioning, it should strive to be a stable one and not a "we are still experimenting" release. IMHO knowing that something doesn't work, or isn't even implemented yet and will be addressed in the next release is OK for an 0.x.y release, but you shouldn't rush towards 1.0.0, already planing to release it "unfinished" and complete it later on in 1.0.1. That IMO kind of misses the point of the versioning semantics.
  - wongarsu 5y ago
    By that logic, would you ever release a 1.0? Imho, just having a stable API that works on x86-64 without optimizations would be enough for a 1.0 release. Having a stable API with highly optimized implementation for x86 and common ARM systems is a lot for a 1.0 release. The limitations on ARM BE could be better documented though.
    - st_goliath 5y ago
      > By that logic, would you ever release a 1.0? Of course! But you need a well defined feature set you want to have for 1.0 and stick to that. The scope of that is up to the people who run the project, I did not make any demands what this specific software should include in their 1.0 release. I did not say that it needs to be perfect, include all bells and whistles imaginable, all possible CPU optimizations and smell nice in order to merit an 1.0 release. What I'm saying is, that you should make an effort to ensure that the implementation of this feature set is somewhat stable. So rather to the contrary, I would also suggest to keep the scope of a 1.0 release smaller than that, just like you suggested: > Imho, just having a stable API that works on x86-64 without optimizations would be enough for a 1.0 release. Releasing an un-optimized reference implementation as 1.0, or maybe only one optimized code path for x86_64 would IMO be perfectly fine. If they decide they really want optimizations for all kinds of CPUs in the 1.0 release, also fine with me. What the scope for 1.0 should or should not be is their choice. And it's also completely besides my point. I'm specifically arguing against doing what pornel seems to imply: My point is, you shouldn't release an implementation you know misbehaves in some cases, because "we can fix it later". I'm a fan of semantic versioning. And I firmly believe that in addition to API & ABI stability, for a major version release, some effort should be taken to iron out the implementation of that API as well. Of course bugs can, and will, crop up later and can be fixed with a patch level release, but IMO you shouldn't rush towards a 1.0 release with a backlog of known bugs for the next release. That's what I meant with "kind of misses the point of the versioning semantics".
jedisct1 5y ago
Looks like the 1.0 tag is only about the Rust implementation. The Changelog only lists Rust things. The algorithm itself hasn't changed, which is great!
cesarb 5y ago
What it being 1.0 means is that the API is stable. There might still be bugs in the implementation (no software is perfect), but once it's reached 1.0, it's hoped that the API is good enough and will not have to change for quite some time. If you look at these release notes for 1.0.0, they were all API changes.
brandmeyer 5y ago
IMO, this shouldn't be a TODO. This should be an #error based on the standard C macros for detecting endianness at compile-time.
- oconnor663 5y ago
  This is a good point, and I'd like to fix it. When I look for endianness macros, it doesn't seem like there's a common standard. Is there an approach you'd recommend?
  - brandmeyer 5y ago
    It seems that you're right - ISO C doesn't have standard macros for this. That said, since this code is ARM-specific, and you are already leveraging the ACLE header <arm_neon.h>, I think that a negative test for __ARM_BIG_ENDIAN is appropriate. https://developer.arm.com/documentation/101028/0012/5--Feature-test-macros https://developer.arm.com/documentation/101028/0012/5--Featu...
    - oconnor663 5y ago
      I just put up https://github.com/BLAKE3-team/BLAKE3/pull/188 https://github.com/BLAKE3-team/BLAKE3/pull/188 for this. Let me know if you have any suggestions about how to test it?

kbumsik 5y ago

I know ARM in principal supports BE, but never heard real BE ARM products yet. Is there one? I am just curious.

tyingq 5y ago
The hardware products support setting endianness at boot time. It's just a matter of OS's supporting BE mode. Netbsd releases big endian versions that work on various ARM boards. Here's an announcement showing it works on an Rpi3 and below, for example: https://mail-index.netbsd.org/port-arm/2020/12/03/msg007117.html https://mail-index.netbsd.org/port-arm/2020/12/03/msg007117.... They said at the time that it's not yet working on the Rpi4 because of issues getting BE mode and UEFI to work together. I don't know that it's used often, but it does exist.
- howinteresting 5y ago
  Why would anyone want to use BE mode at all?
  - tyingq 5y ago
    Handy for testing that your code doesn't have any endian bugs that might crop up if someone tried compiling it on older RISC hardware? Or perhaps very slightly more efficient networking code, since big endian is the default order for many operations there?
gsnedders 5y ago
It used to be much more common than it is nowadays; commonality between desktop systems used for development and production has slowly got rid of it. I know certain TVs and set-top boxes at least used to use big-endian, but mostly them long-ago migrated to little-endian nowadays.
brandmeyer 5y ago
TMS570 dual-lockstep automotive safety controllers are hardwired to be big-endian. Their close cousins the RM57 line look like nearly identical parts that are little-endian. I believe, but cannot prove, that the only difference in silicon is a factory one-time-programmable setting (fuse). "Why would anyone want to build a BE machine?" In this case, they are trying to gain market-share from big-endian POWER microcontrollers.

pornel 5y ago

1.0 is not the last and final version of a product. There can always be 1.0.1 or 1.1 or 2.0. Numbers are plentiful. Rust ecosystem is already overthinking 1.0 releases, which results in tons of crates having 0.x versions while being depended on as de-facto stable.

st_goliath 5y ago
> 1.0 is not the last and final version of a product. Of course, but if you bother to do semantic versioning, it should strive to be a stable one and not a "we are still experimenting" release. IMHO knowing that something doesn't work, or isn't even implemented yet and will be addressed in the next release is OK for an 0.x.y release, but you shouldn't rush towards 1.0.0, already planing to release it "unfinished" and complete it later on in 1.0.1. That IMO kind of misses the point of the versioning semantics.
- wongarsu 5y ago
  By that logic, would you ever release a 1.0? Imho, just having a stable API that works on x86-64 without optimizations would be enough for a 1.0 release. Having a stable API with highly optimized implementation for x86 and common ARM systems is a lot for a 1.0 release. The limitations on ARM BE could be better documented though.
  - st_goliath 5y ago
    > By that logic, would you ever release a 1.0? Of course! But you need a well defined feature set you want to have for 1.0 and stick to that. The scope of that is up to the people who run the project, I did not make any demands what this specific software should include in their 1.0 release. I did not say that it needs to be perfect, include all bells and whistles imaginable, all possible CPU optimizations and smell nice in order to merit an 1.0 release. What I'm saying is, that you should make an effort to ensure that the implementation of this feature set is somewhat stable. So rather to the contrary, I would also suggest to keep the scope of a 1.0 release smaller than that, just like you suggested: > Imho, just having a stable API that works on x86-64 without optimizations would be enough for a 1.0 release. Releasing an un-optimized reference implementation as 1.0, or maybe only one optimized code path for x86_64 would IMO be perfectly fine. If they decide they really want optimizations for all kinds of CPUs in the 1.0 release, also fine with me. What the scope for 1.0 should or should not be is their choice. And it's also completely besides my point. I'm specifically arguing against doing what pornel seems to imply: My point is, you shouldn't release an implementation you know misbehaves in some cases, because "we can fix it later". I'm a fan of semantic versioning. And I firmly believe that in addition to API & ABI stability, for a major version release, some effort should be taken to iron out the implementation of that API as well. Of course bugs can, and will, crop up later and can be fixed with a patch level release, but IMO you shouldn't rush towards a 1.0 release with a backlog of known bugs for the next release. That's what I meant with "kind of misses the point of the versioning semantics".

jedisct1 5y ago

Looks like the 1.0 tag is only about the Rust implementation. The Changelog only lists Rust things. The algorithm itself hasn't changed, which is great!

cesarb 5y ago

What it being 1.0 means is that the API is stable. There might still be bugs in the implementation (no software is perfect), but once it's reached 1.0, it's hoped that the API is good enough and will not have to change for quite some time. If you look at these release notes for 1.0.0, they were all API changes.

brandmeyer 5y ago

IMO, this shouldn't be a TODO. This should be an #error based on the standard C macros for detecting endianness at compile-time.

oconnor663 5y ago
This is a good point, and I'd like to fix it. When I look for endianness macros, it doesn't seem like there's a common standard. Is there an approach you'd recommend?
- brandmeyer 5y ago
  It seems that you're right - ISO C doesn't have standard macros for this. That said, since this code is ARM-specific, and you are already leveraging the ACLE header <arm_neon.h>, I think that a negative test for __ARM_BIG_ENDIAN is appropriate. https://developer.arm.com/documentation/101028/0012/5--Feature-test-macros https://developer.arm.com/documentation/101028/0012/5--Featu...
  - oconnor663 5y ago
    I just put up https://github.com/BLAKE3-team/BLAKE3/pull/188 https://github.com/BLAKE3-team/BLAKE3/pull/188 for this. Let me know if you have any suggestions about how to test it?