3 ms·

> Please note that your APE binary will assimilate itself as a conventional resident of your platform after the first run, so it can be fast and efficient for s

> Please note that your APE binary will assimilate itself as a conventional resident of your platform after the first run, so it can be fast and efficient for subsequent executions. Is my understanding correct that the binary changes itself when first run on the target platform? That sounds like it will trigger a lot of red lights with many automated defensive mechanisms like anti virus.

slimsag 5y ago
Yes, I think that's correct. However, see: https://github.com/jart/cosmopolitan/issues/90 https://github.com/jart/cosmopolitan/issues/90
- jart 5y ago
  Author here. The people who make virus scanners are going to have their heads explode when they learn about redbean's new StoreAsset() function https://justine.lol/redbean/index.html#StoreAsset https://justine.lol/redbean/index.html#StoreAsset where the executable edits itself as though it were MongoDB. It took a pretty heroic hack to work around ETXTBSY where the executable unmaps itself from memory after loading and then remaps itself so it can restore the original APE header and modify the ZIP central directory. https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d735ebbe58e1d9bf1956de39/libc/runtime/openexecutable.S#L25-L189 https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d7... That said, I still upvote on VirusTotal each executable I publish, so you can verify they came from me, provided the executable is byte-for-byte identical to its original form. After you've modified it, it becomes yours, and you can upload your modified version to VirusTotal too and upvote that. In the future, I'd like to have smarter tools for verifying the authenticity of the non-zip bits of APE binaries, but that will come with time.
  - gravypod 5y ago
    As your repo grows do you think you'll eventually need to implement some blaze-like tooling? In previous treads you mentioned compiling the entire repo only takes a minute or two but I'm wondering if the reproducability guarantees would help an effort like this.
    - jart 5y ago
      Author here. Believe it or not, I'm also the author of some of Blaze's coolest features, like its downloader, which reduced TensorFlow build flakes down from 10% to 0%. https://github.com/bazelbuild/bazel/commit/92887a08a55be96ec44e63f0ef0fd287cf49d0cb https://github.com/bazelbuild/bazel/commit/92887a08a55be96ec... The Cosmopolitan Makefile already has Blaze-like tooling. For example, it does strict dependency checking to verify the DEPS list for each static archive specifies all of its directly-reachable dependencies. https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d735ebbe58e1d9bf1956de39/tool/build/package.c#L58-L102 https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d7... See example build config here: https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d735ebbe58e1d9bf1956de39/tool/build/build.mk#L24-L59 https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d7... That was actually one of the first features I wrote for the Cosmopolitan codebase, because it has ~67 static archive .a files, and making sure those all get linked in the correct order and are acyclic would have been nearly impossible without such a tool. I'm sure in the future people will want a more Pythonic build syntax, since with GNU Make it's a bit easy to shoot oneself in the foot. The biggest issue contributors have had so far with the build is that we need to specify in the top-level Makefile a correctly topologically-ordered list of `include foo.mk` lines. Otherwise some pretty counterintuitive errors happen. But one grows used to it after a little pain and the repo feels like second nature. Ultimately, I just really didn't want to invent yet another build system. I'm pretty proud of the fact that (at least for now) I've managed to make GNU Make work so well for such a large repo. Google themselves actually used GNU Make for their codebase until around ~2005 so I'm hoping Cosmopolitan has got at least another decade of use in it.
      - rockwotj 5y ago
        > I'm sure in the future people will want a more Pythonic build syntax > Ultimately, I just really didn't want to invent yet another build system. Is there a reason not to use Bazel initially or in the future when you want a Python syntax?
        jart 5y ago
        No comment.
  - jcelerier 5y ago
    How does that work with Apple's notarization which stores the signature and a hash of your program in Apple's servers ?
    - saagarjha 5y ago
      Presumably com.apple.security.cs.disable-executable-page-protection or similar
GordonS 5y ago
Hmm, on Windows it's common to install binaries to %PROGRAMFILES%, precisely so that non-admins cannot modify them. And in general, users having write access to binaries is frowned upon pretty much everywhere. What I'd like is 2 mechanisms: First, the ability to disable this optimisation altogether. Second, a means to run this optimisation as a distinct step, without executing the rest of the binary. For example, if the "--optimize" flag is used, do the optimisation and then exit.
- deleted 5y ago
  [deleted]
- jart 5y ago
  On Windows APE doesn't need to modify itself because it's an MZ executable. On UNIX the MZ is the prologue to a shell script, which calls printf 'ELF HEADER' >$0; exec $0 to make it conform to the local convention. For subsequent executions it won't need to self-modify so you can simply prime the ape binary once before copying it to /usr/bin if you're an admin. Think of it like a zero-step install wizard that takes a few microseconds.

slimsag 5y ago

Yes, I think that's correct. However, see: https://github.com/jart/cosmopolitan/issues/90 https://github.com/jart/cosmopolitan/issues/90

jart 5y ago
Author here. The people who make virus scanners are going to have their heads explode when they learn about redbean's new StoreAsset() function https://justine.lol/redbean/index.html#StoreAsset https://justine.lol/redbean/index.html#StoreAsset where the executable edits itself as though it were MongoDB. It took a pretty heroic hack to work around ETXTBSY where the executable unmaps itself from memory after loading and then remaps itself so it can restore the original APE header and modify the ZIP central directory. https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d735ebbe58e1d9bf1956de39/libc/runtime/openexecutable.S#L25-L189 https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d7... That said, I still upvote on VirusTotal each executable I publish, so you can verify they came from me, provided the executable is byte-for-byte identical to its original form. After you've modified it, it becomes yours, and you can upload your modified version to VirusTotal too and upvote that. In the future, I'd like to have smarter tools for verifying the authenticity of the non-zip bits of APE binaries, but that will come with time.
- gravypod 5y ago
  As your repo grows do you think you'll eventually need to implement some blaze-like tooling? In previous treads you mentioned compiling the entire repo only takes a minute or two but I'm wondering if the reproducability guarantees would help an effort like this.
  - jart 5y ago
    Author here. Believe it or not, I'm also the author of some of Blaze's coolest features, like its downloader, which reduced TensorFlow build flakes down from 10% to 0%. https://github.com/bazelbuild/bazel/commit/92887a08a55be96ec44e63f0ef0fd287cf49d0cb https://github.com/bazelbuild/bazel/commit/92887a08a55be96ec... The Cosmopolitan Makefile already has Blaze-like tooling. For example, it does strict dependency checking to verify the DEPS list for each static archive specifies all of its directly-reachable dependencies. https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d735ebbe58e1d9bf1956de39/tool/build/package.c#L58-L102 https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d7... See example build config here: https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d735ebbe58e1d9bf1956de39/tool/build/build.mk#L24-L59 https://github.com/jart/cosmopolitan/blob/b8f38cf55d84bac7d7... That was actually one of the first features I wrote for the Cosmopolitan codebase, because it has ~67 static archive .a files, and making sure those all get linked in the correct order and are acyclic would have been nearly impossible without such a tool. I'm sure in the future people will want a more Pythonic build syntax, since with GNU Make it's a bit easy to shoot oneself in the foot. The biggest issue contributors have had so far with the build is that we need to specify in the top-level Makefile a correctly topologically-ordered list of `include foo.mk` lines. Otherwise some pretty counterintuitive errors happen. But one grows used to it after a little pain and the repo feels like second nature. Ultimately, I just really didn't want to invent yet another build system. I'm pretty proud of the fact that (at least for now) I've managed to make GNU Make work so well for such a large repo. Google themselves actually used GNU Make for their codebase until around ~2005 so I'm hoping Cosmopolitan has got at least another decade of use in it.
    - rockwotj 5y ago
      > I'm sure in the future people will want a more Pythonic build syntax > Ultimately, I just really didn't want to invent yet another build system. Is there a reason not to use Bazel initially or in the future when you want a Python syntax?
      - jart 5y ago
        No comment.
- jcelerier 5y ago
  How does that work with Apple's notarization which stores the signature and a hash of your program in Apple's servers ?
  - saagarjha 5y ago
    Presumably com.apple.security.cs.disable-executable-page-protection or similar

GordonS 5y ago

Hmm, on Windows it's common to install binaries to %PROGRAMFILES%, precisely so that non-admins cannot modify them. And in general, users having write access to binaries is frowned upon pretty much everywhere. What I'd like is 2 mechanisms: First, the ability to disable this optimisation altogether. Second, a means to run this optimisation as a distinct step, without executing the rest of the binary. For example, if the "--optimize" flag is used, do the optimisation and then exit.

deleted 5y ago
[deleted]
jart 5y ago
On Windows APE doesn't need to modify itself because it's an MZ executable. On UNIX the MZ is the prologue to a shell script, which calls printf 'ELF HEADER' >$0; exec $0 to make it conform to the local convention. For subsequent executions it won't need to self-modify so you can simply prime the ape binary once before copying it to /usr/bin if you're an admin. Think of it like a zero-step install wizard that takes a few microseconds.