6 ms·

Cool project! The parts I don't like: + Static linking + No extension (shared library) support + The module importers look over-engineered / too complicated

Cool project! The parts I don't like: + Static linking + No extension (shared library) support + The module importers look over-engineered / too complicated + Custom Python distributions + Not sure what Rust gets you here, besides additional complexity + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios) Many years ago, I solved similar problems (Windows-only at the time) with an approach that: + Requires no static linking, absolutely no compilation of Python + Supports packaging any pre-installed stock Python 2 distribution, no modifications except to fix bugs in the standard library + Uses in-memory loading for _everything_ including Python extensions (shared libraries) + Is based on top of a minimal C-bootstrapping layer that can be compiled once and reused + Includes a better loader than the one in py2exe :-] + Delivers an injectable DLL or executable with absolutely no dependencies except OS built-ins + Works on every version of Windows from XP SP0 and beyond The product is commercial (platform for modelling APT attacks and performing automated exfiltration pentests) but I did a presentation [1] which lays out the entire loader and the benefits of my approach in a way that is easy to independently re-engineer. I can see some benefits of static linking on Unix (even though I'd still avoid it there), but I'm convinced that it's absolutely the wrong thing to do for Windows. [1] https://downloads.immunityinc.com/infiltrate-archives/python_deflowered.pdf https://downloads.immunityinc.com/infiltrate-archives/python...

gravypod 7y ago
Why is "Static linking" bad? A majority of software engineers are shifting to statically linked to reduce runtime complexity. Python 2.7 is also deprecated. New projects shouldn't really be supporting it going forward in my opinion. We're coming up on the 2020 complete end of life cycle.
- metroholografix 7y ago
  It depends on the problem. Someone could say that static linking optimizes for runtime costs but for the problems that OP (and me) are trying to solve, I find that it increases runtime complexity since: + It tightly couples independent subsystems together, you can no longer see them (e.g. update) in isolation. Some of those subsystems explode when statically linked due to second-order effects (e.g. memory management). + Requires compilation. + Licensing woes that do not apply with dynamic linking. + Most important: Static linking reduces your runtime flexibility by being the opposite of late-binding and forces you to commit upfront. Many problems can be elegantly solved when one can take additional context into account (e.g. delay solving them until the 'how' becomes clear). On top of that, static linking does not optimize for deployment costs, which can be substantial depending on the application and frequency of component changes. For all these reasons, it should be avoided when possible. The counterpoint is that in practice, it takes a lot more effort to design a system that does not deploy static linking in order to solve the problems described by OP and me. Static linking is thus an easy and quick solution but it comes with many strings attached.
beagle3 7y ago
It's been a while since I looked at this (or at Windows, for that matter), but back in the day one thing that consistently failed for these approaches were thread initialization - after you load a DLL, every new thread gets a call to DllMain with reason DLL_THREAD_ATTACH; You can hook CreateThread and friends, but some threads are not created through those but rather through native APIs. I don't remember the details, but ... I do remember failing to find a working solution (myself or in any of the projects I checked at the time).
- metroholografix 7y ago
  https://docs.microsoft.com/en-us/windows/desktop/api/libloaderapi/nf-libloaderapi-disablethreadlibrarycalls https://docs.microsoft.com/en-us/windows/desktop/api/libload...
  - beagle3 7y ago
    This tells how to stop it from happening. But my problem was that I needed it to happen (e.g. if using msvcrt) and having no way to do so for DLLs “manually” loaded from memory.
    - metroholografix 7y ago
      One way is to use the Windows library loader and simply trick it to load a DLL from memory rather than disk. If you use a method that completely bypasses the OS loader, then you have to do a lot more work to have your DLL behave like a DLL loaded by the OS would.
      - beagle3 7y ago
        How do you trick the loader to load from memory?
        metroholografix 7y ago
        There are many ways but the one I chose is in my presentation that I linked above. Roughly, first you get an idea of the system calls involved when a DLL is loaded. Then you hook the system calls that are responsible for mapping/reading in order to redirect them to a memory buffer. Conceptually, it's straightforward. The devil is in the implementation details as they say.
sametmax 7y ago
> The parts I don't like: > + Static linking It's actually what I like in this project. Dynamic linking causes many issues, especially on Linux, where you never have quite the same libs depending of the distro. If I want a standalone exe, I want static linking. If I want dynamic linking, I'll make a deb or a rpm. > + No extension (shared library) support > + The module importers look over-engineered / too complicated Man, the project just came out, give it time. You're very harsh on this. > + Custom Python distributions Well, yes. That's why it works. Otherwise you are back to require a C compiler on the machine or use dynamic linking. > + Not sure what Rust gets you here, besides additional complexity - link to the custom python distrib, allowing you to use the nice rust toolchain instead of the mess that is the c toolchain or dynamic linking - free rust / python interrop. This is very interesting. - potentially cross compilation and even make the tools stand alone itself > + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios) Python 2.7 is reaching EOL in a few months. It's perfectly reasonable to not support it. No poney.

gravypod 7y ago

Why is "Static linking" bad? A majority of software engineers are shifting to statically linked to reduce runtime complexity. Python 2.7 is also deprecated. New projects shouldn't really be supporting it going forward in my opinion. We're coming up on the 2020 complete end of life cycle.

metroholografix 7y ago
It depends on the problem. Someone could say that static linking optimizes for runtime costs but for the problems that OP (and me) are trying to solve, I find that it increases runtime complexity since: + It tightly couples independent subsystems together, you can no longer see them (e.g. update) in isolation. Some of those subsystems explode when statically linked due to second-order effects (e.g. memory management). + Requires compilation. + Licensing woes that do not apply with dynamic linking. + Most important: Static linking reduces your runtime flexibility by being the opposite of late-binding and forces you to commit upfront. Many problems can be elegantly solved when one can take additional context into account (e.g. delay solving them until the 'how' becomes clear). On top of that, static linking does not optimize for deployment costs, which can be substantial depending on the application and frequency of component changes. For all these reasons, it should be avoided when possible. The counterpoint is that in practice, it takes a lot more effort to design a system that does not deploy static linking in order to solve the problems described by OP and me. Static linking is thus an easy and quick solution but it comes with many strings attached.

beagle3 7y ago

It's been a while since I looked at this (or at Windows, for that matter), but back in the day one thing that consistently failed for these approaches were thread initialization - after you load a DLL, every new thread gets a call to DllMain with reason DLL_THREAD_ATTACH; You can hook CreateThread and friends, but some threads are not created through those but rather through native APIs. I don't remember the details, but ... I do remember failing to find a working solution (myself or in any of the projects I checked at the time).

metroholografix 7y ago
https://docs.microsoft.com/en-us/windows/desktop/api/libloaderapi/nf-libloaderapi-disablethreadlibrarycalls https://docs.microsoft.com/en-us/windows/desktop/api/libload...
- beagle3 7y ago
  This tells how to stop it from happening. But my problem was that I needed it to happen (e.g. if using msvcrt) and having no way to do so for DLLs “manually” loaded from memory.
  - metroholografix 7y ago
    One way is to use the Windows library loader and simply trick it to load a DLL from memory rather than disk. If you use a method that completely bypasses the OS loader, then you have to do a lot more work to have your DLL behave like a DLL loaded by the OS would.
    - beagle3 7y ago
      How do you trick the loader to load from memory?
      - metroholografix 7y ago
        There are many ways but the one I chose is in my presentation that I linked above. Roughly, first you get an idea of the system calls involved when a DLL is loaded. Then you hook the system calls that are responsible for mapping/reading in order to redirect them to a memory buffer. Conceptually, it's straightforward. The devil is in the implementation details as they say.

sametmax 7y ago

> The parts I don't like: > + Static linking It's actually what I like in this project. Dynamic linking causes many issues, especially on Linux, where you never have quite the same libs depending of the distro. If I want a standalone exe, I want static linking. If I want dynamic linking, I'll make a deb or a rpm. > + No extension (shared library) support > + The module importers look over-engineered / too complicated Man, the project just came out, give it time. You're very harsh on this. > + Custom Python distributions Well, yes. That's why it works. Otherwise you are back to require a C compiler on the machine or use dynamic linking. > + Not sure what Rust gets you here, besides additional complexity - link to the custom python distrib, allowing you to use the nice rust toolchain instead of the mess that is the c toolchain or dynamic linking - free rust / python interrop. This is very interesting. - potentially cross compilation and even make the tools stand alone itself > + No support for Python 2.7 (Python 3 drops support for Windows XP which is still a legit platform for many scenarios) Python 2.7 is reaching EOL in a few months. It's perfectly reasonable to not support it. No poney.