3 ms·

Thanks a lot for your comments! Looking forward a lot to your tweeks, performance will definitely need to be optimized in the future. The problem I was trying t

by phreeza 16y ago

Thanks a lot for your comments! Looking forward a lot to your tweeks, performance will definitely need to be optimized in the future. The problem I was trying to adress with the accessors and the whole *View concept is that there are no private members in python, and i was trying to prevent the minds/genomes from "cheating" by reading data they aren't allowed to. If you can think of a better way to do that, I'd love to hear from you.

deleted 16y ago
[deleted]
- deleted 16y ago
  [deleted]
jerf 16y ago
Last I knew, the official word is that there is no way to stop Python code from reading any particular value. Python can even crack into the values contained in closures if you know the right incantations. If you are looking to rigidly enforce that only accepted interfaces are being used to read data in the face of a hostile adversary writing the Python, I believe the current state-of-the-art is that it is impossible. If this is a core feature requirement, then as far as I know your only option is to change languages to something where you can rigidly sandbox. Your nearest option in programming space would be Perl and some variant of Safe (that's a CPAN module name). Otherwise, if you can live with a gentleman's agreement, you're fine. This may make the automatic-online-matchup really tricky, though.
- amalcon 16y ago
  Is this true of all implementations? I know CPython and PyPy have this limitation, but might it be possible to make something work in IronPython, Jython, or some other project I'm as yet unaware of?
  - devinj 16y ago
    PyPy has no such limitation. PyPy has perfect sandboxing.
    - jerf 16y ago
      Is this ready for primetime? The documentation I found on it seems a bit tentative to me, but it may simply have not been updated. It is worth pursuing if so. Combined with standard OS limitation mechanisms, which I would certainly use in addition to this, it might at least be worth a try. I would wait to say PyPy has "perfect" sandboxing until it has been attacked for a while, though. CPython used to have a sandbox module, too. But at least PyPy has been designed with this use case in mind.
      - devinj 16y ago
        PyPy's sandboxing hasn't been as thoroughly analyzed and tested as, say, Javascript's (not by far) or Java's, but the actual construction is simpler and arguably better. The PyPy devs, at least, are fairly convinced of its robustness. Basically they whitelist system calls and all other calls are translated to sandboxed calls during compilation of the sandboxed interpreter, as I recall. This is entirely different from CPython's restricted execution, which was essentially a blacklist of Python calls/attributes-- every attempt at such a system has died a death of a thousand cuts; it really pays to do it from the bottom up like PyPy or E. It might not be ready for prime-time in some very critical systems (like a web browser for example) but it's honestly probably good enough for this, especially in combination with process jailing as you suggest. I might consider trying to patch Cells up to use PyPy's sandbox instead of the non-solution currently implemented (I cried (okay, groaned) when I heard about it-- yeagh), when I have time, and if nobody else does it first. I'm a bit worried anything I write now would get lost or broken in the incoming tide of patches/changes anyway. The main problems with using PyPy's sandboxing is probably with communication to/out of the sandbox, which is apparently problematic. I also anticipate some complaints about relying on PyPy in the first place, which is rather abnormal compared to using CPython-- using and relying on PyPy-specific features is pretty much unheard of.
- phreeza 16y ago
  > This may make the automatic-online-matchup really tricky, though. Another option I was thinking about for this purpose is parsing the python to make sure they don't get up to anything fishy, but I have no idea how hard a Python parser would be. My guess is very. Or we put them into separate proceses, as swolchok suggested.
  - jerf 16y ago
    "Another option I was thinking about for this purpose is parsing the python to make sure they don't get up to anything fishy." If that was safe, it would have been bundled up into a module. This is a Frequently Asked Question for Python, and the answer is, you can't. I thought of process separation, but it won't work. The best performance would be one persistent process that runs the cell computations, but it's too easy for Python code to find places to hide data persistently for communication above and beyond what you expect. For example, would you have thought to block the following? Python 2.4.3 (...) Type "help", ... >>> def f(): ... f.i = f.i + 1 ... print f.i ... >>> f.i = 1 >>> f() 2 >>> f() 3 The people who know the most about Python, including the implementors, have said this impossible. Again, unless someone familiar with the community pops up and says something has changed in the several years since I was hanging out on comp.lang.python. But I doubt it. And mind you, that's one example, and not a very sophisticated one. If you're answer was "no", or indeed anything other than "duh, of course, I've known about that for years", then you don't stand a chance at the sophisticated stuff. You can't fork at cell-run-time, because while a process couldn't write anything back out to the parent, it could still grovel over the parent's data space which could still be used to advantage, and depending on the rest of your code may still make an out-of-band channel available. You could fork a process per cell before anything "incriminating" has been instantiated in the parent process, but that's got a lot of problems even on one machine (including the 32K process limit), let alone a hosted service, and you'd still end up with the possibility of a "cell history" being recorded that could be potentially exploited. Python does not give you the necessary primitives to accomplish this task. I see three options: Live with it (a viable option except online), spawn one process per team and officially bless that level of communication (though it radically changes the nature of the game), or change languages. To be honest, Python is going to cause you other problems on your online server too, such as the extreme difficulty you're going to face in isolating Python functions to a certain amount of CPU time. (Same problem, as long as the Python code cooperates it might be possible (and it might not really) but if it starts hostilely modifying the parent code you lose.) Other languages have support for that too. Unfortunately, you're really bashing on Python's weaknesses here.
ssadler 16y ago
Would it work to store all the state in a python C module?
swolchok 16y ago
The performance is actually not too bad the way it is right now.

by phreeza 16y ago

deleted 16y ago

[deleted]

deleted 16y ago
[deleted]

jerf 16y ago

Last I knew, the official word is that there is no way to stop Python code from reading any particular value. Python can even crack into the values contained in closures if you know the right incantations. If you are looking to rigidly enforce that only accepted interfaces are being used to read data in the face of a hostile adversary writing the Python, I believe the current state-of-the-art is that it is impossible. If this is a core feature requirement, then as far as I know your only option is to change languages to something where you can rigidly sandbox. Your nearest option in programming space would be Perl and some variant of Safe (that's a CPAN module name). Otherwise, if you can live with a gentleman's agreement, you're fine. This may make the automatic-online-matchup really tricky, though.

amalcon 16y ago
Is this true of all implementations? I know CPython and PyPy have this limitation, but might it be possible to make something work in IronPython, Jython, or some other project I'm as yet unaware of?
- devinj 16y ago
  PyPy has no such limitation. PyPy has perfect sandboxing.
  - jerf 16y ago
    Is this ready for primetime? The documentation I found on it seems a bit tentative to me, but it may simply have not been updated. It is worth pursuing if so. Combined with standard OS limitation mechanisms, which I would certainly use in addition to this, it might at least be worth a try. I would wait to say PyPy has "perfect" sandboxing until it has been attacked for a while, though. CPython used to have a sandbox module, too. But at least PyPy has been designed with this use case in mind.
    - devinj 16y ago
      PyPy's sandboxing hasn't been as thoroughly analyzed and tested as, say, Javascript's (not by far) or Java's, but the actual construction is simpler and arguably better. The PyPy devs, at least, are fairly convinced of its robustness. Basically they whitelist system calls and all other calls are translated to sandboxed calls during compilation of the sandboxed interpreter, as I recall. This is entirely different from CPython's restricted execution, which was essentially a blacklist of Python calls/attributes-- every attempt at such a system has died a death of a thousand cuts; it really pays to do it from the bottom up like PyPy or E. It might not be ready for prime-time in some very critical systems (like a web browser for example) but it's honestly probably good enough for this, especially in combination with process jailing as you suggest. I might consider trying to patch Cells up to use PyPy's sandbox instead of the non-solution currently implemented (I cried (okay, groaned) when I heard about it-- yeagh), when I have time, and if nobody else does it first. I'm a bit worried anything I write now would get lost or broken in the incoming tide of patches/changes anyway. The main problems with using PyPy's sandboxing is probably with communication to/out of the sandbox, which is apparently problematic. I also anticipate some complaints about relying on PyPy in the first place, which is rather abnormal compared to using CPython-- using and relying on PyPy-specific features is pretty much unheard of.
phreeza 16y ago
> This may make the automatic-online-matchup really tricky, though. Another option I was thinking about for this purpose is parsing the python to make sure they don't get up to anything fishy, but I have no idea how hard a Python parser would be. My guess is very. Or we put them into separate proceses, as swolchok suggested.
- jerf 16y ago
  "Another option I was thinking about for this purpose is parsing the python to make sure they don't get up to anything fishy." If that was safe, it would have been bundled up into a module. This is a Frequently Asked Question for Python, and the answer is, you can't. I thought of process separation, but it won't work. The best performance would be one persistent process that runs the cell computations, but it's too easy for Python code to find places to hide data persistently for communication above and beyond what you expect. For example, would you have thought to block the following? Python 2.4.3 (...) Type "help", ... >>> def f(): ... f.i = f.i + 1 ... print f.i ... >>> f.i = 1 >>> f() 2 >>> f() 3 The people who know the most about Python, including the implementors, have said this impossible. Again, unless someone familiar with the community pops up and says something has changed in the several years since I was hanging out on comp.lang.python. But I doubt it. And mind you, that's one example, and not a very sophisticated one. If you're answer was "no", or indeed anything other than "duh, of course, I've known about that for years", then you don't stand a chance at the sophisticated stuff. You can't fork at cell-run-time, because while a process couldn't write anything back out to the parent, it could still grovel over the parent's data space which could still be used to advantage, and depending on the rest of your code may still make an out-of-band channel available. You could fork a process per cell before anything "incriminating" has been instantiated in the parent process, but that's got a lot of problems even on one machine (including the 32K process limit), let alone a hosted service, and you'd still end up with the possibility of a "cell history" being recorded that could be potentially exploited. Python does not give you the necessary primitives to accomplish this task. I see three options: Live with it (a viable option except online), spawn one process per team and officially bless that level of communication (though it radically changes the nature of the game), or change languages. To be honest, Python is going to cause you other problems on your online server too, such as the extreme difficulty you're going to face in isolating Python functions to a certain amount of CPU time. (Same problem, as long as the Python code cooperates it might be possible (and it might not really) but if it starts hostilely modifying the parent code you lose.) Other languages have support for that too. Unfortunately, you're really bashing on Python's weaknesses here.

ssadler 16y ago

Would it work to store all the state in a python C module?

swolchok 16y ago

The performance is actually not too bad the way it is right now.