4 ms·

How are interpreters designed in Rust? What parts can be safe and what can't be? What is the internal representation of a Python object? In particular how do y

How are interpreters designed in Rust? What parts can be safe and what can't be? What is the internal representation of a Python object? In particular how do you mutate a Python object while there may be other references to it - is it UnsafeCell, is there some checking? How does the GC work? The GC needs to access and perhaps mutate all Python objects. In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust.

steveklabnik 6y ago
I was curious too, so I poked around! I had never looked at this codebase before this comment, so, I may have some things wrong. > How are interpreters designed in Rust? What parts can be safe and what can't be? Depends on the design. Different interpreters are different. A quick glance at the source code seems to show unsafe in: * A data structure called "boxvec" * Some calls to unreachable_unchecked, get_unchecked, and the like. Wonder what the overheads were here. * Some lock/mutex implementations * A few unsafe functions in the jit. They use cranelift. Bet there's a bunch in there too. Usually this is the classic thing that needs unsafe in interpreters. * The python object data structures; they do some stuff to have a smaller representation * some FFI code in the standard library, calls directly to libc > What is the internal representation of a Python object? https://github.com/RustPython/RustPython/blob/master/vm/src/pyobjectrc.rs#L12-L16 https://github.com/RustPython/RustPython/blob/master/vm/src/... > How does the GC work? Traditionally, Python uses refcounting + cycle detection. A quick glance at their issue tracker implies that as of last month, they had recounting but no cycle checks yet. > In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust. Looks like they have a RefCell<Vec<FrameRef>>, instead.

steveklabnik 6y ago

I was curious too, so I poked around! I had never looked at this codebase before this comment, so, I may have some things wrong. > How are interpreters designed in Rust? What parts can be safe and what can't be? Depends on the design. Different interpreters are different. A quick glance at the source code seems to show unsafe in: * A data structure called "boxvec" * Some calls to unreachable_unchecked, get_unchecked, and the like. Wonder what the overheads were here. * Some lock/mutex implementations * A few unsafe functions in the jit. They use cranelift. Bet there's a bunch in there too. Usually this is the classic thing that needs unsafe in interpreters. * The python object data structures; they do some stuff to have a smaller representation * some FFI code in the standard library, calls directly to libc > What is the internal representation of a Python object? https://github.com/RustPython/RustPython/blob/master/vm/src/pyobjectrc.rs#L12-L16 https://github.com/RustPython/RustPython/blob/master/vm/src/... > How does the GC work? Traditionally, Python uses refcounting + cycle detection. A quick glance at their issue tracker implies that as of last month, they had recounting but no cycle checks yet. > In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust. Looks like they have a RefCell<Vec<FrameRef>>, instead.