6 ms·

Not just Ruby, Python too. Was doing some work of converting [0,1] signals to 16-bit PCM data with 550Hz tone in numpy. Python version took ~15 minutes to gen

Not just Ruby, Python too. Was doing some work of converting [0,1] signals to 16-bit PCM data with 550Hz tone in numpy. Python version took ~15 minutes to generate 5,000 4 second files. Broke out the inner loop into Rust with FFI via ctypes and cut that time to ~10 seconds with nearly identical code.

gravypod 9y ago
Have you tried using Julia for this task? It's compiled and provides many of the math-y facilities you'll need for something like this (I'd assume).
- vvanders 9y ago
  Yeah, I've heard of Julia before but looks like it doesn't have a very clean[1] FFI story. I'm not super-interested in spinning up a full-blown interpreter for an inner loop. Plus I already had Rust installed :). Seriously though Rust was 43 lines of code + cargo build vs 33 lines of Python and a simple C FFI so it was pretty straightforward to drop in. https://docs.julialang.org/en/release-0.4/manual/embedding/ https://docs.julialang.org/en/release-0.4/manual/embedding/
  - chroem- 9y ago
    I think the whole idea behind using Julia in this situation would be that you get good enough performance "out of the box" so that you don't have to resort to using an FFI in the first place.
    - vvanders 9y ago
      But that's also the brilliance of Rust supporting a clean C ABI. I can write in anything I want since every language supports C ABI for system calls. I don't want to have to learn a whole new language and new API at the same time. Also a cursory glance at Julia's numpy support looks like it requires deep copies which is painful when you want things to be fast. I'm sure it's a great language but needing to drag in a whole new VM means it doesn't fit my needs.
      - gravypod 9y ago
        Julia is jit compiled by llvm to your native machine code. Correctly written Julia is within 1.5 the speed of C. It removes the complexity of needing to do the FFI at all.
        deleted 9y ago
        [deleted]
        vvanders 9y ago
        You make it sound like ffi is a nasty 3 letter word ;). It look me 3 lines of code to import the dll, assign the signature and call the function, I'd hardly call that complex.
        mlevental 9y ago
        this is probably a dumb question but did you just compile the rust binary and then expose the functions? how? I don't remember but does rust face header files?
        vvanders 9y ago
        Yup, just declare the function like so: #[no_mangle] pub unsafe extern "C" fn transform_pcm( data: *mut c_short, len: c_uint, ramp_time: c_uint, tone_delta_frame: c_float) { then call from python: gen_pcm_dll = ctypes.cdll.LoadLibrary("gen_pcm_rust/target/release/gen_pcm.dll") gen_pcm_dll.transform_pcm.argtypes = [ctypes.c_void_p, ctypes.c_uint, ctypes.c_uint, ctypes.c_float] gen_pcm_dll.transform_pcm(ptr, len(pcm), ramp_time, tone_delta_frame) Rust doesn't generate headers(although there are helper libs if you want) and you don't need them to FFI.
        the_mitsuhiko 9y ago
        Here is how we do it: https://blog.sentry.io/2017/11/14/evolving-our-rust-with-milksnake https://blog.sentry.io/2017/11/14/evolving-our-rust-with-mil...
    - one-more-minute 9y ago
      This is fine for new projects, but OP already has a large Ruby codebase. "Just rewrite the whole thing in Julia and you won't need FFI" isn't a reasonable ask. Rust's ability to compile to a small C-compatible binary is definitely an advantage here, although Julia will have similar capabilities soon.
      - jamii 9y ago
        > although Julia will have similar capabilities soon. Do you have a link to further info on that?
        one-more-minute 9y ago
        See https://github.com/JuliaComputing/static-julia https://github.com/JuliaComputing/static-julia A lot of this stuff has worked for ages, but it's rough in various places and needs a better interface. I'm not sure to what extent exporting C functions is actually supported in that wrapper script, or whether you'll need to fiddle with compiler options.
    - coldtea 9y ago
      No, but you would have to do with subpar/immature third party ecosystem AND you'd have to port all your Python code to Julia before you can get to resort entirely to it.
Nicholas7900 9y ago
So many young Hackers have been asking for the easiest way to hack an email, Do you know how many email services there are, how many kinds of mail servers, how many forms of custom security? In my time, I've hacked numerous email servers and services, usually incidentally to compromising the rest of a company or entity. It's always been different. Sometimes it´s as simple as replacing a username in a URL. Other times I've had to compromise the entire support system for an email service. It varies. It takes imagination. There's no formula. There's no ¨teaching to hack email¨. There's just understanding how systems work and learning to extrapolate. I will reluctantly say that a public favorite seems to be guessing ¨secret answers¨ to ¨secret questions¨ needed to reset passwords on some systems (For Guccifer, hacking email is easy — prison is hard). I never personally used this technique, since it felt like cheating. It's hard to explain why. But certain passwords, once discovered, made sense for certain people (Ih8K1ds for an extremely unpleasant lady at [redacted],SPQR for a CIA officer [on a system with very low password standards]).So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via compositehacks@gmail.com. I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co" So many people always think i only work for big companies, that i dont work for private individuals, here is my mail """"""compositehacks@gmail.com,""""" feel free to contact me for any hacking services you need, i'll do your job professionally. So many people always ask me if i'm a Hacker or a Cracker, as a matter of fact, I don't really care. You can consider me whatever you like - it's entirely subjective. I've been called a white hat, gray hat, and black hat, depending on the attitude and perspective of the writer, and it's not my place to insist that one or the other is correct. I encourage readers to reach conclusions from their own mind and feelings, not arbitrary segregation. I do what I do, and leave it to others to document it - or not - in the way that suits them best. Distinctions like these work for role playing games, and it's kind of like asking "Is Adrian Lamo True Neutral?" - an interesting intellectual exercise, but with no particular relevance to reality. So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via """""compositehacks@gmail.com"""". I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"
fulafel 9y ago
A 100x difference to native and having an inner loop in Python code sound exceptionally slow for a numpy app. I guess you could not leverage numpy array operations?
- vvanders 9y ago
  Nope, I agree that base numpy operations are snappy. However I needed to both multiply by a sin wave of 550hz(changing input) and taper the edges of the signal on the transition from 0->1 to not generate a ton of harmonics.
FRex 9y ago
What did you do exactly and on how much input data? Do you have that Rust and Python code available somewhere or is it too sensitive? I'm very tempted to compare that with LuaJIT and/or its FFI[0] (just to use structs, not to call any native code) just to see the results. [0] - http://luajit.org/ext_ffi.html http://luajit.org/ext_ffi.html
rplnt 9y ago
Have you looked at cython? Might have been a great compromise. I saw similar gains in my code eventually, but the process to get there was very smooth. At first I could just use original python code and covert it piece by piece (of course sometimes you need to refactor). You can drop GIL too.
kodablah 9y ago
Not just Python, JVM langs too (and other runtimes w/ fast C interfacing, e.g. node w/ neon). JNI is actually incredibly easy with Rust. I've even used some advanced features of the JVM via JVMTI and exposed it via JNI all in Rust as a learning exercise [0]. Contemplating writing a fuzzer using similar tech soon. 0 - https://github.com/cretz/stackparam https://github.com/cretz/stackparam

gravypod 9y ago

Have you tried using Julia for this task? It's compiled and provides many of the math-y facilities you'll need for something like this (I'd assume).

vvanders 9y ago
Yeah, I've heard of Julia before but looks like it doesn't have a very clean[1] FFI story. I'm not super-interested in spinning up a full-blown interpreter for an inner loop. Plus I already had Rust installed :). Seriously though Rust was 43 lines of code + cargo build vs 33 lines of Python and a simple C FFI so it was pretty straightforward to drop in. https://docs.julialang.org/en/release-0.4/manual/embedding/ https://docs.julialang.org/en/release-0.4/manual/embedding/
- chroem- 9y ago
  I think the whole idea behind using Julia in this situation would be that you get good enough performance "out of the box" so that you don't have to resort to using an FFI in the first place.
  - vvanders 9y ago
    But that's also the brilliance of Rust supporting a clean C ABI. I can write in anything I want since every language supports C ABI for system calls. I don't want to have to learn a whole new language and new API at the same time. Also a cursory glance at Julia's numpy support looks like it requires deep copies which is painful when you want things to be fast. I'm sure it's a great language but needing to drag in a whole new VM means it doesn't fit my needs.
    - gravypod 9y ago
      Julia is jit compiled by llvm to your native machine code. Correctly written Julia is within 1.5 the speed of C. It removes the complexity of needing to do the FFI at all.
      - deleted 9y ago
        [deleted]
      - vvanders 9y ago
        You make it sound like ffi is a nasty 3 letter word ;). It look me 3 lines of code to import the dll, assign the signature and call the function, I'd hardly call that complex.
        mlevental 9y ago
        this is probably a dumb question but did you just compile the rust binary and then expose the functions? how? I don't remember but does rust face header files?
        vvanders 9y ago
        Yup, just declare the function like so: #[no_mangle] pub unsafe extern "C" fn transform_pcm( data: *mut c_short, len: c_uint, ramp_time: c_uint, tone_delta_frame: c_float) { then call from python: gen_pcm_dll = ctypes.cdll.LoadLibrary("gen_pcm_rust/target/release/gen_pcm.dll") gen_pcm_dll.transform_pcm.argtypes = [ctypes.c_void_p, ctypes.c_uint, ctypes.c_uint, ctypes.c_float] gen_pcm_dll.transform_pcm(ptr, len(pcm), ramp_time, tone_delta_frame) Rust doesn't generate headers(although there are helper libs if you want) and you don't need them to FFI.
        the_mitsuhiko 9y ago
        Here is how we do it: https://blog.sentry.io/2017/11/14/evolving-our-rust-with-milksnake https://blog.sentry.io/2017/11/14/evolving-our-rust-with-mil...
  - one-more-minute 9y ago
    This is fine for new projects, but OP already has a large Ruby codebase. "Just rewrite the whole thing in Julia and you won't need FFI" isn't a reasonable ask. Rust's ability to compile to a small C-compatible binary is definitely an advantage here, although Julia will have similar capabilities soon.
    - jamii 9y ago
      > although Julia will have similar capabilities soon. Do you have a link to further info on that?
      - one-more-minute 9y ago
        See https://github.com/JuliaComputing/static-julia https://github.com/JuliaComputing/static-julia A lot of this stuff has worked for ages, but it's rough in various places and needs a better interface. I'm not sure to what extent exporting C functions is actually supported in that wrapper script, or whether you'll need to fiddle with compiler options.
  - coldtea 9y ago
    No, but you would have to do with subpar/immature third party ecosystem AND you'd have to port all your Python code to Julia before you can get to resort entirely to it.

Nicholas7900 9y ago

So many young Hackers have been asking for the easiest way to hack an email, Do you know how many email services there are, how many kinds of mail servers, how many forms of custom security? In my time, I've hacked numerous email servers and services, usually incidentally to compromising the rest of a company or entity. It's always been different. Sometimes it´s as simple as replacing a username in a URL. Other times I've had to compromise the entire support system for an email service. It varies. It takes imagination. There's no formula. There's no ¨teaching to hack email¨. There's just understanding how systems work and learning to extrapolate. I will reluctantly say that a public favorite seems to be guessing ¨secret answers¨ to ¨secret questions¨ needed to reset passwords on some systems (For Guccifer, hacking email is easy — prison is hard). I never personally used this technique, since it felt like cheating. It's hard to explain why. But certain passwords, once discovered, made sense for certain people (Ih8K1ds for an extremely unpleasant lady at [redacted],SPQR for a CIA officer [on a system with very low password standards]).So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via compositehacks@gmail.com. I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co" So many people always think i only work for big companies, that i dont work for private individuals, here is my mail """"""compositehacks@gmail.com,""""" feel free to contact me for any hacking services you need, i'll do your job professionally. So many people always ask me if i'm a Hacker or a Cracker, as a matter of fact, I don't really care. You can consider me whatever you like - it's entirely subjective. I've been called a white hat, gray hat, and black hat, depending on the attitude and perspective of the writer, and it's not my place to insist that one or the other is correct. I encourage readers to reach conclusions from their own mind and feelings, not arbitrary segregation. I do what I do, and leave it to others to document it - or not - in the way that suits them best. Distinctions like these work for role playing games, and it's kind of like asking "Is Adrian Lamo True Neutral?" - an interesting intellectual exercise, but with no particular relevance to reality. So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via """""compositehacks@gmail.com"""". I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"

fulafel 9y ago

A 100x difference to native and having an inner loop in Python code sound exceptionally slow for a numpy app. I guess you could not leverage numpy array operations?

vvanders 9y ago
Nope, I agree that base numpy operations are snappy. However I needed to both multiply by a sin wave of 550hz(changing input) and taper the edges of the signal on the transition from 0->1 to not generate a ton of harmonics.

FRex 9y ago

What did you do exactly and on how much input data? Do you have that Rust and Python code available somewhere or is it too sensitive? I'm very tempted to compare that with LuaJIT and/or its FFI[0] (just to use structs, not to call any native code) just to see the results. [0] - http://luajit.org/ext_ffi.html http://luajit.org/ext_ffi.html

rplnt 9y ago

Have you looked at cython? Might have been a great compromise. I saw similar gains in my code eventually, but the process to get there was very smooth. At first I could just use original python code and covert it piece by piece (of course sometimes you need to refactor). You can drop GIL too.

kodablah 9y ago

Not just Python, JVM langs too (and other runtimes w/ fast C interfacing, e.g. node w/ neon). JNI is actually incredibly easy with Rust. I've even used some advanced features of the JVM via JVMTI and exposed it via JNI all in Rust as a learning exercise [0]. Contemplating writing a fuzzer using similar tech soon. 0 - https://github.com/cretz/stackparam https://github.com/cretz/stackparam