41 ms·

The totally safe and sane approach is to write C code that gets passed data via the command line during execution, then vomits results to the command line or ju

The totally safe and sane approach is to write C code that gets passed data via the command line during execution, then vomits results to the command line or just into a memory page. Then just execute the c program with your flags or data in the terminal using ruby and viola, ruby can run C code.

grandempire 2y ago
This. I think many people do not understand Unix processes and don’t realizing how rare it is to need bindings, ffi, and many libraries. How many programs have an https client in them because they need to make one request and didn’t know they could use curl?
- nirvdrum 2y ago
  Can you please elaborate on this because I'm struggling to follow your suggestion. Shelling out to psql every time I want to run an SQL query is going to be prohibitively slow. It seems to me you'd need bindings in almost the exact same cases you'd use a shared library if you were writing in C and that's really all bindings are anyway -- a bridge between the VM and a native library.
  - grandempire 2y ago
    Spawning a process isn't the right tool for ALL X-language communication. But sometimes it is - and the bias tends to be to overlook these opportunities. When you are comfortable using libraries, you make more libraries. When you know how to use programs, you more often make programs. > Shelling out to psql I would recommend using a Postgres connection library, because that's how Postgres is designed. Note that ongoing communication can still work with a multi-process stdin/stdout design. This is how email protocols work. So someone could design a SQL client that works this way. I have absolutely written batch import scripts which simply spawn psql for every query, with great results. > in almost the exact same cases you'd use a shared library That's the thing. Libraries are an entangling relationship (literally in your binary). Programs in contrast have a clean, modular interface. So for example you can choose to load the imagemagick library, or you can spawn imagemagick. Which one is better depends, but most often you don't need the library. Here is a list of examples I have seen solved with a library that were completely unnecessary: - identify the format of an image. - zip some data - post usage analytics to a web server at startup - diff two pieces of data - convert math symbols to images - convert x format to y I have even seen discourse online that suggests that if you are serious about AI your web stack needs to be in python - as if you can't spawn a process for an AI job.
    - nirvdrum 2y ago
      > Spawning a process isn't the right tool for ALL X-language communication. But sometimes it is I'm with you here. > ...many people do not understand Unix processes and don’t realizing how rare it is to need bindings, ffi, and many libraries But, this is a much stronger claim. I can't tell if you're making a meta point or addressing something in the Ruby ecosystem. I mentioned database library bindings because that's far and away the most common usage in the Ruby ecosystem, particularly because of its frequent adoption for web applications. The author is advocating for not using native code at all if you can avoid it. Keep as much code in Ruby as you can and let the JIT optimize it. But, if you do need bindings, it'd be great if you didn't have to write a native extension. There are a lot of ways to shoot yourself in the foot and they complicate the build pipeline. However, historically, FFI has been much slower than writing a native extension. The point of this post is to explore a way to speed up FFI in the cases where you need it. It needs to be taken on faith that the author is doing this work because he either has performance sensitive code or needs to work with a library. Spawning a new process in that case is going to be slower than any of the options explored in the post. Writing and distributing a C/C++/Zig/Rust/Go application as part of a Rails app is a big hammer to swing and complicates deployments (a big part of the reason to move away from native extensions). It's possible the author is just complicating things unnecessarily, but he's demonstrated a clear mastery of the technologies involved so I'm willing to give him the benefit of the doubt. A frequent critique of Ruby is that it's slow. Spawning processes for operations on the hot path isn't going to help that perception. I agree there are cases where shelling out makes sense. E.g., spawning out to use ImageMagick has proven to be a better approach than using bindings when I want to make image thumbnails. But, those are typically handled in asynchronous jobs. I'm all for anything we can do to speed up the hot path and it's remarkable how much performance was easily picked up.
      - grandempire 2y ago
        You interpreted my comment as an attack on the author. If they have a special case they want to optimize FFI for this seems like a great way to do it. > Spawning processes for operations on the hot path isn't going to help that perception. Yes it will. Because then you do the number crunching in C/Java/Rust etc instead of Ruby. And the OS will do multi-core scheduling for free. This is playing to Ruby’s strength as a high level orchestrator. I think you’re vastly overestimating the fork/exec overhead on Linux. If your hot path is this hot you better not be calling Ruby functions. > Writing and distributing a C/C++/Zig/Rust/Go application as part of a Rails app is a big hammer to swing You’re already doing it. Do you have a database? Nginx? Me cached? Imagemagick? Do you use containers? Consider the alternative: is the ideal world the one where all software is available as a ruby package?
fomine3 2y ago
It's slow

grandempire 2y ago

This. I think many people do not understand Unix processes and don’t realizing how rare it is to need bindings, ffi, and many libraries. How many programs have an https client in them because they need to make one request and didn’t know they could use curl?

nirvdrum 2y ago
Can you please elaborate on this because I'm struggling to follow your suggestion. Shelling out to psql every time I want to run an SQL query is going to be prohibitively slow. It seems to me you'd need bindings in almost the exact same cases you'd use a shared library if you were writing in C and that's really all bindings are anyway -- a bridge between the VM and a native library.
- grandempire 2y ago
  Spawning a process isn't the right tool for ALL X-language communication. But sometimes it is - and the bias tends to be to overlook these opportunities. When you are comfortable using libraries, you make more libraries. When you know how to use programs, you more often make programs. > Shelling out to psql I would recommend using a Postgres connection library, because that's how Postgres is designed. Note that ongoing communication can still work with a multi-process stdin/stdout design. This is how email protocols work. So someone could design a SQL client that works this way. I have absolutely written batch import scripts which simply spawn psql for every query, with great results. > in almost the exact same cases you'd use a shared library That's the thing. Libraries are an entangling relationship (literally in your binary). Programs in contrast have a clean, modular interface. So for example you can choose to load the imagemagick library, or you can spawn imagemagick. Which one is better depends, but most often you don't need the library. Here is a list of examples I have seen solved with a library that were completely unnecessary: - identify the format of an image. - zip some data - post usage analytics to a web server at startup - diff two pieces of data - convert math symbols to images - convert x format to y I have even seen discourse online that suggests that if you are serious about AI your web stack needs to be in python - as if you can't spawn a process for an AI job.
  - nirvdrum 2y ago
    > Spawning a process isn't the right tool for ALL X-language communication. But sometimes it is I'm with you here. > ...many people do not understand Unix processes and don’t realizing how rare it is to need bindings, ffi, and many libraries But, this is a much stronger claim. I can't tell if you're making a meta point or addressing something in the Ruby ecosystem. I mentioned database library bindings because that's far and away the most common usage in the Ruby ecosystem, particularly because of its frequent adoption for web applications. The author is advocating for not using native code at all if you can avoid it. Keep as much code in Ruby as you can and let the JIT optimize it. But, if you do need bindings, it'd be great if you didn't have to write a native extension. There are a lot of ways to shoot yourself in the foot and they complicate the build pipeline. However, historically, FFI has been much slower than writing a native extension. The point of this post is to explore a way to speed up FFI in the cases where you need it. It needs to be taken on faith that the author is doing this work because he either has performance sensitive code or needs to work with a library. Spawning a new process in that case is going to be slower than any of the options explored in the post. Writing and distributing a C/C++/Zig/Rust/Go application as part of a Rails app is a big hammer to swing and complicates deployments (a big part of the reason to move away from native extensions). It's possible the author is just complicating things unnecessarily, but he's demonstrated a clear mastery of the technologies involved so I'm willing to give him the benefit of the doubt. A frequent critique of Ruby is that it's slow. Spawning processes for operations on the hot path isn't going to help that perception. I agree there are cases where shelling out makes sense. E.g., spawning out to use ImageMagick has proven to be a better approach than using bindings when I want to make image thumbnails. But, those are typically handled in asynchronous jobs. I'm all for anything we can do to speed up the hot path and it's remarkable how much performance was easily picked up.
    - grandempire 2y ago
      You interpreted my comment as an attack on the author. If they have a special case they want to optimize FFI for this seems like a great way to do it. > Spawning processes for operations on the hot path isn't going to help that perception. Yes it will. Because then you do the number crunching in C/Java/Rust etc instead of Ruby. And the OS will do multi-core scheduling for free. This is playing to Ruby’s strength as a high level orchestrator. I think you’re vastly overestimating the fork/exec overhead on Linux. If your hot path is this hot you better not be calling Ruby functions. > Writing and distributing a C/C++/Zig/Rust/Go application as part of a Rails app is a big hammer to swing You’re already doing it. Do you have a database? Nginx? Me cached? Imagemagick? Do you use containers? Consider the alternative: is the ideal world the one where all software is available as a ruby package?

fomine3 2y ago

It's slow