6 ms·

Impressive! Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an inte

by r0naa 11y ago

Impressive! Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds? To clarify, my metric for "value" is usefulness to other people. That is, without considering the (interesting) learning opportunity that it represent for the author. For example, someone developed a Python interface to the Stanford Core-NLP library (written in Java). Would re-writing the Core NLP library to Python be useful to the community? How to figure what are people needs? I am asking because while I think it would be ton of fun and allow me to learn a lot, I also value building useful software and re-writing a whole system sounds like an overkill except for a few very niche cases.. And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it. Looking forward to hear what HNers think! edit: clarified ambiguities

themckman 11y ago
The README answers this: To learn Rust. Edit: It also mentions not being tied to UNIX and appears to claim it will run on Windows. That's certainly something.
- r0naa 11y ago
  Sorry if I wasn't clear, but I am looking for a more general answer! I would like to know in which case it is useful (to other people) and discuss it's value comparatively to writing interfaces to other languages.
  - seppo0010 11y ago
    I had no intention of making the end result useful, but I run into interesting problems. First, I wanted to make it as pure Rust as possible. I tried to avoid UNIX specific code, and since there is no library with Windows support for asynchronous IO in Rust, I was pushed into spawning thread and blocking waiting for client data. I quickly noticed that the benchmark was way below Redis (around 60% of ops/sec with 50 clients). But then someone point out to me[1] that I was running tests in a machine with two cores, and this actually may be better for machines with multiple cores[2]. I have yet to try it out and benchmark the results. So far Rust API was disappointing for network operations. For example, `TcpStream.read()`[3] and `TcpListener.incoming()`[4] do not have a timeout. Maybe because its development is driven for Servo and not for servers. I have thought about doubling down on multithreading and instead of a global database lock as rsedis is using now, having one per key (or some other arbitrary partition), and having concurrent operations, which is hard to do safely in C. But I have not gotten there yet. [1] https://github.com/jonhoo/rucache/issues/2 https://github.com/jonhoo/rucache/issues/2 [2] https://github.com/jonhoo/volley/ https://github.com/jonhoo/volley/ [3] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream... [4] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListener.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListen...
    - bsaul 11y ago
      Your comment is interesting to me because although i've only looked at Rust from a distance, its seemingly lack of features related to server side code ( threading , non blocking io and networking api) have made me wait before i actually start coding in that language. Seems like those parts still need more work.
      - steveklabnik 11y ago
        It depends on what you're doing. See https://github.com/jonhoo/volley https://github.com/jonhoo/volley, which uses blocking IO and 1:1 threads, and we're still damn fast. There are also libraries like mio that can give you no blocking IO. Rust's standard library is deliberately minimal, exploring the ecosystem a bit is helpful, though admittedly not as easy as it could be. Working on it!
        pron 11y ago
        Right. People should realize that there is no latency difference between blocking and non-blocking IO -- in fact there is a small difference in blocking IO's favor. The difference is in throughput when there is a very large number of concurrent requests, namely more than about 10K and certainly more than 2K. If you're serving under 2K concurrent requests, blocking IO is probably a better choice than nonblocking. Of course, the number of concurrent requests you need to serve depends on the time you spend serving each request; Little's Law and all that.
    - steveklabnik 11y ago
      > Maybe because its development is driven for Servo > and not for servers. This is not true at all. If Rust's development were determined by Servo, we would have kept green threads and implemented struct inheritance by now. The reason timeouts were dropped is in the IO/OS reform RFC: https://github.com/rust-lang/rfcs/blob/8fa971a670f9b9bc30f31bed30b9c3b679ea1ad3/text/0517-io-os-reform.md#tcp https://github.com/rust-lang/rfcs/blob/8fa971a670f9b9bc30f31... > set_timeout has been removed for now (as well as other > timeout-related functions). It is likely that this may > come back soon as a binding to setsockopt to the > SO_RCVTIMEO and SO_SNDTIMEO options. This RFC does not > currently proposed adding them just yet, however. And on UDP: > All timeout support is removed. This may come back in > the form of setsockopt (as with TCP streams) or with > a more general implementation of select. I'm on shaky wifi and my phone, so I can't find a citation for this, but I also believe it was removed due to 1) Rust not having any stable representation of time and 2) needing to shim certain behaviors on some platforms, which we decided wouldn't happen in the first round of stable interfaces. That said, the lack here certainly hurts, and we did manage to stabilize Duration, paving the way for timeouts to return. EDIT: Oh! I forgot that https://github.com/rust-lang/rfcs/pull/1047 https://github.com/rust-lang/rfcs/pull/1047 got merged recently. https://github.com/rust-lang/rust/issues/25818 https://github.com/rust-lang/rust/issues/25818 implemented it. http://doc.rust-lang.org/nightly/std/net/struct.TcpStream.html#method.read_timeout http://doc.rust-lang.org/nightly/std/net/struct.TcpStream.ht... shows it implemented on nightly, so you can actually even do timeouts today, just not on the stable channel.
unoti 11y ago
From a learning perspective, reimplementation has key advantage versus other kinds of projects: the design is completely done, so you can focus exclusively on the mechanics of implementation.
kbenson 11y ago
In this case, it's a learning exercise. Sometimes, it's because there are other benefits, such as making it easier to distribute or easier to tie into specific parts of the algorithm than may be possible by calling a library that provides a high level interface.
nickpsecurity 11y ago
Aside from project's reason, there's one very good reason to re-implement a bunch of stuff in Rust: testing whether it delivers. It's being pushed as a safer, better systems language. So, let's take many different C/C++ apps, rewrite them in Rust, and see what the results are across many metrics. By the end of it, we should know Rust's strengths and weaknesses in real-world coding.
coldtea 11y ago
>Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds? Making it safer and even catching bugs in the original implementation (both things Rust will help with)? Making it integrate seamlesly with the new language's ecosystem? E.g. Lucene is Java, and someone could use that, but there are tons of ports of it in C, C++, Python etc, providing convenience to integrate it with projects in these languages. >And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it. Not necessarily. A project with 10 part time contributors could be matched with a project with 2-3 full time competent hackers for example, or even surpassed.
- frik 11y ago
  > Lucene is Java ... there are tons of ports of it in C, C++, Python, etc. There used to be several ports, though most are dead and/or are several major versions behind. A new C++ or Rust port would be great, though unrealistic given the huge project side.
  - coldtea 11y ago
    >though unrealistic given the huge project side. It could be done with some automated translation, and cleared out from there.
deet 11y ago
This project specifically is probably not useful for others just because it is written in another language (unless it were to succeed in fixing problems or improving security or performance.) Redis is a server application, not a library, so there's a clearly defined and interoperable protocol to bridge the Redis server to other languages already. Writing a Redis client for the other language would be more useful. More generally, as coldtea mentions, making integration into the rest of the language's ecosystem is the primary benefit of rewriting in another language. The value of such a port to others depends on how easy it is to integrate between the two languages, either via libraries or other methods. The harder it is to integrate the two (and the absence of automated translation tools) increases the value of the rewrite to others. Your Core-NLP example is actually an interesting one, because that library has already been ported to other languages... It is available for the C#/F# ecosystem (http://sergey-tihon.github.io/Stanford.NLP.NET/ http://sergey-tihon.github.io/Stanford.NLP.NET/).

by r0naa 11y ago

themckman 11y ago

The README answers this: To learn Rust. Edit: It also mentions not being tied to UNIX and appears to claim it will run on Windows. That's certainly something.

r0naa 11y ago
Sorry if I wasn't clear, but I am looking for a more general answer! I would like to know in which case it is useful (to other people) and discuss it's value comparatively to writing interfaces to other languages.
- seppo0010 11y ago
  I had no intention of making the end result useful, but I run into interesting problems. First, I wanted to make it as pure Rust as possible. I tried to avoid UNIX specific code, and since there is no library with Windows support for asynchronous IO in Rust, I was pushed into spawning thread and blocking waiting for client data. I quickly noticed that the benchmark was way below Redis (around 60% of ops/sec with 50 clients). But then someone point out to me[1] that I was running tests in a machine with two cores, and this actually may be better for machines with multiple cores[2]. I have yet to try it out and benchmark the results. So far Rust API was disappointing for network operations. For example, `TcpStream.read()`[3] and `TcpListener.incoming()`[4] do not have a timeout. Maybe because its development is driven for Servo and not for servers. I have thought about doubling down on multithreading and instead of a global database lock as rsedis is using now, having one per key (or some other arbitrary partition), and having concurrent operations, which is hard to do safely in C. But I have not gotten there yet. [1] https://github.com/jonhoo/rucache/issues/2 https://github.com/jonhoo/rucache/issues/2 [2] https://github.com/jonhoo/volley/ https://github.com/jonhoo/volley/ [3] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream... [4] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListener.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListen...
  - bsaul 11y ago
    Your comment is interesting to me because although i've only looked at Rust from a distance, its seemingly lack of features related to server side code ( threading , non blocking io and networking api) have made me wait before i actually start coding in that language. Seems like those parts still need more work.
    - steveklabnik 11y ago
      It depends on what you're doing. See https://github.com/jonhoo/volley https://github.com/jonhoo/volley, which uses blocking IO and 1:1 threads, and we're still damn fast. There are also libraries like mio that can give you no blocking IO. Rust's standard library is deliberately minimal, exploring the ecosystem a bit is helpful, though admittedly not as easy as it could be. Working on it!
      - pron 11y ago
        Right. People should realize that there is no latency difference between blocking and non-blocking IO -- in fact there is a small difference in blocking IO's favor. The difference is in throughput when there is a very large number of concurrent requests, namely more than about 10K and certainly more than 2K. If you're serving under 2K concurrent requests, blocking IO is probably a better choice than nonblocking. Of course, the number of concurrent requests you need to serve depends on the time you spend serving each request; Little's Law and all that.
  - steveklabnik 11y ago
    > Maybe because its development is driven for Servo > and not for servers. This is not true at all. If Rust's development were determined by Servo, we would have kept green threads and implemented struct inheritance by now. The reason timeouts were dropped is in the IO/OS reform RFC: https://github.com/rust-lang/rfcs/blob/8fa971a670f9b9bc30f31bed30b9c3b679ea1ad3/text/0517-io-os-reform.md#tcp https://github.com/rust-lang/rfcs/blob/8fa971a670f9b9bc30f31... > set_timeout has been removed for now (as well as other > timeout-related functions). It is likely that this may > come back soon as a binding to setsockopt to the > SO_RCVTIMEO and SO_SNDTIMEO options. This RFC does not > currently proposed adding them just yet, however. And on UDP: > All timeout support is removed. This may come back in > the form of setsockopt (as with TCP streams) or with > a more general implementation of select. I'm on shaky wifi and my phone, so I can't find a citation for this, but I also believe it was removed due to 1) Rust not having any stable representation of time and 2) needing to shim certain behaviors on some platforms, which we decided wouldn't happen in the first round of stable interfaces. That said, the lack here certainly hurts, and we did manage to stabilize Duration, paving the way for timeouts to return. EDIT: Oh! I forgot that https://github.com/rust-lang/rfcs/pull/1047 https://github.com/rust-lang/rfcs/pull/1047 got merged recently. https://github.com/rust-lang/rust/issues/25818 https://github.com/rust-lang/rust/issues/25818 implemented it. http://doc.rust-lang.org/nightly/std/net/struct.TcpStream.html#method.read_timeout http://doc.rust-lang.org/nightly/std/net/struct.TcpStream.ht... shows it implemented on nightly, so you can actually even do timeouts today, just not on the stable channel.

unoti 11y ago

From a learning perspective, reimplementation has key advantage versus other kinds of projects: the design is completely done, so you can focus exclusively on the mechanics of implementation.

kbenson 11y ago

In this case, it's a learning exercise. Sometimes, it's because there are other benefits, such as making it easier to distribute or easier to tie into specific parts of the algorithm than may be possible by calling a library that provides a high level interface.

nickpsecurity 11y ago

Aside from project's reason, there's one very good reason to re-implement a bunch of stuff in Rust: testing whether it delivers. It's being pushed as a safer, better systems language. So, let's take many different C/C++ apps, rewrite them in Rust, and see what the results are across many metrics. By the end of it, we should know Rust's strengths and weaknesses in real-world coding.

coldtea 11y ago

>Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds? Making it safer and even catching bugs in the original implementation (both things Rust will help with)? Making it integrate seamlesly with the new language's ecosystem? E.g. Lucene is Java, and someone could use that, but there are tons of ports of it in C, C++, Python etc, providing convenience to integrate it with projects in these languages. >And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it. Not necessarily. A project with 10 part time contributors could be matched with a project with 2-3 full time competent hackers for example, or even surpassed.

frik 11y ago
> Lucene is Java ... there are tons of ports of it in C, C++, Python, etc. There used to be several ports, though most are dead and/or are several major versions behind. A new C++ or Rust port would be great, though unrealistic given the huge project side.
- coldtea 11y ago
  >though unrealistic given the huge project side. It could be done with some automated translation, and cleared out from there.

deet 11y ago

This project specifically is probably not useful for others just because it is written in another language (unless it were to succeed in fixing problems or improving security or performance.) Redis is a server application, not a library, so there's a clearly defined and interoperable protocol to bridge the Redis server to other languages already. Writing a Redis client for the other language would be more useful. More generally, as coldtea mentions, making integration into the rest of the language's ecosystem is the primary benefit of rewriting in another language. The value of such a port to others depends on how easy it is to integrate between the two languages, either via libraries or other methods. The harder it is to integrate the two (and the absence of automated translation tools) increases the value of the rewrite to others. Your Core-NLP example is actually an interesting one, because that library has already been ported to other languages... It is available for the C#/F# ecosystem (http://sergey-tihon.github.io/Stanford.NLP.NET/ http://sergey-tihon.github.io/Stanford.NLP.NET/).