4 ms·

Somewhat off topic, but for anyone more familiar with spaCy: Any idea if spaCy has a sane way to be used in other languages? Specifically Rust, in this case? I

Somewhat off topic, but for anyone more familiar with spaCy: Any idea if spaCy has a sane way to be used in other languages? Specifically Rust, in this case? I'm in need of a decent NLP library to train and or map sentences to intents. Somewhat similar to what wit.ai offers, but it needs to be offline-able. spaCy sounds great, but requiring a runtime (if spaCy indeed does) on the installed system is a no-go for me. Not knowing much about Python, It looks like it is actually compatible with CPython, so could it therefor be compiled to C and imported into Rust?

syllogism 10y ago
spaCy's implemented in Cython, which compiles into C++. So, the end-state at the moment is a .so object that expects to be loaded into Python. Now, you can instead tell Cython to compile into standalone objects, so that they can be imported from C/C++ code. I'm not well versed on this, but if you can call into C/C++ libraries easily from Rust, you should be able to call into spaCy. You'll miss some of the niceties from the Python layer, but the C-level API is pretty well behaved. So, you should be able to get by okay.
- danieldk 10y ago
  Curious: since spaCy is a commercial project (I know that it is open source), did you consider writing it in C or C++, it would make binding to other languages a lot easier. I can imagine that importing the Python runtime can be problematic in some projects. Also, I assume that you will see some Python types in the API when you want to call the parser from C? (My own NN dependency parser is written in Go, since it's mostly for research, but if I wanted to make it embeddable, I'd have implemented it in C++ or Rust with a C API.)
  - syllogism 10y ago
    Well, I like writing in Cython, and I figured I was better off pleasing some people (Python) a lot before trying to please a lot of people a little. It won't be hard to port to C++ when the time comes.
wyldfire 10y ago
> so could it therefor be compiled to C This is not the case in general for Python. If you want to run python from Rust or C you effectively link the CPython interpreter into your program and start executing the Python source code as an embedded string. IIRC some of spaCy is implemented in cython which I think generates C for a CPython extension. This is typically for lower-layer stuff and doesn't provide any real utility for linkage with C/Rust. "Python is so limiting" is an odd way to you express your problem IMO. It is limiting in that you can't just use a foreign-function interface to simply call into it without the baggage of a big runtime/interpreter. But in general it's a super-high level language that I wouldn't describe as "limiting." EDIT: I stand corrected, syllogism's comment indicates that the cython layer might be useful on its own.
- fizzbatter 10y ago
  > "Python is so limiting" is an odd way to you express your problem IMO. You are correct, i'll edit that to better reflect my intent. I worded it poorly, and meant no disrespect to Python :), it just doesn't fit my needs in this case. Appreciate the info!

syllogism 10y ago

spaCy's implemented in Cython, which compiles into C++. So, the end-state at the moment is a .so object that expects to be loaded into Python. Now, you can instead tell Cython to compile into standalone objects, so that they can be imported from C/C++ code. I'm not well versed on this, but if you can call into C/C++ libraries easily from Rust, you should be able to call into spaCy. You'll miss some of the niceties from the Python layer, but the C-level API is pretty well behaved. So, you should be able to get by okay.

danieldk 10y ago
Curious: since spaCy is a commercial project (I know that it is open source), did you consider writing it in C or C++, it would make binding to other languages a lot easier. I can imagine that importing the Python runtime can be problematic in some projects. Also, I assume that you will see some Python types in the API when you want to call the parser from C? (My own NN dependency parser is written in Go, since it's mostly for research, but if I wanted to make it embeddable, I'd have implemented it in C++ or Rust with a C API.)
- syllogism 10y ago
  Well, I like writing in Cython, and I figured I was better off pleasing some people (Python) a lot before trying to please a lot of people a little. It won't be hard to port to C++ when the time comes.

wyldfire 10y ago

> so could it therefor be compiled to C This is not the case in general for Python. If you want to run python from Rust or C you effectively link the CPython interpreter into your program and start executing the Python source code as an embedded string. IIRC some of spaCy is implemented in cython which I think generates C for a CPython extension. This is typically for lower-layer stuff and doesn't provide any real utility for linkage with C/Rust. "Python is so limiting" is an odd way to you express your problem IMO. It is limiting in that you can't just use a foreign-function interface to simply call into it without the baggage of a big runtime/interpreter. But in general it's a super-high level language that I wouldn't describe as "limiting." EDIT: I stand corrected, syllogism's comment indicates that the cython layer might be useful on its own.

fizzbatter 10y ago
> "Python is so limiting" is an odd way to you express your problem IMO. You are correct, i'll edit that to better reflect my intent. I worded it poorly, and meant no disrespect to Python :), it just doesn't fit my needs in this case. Appreciate the info!