9 ms·

> And in the end, the code seems to run "fast enough" and nobody involved really notices that what is running in 750ms really ought to run in something more lik

> And in the end, the code seems to run "fast enough" and nobody involved really notices that what is running in 750ms really ought to run in something more like 200us. Nobody has created a language that is both thousands of times faster than Python and nearly as straightforward to learn and to use. The closest thing I know of might be Julia, but that has its own performance problems and is tied closely to its AI/ML niche. Even within that niche I'm certainly not going to get most data scientists to write their code in C or C++ (or heaven forbid Rust) to solve a performance impediment that they've generally been able to work around. It's great that you've been able to switch to higher-performance languages, but not everyone can do that easily enough to make it worth doing.

divan 4y ago
Go hit a really sweet spot here.
stefanos82 4y ago
> Nobody has created a language that is both thousands of times faster than Python and nearly as straightforward to learn and to use. Not Python-based, but Lua-based is Nelua [1] If you like Lua's syntax, LISP's metaprogramming abilities, and C's performance, well there you have it! [1] https://github.com/edubart/nelua-lang https://github.com/edubart/nelua-lang
morelisp 4y ago
The "iterate from notebook to production" process which is common everywhere but the largest data engineering groups rules out anything with manual memory management from becoming popular with data science work. Some data scientists I know like (or even love) Scala, but that tends to blow up once it's handed over to the data engineers as Scala supports too many paradigms and just a couple DSs will probably manage to find all of them in one program. We use Go extensively for other things, and most data scientists I've worked with sketching ideas in Go liked it a lot, but the library support just isn't there, and it's not really a priority for any of the big players who are all committed to Python wrapper + C/C++/GPU core, or stock Java stacks. (The performance also isn't quite there yet compared to the top C and C++ libraries, but it's improving.)
- ishjoh 4y ago
  I love scala and wish it was more popular. I've made piece with java at this point as it slowly adopts my favorite parts of scala but I miss how concise my code was.
Gibbon1 4y ago
C# is faster than python and as easy to use.
- monkpit 4y ago
  Apparently Jupyter works with .net now. Cool
- dataflow 4y ago
  As nice as it is, C# is definitely not as easy to use as Python.
  - radicalbyte 4y ago
    If you're using an IDE (Rider or Visual Studio) and avoid the Enterprise frameworks, then it's much easier to use than Python. Tooling makes a huge difference, no more digging through the sometimes flakey Python documentation and cursing compatibility issues with random dependencies not supporting Apple Silicon.
    - dataflow 4y ago
      I agree tooling makes a huge difference but I specifically said this with the understanding that you're using C# with Visual Studio. Some stuff will be easier in C#, but a lot of other stuff just isn't as easy as in Python. At the risk of setting up a strawman for people to punch down, try comparing how easy it is to do the equivalent of something like this in C#, and feel free to use as much IDE magic as you'd like: x = [t[1] for t in enumerate(range(1, 50, 4)) if t[0] % 3 == 0][2:] Was it actually easier? There's a million other examples I could write here, but I'm hoping that one-liner will be sufficient for illustration purposes.
      - eterm 4y ago
        Here's a one liner in c# for that: Enumerable.Range(1,50).Where((x,i) => i % 4 == 0).Where(e => e % 3 == 0).Skip(1).Select(e => e+4) Okay, so you might consider that last e+4 cheating and against the spirit, but I couldn't be bothered to spend money upgrading my linqpad to support the latest .net with Enumerable.Chunk which makes taking two at a time easier for the first part. Edit: more in spirit: Enumerable.Range(1,50).Where(e => e % 4 == 0 && e % 3 == 0).Skip(1).Select(e => e + 1)
alar44 4y ago
I think that's my argument. If a developer thinks C or C++ is really that difficult and they can only write effectively in Python, they're a shitty developer and the world seems to be jam packed with them.
V1ndaar 4y ago
I don't know, but imo Nim begs to differ.
lmm 4y ago
I was able to convert a couple of my data scientist colleagues over to using Scala (given that they were writing code for our Spark cluster it seemed like a no-brainer compared to Python or R). It's not thousands of times faster but it might be ten or a hundred times faster, and a lot of the time you can write the very same code aside from punctuation (and even that difference is smaller in Scala 3, although I don't think Spark has moved to that yet).
pjmlp 4y ago
It is called Common Lisp, maybe you have heard of it.
Dylan16807 4y ago
Those speed differences don't come from language, they come from using terribly wrong data structures and algorithms.

divan 4y ago

Go hit a really sweet spot here.

stefanos82 4y ago

> Nobody has created a language that is both thousands of times faster than Python and nearly as straightforward to learn and to use. Not Python-based, but Lua-based is Nelua [1] If you like Lua's syntax, LISP's metaprogramming abilities, and C's performance, well there you have it! [1] https://github.com/edubart/nelua-lang https://github.com/edubart/nelua-lang

morelisp 4y ago

The "iterate from notebook to production" process which is common everywhere but the largest data engineering groups rules out anything with manual memory management from becoming popular with data science work. Some data scientists I know like (or even love) Scala, but that tends to blow up once it's handed over to the data engineers as Scala supports too many paradigms and just a couple DSs will probably manage to find all of them in one program. We use Go extensively for other things, and most data scientists I've worked with sketching ideas in Go liked it a lot, but the library support just isn't there, and it's not really a priority for any of the big players who are all committed to Python wrapper + C/C++/GPU core, or stock Java stacks. (The performance also isn't quite there yet compared to the top C and C++ libraries, but it's improving.)

ishjoh 4y ago
I love scala and wish it was more popular. I've made piece with java at this point as it slowly adopts my favorite parts of scala but I miss how concise my code was.

Gibbon1 4y ago

C# is faster than python and as easy to use.

monkpit 4y ago
Apparently Jupyter works with .net now. Cool
dataflow 4y ago
As nice as it is, C# is definitely not as easy to use as Python.
- radicalbyte 4y ago
  If you're using an IDE (Rider or Visual Studio) and avoid the Enterprise frameworks, then it's much easier to use than Python. Tooling makes a huge difference, no more digging through the sometimes flakey Python documentation and cursing compatibility issues with random dependencies not supporting Apple Silicon.
  - dataflow 4y ago
    I agree tooling makes a huge difference but I specifically said this with the understanding that you're using C# with Visual Studio. Some stuff will be easier in C#, but a lot of other stuff just isn't as easy as in Python. At the risk of setting up a strawman for people to punch down, try comparing how easy it is to do the equivalent of something like this in C#, and feel free to use as much IDE magic as you'd like: x = [t[1] for t in enumerate(range(1, 50, 4)) if t[0] % 3 == 0][2:] Was it actually easier? There's a million other examples I could write here, but I'm hoping that one-liner will be sufficient for illustration purposes.
    - eterm 4y ago
      Here's a one liner in c# for that: Enumerable.Range(1,50).Where((x,i) => i % 4 == 0).Where(e => e % 3 == 0).Skip(1).Select(e => e+4) Okay, so you might consider that last e+4 cheating and against the spirit, but I couldn't be bothered to spend money upgrading my linqpad to support the latest .net with Enumerable.Chunk which makes taking two at a time easier for the first part. Edit: more in spirit: Enumerable.Range(1,50).Where(e => e % 4 == 0 && e % 3 == 0).Skip(1).Select(e => e + 1)

alar44 4y ago

I think that's my argument. If a developer thinks C or C++ is really that difficult and they can only write effectively in Python, they're a shitty developer and the world seems to be jam packed with them.

V1ndaar 4y ago

I don't know, but imo Nim begs to differ.

lmm 4y ago

I was able to convert a couple of my data scientist colleagues over to using Scala (given that they were writing code for our Spark cluster it seemed like a no-brainer compared to Python or R). It's not thousands of times faster but it might be ten or a hundred times faster, and a lot of the time you can write the very same code aside from punctuation (and even that difference is smaller in Scala 3, although I don't think Spark has moved to that yet).

pjmlp 4y ago

It is called Common Lisp, maybe you have heard of it.

Dylan16807 4y ago

Those speed differences don't come from language, they come from using terribly wrong data structures and algorithms.