6 ms·

I've been thinking (and researching) heavily in preparation for a new (offline, desktop) app I want to build. The app has a lot of data wrangling, and probably

by pugio 4y ago

I've been thinking (and researching) heavily in preparation for a new (offline, desktop) app I want to build. The app has a lot of data wrangling, and probably a decent amount of ML. Python seems like the logical choice, but... I've written a lot of Python, taught classes in it, deployed production code... and I still feel a semi-conscious urge to reach for something else whenever I contemplate starting a new project. Something about its approach, syntax, common idioms, always feels just a tad clunkier to me than I'd prefer. The whitespace makes it hard to paste lines into my repl, and (probably the biggest thing) comprehensions ARE NOT SUFFICIENT replacements for easy anonymous inline functions (JS () => , or Ruby's blocks). I like dynamic languages, I really like the advances in tooling with type checking and LSP support. I like dynamic notebooks (either inline in VS Code #%%, or straight Jupyter), the massive package ecosystem (but obviously hate the actual packaging tools), cool tools like Rich... but it just doesn't make me as happy to use as other languages. I'm still trying to articulate why. Maybe I was just ruined by learning Ruby first.

pphysch 4y ago
Unless you have a clear specification for what you are going to build (as in, you have most of your data schemas and processes sketched out), stick with Python and benefit from the development velocity as you figure things out. Then rewrite it later when you need to scale.
- nnadams 4y ago
  I have embraced this workflow completely. I used to be more concerned about which language, but now I find I much more useful to just start immediately in Python. I spend most of the time working out the kinks and edge cases, instead of memory management or other logistics. Maybe ~75% of the time, that's it, no need for further improvement. Recently I chose to rewrite several thousands of lines of Python in Go, because we needed more speed and improved concurrency. Already having a working program and tests in Python was great. After figuring out a few Go-isms, it was a quick couple of days to port it all for big improvements.
  - hsbauauvhabzb 4y ago
    When you say tests are you referring to end to end tests, did you rewrite unit tests or did you do some cross language magic?
    - nnadams 4y ago
      End to end tests mostly. Those were kept static throughout the process as a baseline and for benchmarking. That let us have simple metrics like "test 1 had a 100x speed up" and "test 3 took a way too long to finish before the rewrite." Any unit tests got rewritten along with the test's relevant code. Now that we are sticking with this Go based solution, adding more unit and regression feels more reasonable now.
- alar44 4y ago
  This is what I do. Hack it together in Python to get something running, then rewrite everything in C++.
  - JonChesterfield 4y ago
    This works especially well if you test the python version as you go, then run the same python tests against the C++ replacement. The up front bother of pybind or similar is minor relative to not having to recompile the C++ to iterate on test scripts.
  - trenchgun 4y ago
    Or even better, rewrite the python code in Rust! Like Asahi Lina did. https://asahilinux.org/2022/11/tales-of-the-m1-gpu/ https://asahilinux.org/2022/11/tales-of-the-m1-gpu/
- therealdrag0 4y ago
  Gp was saying their biggest complaint is better in JS and Ruby, so aren’t those just as good for velocity and not for scale?
- nine_k 4y ago
  The same thing can be said about Lisp, Ruby, or JS/TS. Start a REPL, shape things as you would clay. What Python gives you, and what's not easy to get elsewhere, is access to really high performance numerical code, wrapped into quite ergonomic interfaces: Numpy, Scipy, Pytorch, etc. Many other things choose to make Python the glue language, like above, or to embed it (from Gimp to Blender). This makes Python stand out, despite its shortcomings (which are well-known), and make it an even more reasonable interface language for whatever next big thing. Network effect in action.
- mmcnl 4y ago
  Notice the rewrite most often will never come when you do this because it will probably not be needed. And if it comes to it that you do need a rewrite, celebrate the success of your project!
devoutsalsa 4y ago
Elixir is nice is you are looking for something new to try.
- jamal-kumar 4y ago
  I recently discovered it. It's pretty amazing for how incredibly easy it is to build distributed applications with it, that is the part which impressed me the most. It handles concurrency well for sure but same program seamlessly across multiple computers is just awesome. I know it's basically just taking this functionality from being built on top of erlang already but it's still very impressive. Plus like crystal if you already know ruby, having tried to base it's syntax on it, it's an easy thing to pick up if you can grasp the functional aspects too.
  - deleted 4y ago
    [deleted]
angrais 4y ago
>> and (probably the biggest thing) comprehensions ARE NOT SUFFICIENT replacements for easy anonymous inline functions (JS () => , or Ruby's blocks). I'm confused by this statement. As I understand it, list comprehensions exist to transform data. Inline functions in JS would be equal to lambdas in python, I.e. anonymous functions. So what differences between JS and python do you think is missing?
- code_runner 4y ago
  Multi line anonymous functions. An anonymous function is just as good as a true function in JavaScript. Lambdas are incredibly limited and encourage the declaration of a whole separate function that now lives physically far from its use. List comprehensions are also really underpowered and I personally hate the map and filter functions and their goofy list-like types. Check out linq in .net, streams in Java, array in JS, collections in scala to see the expressive power of a fluent interface on collections
  - beckingz 4y ago
    just make a function at that point??
  - Spivak 4y ago
    > declaration of a whole separate function that now lives physically far from its use Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time. > I personally hate the map and filter functions and their goofy list-like types. You mean iterators? There’s nothing else they could possibly return. Map and filter are lazy and work with arbitrary iterables even ones that never end or have no possible way to define bind.
    - Izkata 4y ago
      > Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time. I think a lot of people have no idea nested functions exist in python...
dist1ll 4y ago
+1 for the Elixir recommendation. Whenever I'm looking for ergonomics, I try to look for the functional languages (Haskell, Elixir, Ocaml, Idris, etc.). And since you mentioned dynamic languages and Ruby, I think it might be a really good match for you!
weakfortress 4y ago
It sounds like you're incredibly picky. I moved away from Ruby to Python years ago for 99.9% of my professional work. Ruby to me is write once read nowhere. "Professional" developers invent all sorts of creative DSLs to make code impossible to use. Rails is insufficient for a project of modest complexity owing once again to it's 9 levels of hell worth of DSLs. If you follow the prescribed formulas exactly, to the letter, and never stray from the DSLs given in Rails, and Ruby code in general, is fantastic. Once you start to stray you end up in hell. Python has it's faults. Most are handled by a good editor. I only switch from Python when I need something lower level. With Python a company gains: 1. Simple language with relatively low ramp-up time 2. Reasonably fast with proper tuning. I've been on projects processing impressive numbers of RPS built entirely in Python. 3. Comprehensions. You claim they are not sufficient. With them I rarely need to lambda anything. When I do I have `functools` to do any other work. 4. A massive ecosystem of libraries, resources, etc. A company loses: 1. Projects become difficult to understand at a certain size. This is not properly addressed by even the proper use of MyPy. 2. Somewhat more involved testing process compared to it's contemporary - Ruby. To compare apples-to-apples you can even look at the interpreters themselves. Python's mainline interpreter has been able to run CIRCLES around Ruby's for years now. I can understand using something like Go for systems level work, but there are few languages that give you so much for very little investment. Even dropbox was run entirely on Python for LONG time before it became too unruly. That's INCREDIBLE given the simplicity of the language. What you use on your personal time is irrelevant. Your complaints apparently are not experienced by the swathes of companies switching from almost every language under the sun to Python. It's here to stay. At least until Go's ecosystem catches up to Python's. My belief is Go will be the only language to unseat Python at this point. Python will still be around as a "perl-replacement" for basically ever as far as I am concerned.
xhevahir 4y ago
ptpython does a pretty good job of handling pasted code, in my experience.

by pugio 4y ago

pphysch 4y ago

Unless you have a clear specification for what you are going to build (as in, you have most of your data schemas and processes sketched out), stick with Python and benefit from the development velocity as you figure things out. Then rewrite it later when you need to scale.

nnadams 4y ago
I have embraced this workflow completely. I used to be more concerned about which language, but now I find I much more useful to just start immediately in Python. I spend most of the time working out the kinks and edge cases, instead of memory management or other logistics. Maybe ~75% of the time, that's it, no need for further improvement. Recently I chose to rewrite several thousands of lines of Python in Go, because we needed more speed and improved concurrency. Already having a working program and tests in Python was great. After figuring out a few Go-isms, it was a quick couple of days to port it all for big improvements.
- hsbauauvhabzb 4y ago
  When you say tests are you referring to end to end tests, did you rewrite unit tests or did you do some cross language magic?
  - nnadams 4y ago
    End to end tests mostly. Those were kept static throughout the process as a baseline and for benchmarking. That let us have simple metrics like "test 1 had a 100x speed up" and "test 3 took a way too long to finish before the rewrite." Any unit tests got rewritten along with the test's relevant code. Now that we are sticking with this Go based solution, adding more unit and regression feels more reasonable now.
alar44 4y ago
This is what I do. Hack it together in Python to get something running, then rewrite everything in C++.
- JonChesterfield 4y ago
  This works especially well if you test the python version as you go, then run the same python tests against the C++ replacement. The up front bother of pybind or similar is minor relative to not having to recompile the C++ to iterate on test scripts.
- trenchgun 4y ago
  Or even better, rewrite the python code in Rust! Like Asahi Lina did. https://asahilinux.org/2022/11/tales-of-the-m1-gpu/ https://asahilinux.org/2022/11/tales-of-the-m1-gpu/
therealdrag0 4y ago
Gp was saying their biggest complaint is better in JS and Ruby, so aren’t those just as good for velocity and not for scale?
nine_k 4y ago
The same thing can be said about Lisp, Ruby, or JS/TS. Start a REPL, shape things as you would clay. What Python gives you, and what's not easy to get elsewhere, is access to really high performance numerical code, wrapped into quite ergonomic interfaces: Numpy, Scipy, Pytorch, etc. Many other things choose to make Python the glue language, like above, or to embed it (from Gimp to Blender). This makes Python stand out, despite its shortcomings (which are well-known), and make it an even more reasonable interface language for whatever next big thing. Network effect in action.
mmcnl 4y ago
Notice the rewrite most often will never come when you do this because it will probably not be needed. And if it comes to it that you do need a rewrite, celebrate the success of your project!

devoutsalsa 4y ago

Elixir is nice is you are looking for something new to try.

jamal-kumar 4y ago
I recently discovered it. It's pretty amazing for how incredibly easy it is to build distributed applications with it, that is the part which impressed me the most. It handles concurrency well for sure but same program seamlessly across multiple computers is just awesome. I know it's basically just taking this functionality from being built on top of erlang already but it's still very impressive. Plus like crystal if you already know ruby, having tried to base it's syntax on it, it's an easy thing to pick up if you can grasp the functional aspects too.
- deleted 4y ago
  [deleted]

angrais 4y ago

>> and (probably the biggest thing) comprehensions ARE NOT SUFFICIENT replacements for easy anonymous inline functions (JS () => , or Ruby's blocks). I'm confused by this statement. As I understand it, list comprehensions exist to transform data. Inline functions in JS would be equal to lambdas in python, I.e. anonymous functions. So what differences between JS and python do you think is missing?

code_runner 4y ago
Multi line anonymous functions. An anonymous function is just as good as a true function in JavaScript. Lambdas are incredibly limited and encourage the declaration of a whole separate function that now lives physically far from its use. List comprehensions are also really underpowered and I personally hate the map and filter functions and their goofy list-like types. Check out linq in .net, streams in Java, array in JS, collections in scala to see the expressive power of a fluent interface on collections
- beckingz 4y ago
  just make a function at that point??
- Spivak 4y ago
  > declaration of a whole separate function that now lives physically far from its use Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time. > I personally hate the map and filter functions and their goofy list-like types. You mean iterators? There’s nothing else they could possibly return. Map and filter are lazy and work with arbitrary iterables even ones that never end or have no possible way to define bind.
  - Izkata 4y ago
    > Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time. I think a lot of people have no idea nested functions exist in python...

dist1ll 4y ago

+1 for the Elixir recommendation. Whenever I'm looking for ergonomics, I try to look for the functional languages (Haskell, Elixir, Ocaml, Idris, etc.). And since you mentioned dynamic languages and Ruby, I think it might be a really good match for you!

weakfortress 4y ago

It sounds like you're incredibly picky. I moved away from Ruby to Python years ago for 99.9% of my professional work. Ruby to me is write once read nowhere. "Professional" developers invent all sorts of creative DSLs to make code impossible to use. Rails is insufficient for a project of modest complexity owing once again to it's 9 levels of hell worth of DSLs. If you follow the prescribed formulas exactly, to the letter, and never stray from the DSLs given in Rails, and Ruby code in general, is fantastic. Once you start to stray you end up in hell. Python has it's faults. Most are handled by a good editor. I only switch from Python when I need something lower level. With Python a company gains: 1. Simple language with relatively low ramp-up time 2. Reasonably fast with proper tuning. I've been on projects processing impressive numbers of RPS built entirely in Python. 3. Comprehensions. You claim they are not sufficient. With them I rarely need to lambda anything. When I do I have `functools` to do any other work. 4. A massive ecosystem of libraries, resources, etc. A company loses: 1. Projects become difficult to understand at a certain size. This is not properly addressed by even the proper use of MyPy. 2. Somewhat more involved testing process compared to it's contemporary - Ruby. To compare apples-to-apples you can even look at the interpreters themselves. Python's mainline interpreter has been able to run CIRCLES around Ruby's for years now. I can understand using something like Go for systems level work, but there are few languages that give you so much for very little investment. Even dropbox was run entirely on Python for LONG time before it became too unruly. That's INCREDIBLE given the simplicity of the language. What you use on your personal time is irrelevant. Your complaints apparently are not experienced by the swathes of companies switching from almost every language under the sun to Python. It's here to stay. At least until Go's ecosystem catches up to Python's. My belief is Go will be the only language to unseat Python at this point. Python will still be around as a "perl-replacement" for basically ever as far as I am concerned.

xhevahir 4y ago

ptpython does a pretty good job of handling pasted code, in my experience.