11 ms·
More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python
by timothycrosley 7y ago
More and more I want someone to create a new language that amounts to a strict subset of Python, with mypy built-in, and is compilable into machine code. Python has by far my favorite syntax, community, and in my experience leads to the greatest productivity. There just happens to be a lot of overly dynamic features, that aren't even used by most, but used just enough to hold back optimization and structural improvement.
- TylerE 7y agonim
- timothycrosley 7y agoI've been tracking nim, and would agree it's the most promising so far! I feel though that it's trying to be too flexible in many ways. Examples of this include allowing multiple different garbage collectors and encouraging heavy ast manipulation. I'm also afraid it is different enough to keep it from attracting a significant amount of developers from the Python community. Nonetheless, it's something I plan on using and contributing to, since it's the best option so far.
- nimmer 7y ago> allowing multiple different garbage collectors How's that a problem?
- carapace 7y agoCython? Nuitka?
- timothycrosley 7y agoI use Cython a lot! But mostly to speed up existing Python code, and build C-extensions faster. I don't see it as a strict subset of Python or a new language to build a community around. Nuitka I just started experimenting with to build standalone Python executable, and I really like the direction and roadmap they are following. In the end though both of these technologies seem like ways to somewhat speedup existing Python code and not attempts to introduce a strict language subset that would allow the greatest amount of optimization, and finally fix long running issues, like the inability to have multiple versions of a package installed.
- carapace 7y agoWhat about RPython? https://rpython.readthedocs.io/en/latest/rpython.html https://rpython.readthedocs.io/en/latest/rpython.html
- timothycrosley 7y agoAs far as I understand it, RPython isnt' really meant for actually writing programs in: > Do I have to rewrite my programs in RPython? > No, and you shouldn’t try. First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library, then it is not RPython to start with. You should only look at RPython if you try to write your own interpreter. See: https://rpython.readthedocs.io/en/latest/faq.html https://rpython.readthedocs.io/en/latest/faq.html
- carapace 7y agoHmm, still it might make a good foundation for the sort of thing you're talking about, eh? Cheers.
- timothycrosley 7y agoFor sure!
- scrollaway 7y agoPython has some of my favourite syntax as well but I absolutely hate its annotations. TypeScript got typings right. I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like. And of course if something changes the syntax, better anonymous functions will be the absolute first thing I would look for...
- timothycrosley 7y ago> TypeScript got typings right. I have not used TypeScript, but looking at it's documentation the syntax for type annotations look identical. Would you be willing to expand on why you think its approach is better / how it's different?
- scrollaway 7y agoNo importing basic types, using binary operators instead of awful things like Union and bracket accessors and what not, inline interfaces... Try it a bit. it truly is enjoyable. Fifteen years of python and I'm still enjoying TypeScript more. Mypy is limited by annotations having to be compatible python syntax.
- timothycrosley 7y ago> No importing basic type This at least is being solved in Python: https://www.python.org/dev/peps/pep-0563/ https://www.python.org/dev/peps/pep-0563/ > inline interfaces Typed dicts in 3.8 look pretty similar at cursory glance https://www.python.org/dev/peps/pep-0589/ https://www.python.org/dev/peps/pep-0589/ I'll definitely play around with it more!
- scrollaway 7y agoWatch the difference: A variable that can be an object with two elements (one of them being a list of strings), or a tuple of exactly two strings. // typescript let t: {a: string[], b: number} | [string, string] # python 3.8 from typing import TypedDict, Tuple, Union class SomeTypedDict(TypedDict): a: List[str] b: Union[float, int] t: Union[SomeTypedDict, Tuple[str, str]] I had to google a bunch to figure out how to write the Python version, whereas the typescript one was completely natural to write. It takes one line and requires no imports. The interface is inlined. All of this also makes it more readable when you come across it eg. in an IDE tooltip.
- bratao 7y agoThere is https://github.com/python/mypy/tree/master/mypyc https://github.com/python/mypy/tree/master/mypyc that I think is a great idea and approach
- staticassertion 7y agohttps://github.com/python/mypy/tree/master/mypyc https://github.com/python/mypy/tree/master/mypyc This may interest you. This is probably going to be the 'official' way to get what you're talking about.
- timothycrosley 7y agoThank you for sharing! This looks really promising, I'll try to think of ways I can contribute to the project.
- ledauphin 7y agoyeah, I'm excited about this approach. but it's a long way from being a realistic approach for folks outside Dropbox, it seems.
- staticassertion 7y agoIt's a long way from being a realistic approach for anyone, including Dropbox, really.
- timothycrosley 7y agoWhat would you say are the biggest blockers to becoming realistic? I saw on the README they need tools in the Python ecosystem to start utilizing them, which I can help with starting with isort, beyond that I'd want to do whatever I can to help the project succeed.
- chubot 7y agoProbably the biggest issue is that it can't run many libraries and frameworks because they use a lot of dynamic features, i.e. reflection and metaprogramming. To be more specific: getattr, operator overloading, descriptors, heterogeneous dicts, decorators, etc. Type checking and metaprogramming are fundamentally at odds [1]. Dynamic languages like Python have more of a focus on the latter. They later added type checking, but it comes at the "cost" of ruling out the more idiomatic metaprogramming and reflection features. In other words, static typing makes your source code bigger. Well, optional typing to some degree lets you have the best of both worlds -- you can skip type checking of the hard parts. But optional typing doesn't let you compile your program to make it faster -- you need a fully-typed program for that. ---- I'm doing something similar to mypyc with https://www.oilshell.org/ https://www.oilshell.org/ (I actually visited Dropbox and chatted with them about it back in the spring.) The difference is that I'm compiling Oil's Python source to C++ rather than to Python-C extension modules. So it doesn't depend on the Python runtime. It's not done but it's working well so far, and it's given me a lot of appreciation for which dynamic features Python programs actually use! (both my own and others) Also note that mypyc was used to speed up mypy, which is a type checker. A type checker is a very particular kind of program that's different than 99% of the use cases of Python. So success on speeding it up is super impressive but it's not clear it generalizes. The same is true for Oil -- my translation work doesn't generalize to arbitrary Python programs. Lots of people have died on that hill because it's really hard. You have a hard tradeoff between the kinds of Python programs you can support and the speedup you can give them. There are 10-20 projects over the last 2 decades at various points along that spectrum. In addition to mypyc, Oil's strategy was also inspired by Shed Skin, which is an impressive but mostly dormant Python-to-C++ compiler. ---- So in short I would say the problem is that nobody will be able to agree on a subset. You will have a lot of different fragments of Python geared toward particular use cases. But Python will very often be more appealing than any of those fragments because it has a bigger ecosystem. One thing that I've appreciated more and more while designing a language is how much the network effects and inertia matter. It's why we're still using C and C++ after almost 50 years. I'm sure every day there is still a lot more C++ written than Go, Rust, Swift, and D combined, etc. Python has a similar network effect and it will be around basically forever in its current form. Software doesn't really get rewritten or reduced -- more stuff just gets added on top. [1] I wrote some posts about that tradeoff here: http://www.oilshell.org/blog/tags.html?tag=metaprogramming#metaprogramming http://www.oilshell.org/blog/tags.html?tag=metaprogramming#m...
- weberc2 7y agoSounds like Go. ;) This is a cheeky remark, but I use Python and Go, and Go very much feels like an improved Python in most ways. Especially when it comes to static analysis, build tooling, distribution, performance, etc. In particular, I love that there are no venvs, pipenvs, virtualenvs, pyenvs, wheels, eggs, setuptools, easy_installs, etc.
- timothycrosley 7y agoI hate the fact that you may be right, because I really don't like Go in many ways: - I hate it's module system and package eco-system story. - I don't like its syntax. - I don't like its error handling. - I'd much prefer gradual typing. - I want to maintain the ability to use interactive interpreters. - I don't like the fact that instead of being community driven it is Google driven. But, anecdotally, I see go being used as a second language to Python more than anything else and at an ever accelerating rate.
- weberc2 7y agoThese are all fair points. I really enjoy Python, but there are too many things I fight with on a regular basis that simply aren’t issues in Go. It could be so much better if (1) there was a better type system (mypy is unnecessarily shoehorned into the syntax and still very broken—can’t even express recursive types like JSON), (2) a good way to constrain the dynamism so performance could be improved, and (3) a better environment/package management and distribution story (so far pantsbuild.org and PEX files are the best I’ve found). Then there are a long tail of more minor issues, like async/await vs goroutines, real parallelism, etc.
- timothycrosley 7y ago> type-system I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9. > Good way to constrain the dynamism so performance could be improved Couldn't agree more! > environment I find poetry a joy to use. If you want to bypass venvs all together, there's a lot of work to make that a reality, such as https://github.com/David-OConnor/pyflow https://github.com/David-OConnor/pyflow. > packaging Python in 3.5 added complete zip app support, which has improved this dramatically from my perspective. Extended by things like shiv https://github.com/linkedin/shiv https://github.com/linkedin/shiv make it fairly complete. > async/await This is interesting to me. I prefer async/await in general, because it has become a standard across programming languages and I find it really easy to reason about. I also find channels to be too widely seen as a cure-all, when the only study so far has shown they actually led to an increase bug count. But I don't discount the value of real-parallelism, and am glad to see that Python has been pushing harder on that lately, with things like subshells that allow bypassing the GIL on a single thread.
- totalperspectiv 7y agoI'm hoping someone with more experience will chime in, but what about rpython / pypy?
- cpeterso 7y agoI like Python, but I often wonder how many developers use Python because they actually use dynamic language features versus just liking the languages' clean syntax and library ecosystem. I'm surprised languages that offer both REPL (for development) and AOT native compilation (for production), like OCaml, are not more popular. Evidence that syntax matters, I guess. :) mypy and mypyc are interesting but their compile-time checks and optimizations are still hampered by Python's dynamic language semantics.
- Sophistifunk 7y agoI think a great deal of this sort of thing could be done by just doing some eval in a dynamic state before you stop the vm and compile its stable state, rather than the actual source code.
- dgoldstein0 7y agoI think you missed part of the point of what the article was trying to say - or rather, what they hoped to do with this strict python. One of those things being some form of hot code loading. A snapshot of the state can't be incrementally rebuilt - it's very much all or nothing; whereas if we know or modules are side effect free, or at least some useful part of module loading is, we could cache that part and get faster start up times on incremental changes.
- angry_octet 7y agoSounds like they need Erlang.
- peteradio 7y agoNames matter and OCaml is a crappy name.
- angry_octet 7y agoIf you think OCaml is bad, it used to be Caml Special Light, a play on Camel cigarette naming.
- Nimitz14 7y agoYeah I was hoping Nim would be it but I don't like the syntax they use.
- jsmeaton 7y agoFor the sites I typically work on it’s very hard to give up the Django admin and all of the features it provides. At the same time, I’d love a stronger type system to avoid a bunch of the pitfalls that the dynamism of python has. So count me in.
- paulie_a 7y agoI completely agree. With python I need ten packages. With the shit show of JavaScript I need 100 conflicting packages. Why bother on a backend framework like js. it's a worthless language for backend development
- chenzhekl 7y agoHow about Nim? It has Python-like syntax and is as fast as C. https://nim-lang.org/ https://nim-lang.org/
- timothycrosley 7y agoAnswered this below: > I've been tracking nim, and would agree it's the most promising so far! I feel though that it's trying to be too flexible in many ways. Examples of this include allowing multiple different garbage collectors and encouraging heavy ast manipulation. I'm also afraid it is different enough to keep it from attracting a significant amount of developers from the Python community. Nonetheless, it's something I plan on using and contributing to, since it's the best option so far. Though, now that another commenter pointed out mypyc: https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc I believe I'll invest my limited free-time in that project instead, as it will allow me to stay within the Python community and eco-system that I love so much.
- Jefro118 7y agoJust in case it's of interest to anyone reading this, I interviewed the designer of Nim, Andreas, about his design choices and what he learned from Python and the C family here: https://sourcesort.com/interview/andreas-rumpf-on-creating-and-growing-nim https://sourcesort.com/interview/andreas-rumpf-on-creating-a... Gives some good insight into where Nim is going in the future too.
- strokirk 7y agoIt's certainly interesting to use! However, it's type checker still have a lot of work to go, since you can easily segfault due to using a nil reference.
- thelastbender12 7y agoVery much this! For numerical computing, Numba + llvmlite attempts to do it. I don't know however if this approach could be extended to other domains - say making a web framework. Given, python classes let you do so much tinkering, any attempts to port existing code will probably need a lot of rewriting?
- moksly 7y agoAside from having a two decade history of using C#, types is the only thing preventing us from going full Python. Even so, the dynamic types in Python are more often a benefit than a disadvantage because Python is so great at handling them automatic. We build our employee database, and from there our IDM, from a singel XML file in a really shitty format + three txt files in even worse formats (they are single line output files from an old mainframe system predating sap). We used to do it in a rather complicated Microsoft SSIS workflow with a lot of C# services. All in all it’s a 30 minute nightly runtime. I recently replaced it with around 500 lines of Python and a 1-5 minute Runtime (sometimes at the beginning of a school year we’ll see changes to around 1000 positions). Python eats the XML like it wasn’t shit. It takes things like terrible date formats, we’re talking the output of a SAP free-text box shitty, and ports then seamlessly into a SQL date field. This alone was a nightmare in C# and Python just does it. Still, after two decades of strict types it feels dangerous.
- deleted 7y ago[deleted]
- Rotareti 7y agoI can imagine the next big programming language will be one that is split into two language-variants: the "low-level-variant" and the "high-level-variant". The high-level-variant is a dynamic language with optional typing, which is good for scripting, fast prototyping, fast time-to-market, etc. The low-level-variant is similar to the high-level-variant (same syntax, same features mostly, same documentation), but it has no garbage collector, typing is mandatory and it runs fast like C/C++/Rust. Compiled packages that are written in the low-level-variant can be used from the high-level-variant without additional effort at all. The tooling to achieve this comes with the language. A language like this would be insane, IMHO.
- nikki93 7y agoA key consideration here would probably be the expression and passing around of managed instances spawned in the high-level variant through low-level code. Would you explicitly retain and release them? -- etc. I think it should be an ergonomic solution for this language to provide an edge over just using C / C++ / etc. with Lua / Python / etc.
- alex7o 7y agoYou can say that this is typescript and assemblyscript, they have the same syntax but one of them is compiled natively (wasm).
- Rotareti 7y agoSomething in that direction, but I'd imagine it more like a Rust with a high-level-variant than a JS/TS with a low-level-variant ;)