35 ms·
Pyre: Fast Type Checking for Python
- cup-of-tea 8y agoIf you're going to go to all the effort of declaring types, why not use a language that can take advantage of that information to get faster and more efficient code?
- always_good 8y agoBecause you already have Python code. Or because throwing out the baby with the bathwater isn't a smart business decision.
- thrawaylost112 8y agoLately, facebook has lots of NIH.
- philipov 8y agoCan't read this; can we get a non-facebook link please?
- deleted 8y ago[deleted]
- phooji 8y agohttps://pyre-check.org/ https://pyre-check.org/ roughly covers it.
- iffycan 8y agoThere's https://pyre-check.org/ https://pyre-check.org/
- seltzered_ 8y agoHere: https://github.com/zachlatta/pyre https://github.com/zachlatta/pyre :)
- ericpts 8y agoI am particularly curious why they chose to start from scratch instead of cooperating with Dropbox on mypy.
- ben509 8y agoThey didn't start entirely from scratch, they've been doing this with PHP already: > Internally, Pyre's high-level architecture is similar to that of Hack, Facebook's type checker for PHP. If they had a performant codebase to start with, this makes a lot of sense.
- stochastic_monk 8y agoThe same reason everyone has a deep learning platform. It’s about developer mindshare and industry dominance rather than honestly thinking they’ll make a better framework starting from scratch rather than improving someone else’s.
- oblio 8y agoAlso the same reason everyone has a VM, programming language, back end framework or front end framework. Apple: LLVM (I know I’m stretching the definition here :) ); Objective-C, Swift; N/A; Cocoa. Microsoft: .NET CLR; Visual Basic, C#, F#; ASP.NET; N/A. Facebook: HHVM; Hack; N/A; React. Google: Go, Dart; Golang, Dartlang; GWT, Guava; Angular; Android, Flutter. Oracle: JVM; Java; APEX; N/A. It seems that for some reason just Amazon doesn’t want to play :)
- mraison 8y agoWebsite: https://pyre-check.org/ https://pyre-check.org/ Github: https://github.com/facebook/pyre-check https://github.com/facebook/pyre-check
- danso 8y agoOT: I thought the docs for pyre-check.org looked really nice. Didn't look like a Jekyll or Sphinx template. Looked at the source code and it's another FB library, Docusaurus, which uses React and Markdown: https://docusaurus.io/ https://docusaurus.io/ Had been briefly discussed a few months ago on HN [0] but I guess I missed it. Always looking for some variety in static site/docs deployment! [0] https://news.ycombinator.com/item?id=15924779 https://news.ycombinator.com/item?id=15924779
- okl 8y agoI use mypy[0] for PEP-484 type checking on a regular basis in large projects. PyCharm[1] supports PEP-484 type checking as well and displays errors in the editor. -- [0] - http://mypy-lang.org/ http://mypy-lang.org/ [1] - https://www.jetbrains.com/pycharm/ https://www.jetbrains.com/pycharm/
- mylons 8y agopycharm is so good.
- gipp 8y agoMypy is the standard, but too slow to integrate into editors (more appropriate for CI). PyCharm built its own implementation for this reason, but of course it's proprietary. Pyre's trying to get typecheck integration into other editors.
- sametmax 8y agoI got mypy plugged in on VSCode. It works very well. You don't get error reporting at the very second you type things but it never bothered me. I was more annoyed that it was practically unusable for years. Saner defaults, bug fixes and new types made it finally useful in late 2017.
- klibertp 8y agoWhy do you think it's too slow? I have it integrated with Emacs' flymake and it works pretty well. It takes less than a second to run mypy and display its results (with red underline or whatever else you have configured). From my experience, Visual Studio with F# takes similar time to type-check and show errors.
- neuland 8y agoIt's cool to see another type checker! I haven't had any big issues with MyPy, but competition is good in a healthy ecosystem. The language server / editor integrations are an interesting goal and that seems to be why they have the watchman stuff in there. I was initially kinda put off by that, as tools that watch for file events never end up working for me because I save like every 10 seconds.
- phooji 8y agoYou can run Pyre without using watchman if your editor-of-choice supports LSP -- the two modes are basically complementary. You can also run `pyre check` in non-incremental mode, though that may be slower depending on the size of your codebase.
- neuland 8y agoSo you're saying the LSP portion checks the file(s) you're currently editing, whereas the watchman (also incremental) check or the non-incremental check do the whole project? Edit: for the active file / LSP part, I mean it shows errors mainly for the active file.
- sinancepel 8y agoExactly - the LSP portion of pyre works for files you have open in an editor, but might miss changes due to a rebase or files you edit on the terminal. The watchman integration is there to make sure that pyre's aware of changes outside your editor. Edit: pyre will show type errors for all files in your repository, not just the ones you have open.
- neuland 8y agoAh, that's nice. So you can change the type of a function in one file that you are editing, and pyre will show you all the other places that are now using the wrong types.
- mraison 8y ago
- ben509 8y agoIt should be easy enough to put together a flycheck plugin for emacs users. I'll post a comment here when I'm done if you're interested.
- phooji 8y agoPlease send us a pull request if this is something you'd like this to live in the pyre-check repo!
- kmonad 8y agoI would be very interested. Thanks
- kirubakaran 8y agoYou could make a new HN submission for this, not just a comment. I bet a lot of people will be interested in trying it and also discussing it.
- mlevental 8y agowhat i don't understand about stuff like this (and honestly typescript as well) is if there aren't definitions for type schemas then this doesn't work right? like none of these tools will introspect or something and figure out which types i really mean without me annotating with some kind of crazy sum/tuple type definition (e.g. foo() -> Object{a:b,c:d} or whatever it is in typescript). maybe i'm wrong and there's something different here?
- orf 8y agoPyCharm, and perhaps other tools/editors, does introspect the code. It's not too hard to work out what protocols or interfaces an argument requires, e.g if you use it in a for loop it needs to be iterable, if you index or slice it then it needs to support that, if you add it to an integer etc etc. It's not about the concrete types but instead about what protocols it supports.
- hobofan 8y agoEven without you explicitly defining types, there is already quite a bit such tools can help you with since they usually come with type annotations for the standard library (or equivalent) of their languages.
- mlevental 8y ago>there is already quite a bit such tools example?
- hobofan 8y ago0.to_string() / 5 map([1, 2, 3], lambda n: n * 2).some_field_that_doesnt_exist A type checker would find both mistakes very easily, by only using the information from the standard library.
- ben509 8y agoTypeshed[1] identifies the types of all builtins for all current (2.7+) versions of python. [1] https://github.com/python/typeshed/ https://github.com/python/typeshed/
- hungerstrike 8y agoDon't you think it's a bit tone-deaf to be linking to Facebook here this week? Why wouldn’t you link to the projects fucking website? Seriously...why?
- alangpierce 8y agoThe link is to the announcement of the project. The announcement is a public Facebook post, which seems reasonable given that Facebook developed the project.
- sctb 8y agoYou've been posting a lot of unsubstantive comments, could you please re-read the guidelines? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- hungerstrike 8y agoSeriously? Another user posted "can't read this; can we get a non-facebook link please" - is that substantive? Or, are you just picking on my comment because I phrased my concern differently? Also, "a lot of unsubstantive comments" is false. Two of my recent comments created large sub-threads. Two others were downvoted simply because people didn't agree with my opinion. So, I think it's fair to say that you're ignoring plenty of "unsubstantive" comments and basically picking on people who you disagree with.
- nebgnahz 8y agoSo, there is a trend to build static type checker for popular managed dynamic languages? Facebook also built Flow, a type checker for Javascript [1]. [1] https://flow.org/ https://flow.org/
- lacksconfidence 8y agoThey also transformed php into a very similar language (hacklang) and built a type checker for that.
- m0meni 8y agoAlso they're rewriting messenger in basically OCaml (Reason[0]) i.e. a statically typed FP language. [0]: https://reasonml.github.io/ https://reasonml.github.io/
- dons 8y agoAnd "eradicate" for Java... http://fbinfer.com/docs/eradicate.html http://fbinfer.com/docs/eradicate.html we love types. They help us ship stuff faster.
- laylomo2 8y agoThis should be upvoted higher. ReasonML is a fantastically beautiful language.
- weberc2 8y agoYeah, it just needs to work on its tooling and library story. Last time I tried, the OCaml toolchain was such a big pain that the Reason community basically told me to give up and compile to Node. And then there are relatively few native-ReasonML libraries, so you have to figure out how to integrate with OCaml which usually means learning how to _read_ OCaml without clawing your eyes out (I'm only sort of joking). I'm rooting for Reason, but it has a few nontrivial hills to climb before it's practical.
- Rotareti 8y ago
- jedberg 8y agoThe Python stuff is cool, but I was actually more excited about asciinema[0], the tool they used to make the demo video. It actually makes ascii move around so that someone can pause the "video" and copy/paste out of it! Thats so cool! [0] https://asciinema.org https://asciinema.org
- kevin_thibedeau 8y agoAsciinema has the fatal flaw that it is dependent on their server. When they go under so do your recordings. There are other options out there that can handle playback of traditional console recordings without such dependencies.
- devy 8y agoPEP-484 Type Hints are for Python 3.5+, sorry if you are running on Python 2.x like majority of the Google python codebase does. Update: for the downvoters, the PEP listed the python version for 3.5 here https://www.python.org/dev/peps/pep-0484/ https://www.python.org/dev/peps/pep-0484/. Yes it does have a python 2.7 section but after Python 2.7 is discontinued pretty sure Python core team won't support type hints on 2.x. On top of that, not to mention the fact the type hint suggested on 2.x (as a comment) is different from the official 3.x syntax.
- eesmith 8y agoThat PEP has a section titled "Suggested syntax for Python 2.7 and straddling code", https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code https://www.python.org/dev/peps/pep-0484/#suggested-syntax-f... > Some tools may want to support type annotations in code that must be compatible with Python 2.7. For this purpose this PEP has a suggested (but not mandatory) extension where function annotations are placed in a # type: comment. mypy supports those annotations for Python 2 code. Quoting http://mypy.readthedocs.io/en/latest/faq.html#how-do-i-type-check-my-python-2-code http://mypy.readthedocs.io/en/latest/faq.html#how-do-i-type-... : > How do I type check my Python 2 code? > You can use a comment-based function annotation syntax and use the --py2 command-line option to type check your Python 2 code. You’ll also need to install typing for Python 2 via pip install typing.
- epage 8y agoI fully endorse this. We've been using mypy on our Python2/Python3 project successfully. it is helpful for the type checking but I'm most interested in giving good data to IDEs for code completion.
- joshuamorton 8y agoCurious, do you mean google internal or external code? Google's been developing pytype for quite some time, which is used internally.
- 8y ago
- pspeter3 8y agoHas anyone tried adding this to Bazel as a test?
- ambivalence 8y agoThat shouldn't be hard to integrate. At Facebook we have it integrated with Buck which is pretty similar to Bazel.
- pspeter3 8y agoAwesome, thanks! Sometimes tools require a global context which makes it hard to split work into packages
- wocram 8y agoDo you mean to check Skylark or python built with bazel?
- pspeter3 8y agoI mean using Pyre as something like a py_test. I assume Google's new Go based Skylark evaluator could test Skylark
- seanwilson 8y agoIt's interesting how static type systems are slowly gaining more adoption now by being bolted on to popular dynamic languages. It feels like a weird approach compared to using a language that was designed from the ground up to have a strong static type system but it's practical in terms of easing people into it.
- aero142 8y agoI think describing these as static type systems is a bit limiting. I prefer to think of them as static code analyzers. They don't change anything about the representations in memory, but they do validate invariants about the code. I actually hope we start seeing systems that validate invariants that are not type based like "SQL Injection Safe SQL Query" and "Range between 10-100".
- dragonwriter 8y ago> I actually hope we start seeing systems that validate invariants that are not type based like "SQL Injection Safe SQL Query" and "Range between 10-100". Both of those can be types in a rich enough type system; the latter is a fairly common example for dependent types, for instance.
- hellofunk 8y agoAgreed, it's like saying C++ has a borrow checker like Rust because memory sanitization tools exist.
- kbd 8y agoFWIW, static languages have added dynamic typing features as well: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/types/using-type-dynamic https://docs.microsoft.com/en-us/dotnet/csharp/programming-g...
- jerf 8y agoBut I suspect whereas the static type additions to Python will tend to spread throughout your entire system (the more you annotate, the more useful the feature is), the dynamic features of static languages tend to be used in specific places for specific reasons (the less you use the feature and the more you encapsulate it so the dynamism doesn't "escape", the better it works with the program as a whole).
- sjroot 8y agoDoes this comply with the type hinting format outlined in Python 3's typing module? [1] [1] https://docs.python.org/3/library/typing.html#module-typing https://docs.python.org/3/library/typing.html#module-typing
- sinancepel 8y agoYes! Pyre adopts types from the typing module, and relies on typeshed[1] to infer types for standard library functions. [1]: https://github.com/python/typeshed https://github.com/python/typeshed
- m0meni 8y agoI'm curious what the proponents of dynamic typing over static typing think about the recent push towards adding type systems to the two biggest dynamic languages out there: Python and JS.
- brightball 8y agoIMO it’s that people coming from static languages to dynamic languages want them. I honestly can’t think of anybody in my career who’s been comfortable with dynamic languages that had a desire to move to static. It’s such an impediment to the entire programming style that it doesn’t naturally happen. On the flip side, I know plenty of static typing people who don’t seem to think they can function without it. No matter what your preferred language, Python and JavaScript are almost unavoidable and because of that I think you see a lot of stuff like this brought to the table to help make people more comfortable. Python for sysadmin, math, ML, etc. Javascript for browser. You see a lot of the same thing in the Elixir community with its gradual type system. I can’t tell you how many times I’ve seen the discussion from people who want it to be statically typed, but neglect that all of the guarantees from it go out the window with distributed nodes.
- m0meni 8y agoCould you elaborate on that last point regarding elixir and why type systems don't work for it? I couldn't find anything when I did a cursory search on google.
- jerf 8y agoI like to use a cellular metaphor for programs, where you convert incoming data to either internal representations or errors as soon as possible, and delay converting to external representations as long as possible on the way out back into the "unfiltered world", but inside the cell things are considered "clean" and trusted. Type systems fit this point of view very naturally, because they can define that "clean view" of the world and enforce that you do the conversions early, because otherwise you won't have the target types. However, a "cell" has to be local, within a context where for instance it can count on having a stable view of the types and what they can do. (This being Erlang/Elixir, we can talk about upgrading running processes, but in that case, you have to provide a conversion function as well, though the default and common one is the identity conversion. Still, there is a conversion process.) Once you have a system with multiple contexts, which you can't help but have if you're crossing machines but even just across OS processes you ought to behave as if you have a separate context, the type system is less helpful. You can't take a message from a foreign node of type X!foreign and simply assume it's of type X!local, because you do not know in the general case whether the foreign and local concept of X is the same thing. Perhaps they are different versions. Perhaps they aren't even the same program. At a sufficiently large scale or with sufficiently bad luck, perhaps the foreign node is actively lying to you in a attempt to hack you. You end up having to serialize from the foreign node to the local one, and then deserialize on the local node, in order to communicate. Especially in a world where nodes can upgrade their code without so much as dropping their TCP connection to you or something. So there is a point of view from which having strong types at the language level isn't all that helpful to this process, because even having strong types doesn't let you escape from these issues. You can do all sorts of automated stuff to try to escape from the code boilerplate aspects of this problem, but you can not fully wrap the semantic problem of "this message, no matter how hard we try to abstract this away for you, may be invalid by local standards". How much this matters to you depends on how distributed your system is. If you've got a 3-node Erlang/Elixir cluster that you fully control all the code for, you might be able to get away with just ignoring the issue. Like many other programming issues, at small enough scales it doesn't matter and you can just ignore it. But as you scale up it becomes an inevitable problem you must program for; consider something like an AWS API for S3 or something, served by a few different servers and consumed by thousands of clients using every possible version of every SDK Amazon has ever put out and who knows how many home-grown implementations. Types can define the communication contract at that level, but provide zero guarantees to a server about what it's going to receive.
- galonk 8y agoAnyone know how this project's capabilities compare the the PEP-484 checking built-in to IDEA/PyCharm? That implementation goes "above and beyond" inferring types based on use when there's no explicit annotations. I'm wondering if other projects are doing that kind of thing.
- shannonzhu 8y agoWe haven't yet released a type inference feature, but have been building and using one internally. It helps by adding type annotations directly, based on what the type checker is able to infer. We're hoping to release this feature in the future. Also, MonkeyType will allow you to add types to your Python based on types inferred at runtime: https://github.com/Instagram/MonkeyType https://github.com/Instagram/MonkeyType
- tjpnz 8y agoWhy would I want to use this over MyPy which has this problem largely solved?
- phooji 8y agoPyre is faster in some circumstances. With that said, the real answer is 'it depends', and it may well be the case that mypy serves your needs better.
- pcx 8y agoArgh, would love to have one for Ruby too! I am a big fan of Flowtype, definitely going to check out Pyre!
- bhahn 8y agoOn mobile web, pressing back from the note sends me to Facebook's homepage, not to hn. Anyone else notice this? This is on top of the pop-up modal prodding me to add a shortcut to Facebook on my home screen. These are the kind of annoyances and "tactics" that make me dislike Facebook and want to stop using them. Aside from that, pyre looks like a super useful tool. Hopefully it has fewer rough edges than mypy.
- vjeux 8y agoThis is a bug, could you send me the details at vjeux@fb.com? It goes back to HN for me.
- kodablah 8y agoSomewhat off-topic: Does anyone else feel the need to avoid products purely because of their maker regardless of quality? I like some Nestle products but avoid them because of their maker. I am also avoiding GraalVM for the same reasons. Some things we can't avoid of course (especially those with momentum or without equal), but if I can avoid things built by companies I don't like, I will (digitally and in meatspace). Am I unreasonable?
- dijit 8y agoProbably, but there is a lot to be said for having a moral sense. I will never use Oracle products or allow them to bleed into my infrastructure for any reason for example. It's possible that this has saved my company a lot of money, it's also possible that it was completely unjustified, but given the trend of the company in question- I believe it was a positive decision. Make choices, the best you can in the moment, you can usually revisit them later.
- teddyh 8y agoRemember that “quality” does not merely mean “works right now at this very moment”, but also should mean “is safe to build on and count on to still be there and developed in a reasonable direction for many years to come”. There are many companies which should not get an ounce of that kind of trust, regardless of how useful their products are right now; very many simply can’t be trusted, for a myriad of different reasons.
- hayd 8y agoHas anyone tried installing this? I get: $ pip install pyre-check Collecting pyre-check Could not find a version that satisfies the requirement pyre-check (from versions: ) No matching distribution found for pyre-check Just me? (I am on python 3.6 and pip(3).)
- bpicolo 8y agoWorking fine for me. Are you using public pypi?
- fermigier 8y agoMe too. I guess they provide binaries only for some platforms, and yours (and mine) is not supported. I tried to build from source but got another error after some time: File "ast/astStatement.ml", line 289, characters 41-74: Error: Unbound module Recognized (NB: testing on Mac OS 10.10).
- tuvistavie 8y agoChanging `INCLUDES = . parser ....` to `INCLUDES = . external parser ...` in the Makefile fixed compilation for me.
- sinancepel 8y agoIf you're having trouble with pip, I'd recommend trying `pip3.6 install pyre-check`.
- hayd 8y agoAh, I was running this on a personal macbook which is apparently on an old osx version. This seemed to be the issue: https://github.com/facebook/pyre-check/issues/11 https://github.com/facebook/pyre-check/issues/11 (Installed fine on an my work machine running OSX 10.13.3.)
- Bromskloss 8y agoIs there any chance of using this in conjunction with Coconut [0]? [0] http://coconut-lang.org/ http://coconut-lang.org/
- ambivalence 8y agoNo. Supporting an entire new AST and type system (pattern matching) is a large investment. It can be made if there's enough of a userbase out there to justify it. That doesn't seem to be the case. Not yet at least.
- mcintyre1994 8y agoOff topic but thanks for sharing this - it looks awesome!
- fithisux 8y agoPython is a very good language for academic use. It is a more featureful FOSS system that can be used for RAD or matlab like purposes. Lack of types is not a serious problem for small codebases. But when the code base passes the magical 10000 line spot, type problems start to become serious. I hate the annotations about types in comments it is not maintenable. Either you create a codebase strictly modularized respecting the lack of types and the 10000LOC speed limit or you change language. Golang is not fancy but it is really pragmatical and could appeal to the dynamic language crowd. The other choices are Java/D/Scala. Or one can use Python as a glue language (superpowered C written modules used by Python). This is the problem/solution with Lisp too. If you go dynamic, you should be very careful, but there are gains. Don't forget to write tests unless you intend for a Matlab style experience.
- some_account 8y agoIt's very easy to use rust for code that needs to be fast, and call them from python.
- hellofunk 8y agoI like Rust as much as the next HN enthusiast, but there's a reason there's not a lot of Rust code actually out in the wild, or jobs much available, or libraries with more than one creator/maintainer... and it's because of all the words you could use to describe Rust, the last one on that list would be "easy." Anything non-trivial in Rust is a lesson in extreme patience.
- ApolloFortyNine 8y agoIt's more because every place of employment uses the language of the lowest common denominator: either C++ or Java. A new language will have few people who can support the code. An old language will have many.
- stevesimmons 8y agoPython scales to large code bases too. I work in a decade-old Python codebase that is now up to 30m LOC. Our 2000+ developers are doing 15k commits a week, with continuous deployment. (I gave a talk on this at PyData London two weeks ago)
- cx1000 8y agoHypothesis 1: All successful dynamically-typed programs eventually grow large enough to get rewritten or retrofitted with static types. Hypothesis 2: All successful statically-typed languages eventually grow large enough to have a dynamically typed scripting language embedded in them. https://twitter.com/munificentbob/status/988555364252631040 https://twitter.com/munificentbob/status/988555364252631040
- dudul 8y agoWhile I see a lot of examples of 1), I can't think of one for 2). Which successful statically-typed language has that?
- cx1000 8y agoC and C++ are the basis for Python (CPython) and JavaScript, respectively. Lua was written in C.
- philwelch 8y agoLua is an example for the C-type languages, and Groovy is an example for Java.
- vorg 8y agoBeanshell is another example for Java, and did it before Apache Groovy did. Beanshell added features like optional typing and terse property syntax to Java, then Groovy extended Beanshell by adding closures to Java. Java has made these scripting languages less relevant lately because of more recent features like lambdas and inferred types.
- outworlder 8y agoI can't think of 2 as written. But big, successful programs will often do (2).
- naasking 8y agoC# has had a dynamic type built in for years now. It's literally a dynamically typed language embedded within the statically typed C#,which grew out of the work on gradual type systems.
- mehrdadn 8y agoThis doesn't work on Windows? I'm getting an error installing it: > python --version Python 3.6.1 > python -m pip install pyre-check Collecting pyre-check Could not find a version that satisfies the requirement pyre-check (from versions: ) No matching distribution found for pyre-check
- masklinn 8y agoLooking at the published package, they're binary wheels and only published for linux/AMD64 and OSX 10.13/AMD64. I guess it's manual installation for everybody else.
- mehrdadn 8y agoWow. Do you know how they even build this thing? The instructions I see just keep saying pip install everywhere. I don't see any setup.py/configure/Makefile/etc.
- robertlagrant 8y ago+1
- newscracker 8y agoI'm curious. Why the name "pyre" and how did it get that name? I get the "py" at the beginning to refer to python. What dead things are being symbolically (or otherwise) being burned here? Edit: I did skim through the comments here and also skimmed through the official site and the GitHub repo. I didn't find anything about this name.
- jwilk 8y ago"re" of course stands for regular expression... Oh wait, no. The person who came up with this name didn't think it through.
- newscracker 8y agoRegular expressions is what I thought when I just saw the name, but it didn't make sense in this context.
- edward 8y agoThey accepted my spelling correction pull request. https://github.com/facebook/pyre-check/pull/3 https://github.com/facebook/pyre-check/pull/3
- airstrike 8y agoNext up is re-recording the video at https://pyre-check.org/ https://pyre-check.org/ so the coder uses vim more fluently.
- leshow 8y agoI was under the impression that Ocaml was single threaded? How is it that they have type checker that runs in parallel?
- verlaguet 8y agoIt’s because it is built on top of the infrastructure that was introduce for Hack. The system first starts by allocating a large area in shared memory (through a call to mmap). That area is very large but because most of it wont be written to it’s ok in practice. After that, the program forks as many times as there are cores. Each of those cores are called “workers”. The first program is now the master. The master and the workers communicate through pipes, but that is only used for synchronization. The lion share of the data goes through that shared memory that was mmaped at the beginning. There are 2 main things shared in that area. 1- atomic hashtables of serialized ocaml objects (you can define as many as you like with a functor) 2- an atomic table of dependencies. Each workers can read and write to those tables without locks (but only the master can remove from them). It turns out that that setup works well in practice. Serializing/deserializing costs are mitigated by a cache of deserialized values for each core. And this way each core can manage there memory (the gc does not need to scan the shared memory). Because that setup was working well in practice, it was reused by Flow and now Pyre. We refer to that setup internally as the “Hack infrastructure”.
- sinancepel 8y agoYou can have multi-threaded OCaml (our server actually has two threads), but only one can run at a time currently. We solve this problem by having a multi-process architecture where workers only communicate through a lock-free hash table in shared memory.
- leshow 8y agoIs it fair to say that it runs in parallel if only one thread may run at a time? Isn't that concurrent, but not parallel?
- platinumrad 8y ago
- zeth___ 8y agoPython has not types. Let me repeat: Python has no types. It has built in classes that used to be types 15 years ago, but no types. [0] 90% of the holy war between strongly and weakly typed in python would be resolved if we just removed all references for types from Python, and renamed TypeError to UnsupportedOperand. Pep 484 [1] is about class hints, and this software builds on top of that. That has it's place, but it's an ugly hack that should not become a main feature of the language. [0] https://www.python.org/download/releases/2.2/descrintro/#introduction https://www.python.org/download/releases/2.2/descrintro/#int... [1] https://www.python.org/dev/peps/pep-0484/#type-definition-syntax https://www.python.org/dev/peps/pep-0484/#type-definition-sy...
- miggyrozay 8y agoHoping this gets us closer to pycharm-like type checking for jupyterlab/notebooks. Or does that already exist?
- mdaniel 8y agohttps://www.jetbrains.com/help/pycharm/ipython-notebook-support.html https://www.jetbrains.com/help/pycharm/ipython-notebook-supp... I've tried it back when we were using PySpark and it did what I expected, but I am not the heaviest consumer of Jupyter notebooks to be able to say if it's 80/20 or 100% of what one would expect
- augbog 8y agoI'll be honest -- we've been using Flow for JavaScript on our team for a while and we can't get over a lot of the problems we run into with Flow. We have been considering moving to Typescript. I can't really see how adding a static type checker to Python will be any better. I know it's probably better than nothing but if the same problems carry through, it'll be a pain.
- weberc2 8y agoPython’s type checking story is pretty painful right now, partly because the typing library has a lot of confusing corners, MyPy (the reference type checker implementation) is pretty buggy, the rules are limiting (no support for interfaces/protocols and no recursive types), and magical code like SQLAlchemy simply can’t be type-stubbed. These are all tractable problems, so I don’t see a theoretical problem; however without more info on your flow issues I can’t make a good comparison.
- agravier 8y agoGenerally true but MyPy has some support for protocols http://mypy.readthedocs.io/en/latest/class_basics.html#protocols-and-structural-subtyping http://mypy.readthedocs.io/en/latest/class_basics.html#proto...
- weberc2 8y agoWhoa, I didn’t realize structural subtyping had landed already. Thanks for sharing!
- uryga 8y agoI'd be happy to be corrected on this, but the documentation seems a bit... bare? I've used mypy, and I'm interested in types more complicated than `foo() -> int`. I'd like to see how pyre handles generic stuff, unions etc... Maybe that's all rolled into "we support PEP X", but it'd be nice to see some examples. (I might have somehow missed the full docs - in that case, if anyone has a link, I'd be grateful.)
- sinancepel 8y agoIf you're interested in the details of how pyre handles generics, unions, etc., you can take a look at the end-to-end typechecking tests, which provide lots of examples: https://github.com/facebook/pyre-check/blob/master/analysis/test/analysisTypeCheckTest.ml https://github.com/facebook/pyre-check/blob/master/analysis/...
- uryga 8y agoThanks! I'll dig into it later, as I've been looking for a tool like his, and it so happens that the details are interesting to me :) (I built my own namedtuple-style library for sum types, go figure). It didn't seem like the easiest reading though - that could make some potential users bounce... Are you working on docs that expose these features in a friendlier way? edit: Also, was I correct in that it's basically "The stuff that's specified in the PEP and works in mypy should work"?
- elbear 8y agoI'm glad to see more mind-share for static type checking in the Python community. At the moment, types aren't as popular in the Python world as they are in the JavaScript world where you can find TypeScript type definitions for most libraries. I hope adding types to your library becomes the norm in Python as well, because as it stands, very few Python libraries have types defined. This means you can only have type guarantees in the code you write and not in that of your dependencies.
- Rapzid 8y agoThis is a great development for Python. The language services ecosystem was in the ICU, but perhaps now it's just on life support. VS Code, my editor of choice, supposedly has a pretty good python experience.. That was not my experience; slow! Just pulling tensorflow into a project would introduce serious save lag. Lag that's pretty much omnipresent to a lesser degree. Save and wait for syntax error updating... And wait.. Pylnint and Jedi were just not up to the task. A speedy language server is just what the doctor ordered. Still, annotation/inference experience falls apart completely when interfacing with libraries like SQL Alchemy or boto3. Supposedly writing custom plugins is the way, but that never gets done! How will pyre handle stuff like this, or is it just asking too much? Perhaps a combination of better tooling and better library authoring, eschewing the temptation of meta programming and dynanimo, to play well with type hints will be necessary? I'm excited for these new tools and will give it a play. However I honestly hope I never have to work in a python-forward environment again.
- FnuGk 8y agoHow is this different from the build in optional types in python 3?
- luckycharms810 8y agoWhat’s interesting about this announcement is that they not only announced a type checking tool, they are also open sourcing a tool that will apply a patch and annotate your code with types for you. I tried a demo of this at the Instagram booth at pycon yesterday. They basically had a package that would run your tests for you, and using the coverage that you created, it could infer the types while executing your code. Now if the point of type annotations is purely to check for bugs that are avoided in statically typed languages it seems like you should be able to just combine the two tools and catch type based bugs without ever applying the annotation. I think the type checking community also has quite a ways to go when it comes to asynchronous code. It becomes quite worthless to check that a future is being returned by a function, but it could be more helpful to know what that future will eventually yield.
- kasbah 8y agoSurely `await f()` should be a type error if `f` doesn't return a future and checking for that seems useful. I am not up to speed on Python's types but in most type systems you can do something like `f : Future<Int>` which _will_ tell you what the future will eventually yield an `Int`.
- lostmsu 8y agoI use PTVS for the same purpose. (Python Tools for Visual Studio) They do quite a bit of type inferring as well. It is also open source.
- neves 8y agoHas anyone integrated it with Visual Studio Code?