11 ms·

Not typing is kinda a crazy way to write programs when you think about it. It can be beneficial in certain niche use cases like data science where code is alway

Not typing is kinda a crazy way to write programs when you think about it. It can be beneficial in certain niche use cases like data science where code is always terrible. For everything else, types are a huge boon for the developer and everyone consuming their work

waitlist 5y ago
I think data science is a perfect example in favor of types -- the code is often terrible because of the lack of typing. Pandas has notoriously poor developer ergonomics, and I recall painfully poring over type errors across the board -- lists, dataframes, numpy arrays, etc. are all iterables, so they can be interchanged in some contexts, but not in others. Had I had MyPy back when I was working in data science, I would've saved countless hours and headaches.
- bayesian_horse 5y ago
  What Pandas does is notoriously hard to fit into a compile-time type system. Certainly too hard to go into the brains of scientists who didn't grow up coding. No, the code in data science isn't bad because of the lack of typing. The code is "bad" mostly because those writing it are relatively fresh from starting to program. Also there is more pressure to make things possible, often just to run it once, and neglect repeatability or scaling to larger code bases. Different emphasis. That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain.
  - arinlen 5y ago
    > What Pandas does is notoriously hard to fit into a compile-time type system. Certainly too hard to go into the brains of scientists who didn't grow up coding. I'm not sure if that's true. Doesn't Pandas handle ETL and some anaysis? There is nothing inherent to ETL that makes it a hard problem with compiled languages. In your opinion, what does Pandas do that's hard to do with compile-time languages?
    - bayesian_horse 5y ago
      I didn't say "with compile-time languages" but "with compile-time type systems". And many similar tools in a statically typed language will necessarily create a way to have one static type that doesn't care what the data inside actually looks like. This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor. Worse, most often you don't know (or want to know) some of the dimensions or even dimensionality of some of the objects. Then it is impossible to check all of this at compile time.
      - arinlen 5y ago
        > This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor. That doesn't sound like a Python problem. Instead, it sounds like the natural consequence of numpy being designed in a way where their data types aren't organized into subtypes, and leave that as runtime properties. This is a natural reflection of numpy's take on vectors, matrices, and tensors, which in terms of types are just big arrays with runtime properties. To put things in perspective, in C++, Eigen supports static dense vectors and matrices whose size is specified and known at compile-time. I'm sure Python doesn't impose addition static type constraints than C++.
        bayesian_horse 5y ago
        Of course it's not a Python problem, all similar tools have the same "problem" that they can't easily fit that stuff into their type systems, so they invent some way to not care about it.
    - HelloNurse 5y ago
      It isn't a matter of "compile time": explicit type declarations and definitions can often be formally sound but practically worthless. Significant types in ETL-style applications typically come from outside (e.g. a certain CSV column in the input file contains a date in YYYYMMDD format, or maybe YYYYDDMM, figure it out, and don't forget time zones or your accounting will go wrong). Then types are mostly complex but obvious and easily deducted (e.g. multiplying matrices of compatible shapes necessarily gives a matrix of a certain shape, why should the program say anything more detailed or lower-level than "do a matrix multiplication" or "do a tensor product"?); they are an often dynamic and unpredictable property of the data, not a useful abstraction.
  - krageon 5y ago
    > Pandas does is notoriously hard to fit into a compile-time type system What, mutate data with known structure? What exactly do you imagine is hard about this, let alone notoriously so?
    - qsort 5y ago
      Not hard per se, but extremely unergonomic. A pandas dataframe types to something like Iterable[SomeKindOfProductType[int, str, str, ... (78 other columns)]]. The formal type of a dataframe in the middle of a transformation is... not very useful to know.
      - smallerfish 5y ago
        You wouldn't typically type tabular data in any language. Give it (at most) a DataFrame<StoreTransaction> type if you must, where StoreTransaction describes the structure of a row - maybe declaring only columns that model generation code was doing typed operations on (e.g. numerics vs strings) to avoid the need for reflection.
        bayesian_horse 5y ago
        Either you type the tabular data at compile time or you don't get type checking of tabular data at compile time. The number and types of the columns aren't necessarily known at compile time. Which leads to runtime errors. Even in a statically typed language, such dataframes are a kind of "dynamic typing escape hatch". As complexity of a software increases, such mechanismus of dynamic typing and throwing runtime errors creep up all over the place.
        smallerfish 5y ago
        Sure. If you're dealing with untyped data at runtime you can't type it at compile time. Not a new issue, and handled all the time in otherwise typed languages.
      - int_19h 5y ago
  - NumberCruncher 5y ago
    > That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain. This resonates with my experience. I had the opportunity to work on a DS codebase written entirely in Scala with all the typing, parallelism, actor model, whatnot. Basically I joined the company because of this technical factor. It was fun until I figured out that DS was "typed IF-THEN-ELSE written by Java devs in Scala returning stuff the users complain about with high reliability within milliseconds". Now I am happy to be back to the single threaded untyped Python world. Still no bugs in production, because we validate all requests to death, have unit tests and integration tests running on real data not on mokups. Basically we follow the principle: if the integration test passes, our typing is just right, or at least good enough. Funnily all the typing errors we catch are caused by wrongly typed data, coming from the productive system written in a typed programming language... what a strange world.
    - bostik 5y ago
      > integration tests running on real data not on mokups. I can see you are enjoying the life outside of a highly regulated industry. Having certain kinds of production data in tests (or feeding that to test environment) would be a major audit finding in any finance or healthcare company. Makes for both a blessing and a curse.
      - bayesian_horse 5y ago
        You can anonymize such data or get the necessary agreements for a small subset. All of which is tricky, but not impossible.
        dragonwriter 5y ago
        > You can anonymize such data ...thereby destroying the production-like features for which you want it for testing, which you then need to recreate and reintroduce, so you might as well just synthesize test data in the first place, since that's what you end up doing anyway, in effect.
  - jghn 5y ago
    > make things possible, often just to run it once This is the largest difference. When there's no expectation of code lasting beyond a very short lifespan, why go through the effort to future proof things, improve maintainability, have better ergonomics, etc?
    - bogwog 5y ago
      Because science doesn't work unless experimental results are reproducible.
      - jghn 5y ago
        Sure. And once you land on something worth keeping, you clean it up then. But between time point 0 and then, a whole lot of code gets written to be run very few times.
  - int_19h 5y ago
    > What Pandas does is notoriously hard to fit into a compile-time type system. Sounds a lot like F# type providers to me. https://thesharperdev.com/introduction-to-fsharp-type-providers/ https://thesharperdev.com/introduction-to-fsharp-type-provid...
    - bayesian_horse 5y ago
      As I said, it is hard. And good luck teaching beginning scientists F#.
- mountainriver 5y ago
  Totally agree, I would love for data scientists to use types and a lot of other good practices. I work primarily in mlops and have been trying to do this for some time, it just mostly feels like a losing game. The output of most analytics is a report. For longer lived processes it seems easier to just hand off
bayesian_horse 5y ago
Have you tried it? Python has gained most of its popularity before type hints, and I would (wildly) guess 99% of Python programmers don't even use type checking at all. Static typing is completely viable and I hear those arguments mostly from those who didn't use Python for a long time. The added productivity makes up for a little more debugging while the program is running. I'd also posit that many Python codebases are somewhat less "untyped" than one might assume. Django for example does a lot in its ORM and Form classes to make sure the right stuff goes in the right slot.
- smohare 5y ago
  If you mean “Python programmers” to be random sysadmins that write hack code residing on a random EC2 instance then maybe you’re right. But most I get the sense you haven’t looked at many Python libs lately. The added productivity of non-typed Python is such a ridiculous myth to anyone who has to maintain significant Python code bases. Sure, it makes you more productive for one-off exercises but the moment you’re having someone else inspect your code the productivity gains vanish. This largely true of dynamic languages more generally except in niche caches (e.g., ORMs) where types are strongly implicit. I say all this as a mathematician that is used to reading papers full of symbols packed with implicit information. The code I’ve read from gung-ho dynamic typing advocates is some of the most painful drivel I’ve ever had to wade through.
  - bayesian_horse 5y ago
    Python libraries aren't representative of Python code. The majority of Python programmers (not "just sysadmins" wherever that slur comes from) never publish or contribute to any libraries. I can compare Python's productivity to other languages, and it beats all of them so far. Another lesson I learned from working with statically typed languages: The quality of the code is more dependent on who writes it than on the language or even the ecosystem. But people are productive much sooner in Python than they are in C# or F#, and then we can work on the foot-gunning. Python's productivity isn't a myth. This is proven by its popularity and by successful projects across academia and industry alike.
antipaul 5y ago
As a data scientist, you make a strong point. For me, python is mostly a calculator, and it seems faster to try a calculation and immediately see if it runs and/or looks correct, rather than try to implement type hints et al (though I still use type hints occasionally).
deleted 5y ago
[deleted]
eweise 5y ago
Totally agree. I'm on a project using python for the first time and really can't believe how horrible it is to not have type checking, especially given that it takes time to run the code as a glue job.
wheelerof4te 5y ago
Yeah, especially types like: List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]] Oh, so I am supposed to make my own data structures now, just for typing? How about no?
- animal_spirits 5y ago
  If you find that you are returning silly types like that, then it is an indicator you are writing your function badly and should refactor it to be simpler
  - wheelerof4te 5y ago
    That's not a silly type, it is just a silly type annotation. It's easy to unpack the tuple inside that list. What is not easy is type-hinting it. What if I want it to be both an list and an iterator? Well, I would have to use Union. I'll leave it to you to write that Union "type".
    - mountainriver 5y ago
      yeah it speaks to your ability to create code people can understand, that doesn't change with types, they just illuminate it
      - wheelerof4te 5y ago
        I, too, can write code that everyone understands! Behold it's beauty: Union[List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]], Iterator[Tuple[Tuple[str, int, str], Tuple[int, float, float]]]] /s
    - saila 5y ago
      As of Python 3.10, you can use `T | U` instead of Union. Starting from 3.8 (I think), you can also directly use dict[T, U] and list[T] rather than having to import Dict and List.
      - 5y ago

waitlist 5y ago

I think data science is a perfect example in favor of types -- the code is often terrible because of the lack of typing. Pandas has notoriously poor developer ergonomics, and I recall painfully poring over type errors across the board -- lists, dataframes, numpy arrays, etc. are all iterables, so they can be interchanged in some contexts, but not in others. Had I had MyPy back when I was working in data science, I would've saved countless hours and headaches.

bayesian_horse 5y ago
What Pandas does is notoriously hard to fit into a compile-time type system. Certainly too hard to go into the brains of scientists who didn't grow up coding. No, the code in data science isn't bad because of the lack of typing. The code is "bad" mostly because those writing it are relatively fresh from starting to program. Also there is more pressure to make things possible, often just to run it once, and neglect repeatability or scaling to larger code bases. Different emphasis. That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain.
- arinlen 5y ago
  > What Pandas does is notoriously hard to fit into a compile-time type system. Certainly too hard to go into the brains of scientists who didn't grow up coding. I'm not sure if that's true. Doesn't Pandas handle ETL and some anaysis? There is nothing inherent to ETL that makes it a hard problem with compiled languages. In your opinion, what does Pandas do that's hard to do with compile-time languages?
  - bayesian_horse 5y ago
    I didn't say "with compile-time languages" but "with compile-time type systems". And many similar tools in a statically typed language will necessarily create a way to have one static type that doesn't care what the data inside actually looks like. This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor. Worse, most often you don't know (or want to know) some of the dimensions or even dimensionality of some of the objects. Then it is impossible to check all of this at compile time.
    - arinlen 5y ago
      > This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor. That doesn't sound like a Python problem. Instead, it sounds like the natural consequence of numpy being designed in a way where their data types aren't organized into subtypes, and leave that as runtime properties. This is a natural reflection of numpy's take on vectors, matrices, and tensors, which in terms of types are just big arrays with runtime properties. To put things in perspective, in C++, Eigen supports static dense vectors and matrices whose size is specified and known at compile-time. I'm sure Python doesn't impose addition static type constraints than C++.
      - bayesian_horse 5y ago
        Of course it's not a Python problem, all similar tools have the same "problem" that they can't easily fit that stuff into their type systems, so they invent some way to not care about it.
  - HelloNurse 5y ago
    It isn't a matter of "compile time": explicit type declarations and definitions can often be formally sound but practically worthless. Significant types in ETL-style applications typically come from outside (e.g. a certain CSV column in the input file contains a date in YYYYMMDD format, or maybe YYYYDDMM, figure it out, and don't forget time zones or your accounting will go wrong). Then types are mostly complex but obvious and easily deducted (e.g. multiplying matrices of compatible shapes necessarily gives a matrix of a certain shape, why should the program say anything more detailed or lower-level than "do a matrix multiplication" or "do a tensor product"?); they are an often dynamic and unpredictable property of the data, not a useful abstraction.
- krageon 5y ago
  > Pandas does is notoriously hard to fit into a compile-time type system What, mutate data with known structure? What exactly do you imagine is hard about this, let alone notoriously so?
  - qsort 5y ago
    Not hard per se, but extremely unergonomic. A pandas dataframe types to something like Iterable[SomeKindOfProductType[int, str, str, ... (78 other columns)]]. The formal type of a dataframe in the middle of a transformation is... not very useful to know.
    - smallerfish 5y ago
      You wouldn't typically type tabular data in any language. Give it (at most) a DataFrame<StoreTransaction> type if you must, where StoreTransaction describes the structure of a row - maybe declaring only columns that model generation code was doing typed operations on (e.g. numerics vs strings) to avoid the need for reflection.
      - bayesian_horse 5y ago
        Either you type the tabular data at compile time or you don't get type checking of tabular data at compile time. The number and types of the columns aren't necessarily known at compile time. Which leads to runtime errors. Even in a statically typed language, such dataframes are a kind of "dynamic typing escape hatch". As complexity of a software increases, such mechanismus of dynamic typing and throwing runtime errors creep up all over the place.
        smallerfish 5y ago
        Sure. If you're dealing with untyped data at runtime you can't type it at compile time. Not a new issue, and handled all the time in otherwise typed languages.
    - int_19h 5y ago
- NumberCruncher 5y ago
  > That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain. This resonates with my experience. I had the opportunity to work on a DS codebase written entirely in Scala with all the typing, parallelism, actor model, whatnot. Basically I joined the company because of this technical factor. It was fun until I figured out that DS was "typed IF-THEN-ELSE written by Java devs in Scala returning stuff the users complain about with high reliability within milliseconds". Now I am happy to be back to the single threaded untyped Python world. Still no bugs in production, because we validate all requests to death, have unit tests and integration tests running on real data not on mokups. Basically we follow the principle: if the integration test passes, our typing is just right, or at least good enough. Funnily all the typing errors we catch are caused by wrongly typed data, coming from the productive system written in a typed programming language... what a strange world.
  - bostik 5y ago
    > integration tests running on real data not on mokups. I can see you are enjoying the life outside of a highly regulated industry. Having certain kinds of production data in tests (or feeding that to test environment) would be a major audit finding in any finance or healthcare company. Makes for both a blessing and a curse.
    - bayesian_horse 5y ago
      You can anonymize such data or get the necessary agreements for a small subset. All of which is tricky, but not impossible.
      - dragonwriter 5y ago
        > You can anonymize such data ...thereby destroying the production-like features for which you want it for testing, which you then need to recreate and reintroduce, so you might as well just synthesize test data in the first place, since that's what you end up doing anyway, in effect.
- jghn 5y ago
  > make things possible, often just to run it once This is the largest difference. When there's no expectation of code lasting beyond a very short lifespan, why go through the effort to future proof things, improve maintainability, have better ergonomics, etc?
  - bogwog 5y ago
    Because science doesn't work unless experimental results are reproducible.
    - jghn 5y ago
      Sure. And once you land on something worth keeping, you clean it up then. But between time point 0 and then, a whole lot of code gets written to be run very few times.
- int_19h 5y ago
  > What Pandas does is notoriously hard to fit into a compile-time type system. Sounds a lot like F# type providers to me. https://thesharperdev.com/introduction-to-fsharp-type-providers/ https://thesharperdev.com/introduction-to-fsharp-type-provid...
  - bayesian_horse 5y ago
    As I said, it is hard. And good luck teaching beginning scientists F#.
mountainriver 5y ago
Totally agree, I would love for data scientists to use types and a lot of other good practices. I work primarily in mlops and have been trying to do this for some time, it just mostly feels like a losing game. The output of most analytics is a report. For longer lived processes it seems easier to just hand off

bayesian_horse 5y ago

Have you tried it? Python has gained most of its popularity before type hints, and I would (wildly) guess 99% of Python programmers don't even use type checking at all. Static typing is completely viable and I hear those arguments mostly from those who didn't use Python for a long time. The added productivity makes up for a little more debugging while the program is running. I'd also posit that many Python codebases are somewhat less "untyped" than one might assume. Django for example does a lot in its ORM and Form classes to make sure the right stuff goes in the right slot.

smohare 5y ago
If you mean “Python programmers” to be random sysadmins that write hack code residing on a random EC2 instance then maybe you’re right. But most I get the sense you haven’t looked at many Python libs lately. The added productivity of non-typed Python is such a ridiculous myth to anyone who has to maintain significant Python code bases. Sure, it makes you more productive for one-off exercises but the moment you’re having someone else inspect your code the productivity gains vanish. This largely true of dynamic languages more generally except in niche caches (e.g., ORMs) where types are strongly implicit. I say all this as a mathematician that is used to reading papers full of symbols packed with implicit information. The code I’ve read from gung-ho dynamic typing advocates is some of the most painful drivel I’ve ever had to wade through.
- bayesian_horse 5y ago
  Python libraries aren't representative of Python code. The majority of Python programmers (not "just sysadmins" wherever that slur comes from) never publish or contribute to any libraries. I can compare Python's productivity to other languages, and it beats all of them so far. Another lesson I learned from working with statically typed languages: The quality of the code is more dependent on who writes it than on the language or even the ecosystem. But people are productive much sooner in Python than they are in C# or F#, and then we can work on the foot-gunning. Python's productivity isn't a myth. This is proven by its popularity and by successful projects across academia and industry alike.

antipaul 5y ago

As a data scientist, you make a strong point. For me, python is mostly a calculator, and it seems faster to try a calculation and immediately see if it runs and/or looks correct, rather than try to implement type hints et al (though I still use type hints occasionally).

deleted 5y ago

[deleted]

eweise 5y ago

Totally agree. I'm on a project using python for the first time and really can't believe how horrible it is to not have type checking, especially given that it takes time to run the code as a glue job.

wheelerof4te 5y ago

Yeah, especially types like: List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]] Oh, so I am supposed to make my own data structures now, just for typing? How about no?

animal_spirits 5y ago
If you find that you are returning silly types like that, then it is an indicator you are writing your function badly and should refactor it to be simpler
- wheelerof4te 5y ago
  That's not a silly type, it is just a silly type annotation. It's easy to unpack the tuple inside that list. What is not easy is type-hinting it. What if I want it to be both an list and an iterator? Well, I would have to use Union. I'll leave it to you to write that Union "type".
  - mountainriver 5y ago
    yeah it speaks to your ability to create code people can understand, that doesn't change with types, they just illuminate it
    - wheelerof4te 5y ago
      I, too, can write code that everyone understands! Behold it's beauty: Union[List[Tuple[Tuple[str, int, str], Tuple[int, float, float]]], Iterator[Tuple[Tuple[str, int, str], Tuple[int, float, float]]]] /s
  - saila 5y ago
    As of Python 3.10, you can use `T | U` instead of Union. Starting from 3.8 (I think), you can also directly use dict[T, U] and list[T] rather than having to import Dict and List.
    - 5y ago