3 ms·

Am I the only one to stick with the std lib, read the docs and changelogs carefully, and implement functions I really need the way my application makes use of t

Am I the only one to stick with the std lib, read the docs and changelogs carefully, and implement functions I really need the way my application makes use of them? I learned the hard way, that dependencies kill projects. Not saying this isn't great, thanks for creating it! It does have its use cases, of course.

xandrius 1y ago
Creating from scratch also creates hidden debt, it's just moved onto yourself. Especially when working with dates and timezones.
- raverbashing 1y ago
  A library that goes "poof" when you need to upgrade it is also a hidden debt
  - scott_w 1y ago
    A member of staff who goes “poof” as the only one who understands your wrapper library that has a critical bug is a more common kind of hidden debt.
  - 0xCE0 1y ago
    There is also a distinction to be made between "technical" and "political" dependencies. Technical dependencies usually track some spec or common consensus, and political dependencies are e.g. timedate-library. Political dependencies are almost like business logic, because they have to track changing political decision made all over the world (or be local to some country). Timedates are hard, and units may require even harder historical/present "political" tracking as how they are defined, and I would never want to maintain this kind of dependency: https://github.com/ryantenney/gnu-units/blob/master/units.dat https://github.com/ryantenney/gnu-units/blob/master/units.da... And what comes to timedate problems, I try to keep it simple if the project allows: store and operate with UTC timestamps everywhere and only temporarily convert to to local time (DSTs applied, if such) when displaying it in user-facing UI. This functionality/understanding can be locked into own 20-line microlibrary-dependency, which forces its responsible person to understand country's timezone and when e.g. DST changes and where/how/who decides DST changes and what APIs is used to get UTC time (and of course, its dependencies, e.g. NTP, and its dependencies. e.g. unperturbed ground-state hyperfine transition frequency of the caesium-133 atom, which result is then combined with the Pope Gregory XIII's Gregorian calendar, which is a type of solar calendar mixed with religious event for fixing the time, which which is then finally corrected by rewinding/forwarding "the clock" because its too bright or dark for the politicians).
  - xandrius 1y ago
    A service goes poof, a library either slowly deteriorates or breaks just as easily as a self-written one (if an underlying platform breaks it for some reason). The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing. I would still rather using a library for dates, a million times so.
    - raverbashing 1y ago
      > a library either slowly deteriorates or breaks just as easily as a self-written one Yes, I agree with this > The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing. Libraries that have 100 people collaborating on it are very few Most likely you'll have to vendor it and fix whatever issues you have. Even worse when it's a dependency of a dependency and you also use it, so, let's say a dependency of yours bumps the version but this breaks your code. (Not sure this breaks only in python or in js as well, but it's possibly that it does)
      - no_wizard 1y ago
        Honest question: why is it so common for software developers to not upgrade their dependencies on regular intervals? I can’t for the life of me figure out why. If you update everything incrementally you bypass the upgrade version problem when you’re so far behind that so much has changed that it becomes an overwhelming burden. I think frozen dependencies are a big anti pattern, and places where I work that regularly update their deps tended to have better software practices generally
        crote 1y ago
        Because time spent updating dependencies won't contribute to adding new features. Besides, any update risks breaking stuff. Not freezing dependencies isn't an option, because that means any commit can cause breakage in a completely unrelated part of the codebase, in a way which can be extremely confusing to debug. And you don't really want to install the very newest versions either, better wait a week or two for someone else to run into bugs and release a .1 version. The sweet spot is somewhere in the middle: update often enough to avoid updates becoming a massive burden, stick with fixed versions between updates. I reckon it's best to just schedule some dedicated time for it every month or two.
- brookst 1y ago
  I cannot imagine having the spare time to invest in building date/time foundations and maintaining them through changes to DST timing and country/time zone changes. The only crazier idea I can think of is implementing character encoding conversions myself.
  - crazygringo 1y ago
    Seriously. Nobody needs a package for "left-pad", which is the most infamous example. But there are a lot of areas where it's not a good use of your time to reinvent the wheel. There's no moral virtue in writing everything yourself. And popular packages have gone through a more bug-finding and bug-fixing than your personal code probably ever will.
pkkm 1y ago
I'm not the creator, the credit for that goes to Arie Bovenberg. I just wanted to show this to people.
EdwardDiego 1y ago
There are so many footguns in the datetime lib. That's why I use a Flake8 plugin to prohibit especially egregious footguns. https://github.com/jkittner/flake8-ban-utcnow https://github.com/jkittner/flake8-ban-utcnow
- raverbashing 1y ago
  Honestly yeah who in tarnation created that function and called it utcnow These things are really frustrating
  - mark-r 1y ago
    Yeah, utcnow is completely broken. They should have fixed it when they created datetime.timezone.utc, but they didn't. The recommendation is to use datetime.datetime.now(datetime.timezone.utc) instead. utcnow should be deprecated and eventually removed.
    - jessekv 1y ago
      I recently learned that there is datetime.UTC which is an alias of datetime.timezone.utc It helps a bit with the verbosity: from datetime import datetime, UTC datetime.now(UTC)
      - maleldil 1y ago
        That's from 3.11 onwards.
        deleted 1y ago
        [deleted]
stavros 1y ago
> Am I the only one to stick with the std lib, read the docs and changelogs carefully I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time. I don't want footguns in my code, I don't want code I have to write and test myself, and I don't want to have to become an expert in a domain before I can write something that serves my purpose. For the datetime library, specifically, I'm switching to whenever for everything, because I've been bitten by conversions and naive/aware datetime confusion too many times.
- globular-toast 1y ago
  > I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time. This kinda sums up the sorry state of software engineering. People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI. Taking on a dependency does not relieve you of reading docs, but it also adds a further burden as you now need to trust the code. The stdlib is much more heavily tested and documented than any 3rd party library will be.
  - mr_mitm 1y ago
    > People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI. That's a straw man argument. No one said "blindly". You can very well carefully consider the pros and cons of adding a dependency and arrive at the conclusion that it makes sense. Many PyPI packages are in the Debian stable repositories, you could use that as an additional barrier as well.
    - freehorse 1y ago
      To be fair, comment^^ actually said choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency" which is what comment^ answers to, which to me actually sounds like that the added dependency comes in place of "reading docs/changelogs carefully". I think it matters a lot how much one can trust a 3rd party library, how much it is used, how much it is maintained etc. Moreover, it also matters how central and important this is to what you are actually doing, for example if the datetimes I come across are pretty much specific and with limited scope, I would probably care about reading docs less than if I am reading data with datetimes that may be totally wild. Furthermore, there are some libraries that are just considered the standard in some fields. If I use python I call pandas to read csvs, I am obviously not gonna write my own csv parser. It will also make your code more readable for others that already know and use the same libraries if they are so standard. But that's probably a very small minority of libraries that people actually use in certain languages.
sgarland 1y ago
You are a sad minority, IME. I’m right there with you. I extended the uuid library to generate UUIDv7, based off of the RFC. It’s pretty easy to implement, as it turns out. Overruled, because “we don’t want to have to maintain additional code.” As if the ABI for bitshifts is going to change?!
- ljm 1y ago
  There’s an out of sight, out of mind mentality with dependencies. As long as there is a conscious decision to build or ‘buy’, it’s fine. I think some people can be a little too careless with adding dependencies though, not realising they can have an equal if not greater maintenance burden.
mvanbaak 1y ago
As others stated, there are many rough edges and footguns in the stdlib. BUT ... in my (and yours apparently) opinion, it's a matter of knowing those edges/guns, and work with them. Like you, I also prefer to create my own code around those instead of bringing in some library that brings in their own foot guns and possibly sub-dependencies and and and...
- matsemann 1y ago
  So your projects end up with their own "lib" of scattered time functions, possibly with new small bugs. I'd then rather have a proper well-tested and maintained library.
  - mvanbaak 1y ago
    The "lib" one needs for date time functions is as good as non-existent. But yeah, you will have to create tests for the codepath, instead of relying on the tests the library maintainers create.
    - scott_w 1y ago
      Given the list of issues (most of which I had never heard of) that the library author describes with datetime, I think I trust his tests over the ones I’d write. Given how many tests I see fail on 29th February at the companies I’ve been at, I don’t trust my colleagues’ tests either!
      - brookst 1y ago
        In a previous job where we had complete hell with an internal datetime lib under ambiguous ownership, a colleague joked that the original authors had taken early retirement when they turned 2,147,483,647 years old.
mr_mitm 1y ago
Are you saying you never pull in dependencies? Why stop there, why not re-implement the std lib as well? Surely there is a sensible middle ground: If you only need a small part of a dependency, consider implementing it. If you make heavy use of a dependency and want to benefit of years if not decades of dedicated developers testing and maturing its code, with a large community who has already stepped in all pitfalls you might step into and collectively encountered all the edge cases, just use the dependency.
dmos62 1y ago
Curious about examples of projects being killed by dependencies.
- michaelt 1y ago
  While I've never seen a project killed by dependencies, I've certainly seen projects stuck on treadmill of constant dependency updates. You know, they import 5 libraries, each of which imports 5 more libraries, each of which imports 5 more libraries, and suddenly they're buried in 'critical' updates because there's a denial-of-service bug in the date parser used by the yaml parser used by the configuration library used by the logging library used by the application.
- ljm 1y ago
  Not killed IME but bloated and dragged down by tech debt. E.g the JS project that uses the stdlib Date API, and pulls in moment.js, and also uses date-fns. Or the one that pulls in bits and pieces of lodash, ramda, and other functional libraries. And maybe it uses native fetch and axios depending on the current phase of the moon. They don’t die but time is wasted in code review trying to understand if there is any kind of deliberate approach behind the scattershot application of packages with duplicated purposes. (picking on JS is perhaps unfair but it’s probably the most egregious example of dependency hell)
  - MrJohz 1y ago
    The Date example is possibly an even better example of why dependencies that get hard problems right are so important. If Python's datetime library is bad, Date is truly terrible. Every time I have used it I have regretted it long term (including when using it in combination with date-fns in the hope that that makes it usable). And in the end, trying to keep things simple with Date has caused more technical debt than using a sensible dependency. Some problems simply require using the right tools. They aren't necessarily hard, but they will be if you try to hammer a nail in with a screwdriver. The Date API, and to a certain extent Python's datetime module, are screwdrivers for a nail-shaped problem. The rest of your example seem to have more to do with bad dependency practices than using dependencies in the first place. If you are going to include a dependency, think about it, consider whether it's worth it, document that decision, and then consistently use that dependency. Just because you've seen projects use dependencies poorly doesn't mean dependents are bad by themselves.
foolfoolz 1y ago
this is a great idea if you want to slow down your project. most projects start with few rules and “best practices” like this. everyone is free to pull in dependencies as needed. because they are needed. but then once the project grows larger, those who have been around longer want to reverse course and gatekeep dependencies. but this is the opposite of what helped the project grow initially. and later contributors have a harder time making similar progress because they have to fight to add basic libraries. ensuring that efficiency per engineer goes down
johnfn 1y ago
I think this is fairly unrealistic. Does all your datetime manipulation involve proper use of the fold parameter as indicated in the article?
BiteCode_dev 1y ago
Functions that you have to document, test and maintain of course. You do that, right? And all the people in your team, they do that and will keep doing that once you leave, right? And they all understand the business domain and all the pitfalls that come with it and have the skill, time, and resources to take care of it, right? And this for every single problem: time, text, maths, network, parsing, formatting, validating, authenticating...

xandrius 1y ago

Creating from scratch also creates hidden debt, it's just moved onto yourself. Especially when working with dates and timezones.

raverbashing 1y ago
A library that goes "poof" when you need to upgrade it is also a hidden debt
- scott_w 1y ago
  A member of staff who goes “poof” as the only one who understands your wrapper library that has a critical bug is a more common kind of hidden debt.
- 0xCE0 1y ago
  There is also a distinction to be made between "technical" and "political" dependencies. Technical dependencies usually track some spec or common consensus, and political dependencies are e.g. timedate-library. Political dependencies are almost like business logic, because they have to track changing political decision made all over the world (or be local to some country). Timedates are hard, and units may require even harder historical/present "political" tracking as how they are defined, and I would never want to maintain this kind of dependency: https://github.com/ryantenney/gnu-units/blob/master/units.dat https://github.com/ryantenney/gnu-units/blob/master/units.da... And what comes to timedate problems, I try to keep it simple if the project allows: store and operate with UTC timestamps everywhere and only temporarily convert to to local time (DSTs applied, if such) when displaying it in user-facing UI. This functionality/understanding can be locked into own 20-line microlibrary-dependency, which forces its responsible person to understand country's timezone and when e.g. DST changes and where/how/who decides DST changes and what APIs is used to get UTC time (and of course, its dependencies, e.g. NTP, and its dependencies. e.g. unperturbed ground-state hyperfine transition frequency of the caesium-133 atom, which result is then combined with the Pope Gregory XIII's Gregorian calendar, which is a type of solar calendar mixed with religious event for fixing the time, which which is then finally corrected by rewinding/forwarding "the clock" because its too bright or dark for the politicians).
- xandrius 1y ago
  A service goes poof, a library either slowly deteriorates or breaks just as easily as a self-written one (if an underlying platform breaks it for some reason). The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing. I would still rather using a library for dates, a million times so.
  - raverbashing 1y ago
    > a library either slowly deteriorates or breaks just as easily as a self-written one Yes, I agree with this > The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing. Libraries that have 100 people collaborating on it are very few Most likely you'll have to vendor it and fix whatever issues you have. Even worse when it's a dependency of a dependency and you also use it, so, let's say a dependency of yours bumps the version but this breaks your code. (Not sure this breaks only in python or in js as well, but it's possibly that it does)
    - no_wizard 1y ago
      Honest question: why is it so common for software developers to not upgrade their dependencies on regular intervals? I can’t for the life of me figure out why. If you update everything incrementally you bypass the upgrade version problem when you’re so far behind that so much has changed that it becomes an overwhelming burden. I think frozen dependencies are a big anti pattern, and places where I work that regularly update their deps tended to have better software practices generally
      - crote 1y ago
        Because time spent updating dependencies won't contribute to adding new features. Besides, any update risks breaking stuff. Not freezing dependencies isn't an option, because that means any commit can cause breakage in a completely unrelated part of the codebase, in a way which can be extremely confusing to debug. And you don't really want to install the very newest versions either, better wait a week or two for someone else to run into bugs and release a .1 version. The sweet spot is somewhere in the middle: update often enough to avoid updates becoming a massive burden, stick with fixed versions between updates. I reckon it's best to just schedule some dedicated time for it every month or two.
brookst 1y ago
I cannot imagine having the spare time to invest in building date/time foundations and maintaining them through changes to DST timing and country/time zone changes. The only crazier idea I can think of is implementing character encoding conversions myself.
- crazygringo 1y ago
  Seriously. Nobody needs a package for "left-pad", which is the most infamous example. But there are a lot of areas where it's not a good use of your time to reinvent the wheel. There's no moral virtue in writing everything yourself. And popular packages have gone through a more bug-finding and bug-fixing than your personal code probably ever will.

pkkm 1y ago

I'm not the creator, the credit for that goes to Arie Bovenberg. I just wanted to show this to people.

EdwardDiego 1y ago

There are so many footguns in the datetime lib. That's why I use a Flake8 plugin to prohibit especially egregious footguns. https://github.com/jkittner/flake8-ban-utcnow https://github.com/jkittner/flake8-ban-utcnow

raverbashing 1y ago
Honestly yeah who in tarnation created that function and called it utcnow These things are really frustrating
- mark-r 1y ago
  Yeah, utcnow is completely broken. They should have fixed it when they created datetime.timezone.utc, but they didn't. The recommendation is to use datetime.datetime.now(datetime.timezone.utc) instead. utcnow should be deprecated and eventually removed.
  - jessekv 1y ago
    I recently learned that there is datetime.UTC which is an alias of datetime.timezone.utc It helps a bit with the verbosity: from datetime import datetime, UTC datetime.now(UTC)
    - maleldil 1y ago
      That's from 3.11 onwards.
      - deleted 1y ago
        [deleted]

stavros 1y ago

> Am I the only one to stick with the std lib, read the docs and changelogs carefully I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time. I don't want footguns in my code, I don't want code I have to write and test myself, and I don't want to have to become an expert in a domain before I can write something that serves my purpose. For the datetime library, specifically, I'm switching to whenever for everything, because I've been bitten by conversions and naive/aware datetime confusion too many times.

globular-toast 1y ago
> I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time. This kinda sums up the sorry state of software engineering. People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI. Taking on a dependency does not relieve you of reading docs, but it also adds a further burden as you now need to trust the code. The stdlib is much more heavily tested and documented than any 3rd party library will be.
- mr_mitm 1y ago
  > People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI. That's a straw man argument. No one said "blindly". You can very well carefully consider the pros and cons of adding a dependency and arrive at the conclusion that it makes sense. Many PyPI packages are in the Debian stable repositories, you could use that as an additional barrier as well.
  - freehorse 1y ago
    To be fair, comment^^ actually said choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency" which is what comment^ answers to, which to me actually sounds like that the added dependency comes in place of "reading docs/changelogs carefully". I think it matters a lot how much one can trust a 3rd party library, how much it is used, how much it is maintained etc. Moreover, it also matters how central and important this is to what you are actually doing, for example if the datetimes I come across are pretty much specific and with limited scope, I would probably care about reading docs less than if I am reading data with datetimes that may be totally wild. Furthermore, there are some libraries that are just considered the standard in some fields. If I use python I call pandas to read csvs, I am obviously not gonna write my own csv parser. It will also make your code more readable for others that already know and use the same libraries if they are so standard. But that's probably a very small minority of libraries that people actually use in certain languages.

sgarland 1y ago

You are a sad minority, IME. I’m right there with you. I extended the uuid library to generate UUIDv7, based off of the RFC. It’s pretty easy to implement, as it turns out. Overruled, because “we don’t want to have to maintain additional code.” As if the ABI for bitshifts is going to change?!

ljm 1y ago
There’s an out of sight, out of mind mentality with dependencies. As long as there is a conscious decision to build or ‘buy’, it’s fine. I think some people can be a little too careless with adding dependencies though, not realising they can have an equal if not greater maintenance burden.

mvanbaak 1y ago

As others stated, there are many rough edges and footguns in the stdlib. BUT ... in my (and yours apparently) opinion, it's a matter of knowing those edges/guns, and work with them. Like you, I also prefer to create my own code around those instead of bringing in some library that brings in their own foot guns and possibly sub-dependencies and and and...

matsemann 1y ago
So your projects end up with their own "lib" of scattered time functions, possibly with new small bugs. I'd then rather have a proper well-tested and maintained library.
- mvanbaak 1y ago
  The "lib" one needs for date time functions is as good as non-existent. But yeah, you will have to create tests for the codepath, instead of relying on the tests the library maintainers create.
  - scott_w 1y ago
    Given the list of issues (most of which I had never heard of) that the library author describes with datetime, I think I trust his tests over the ones I’d write. Given how many tests I see fail on 29th February at the companies I’ve been at, I don’t trust my colleagues’ tests either!
    - brookst 1y ago
      In a previous job where we had complete hell with an internal datetime lib under ambiguous ownership, a colleague joked that the original authors had taken early retirement when they turned 2,147,483,647 years old.

mr_mitm 1y ago

Are you saying you never pull in dependencies? Why stop there, why not re-implement the std lib as well? Surely there is a sensible middle ground: If you only need a small part of a dependency, consider implementing it. If you make heavy use of a dependency and want to benefit of years if not decades of dedicated developers testing and maturing its code, with a large community who has already stepped in all pitfalls you might step into and collectively encountered all the edge cases, just use the dependency.

dmos62 1y ago

Curious about examples of projects being killed by dependencies.

michaelt 1y ago
While I've never seen a project killed by dependencies, I've certainly seen projects stuck on treadmill of constant dependency updates. You know, they import 5 libraries, each of which imports 5 more libraries, each of which imports 5 more libraries, and suddenly they're buried in 'critical' updates because there's a denial-of-service bug in the date parser used by the yaml parser used by the configuration library used by the logging library used by the application.
ljm 1y ago
Not killed IME but bloated and dragged down by tech debt. E.g the JS project that uses the stdlib Date API, and pulls in moment.js, and also uses date-fns. Or the one that pulls in bits and pieces of lodash, ramda, and other functional libraries. And maybe it uses native fetch and axios depending on the current phase of the moon. They don’t die but time is wasted in code review trying to understand if there is any kind of deliberate approach behind the scattershot application of packages with duplicated purposes. (picking on JS is perhaps unfair but it’s probably the most egregious example of dependency hell)
- MrJohz 1y ago
  The Date example is possibly an even better example of why dependencies that get hard problems right are so important. If Python's datetime library is bad, Date is truly terrible. Every time I have used it I have regretted it long term (including when using it in combination with date-fns in the hope that that makes it usable). And in the end, trying to keep things simple with Date has caused more technical debt than using a sensible dependency. Some problems simply require using the right tools. They aren't necessarily hard, but they will be if you try to hammer a nail in with a screwdriver. The Date API, and to a certain extent Python's datetime module, are screwdrivers for a nail-shaped problem. The rest of your example seem to have more to do with bad dependency practices than using dependencies in the first place. If you are going to include a dependency, think about it, consider whether it's worth it, document that decision, and then consistently use that dependency. Just because you've seen projects use dependencies poorly doesn't mean dependents are bad by themselves.

foolfoolz 1y ago

this is a great idea if you want to slow down your project. most projects start with few rules and “best practices” like this. everyone is free to pull in dependencies as needed. because they are needed. but then once the project grows larger, those who have been around longer want to reverse course and gatekeep dependencies. but this is the opposite of what helped the project grow initially. and later contributors have a harder time making similar progress because they have to fight to add basic libraries. ensuring that efficiency per engineer goes down

johnfn 1y ago

I think this is fairly unrealistic. Does all your datetime manipulation involve proper use of the fold parameter as indicated in the article?

BiteCode_dev 1y ago

Functions that you have to document, test and maintain of course. You do that, right? And all the people in your team, they do that and will keep doing that once you leave, right? And they all understand the business domain and all the pitfalls that come with it and have the skill, time, and resources to take care of it, right? And this for every single problem: time, text, maths, network, parsing, formatting, validating, authenticating...