3 ms·
This is the second time I've ended up reading this post, and I confess I still don't exactly understand what was going on here. There's an issue with PDFs missi
by MrJohz 3y ago
This is the second time I've ended up reading this post, and I confess I still don't exactly understand what was going on here. There's an issue with PDFs missing their contents, but also timeouts and a missing method exception? The problem seems to have come from upgrading some dependencies at some point, but the solution is also an upgrade, as far as I can tell? And then the cache issues at the end sound very much like a red flag in terms of managing JS dependencies - as if they're trying to cache the node_modules directory in some way that isn't properly configured.
That said, the main sense I get is that the team aren't so experienced with Javascript, and therefore working with it is full of surprises because they just aren't used to it. I found the line about bugs in C code being "your fault" particularly amusing: having occasionally had to touch C and C++ code, I would have said the complete opposite! Javascript, for me, is very predictable - if something goes wrong, then it's usually because I made a clear mistake. But with C, I end up trying to debug chains of Makefile and CMake problems that have no rhyme or reason to them.
In other words: as someone inexperienced in C, it is a chaotic, magical language and ecosystem to me, in exactly the same way that the author describes Javascript being to them.
More practically, it sounds like the team need some regression tests for this case that they can run next time they try and upgrade their dependencies. They also need to make sure that caching is working properly for their Javascript code - rerunning the CI multiple times on the same commit should always produce exactly the same dependency tree every time. Probably that means caching the right directory and using NPM CI to clear and rebuild node_modules from scratch every time.
More generally, the author complains about a lack of type checks. In this case, it sounds like the missing method was somewhere in a dependency, so this may not have helped, but: Typescript! It exists, it adds types, it makes a lot of things easier. Easy mode Typescript is to just start writing `.ts` files and use the `tsx` package to automatically compile and execute everything - there are other approaches that they could explore, but this is probably the simplest.
- stana 3y agoStuff that gets pulled in when you do 'npm install' is scary for a more traditional server-side developer. Python for example comes with a lot of built-in libs maintained by the core python team. These could get you pretty far before you start pulling in libs from random authors.
- MrJohz 3y agoBut that's exactly the sort of thing I'm talking about. We find scary what we're unfamiliar with. To be clear, I agree that the Javascript ecosystem isn't perfect, particularly when it comes to supply chain issues. But in a lot of ways it's more secure than Python's typical dependency setup. By default, NPM uses a lock file, which enforces that specific versions of existing packages are used unless explicitly updated. This means that you always know which versions of which dependencies you're currently using, it allows you to upgrade different parts of your dependency tree independently, it ensures that when you remove a dependency, all of its transitive dependencies also get removed, and it allows for different dependencies for production and development environments, and for different portions of a given project. Doing all - or even part - of that in Python is difficult. There's no built-in way, and so people tend to adopt third-party tools or roll their own scripts. The roll-your-own process, in my experience, almost inevitably leads to some sort of failure, usually a certain updating unexpectedly and breaking everything, and then not being able to roll back to a previous version of that dependency. The third party tools (Poetry, Pip-Tools, etc) work better, but are often difficult to integrate more widely into the Python ecosystem. I don't want to make this about Python vs Javascript - both are tools that make different tradeoffs and can be valuable in different ways. The point here is that, as software developers, we often end up with what are essentially superstitions about other programming languages - we describe things we don't know or don't understand as "magical" or "scary", because that's easier than admitting we're just not that well informed. And while it's okay not to be informed about everything, in situations like this, it can lead to bad outcomes. In the article, the author ends up blaming the situation on the magic and weirdness of Javascript, but it sounds like the issue had more to do with them not understanding how best to do package management in Javascript.
- skyguy94 3y agoThis exactly. Javascript needs a standard library.
- fendy3002 3y agoExcept for lodash, axios and dayjs (moment), I found that I don't really need much more core functionality for backend development. And if you want reproducible dependencies, use npm ci not npm install, or use yarn. Best combined with replicated self hosted npm repository.
- golergka 3y ago> I deleted all cache on my local environment, all node_modules references, all lock files. I ran an installation process once again and exported again… it didn’t work. > The next message I sent: “We found it!” May be I misunderstand something, but from the post it sounds that they don't even commit their package-lock.json. They're using node like its 2012.
- shp0ngle 3y agoYeah I still don't understand what was wrong exactly from the article and how is PDF and the two libraries related.