5 ms·

A JavaScript Nightmare

47 points by tempodox 3y ago

its-summertime 3y ago
Does C not have these issues?
- tempodox 3y ago
  Try getting “xxx is not a function” at runtime in C.
  - habitue 3y ago
    Exactly, you just get a null pointer dereference segfault.
    - wiseowise 3y ago
      Without any stacktrace.
      - habitue 3y ago
        Tbf, you might get a core dump if you have them turned on, which is... Kinda like a stack trace
        rezonant 3y ago
        Now enjoy using gdb to debug it
EarthLaunch 3y ago
This was posed 20 hours ago: https://news.ycombinator.com/item?id=39724247 https://news.ycombinator.com/item?id=39724247 Not sure why I can't downvote articles, ah well.
- rezonant 3y ago
  There is no downvote for submissions, only flag.
alessioalex 3y ago
Shouldn’t you use fixed versions and use something like ‘npm ci’ to install the exact dependency tree? Feels more like an oversight from the developers rather than a JavaScript problem. When you want to upgrade, only then do you run all regression tests and do so.
- zaSmilingIdiot 3y ago
  I agree with this. But dont just use `npm ci`on prod builds since that would typically include all the dev dependencies as well in your production builds, which is not usually desirable. It might be possible to add the `--only=production` flag to npm ci? But otherwise, as pointed out, pinned versions are needed for all dependencies.
  - MrJohz 3y ago
    That flag exists (or an equivalent, I forget what the exact syntax is).
  - rezonant 3y ago
    "npm ci" omits dev dependencies by default. However, "npm install" also adheres to your package-lock.json, and does not update things on you. npm ci is just npm install with a few tweaks that make it better to use for CI and builds, see here: https://docs.npmjs.com/cli/v8/commands/npm-ci https://docs.npmjs.com/cli/v8/commands/npm-ci
    - fendy3002 3y ago
      Last time I know during node 16, npm install updates your package-lock.json, but for patch version number (unsure about minor version) only and if the package.json is using relative version number. Definitely use npm ci or yarn if you want to adhere to lock files.
      - rezonant 3y ago
        > if the package.json is using relative version number What do you mean by this? You can specify dependency versions using "^" which means "I accept new minor and patch versions" or "~" which says "I accept new patch versions", or a bare version to say I want an exact version; but none of those have anything to do with a bare "npm install" command. They are relevant when you install a new package, with "npm install <package>". That is a totally different operation, and it absolutely can update versions when needed. However, when you just do "npm install" it does not update your package-lock.json to the latest version that matches your semver requirements in package.json. That will only happen if you install a new package that needs it or you use `npm update` or some other command. This is easy to test but it's also intuitively correct- you don't expect to run `npm install` and have an uncommitted change to `package-lock.json` appear, and that does not happen in normal usage. It is kind of unfortunate that "npm install" and "npm install <package>" use the same verb of "install" for two different things- other package managers don't make this mistake and it avoids the sort of misconception we're running into here.
hyfgfh 3y ago
[flagged]
- hyfgfh 3y ago
  To be fair, JavaScript is harder that most people think, the problem is that everyone thinks they are an expert, so we end up with this kind of problems. There is a lack of engineering discipline and too much hacking!
  - leptons 3y ago
    Javascript is way easier than most people think. The problems come when they try to force their preconceived notions on it, and/or they don't keep it simple. Sure, complex things can be done with Javascript but as you said, it takes discipline.
- wiseowise 3y ago
  >> walking down the road >> ground suddenly became air out of nowhere > Lol, skill issue!
yen223 3y ago
tldr: A dependency wasn't pinned, the dependency changed, and stuff broke. Author thinks Javascript sucks. I will say this though: it is infuriating that package managers (every single one I've used, not just NPM) don't default to pinning exact dependency versions, and instead let it "automatically" update minor versions.
- pier25 3y ago
  The package wasn't following semantic versioning. Only major versions should introduce breaking changes.
  - yen223 3y ago
    The only change that isn't "breaking" is one that only adds functionality. Otherwise, it changes behavour, and has the potential to break your software. In practice, almost all changes should be major version bumps. Anyway that's besides the point. Let me control when version changes happen!
    - treflop 3y ago
      I used to maintain some open source libraries and absolutely no change we would release would break anything and definitely not over a minor version. We even did a major rewrite that completely changed the API and we broke nothing, but it did mean your project got a lot of deprecation errors. And our docs would mention both the old and new way so if you were searching the Internet for help, you wouldn't get confused when you saw wildly completely examples. It's all about craftmanship.
    - brazzy 3y ago
      >The only change that isn't "breaking" is one that only adds functionality. Ha! Think again. On of the more annoying problems I had to analyze was when a new PHP version added a function that already existed in my application (this was before namespaces). The runtime did not like that.
    - rezonant 3y ago
wiseowise 3y ago
> not using TypeScript for any serious product Well, here is your problem.
- crummy 3y ago
  Would Typescript have solved this though? They still would have been using npm etc
  - wiseowise 3y ago
    > this.subset.encodeStream is not a function Yes?
    - PUSH_AX 3y ago
      This was a runtime issue in a third party lib, so no.
      - wiseowise 3y ago
        If the library used TypeScript, it wouldn’t have the issue. The point still stands.
        PUSH_AX 3y ago
        Debatable, and I don’t know how that helps the author. I’m getting huge “silver bullet” sentiment from your replies too.
        wiseowise 3y ago
        > > obvious problem that was solved decades ago by static typing > getting huge “silver bullet” sentiment from your replies ???
        PUSH_AX 3y ago
        I didn’t write the first quote, but a comparison to an opinionated strongly typed language where types are a first class citizen, and a typing sandbox with nonsense constructs enabled by antipatterns such as unions (‘string|number’ anyone?)… these are not the same things.
MichaelRo 3y ago
I didn't quite understand why they had to debug on the spot, thus working like 36 hours uninterrupted or so. A sane working environment would file a priority 0 bug and address it during working hours. I'm not throwing a rock but oftentimes this sort of heroic behavior is self-inflected. Also why is heroic debugging required on something that is not production? The article mentions it was a "quick demo". Then who gives a shit? There's time to investigate. And this would/should NEVER happen to production. The whole point of testing and staging/UAT environment is to catch this sort of problems. Again, a normal day at work in software engineering and not something to kill yourself living on amphetamine like a commando behind enemy lines in WW2.
- peschu 3y ago
  it‘s called „motivation“
  - atoav 3y ago
    Jup, and if your employer doesn't stop you or even expects such behavior that is called a bad work culture. Sure, I would also do long programming sessions on my own things every now and then, but those happen because I am in the flow and it feels like more work to come back into it the other day. But those are my own projects. I don't owe this to my employer. If they want that they could offer me double my current salary and even then I would insist on only doing it when it makes sense to me.
    - number6 3y ago
      How would the employer know?
      - atoav 3y ago
        In my case my superior comes by in my office or sees I made a commit and tells me I should stop working (I tend to forget to look at the time if I am in a flow). But I also live in Europe and my superiors want me to come in with a fresh mind, instead of burning myself for things that should be normal work.
rednafi 3y ago
[dead]
leptons 3y ago
> One tiny version mismatch or breaking change can bring your mission-critical application to a standstill, as we painfully experienced. This is true in many other languages, not just Javascript. This isn't a "Javascript nightmare", this is a dependency nightmare.
- jillesvangurp 3y ago
  In a statically compiled language, when APIs change, you get compile errors that show up at build time. It's usually pretty obvious. With dynamically typed errors, you get weird run-time failures that can be well hidden until you trigger a use case that actually needs the bit that changed. That combined with poor test coverage means a lot of potential for things to slip through build and QA processes. I use Kotlin, both on the JVM and in the browser (via kotlin-js). Dependencies are equally complex as in javascript but it's much easier to find out if stuff breaks or not. I update often to minimize the amount of changes I have to deal with. This may sound counter intuitive but integration testing small deltas is exponentially easier and less risky than big deltas. My attitude to having big updates lurking is that they represent technical debt. Just because you don't know you have a problem because you haven't updated yet doesn't mean you don't have a problem. By not updating, you just increase your ignorance of technical debt. It might be piling up without you realizing. And you might be making things worse by working against obsolete APIs. Upgrading libraries often means I typically only have to deal with a handful of minor updates for libraries that are mostly unproblematic. And if stuff breaks, it's easier to pin point the source of the breakage. And I find out as early as possible about problems I need to deal with. If I don't know about it, I can't deal with it either. Sometimes downgrading a library temporarily is needed. But then you document why and stay on top of issue trackers and upstream development to see when you can upgrade or you start looking for alternatives. Occasionally, some API breaking changes happen and then I fix it (because the compiler tells me stuff is not compiling). If I don't update for a year, I probably have a lot of breakage on my hands and a lot of work fixing things. Much harder and much more time consuming. And risky. All I know then is that I probably have a lot of problems (i.e. technical debt) on my hands that I don't know about.
  - rezonant 3y ago
MrJohz 3y ago
This is the second time I've ended up reading this post, and I confess I still don't exactly understand what was going on here. There's an issue with PDFs missing their contents, but also timeouts and a missing method exception? The problem seems to have come from upgrading some dependencies at some point, but the solution is also an upgrade, as far as I can tell? And then the cache issues at the end sound very much like a red flag in terms of managing JS dependencies - as if they're trying to cache the node_modules directory in some way that isn't properly configured. That said, the main sense I get is that the team aren't so experienced with Javascript, and therefore working with it is full of surprises because they just aren't used to it. I found the line about bugs in C code being "your fault" particularly amusing: having occasionally had to touch C and C++ code, I would have said the complete opposite! Javascript, for me, is very predictable - if something goes wrong, then it's usually because I made a clear mistake. But with C, I end up trying to debug chains of Makefile and CMake problems that have no rhyme or reason to them. In other words: as someone inexperienced in C, it is a chaotic, magical language and ecosystem to me, in exactly the same way that the author describes Javascript being to them. More practically, it sounds like the team need some regression tests for this case that they can run next time they try and upgrade their dependencies. They also need to make sure that caching is working properly for their Javascript code - rerunning the CI multiple times on the same commit should always produce exactly the same dependency tree every time. Probably that means caching the right directory and using NPM CI to clear and rebuild node_modules from scratch every time. More generally, the author complains about a lack of type checks. In this case, it sounds like the missing method was somewhere in a dependency, so this may not have helped, but: Typescript! It exists, it adds types, it makes a lot of things easier. Easy mode Typescript is to just start writing `.ts` files and use the `tsx` package to automatically compile and execute everything - there are other approaches that they could explore, but this is probably the simplest.
- stana 3y ago
  Stuff that gets pulled in when you do 'npm install' is scary for a more traditional server-side developer. Python for example comes with a lot of built-in libs maintained by the core python team. These could get you pretty far before you start pulling in libs from random authors.
flimsypremise 3y ago
This was very funny to read because as someone who is actually familiar with these JS builds I immediately guessed that a dependency had changed. That's literally the first thing I'd check if an app is behaving differently in a remote environment than it is locally. I'd recommend the OP learn some JS best practices and apply them rather than complaining that JS isn't C.
- phanimahesh 3y ago
  I'm honestly split on this. It is obvious that they are not experienced with js ecosystem, but is it fair to frame it as they obviously haven't been burned enough by the ecosystem to internalise the common issues? Dependency silent upgrades and dependency hell and bloat are not unique to javascript, but it feels like the js community has a perverse fetish for fragile build systems and dependencies. Ofcourse, the community is not one organised entity and this is merely the result of multiple individual decisions taken independently, but still. I can not help but empathisewith the author despite seeing that they aren't locking deps, not doing reproducible and rollback-capable deployments via docker or artifact tarballs, don't seem to have robust ci, or good work life balance.. Is it elitist to remark "haha, don't you know you should smerp the blergh?" Is it not atleast somewhat crazy that we normalise rube-goldberg-esque contraptions? I don't even know what I'm getting at, at this point honestly. Js gives me very mixed feelings. I love the accessibility amd universally available execution runtime (browser), and understand why it hassome of the warts it does, but it is also honestly a scary ecosystem with hyper fragile tooling. I've personally burnt multiple full days on webpack config tweaks and it all feels so unnecessary and pointless. JS almost gives me existential crisis at times.
  - lukevp 3y ago
    I think the problem is, JS is actually quite a minimal language as far as standard library is concerned. Node is a good runtime (as is deno), but imho, they really missed the boat on building a standard library and making it a real ecosystem. .NET, Java, and Python, arguably the top most used languages besides JS, have an insanely more extensive standard library than JS does. Packages built by 3rd parties are the Wild West, and unless they’re a full-on company of dedicated, experienced engineers, they’re gonna break semver best practices. They’ll also design bad APIs for interacting with their packages sometimes, and there will be a total lack of consistency across the ecosystem, which is what we have currently. If I had more free time, building a stdlib for JS would be high on the list of “help the world” activities.
phendrenad2 3y ago
The problem philosophy that a few HTTP requests need to be wrapped in a library that includes hundreds of other libraries is not a JavaScript problem. It's a social problem.
lxe 3y ago
Why didn't you check the logs as the first thing to understand what's happening -- a standard operational practice? And then to prevent this, you just lock the dependencies via a lockfile -- a well documented solution. This is just another amateur "javascript bad" post.
- rezonant 3y ago
  Agreed, when they popped out this "such and such isn't a function" I lost the thread. You would know about that before you even started if you just read some logs. Unless they don't have logs, in which case a choice of language isn't going to solve their problems.
  - fendy3002 3y ago
    Or not handling uncaughtException or unhandledPromiseRejection, which may be common mistakes.
    - rezonant 3y ago
      Handling uncaughtException and unhandledPromiseRejection is a common mistake. You should not do it unless you have a good reason. Your program should die for science, and you should fix the place where you didn't handle that exception or rejection. Unless of course your handler results in dying for science, then it's fine (ie for sending to an observability service first)
      - fendy3002 3y ago
        Yes, though you still log them though and then exit the program
        rezonant 3y ago
        Ninja'd, see my edit. Fine to log and die if needed of course
- 3y ago
preommr 3y ago
> @smithy/util-stream@2.2.0 > @smithy/is-array-buffer@2.1.1 Are these direct deps? Because just a cursory look raises all kinds of red flags. There's a disclaimer for the former warning about it being in developer preview, and the rest of the README has a bunch of notes about edge cases. I get that the aws-sdk uses them, but if it works for other people it should be fine. Having these as direct deps and also calling in complex libs like the aws-sdk honestly seems like a recipe for disaster.
CraigJPerry 3y ago
I’m not following the resolution, they were doing an “npm install” in prod and they fixed it by doing an “npm ci” instead?
throw156754228 3y ago
The dependency update (i.e package.json lock file change) as I understand it was unrelated to their feature, and should have gone in on its own, and it should have broken CI if it was a serious piece of functionality that warranted a late night debugging session. It sounds to me they rolled in the dependency update with their features, then had a massive surface area to cover when it flaked out.
- rezonant 3y ago
  Exactly, and since this is AWS SDK for S3 we're taking about, and I depend on that same SDK in my own projects and they aren't broken (including using some of the more advanced streaming parts of it), I'm guessing there was a major version update applied somewhere in the dependency tree when it shouldn't have been.
rezonant 3y ago
This sounds like a team ignoring a huge amount of best practices for JavaScript development. Some combination of the following were not observed: - Use Typescript. - Practice version locking on your dependencies and make sure it works correctly throughout your processes - Do not commit vendored dependencies - Do not commit compiled JavaScript - Use a CI process for deployment which runs your tests - Have tests - Prefer (or require) types for your dependencies, or dependencies written in Typescript - Prefer dependencies that ship the types in package (and not via DefinitelyTyped) Some additional thoughts, they act like running out of memory is something they need to actively handle in the codebase-- it's not in this case. It shows a lack of familiarity with how JavaScript runtimes work which hints at a team which is very inexperienced with the platform. JavaScript runtimes enter GC lockup and ultimately crash if there is no memory, so this is a logging problem, not an error handling problem. There is no mechanism for the runtime to inform the app that an allocation failed.
dfgdfg34545456 3y ago
The real question the OP should be asking is not which language to use, but how this change managed to pass unit and e2e tests and be deployed to master in the first place. Fair enough javascript has its sharp bits, but those should show up in proper commercial level of unit and integration testing.
not_me_ever 3y ago
JavaScript A Nightmare

47 points by tempodox 3y ago

its-summertime 3y ago

Does C not have these issues?

tempodox 3y ago
Try getting “xxx is not a function” at runtime in C.
- habitue 3y ago
  Exactly, you just get a null pointer dereference segfault.
  - wiseowise 3y ago
    Without any stacktrace.
    - habitue 3y ago
      Tbf, you might get a core dump if you have them turned on, which is... Kinda like a stack trace
      - rezonant 3y ago
        Now enjoy using gdb to debug it

EarthLaunch 3y ago

This was posed 20 hours ago: https://news.ycombinator.com/item?id=39724247 https://news.ycombinator.com/item?id=39724247 Not sure why I can't downvote articles, ah well.

rezonant 3y ago
There is no downvote for submissions, only flag.

alessioalex 3y ago

Shouldn’t you use fixed versions and use something like ‘npm ci’ to install the exact dependency tree? Feels more like an oversight from the developers rather than a JavaScript problem. When you want to upgrade, only then do you run all regression tests and do so.

zaSmilingIdiot 3y ago
I agree with this. But dont just use `npm ci`on prod builds since that would typically include all the dev dependencies as well in your production builds, which is not usually desirable. It might be possible to add the `--only=production` flag to npm ci? But otherwise, as pointed out, pinned versions are needed for all dependencies.
- MrJohz 3y ago
  That flag exists (or an equivalent, I forget what the exact syntax is).
- rezonant 3y ago
  "npm ci" omits dev dependencies by default. However, "npm install" also adheres to your package-lock.json, and does not update things on you. npm ci is just npm install with a few tweaks that make it better to use for CI and builds, see here: https://docs.npmjs.com/cli/v8/commands/npm-ci https://docs.npmjs.com/cli/v8/commands/npm-ci
  - fendy3002 3y ago
    Last time I know during node 16, npm install updates your package-lock.json, but for patch version number (unsure about minor version) only and if the package.json is using relative version number. Definitely use npm ci or yarn if you want to adhere to lock files.
    - rezonant 3y ago
      > if the package.json is using relative version number What do you mean by this? You can specify dependency versions using "^" which means "I accept new minor and patch versions" or "~" which says "I accept new patch versions", or a bare version to say I want an exact version; but none of those have anything to do with a bare "npm install" command. They are relevant when you install a new package, with "npm install <package>". That is a totally different operation, and it absolutely can update versions when needed. However, when you just do "npm install" it does not update your package-lock.json to the latest version that matches your semver requirements in package.json. That will only happen if you install a new package that needs it or you use `npm update` or some other command. This is easy to test but it's also intuitively correct- you don't expect to run `npm install` and have an uncommitted change to `package-lock.json` appear, and that does not happen in normal usage. It is kind of unfortunate that "npm install" and "npm install <package>" use the same verb of "install" for two different things- other package managers don't make this mistake and it avoids the sort of misconception we're running into here.

hyfgfh 3y ago

[flagged]

hyfgfh 3y ago
To be fair, JavaScript is harder that most people think, the problem is that everyone thinks they are an expert, so we end up with this kind of problems. There is a lack of engineering discipline and too much hacking!
- leptons 3y ago
  Javascript is way easier than most people think. The problems come when they try to force their preconceived notions on it, and/or they don't keep it simple. Sure, complex things can be done with Javascript but as you said, it takes discipline.
wiseowise 3y ago
>> walking down the road >> ground suddenly became air out of nowhere > Lol, skill issue!

yen223 3y ago

tldr: A dependency wasn't pinned, the dependency changed, and stuff broke. Author thinks Javascript sucks. I will say this though: it is infuriating that package managers (every single one I've used, not just NPM) don't default to pinning exact dependency versions, and instead let it "automatically" update minor versions.

pier25 3y ago
The package wasn't following semantic versioning. Only major versions should introduce breaking changes.
- yen223 3y ago
  The only change that isn't "breaking" is one that only adds functionality. Otherwise, it changes behavour, and has the potential to break your software. In practice, almost all changes should be major version bumps. Anyway that's besides the point. Let me control when version changes happen!
  - treflop 3y ago
    I used to maintain some open source libraries and absolutely no change we would release would break anything and definitely not over a minor version. We even did a major rewrite that completely changed the API and we broke nothing, but it did mean your project got a lot of deprecation errors. And our docs would mention both the old and new way so if you were searching the Internet for help, you wouldn't get confused when you saw wildly completely examples. It's all about craftmanship.
  - brazzy 3y ago
    >The only change that isn't "breaking" is one that only adds functionality. Ha! Think again. On of the more annoying problems I had to analyze was when a new PHP version added a function that already existed in my application (this was before namespaces). The runtime did not like that.
  - rezonant 3y ago

wiseowise 3y ago

> not using TypeScript for any serious product Well, here is your problem.

crummy 3y ago
Would Typescript have solved this though? They still would have been using npm etc
- wiseowise 3y ago
  > this.subset.encodeStream is not a function Yes?
  - PUSH_AX 3y ago
    This was a runtime issue in a third party lib, so no.
    - wiseowise 3y ago
      If the library used TypeScript, it wouldn’t have the issue. The point still stands.
      - PUSH_AX 3y ago
        Debatable, and I don’t know how that helps the author. I’m getting huge “silver bullet” sentiment from your replies too.
        wiseowise 3y ago
        > > obvious problem that was solved decades ago by static typing > getting huge “silver bullet” sentiment from your replies ???
        PUSH_AX 3y ago
        I didn’t write the first quote, but a comparison to an opinionated strongly typed language where types are a first class citizen, and a typing sandbox with nonsense constructs enabled by antipatterns such as unions (‘string|number’ anyone?)… these are not the same things.

MichaelRo 3y ago

I didn't quite understand why they had to debug on the spot, thus working like 36 hours uninterrupted or so. A sane working environment would file a priority 0 bug and address it during working hours. I'm not throwing a rock but oftentimes this sort of heroic behavior is self-inflected. Also why is heroic debugging required on something that is not production? The article mentions it was a "quick demo". Then who gives a shit? There's time to investigate. And this would/should NEVER happen to production. The whole point of testing and staging/UAT environment is to catch this sort of problems. Again, a normal day at work in software engineering and not something to kill yourself living on amphetamine like a commando behind enemy lines in WW2.

peschu 3y ago
it‘s called „motivation“
- atoav 3y ago
  Jup, and if your employer doesn't stop you or even expects such behavior that is called a bad work culture. Sure, I would also do long programming sessions on my own things every now and then, but those happen because I am in the flow and it feels like more work to come back into it the other day. But those are my own projects. I don't owe this to my employer. If they want that they could offer me double my current salary and even then I would insist on only doing it when it makes sense to me.
  - number6 3y ago
    How would the employer know?
    - atoav 3y ago
      In my case my superior comes by in my office or sees I made a commit and tells me I should stop working (I tend to forget to look at the time if I am in a flow). But I also live in Europe and my superiors want me to come in with a fresh mind, instead of burning myself for things that should be normal work.

rednafi 3y ago

[dead]

leptons 3y ago

> One tiny version mismatch or breaking change can bring your mission-critical application to a standstill, as we painfully experienced. This is true in many other languages, not just Javascript. This isn't a "Javascript nightmare", this is a dependency nightmare.

jillesvangurp 3y ago
In a statically compiled language, when APIs change, you get compile errors that show up at build time. It's usually pretty obvious. With dynamically typed errors, you get weird run-time failures that can be well hidden until you trigger a use case that actually needs the bit that changed. That combined with poor test coverage means a lot of potential for things to slip through build and QA processes. I use Kotlin, both on the JVM and in the browser (via kotlin-js). Dependencies are equally complex as in javascript but it's much easier to find out if stuff breaks or not. I update often to minimize the amount of changes I have to deal with. This may sound counter intuitive but integration testing small deltas is exponentially easier and less risky than big deltas. My attitude to having big updates lurking is that they represent technical debt. Just because you don't know you have a problem because you haven't updated yet doesn't mean you don't have a problem. By not updating, you just increase your ignorance of technical debt. It might be piling up without you realizing. And you might be making things worse by working against obsolete APIs. Upgrading libraries often means I typically only have to deal with a handful of minor updates for libraries that are mostly unproblematic. And if stuff breaks, it's easier to pin point the source of the breakage. And I find out as early as possible about problems I need to deal with. If I don't know about it, I can't deal with it either. Sometimes downgrading a library temporarily is needed. But then you document why and stay on top of issue trackers and upstream development to see when you can upgrade or you start looking for alternatives. Occasionally, some API breaking changes happen and then I fix it (because the compiler tells me stuff is not compiling). If I don't update for a year, I probably have a lot of breakage on my hands and a lot of work fixing things. Much harder and much more time consuming. And risky. All I know then is that I probably have a lot of problems (i.e. technical debt) on my hands that I don't know about.
- rezonant 3y ago

MrJohz 3y ago

This is the second time I've ended up reading this post, and I confess I still don't exactly understand what was going on here. There's an issue with PDFs missing their contents, but also timeouts and a missing method exception? The problem seems to have come from upgrading some dependencies at some point, but the solution is also an upgrade, as far as I can tell? And then the cache issues at the end sound very much like a red flag in terms of managing JS dependencies - as if they're trying to cache the node_modules directory in some way that isn't properly configured. That said, the main sense I get is that the team aren't so experienced with Javascript, and therefore working with it is full of surprises because they just aren't used to it. I found the line about bugs in C code being "your fault" particularly amusing: having occasionally had to touch C and C++ code, I would have said the complete opposite! Javascript, for me, is very predictable - if something goes wrong, then it's usually because I made a clear mistake. But with C, I end up trying to debug chains of Makefile and CMake problems that have no rhyme or reason to them. In other words: as someone inexperienced in C, it is a chaotic, magical language and ecosystem to me, in exactly the same way that the author describes Javascript being to them. More practically, it sounds like the team need some regression tests for this case that they can run next time they try and upgrade their dependencies. They also need to make sure that caching is working properly for their Javascript code - rerunning the CI multiple times on the same commit should always produce exactly the same dependency tree every time. Probably that means caching the right directory and using NPM CI to clear and rebuild node_modules from scratch every time. More generally, the author complains about a lack of type checks. In this case, it sounds like the missing method was somewhere in a dependency, so this may not have helped, but: Typescript! It exists, it adds types, it makes a lot of things easier. Easy mode Typescript is to just start writing `.ts` files and use the `tsx` package to automatically compile and execute everything - there are other approaches that they could explore, but this is probably the simplest.

stana 3y ago
Stuff that gets pulled in when you do 'npm install' is scary for a more traditional server-side developer. Python for example comes with a lot of built-in libs maintained by the core python team. These could get you pretty far before you start pulling in libs from random authors.

flimsypremise 3y ago

This was very funny to read because as someone who is actually familiar with these JS builds I immediately guessed that a dependency had changed. That's literally the first thing I'd check if an app is behaving differently in a remote environment than it is locally. I'd recommend the OP learn some JS best practices and apply them rather than complaining that JS isn't C.

phanimahesh 3y ago
I'm honestly split on this. It is obvious that they are not experienced with js ecosystem, but is it fair to frame it as they obviously haven't been burned enough by the ecosystem to internalise the common issues? Dependency silent upgrades and dependency hell and bloat are not unique to javascript, but it feels like the js community has a perverse fetish for fragile build systems and dependencies. Ofcourse, the community is not one organised entity and this is merely the result of multiple individual decisions taken independently, but still. I can not help but empathisewith the author despite seeing that they aren't locking deps, not doing reproducible and rollback-capable deployments via docker or artifact tarballs, don't seem to have robust ci, or good work life balance.. Is it elitist to remark "haha, don't you know you should smerp the blergh?" Is it not atleast somewhat crazy that we normalise rube-goldberg-esque contraptions? I don't even know what I'm getting at, at this point honestly. Js gives me very mixed feelings. I love the accessibility amd universally available execution runtime (browser), and understand why it hassome of the warts it does, but it is also honestly a scary ecosystem with hyper fragile tooling. I've personally burnt multiple full days on webpack config tweaks and it all feels so unnecessary and pointless. JS almost gives me existential crisis at times.
- lukevp 3y ago
  I think the problem is, JS is actually quite a minimal language as far as standard library is concerned. Node is a good runtime (as is deno), but imho, they really missed the boat on building a standard library and making it a real ecosystem. .NET, Java, and Python, arguably the top most used languages besides JS, have an insanely more extensive standard library than JS does. Packages built by 3rd parties are the Wild West, and unless they’re a full-on company of dedicated, experienced engineers, they’re gonna break semver best practices. They’ll also design bad APIs for interacting with their packages sometimes, and there will be a total lack of consistency across the ecosystem, which is what we have currently. If I had more free time, building a stdlib for JS would be high on the list of “help the world” activities.

phendrenad2 3y ago

The problem philosophy that a few HTTP requests need to be wrapped in a library that includes hundreds of other libraries is not a JavaScript problem. It's a social problem.

lxe 3y ago

Why didn't you check the logs as the first thing to understand what's happening -- a standard operational practice? And then to prevent this, you just lock the dependencies via a lockfile -- a well documented solution. This is just another amateur "javascript bad" post.

rezonant 3y ago
Agreed, when they popped out this "such and such isn't a function" I lost the thread. You would know about that before you even started if you just read some logs. Unless they don't have logs, in which case a choice of language isn't going to solve their problems.
- fendy3002 3y ago
  Or not handling uncaughtException or unhandledPromiseRejection, which may be common mistakes.
  - rezonant 3y ago
    Handling uncaughtException and unhandledPromiseRejection is a common mistake. You should not do it unless you have a good reason. Your program should die for science, and you should fix the place where you didn't handle that exception or rejection. Unless of course your handler results in dying for science, then it's fine (ie for sending to an observability service first)
    - fendy3002 3y ago
      Yes, though you still log them though and then exit the program
      - rezonant 3y ago
        Ninja'd, see my edit. Fine to log and die if needed of course
3y ago

preommr 3y ago

> @smithy/util-stream@2.2.0 > @smithy/is-array-buffer@2.1.1 Are these direct deps? Because just a cursory look raises all kinds of red flags. There's a disclaimer for the former warning about it being in developer preview, and the rest of the README has a bunch of notes about edge cases. I get that the aws-sdk uses them, but if it works for other people it should be fine. Having these as direct deps and also calling in complex libs like the aws-sdk honestly seems like a recipe for disaster.

CraigJPerry 3y ago

I’m not following the resolution, they were doing an “npm install” in prod and they fixed it by doing an “npm ci” instead?

throw156754228 3y ago

The dependency update (i.e package.json lock file change) as I understand it was unrelated to their feature, and should have gone in on its own, and it should have broken CI if it was a serious piece of functionality that warranted a late night debugging session. It sounds to me they rolled in the dependency update with their features, then had a massive surface area to cover when it flaked out.

rezonant 3y ago
Exactly, and since this is AWS SDK for S3 we're taking about, and I depend on that same SDK in my own projects and they aren't broken (including using some of the more advanced streaming parts of it), I'm guessing there was a major version update applied somewhere in the dependency tree when it shouldn't have been.

rezonant 3y ago

This sounds like a team ignoring a huge amount of best practices for JavaScript development. Some combination of the following were not observed: - Use Typescript. - Practice version locking on your dependencies and make sure it works correctly throughout your processes - Do not commit vendored dependencies - Do not commit compiled JavaScript - Use a CI process for deployment which runs your tests - Have tests - Prefer (or require) types for your dependencies, or dependencies written in Typescript - Prefer dependencies that ship the types in package (and not via DefinitelyTyped) Some additional thoughts, they act like running out of memory is something they need to actively handle in the codebase-- it's not in this case. It shows a lack of familiarity with how JavaScript runtimes work which hints at a team which is very inexperienced with the platform. JavaScript runtimes enter GC lockup and ultimately crash if there is no memory, so this is a logging problem, not an error handling problem. There is no mechanism for the runtime to inform the app that an allocation failed.

dfgdfg34545456 3y ago

The real question the OP should be asking is not which language to use, but how this change managed to pass unit and e2e tests and be deployed to master in the first place. Fair enough javascript has its sharp bits, but those should show up in proper commercial level of unit and integration testing.

not_me_ever 3y ago

JavaScript A Nightmare