5 ms·

This is great news, but aren't there bigger fish to fry with really supporting node.js & npm on Windows? The 255 character path limit bug has been around for a

by tdicola 12y ago

This is great news, but aren't there bigger fish to fry with really supporting node.js & npm on Windows? The 255 character path limit bug has been around for a long time and still unresolved/broken on Windows (see https://github.com/joyent/node/issues/6960 https://github.com/joyent/node/issues/6960 and https://github.com/npm/npm/issues/5641 https://github.com/npm/npm/issues/5641). Suggestions to fix it on the Windows side have been closed too: http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2156195-fix-260-character-file-name-length-limitation http://visualstudio.uservoice.com/forums/121579-visual-studi... Would be nice for people to stop throwing it around like a hot potato and fix it for good. Although it sounds like an edge case, it's actually really common for some libraries to his this limit and be unusable on Windows.

mousetraps 12y ago
See comments on this thread re: max_path problems, workarounds, and how we help mitigate it in NTVS: https://news.ycombinator.com/item?id=9265694 https://news.ycombinator.com/item?id=9265694 Otherwise, yeah - we don't like it either... here's the not-closed uservoice request ;-) https://windows.uservoice.com/forums/265757-windows-feature-suggestions/suggestions/6573539-remove-the-long-standing-max-path-limitation https://windows.uservoice.com/forums/265757-windows-feature-... EDIT: Re: fish: Indeed - we have some big fish to fry, but luckily msft is a big company that can fry multiple fish at a time. Personally, I'm excited to help improve the Node tooling story by frying this fish, and passing our learnings and feedback onto the the other teams @ msft so that they can improve their recipes. Ultimately, we strive to have a complementary set of fried fish. And maybe some vegetables too... :)
- Already__Taken 12y ago
  I thought connect was the official way to complain about MS things?
  - mousetraps 12y ago
    Technically... - connect is for filing bugs - uservoice is for "ideas and suggestions" How people use it, of course, is a completely different story... And then some teams (like us) have public issue trackers, which are the best approach if available because everything goes directly to the team: https://nodejstools.codeplex.com/workitem/list/basic https://nodejstools.codeplex.com/workitem/list/basic
    - Already__Taken 12y ago
      Ah I see. Yes I put a powershell bug/missing feature request on connect and looked through some other tickets. I don't envy whose reading that. I think Ms is doomed to get people complaining about unrelated junk any time they open a platform up.
      - jyrkesh 12y ago
        Any ideas for how a problem like that can be solved? User feedback on a closed-source project at massive scale?
        Already__Taken 12y ago
        I don't know. I guess you've just got to vet credentials? To avoid having a team to man the identity service perhaps; If you need developer feedback maybe require the UX to be creating Pull requests describing the issue? Kind of like how pullup.io works?
- gnoway 12y ago
  Why can't you (or node?) use the \\?\ syntax?
  - michaelx386 12y ago
    For anyone else like me who didn't know about this \\?\ syntax: http://stackoverflow.com/questions/21194530/what-does-mean-when-prepended-to-a-file-path http://stackoverflow.com/questions/21194530/what-does-mean-w... https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
  - bmeck 12y ago
    Node uses the Unicode functions so no need, but lots of tools in windows break (cmd.exe notably) so things like child processes don't work on long filepaths. You can actually watch `npm` write to files past the limit by using the Unicode functions, but trying to access them via child processes generally fails (and things like IDEs can't see them). The other problem is that using the longer paths turns of filepath expansion. * https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
  - mousetraps 12y ago
    I posted this in the comments in the blog as well: With regards to the technicalities of max_path itself... the issue is with the .NET file I/O API, which doesn't allow you to use \\?\ paths to work around it, and is too big for our team in particular to do much about. Here's a good article that might provide a little more context (it's a little old, and these "plans" they speak of never actually materialized.) http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-path... That said... we don't like it either, and we are still thinking about what we can do.
xenadu02 12y ago
MAX_PATH is the gift that keeps on giving. Thanks to Microsoft's fanatical dedication to compatibility I don't see how it will ever get fixed. I think they could take a lesson from Apple here and have DLLS/EXEs contain SDK versions against which they are built, then mandate that Windows 10 apps must support a larger MAX_PATH. Of course if they were into learning lessons, they'd have done fat binaries so a single install could easily support 64-bit and 32-bit (or a single DLL could contain 64-bit and 32-bit versions). For those not in the know, MAX_PATH and PATH_MAX (ugh) are defined in the Windows headers as 255. Because it was public people copied that constant and spread it around. So many apps and libraries out there are hard-coded to expect 255 that even if you change it in the Windows headers a million things will still be broken. Even if you overcame that limitation, you'd have a situation where paths no longer worked reliably if they got passed around (or worse, long paths would trigger buffer overflows in poorly written apps and drivers). You could fix it by passing apps not declared as large-path aware a shortened virtual path but the testing/edge case burden would be enormous. So this is an example of a really quick decision one programmer probably made by themselves many years ago that we are forced to live with because the pain of changing it is too great compared to the benefit we'd receive.
- ygra 12y ago
  MAX_PATH is a #define, so it just gets literally substituted when compiling, there is no need for people copying it. After compilation it effectively is a hard-coded constant, indistinguishable from literally writing 260 (it's not 255).
- malka 12y ago
  Since the problem seems to be mostly with cmd.exe (and tools that use it), could not Microsoft ship a cmd.exe (with a modified name, so it does not break old stuff) that handles long path name by using the unicode API ?
rdtsc 12y ago
> The 255 character path limit bug has been around for a long time and still unresolved/broken on Windows I don't know, I was cussing out both node.js and Windows. "Let's put all the dependencies in subtree even if they repeat 20 times throughout or paths start wrapping around 3 times at the terminal".
BinaryIdiot 12y ago
This! This really needs to be fixed! My first experience of getting a node application running on Windows ran into this very quickly due to the crazy dependency tree npm creates. While npm could certainly be improved this limit is so old it really needs to be fixed.
yummybear 12y ago
Isn't this a bit like saying "It's great we landed on the moon and all, but shouldn't we concentrate on stuff like curing cancer."?
- Drakim 12y ago
  It's more a bit like saying "It's great we landed on the moon and all, but shouldn't we concentrate on making spaceships safe first?"
- Already__Taken 12y ago
  Consider a fresh user installs nodejs, runs npm install <someproject>, runs npm run start. Immediately they get a problem. Doesn't look great for node and it shouldn't be their problem. More like "We landed on the moon but sometimes the door won't open"
bmeck 12y ago
Once again, these limitations are not with Windows itself (see all the C++ Unicode file functions / file longpaths). In general Node supports long paths just fine (you can watch `npm` write past the 260 [wtf?] char limit). The limitation is in other programs trying to respect MAXPATH even when they should be moving to Unicode functions for compatibility reasons / lack of knowledge. It should be noted that compatibility breaks not just from long paths causing errors, but automatic file expansion does not occur on long paths. So we get into even more special snowflake situations where sometimes files with expansions work with older functions but not with newer ones. But the fact that Unicode functions don't expand adds an interesting problem for us; the MAXPATH respecting (and outdated) _A filesystem functions can cause a collision for non-expanded paths we get to fun things where you can have different FILE objects if you use old _A functions vs _W functions. The inverse is also true where you can collide by moving from _W to _A. So! If we do start using old _A functions we actually can cause problems by directing to new files that would cause expansion because of file expansion being turned on. * https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... * https://msdn.microsoft.com/en-us/library/ms813802.aspx https://msdn.microsoft.com/en-us/library/ms813802.aspx PS. Don't get me started on FS permissions...

by tdicola 12y ago

mousetraps 12y ago

See comments on this thread re: max_path problems, workarounds, and how we help mitigate it in NTVS: https://news.ycombinator.com/item?id=9265694 https://news.ycombinator.com/item?id=9265694 Otherwise, yeah - we don't like it either... here's the not-closed uservoice request ;-) https://windows.uservoice.com/forums/265757-windows-feature-suggestions/suggestions/6573539-remove-the-long-standing-max-path-limitation https://windows.uservoice.com/forums/265757-windows-feature-... EDIT: Re: fish: Indeed - we have some big fish to fry, but luckily msft is a big company that can fry multiple fish at a time. Personally, I'm excited to help improve the Node tooling story by frying this fish, and passing our learnings and feedback onto the the other teams @ msft so that they can improve their recipes. Ultimately, we strive to have a complementary set of fried fish. And maybe some vegetables too... :)

Already__Taken 12y ago
I thought connect was the official way to complain about MS things?
- mousetraps 12y ago
  Technically... - connect is for filing bugs - uservoice is for "ideas and suggestions" How people use it, of course, is a completely different story... And then some teams (like us) have public issue trackers, which are the best approach if available because everything goes directly to the team: https://nodejstools.codeplex.com/workitem/list/basic https://nodejstools.codeplex.com/workitem/list/basic
  - Already__Taken 12y ago
    Ah I see. Yes I put a powershell bug/missing feature request on connect and looked through some other tickets. I don't envy whose reading that. I think Ms is doomed to get people complaining about unrelated junk any time they open a platform up.
    - jyrkesh 12y ago
      Any ideas for how a problem like that can be solved? User feedback on a closed-source project at massive scale?
      - Already__Taken 12y ago
        I don't know. I guess you've just got to vet credentials? To avoid having a team to man the identity service perhaps; If you need developer feedback maybe require the UX to be creating Pull requests describing the issue? Kind of like how pullup.io works?
gnoway 12y ago
Why can't you (or node?) use the \\?\ syntax?
- michaelx386 12y ago
  For anyone else like me who didn't know about this \\?\ syntax: http://stackoverflow.com/questions/21194530/what-does-mean-when-prepended-to-a-file-path http://stackoverflow.com/questions/21194530/what-does-mean-w... https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- bmeck 12y ago
  Node uses the Unicode functions so no need, but lots of tools in windows break (cmd.exe notably) so things like child processes don't work on long filepaths. You can actually watch `npm` write to files past the limit by using the Unicode functions, but trying to access them via child processes generally fails (and things like IDEs can't see them). The other problem is that using the longer paths turns of filepath expansion. * https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- mousetraps 12y ago
  I posted this in the comments in the blog as well: With regards to the technicalities of max_path itself... the issue is with the .NET file I/O API, which doesn't allow you to use \\?\ paths to work around it, and is too big for our team in particular to do much about. Here's a good article that might provide a little more context (it's a little old, and these "plans" they speak of never actually materialized.) http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-path... That said... we don't like it either, and we are still thinking about what we can do.

xenadu02 12y ago

MAX_PATH is the gift that keeps on giving. Thanks to Microsoft's fanatical dedication to compatibility I don't see how it will ever get fixed. I think they could take a lesson from Apple here and have DLLS/EXEs contain SDK versions against which they are built, then mandate that Windows 10 apps must support a larger MAX_PATH. Of course if they were into learning lessons, they'd have done fat binaries so a single install could easily support 64-bit and 32-bit (or a single DLL could contain 64-bit and 32-bit versions). For those not in the know, MAX_PATH and PATH_MAX (ugh) are defined in the Windows headers as 255. Because it was public people copied that constant and spread it around. So many apps and libraries out there are hard-coded to expect 255 that even if you change it in the Windows headers a million things will still be broken. Even if you overcame that limitation, you'd have a situation where paths no longer worked reliably if they got passed around (or worse, long paths would trigger buffer overflows in poorly written apps and drivers). You could fix it by passing apps not declared as large-path aware a shortened virtual path but the testing/edge case burden would be enormous. So this is an example of a really quick decision one programmer probably made by themselves many years ago that we are forced to live with because the pain of changing it is too great compared to the benefit we'd receive.

ygra 12y ago
MAX_PATH is a #define, so it just gets literally substituted when compiling, there is no need for people copying it. After compilation it effectively is a hard-coded constant, indistinguishable from literally writing 260 (it's not 255).
malka 12y ago
Since the problem seems to be mostly with cmd.exe (and tools that use it), could not Microsoft ship a cmd.exe (with a modified name, so it does not break old stuff) that handles long path name by using the unicode API ?

rdtsc 12y ago

> The 255 character path limit bug has been around for a long time and still unresolved/broken on Windows I don't know, I was cussing out both node.js and Windows. "Let's put all the dependencies in subtree even if they repeat 20 times throughout or paths start wrapping around 3 times at the terminal".

BinaryIdiot 12y ago

This! This really needs to be fixed! My first experience of getting a node application running on Windows ran into this very quickly due to the crazy dependency tree npm creates. While npm could certainly be improved this limit is so old it really needs to be fixed.

yummybear 12y ago

Isn't this a bit like saying "It's great we landed on the moon and all, but shouldn't we concentrate on stuff like curing cancer."?

Drakim 12y ago
It's more a bit like saying "It's great we landed on the moon and all, but shouldn't we concentrate on making spaceships safe first?"
Already__Taken 12y ago
Consider a fresh user installs nodejs, runs npm install <someproject>, runs npm run start. Immediately they get a problem. Doesn't look great for node and it shouldn't be their problem. More like "We landed on the moon but sometimes the door won't open"

bmeck 12y ago

Once again, these limitations are not with Windows itself (see all the C++ Unicode file functions / file longpaths). In general Node supports long paths just fine (you can watch `npm` write past the 260 [wtf?] char limit). The limitation is in other programs trying to respect MAXPATH even when they should be moving to Unicode functions for compatibility reasons / lack of knowledge. It should be noted that compatibility breaks not just from long paths causing errors, but automatic file expansion does not occur on long paths. So we get into even more special snowflake situations where sometimes files with expansions work with older functions but not with newer ones. But the fact that Unicode functions don't expand adds an interesting problem for us; the MAXPATH respecting (and outdated) _A filesystem functions can cause a collision for non-expanded paths we get to fun things where you can have different FILE objects if you use old _A functions vs _W functions. The inverse is also true where you can collide by moving from _W to _A. So! If we do start using old _A functions we actually can cause problems by directing to new files that would cause expansion because of file expansion being turned on. * https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... * https://msdn.microsoft.com/en-us/library/ms813802.aspx https://msdn.microsoft.com/en-us/library/ms813802.aspx PS. Don't get me started on FS permissions...