5 ms·

The saddest script I ever wrote

23 points by alexeiz 6y ago

yurishimo 6y ago
We had something kind of similar for a large e-commerce project. Docker was loading the site super slow with a mapped drive so we had the brilliant idea to rsync the files from OSX into the container using a file watcher. This meant the page loads were faster , but it could take even longer for the watcher to diff the changes and sync them back up to the container. Pretty annoying. At some point, the container got optimized to not need the sync anymore for acceptable load times. Nobody is sure what we changed to help that out. ️
krick 6y ago
This is... beautiful. I wonder though, how different are the file edited with ed and the one edited with vi. Maybe ed fails to update the "date modified" under these conditions?
- kjetijor 6y ago
  ed might do truncate/write, and vim might do create/write/rename. New file, cache miss.
  - alexlarsson 6y ago
    Yeah, this is obviously a replace vs in-place modify difference. Could replace the script with a simple copy-to-temp+rename-back.
    - cryptonector 6y ago
      Or not use ed(1).
      - alexlarsson 6y ago
        Personally I use magnets to flip bits around, ed(1) seems a bit to bloated for me.
        diegoperini 6y ago
        Who needs magnets when you have cosmic rays?
  - raverbashing 6y ago
    Yeah I feel the right way to solve this bug (in the compiler) would be to stat the file. Or maybe docker mounts it with -noatime (or something similar?)
kalium-xyz 6y ago
NixOS fixes the issue the author was using docker for. systems with explicit well managed state make development much less of a hell.
- FeepingCreature 6y ago
  Note: this depends crucially on your appraisal of the Nix language and ecosystem as "less hell".
  - deleted 6y ago
    [deleted]
  - kalium-xyz 6y ago
    Its an investment but there are no nasty surprises to it or security issues, just a lotta gotchas. Things get better by the day in terms of editor support and tooling integration so keep watch c:
    - vaibhavsagar 6y ago
      I love Nix and NixOS, but having "a lotta gotchas" doesn't square with "no nasty surprises".
      - kalium-xyz 6y ago
        Maybe im using the term wrong, I mean there is a steep learning curve but its nothing that will cause you serious issues like randomly deleting data or “how do I un-rm a file?”. The complexity is in using the tooling and not in the commands being idiotic and having dangerous side effects, if anything less so than other OSes
jmiskovic 6y ago
Off topic: Page starts with "snail text" logo and ends with quirky upside down link "What's snail text?". Sure, I'll bite. I press the link, and get subscription form without any explanation what I would be subscribing to. And then in bottom of page there's a second bait, "What's snail text?". What indeed. It is launching soon. And it has a twitter account. So that is known. Maybe one day we'll find out more.
- croo 6y ago
  I went to the twitter account then find the blog and checked over the blog post titles. I still don't have a clue about what snail text is. What indeed.
- epse 6y ago
  The first page of their blog says it's "the solution for busy people to sync calendars over sms"
ehnto 6y ago
> because I would not be thwarted by crappy software Oh mate, I don't think that's the issue here! This is like breaking your host OS and then making it pantomime it's own replacement so you can pretend you still have a working OS!
progre 6y ago
> One day about a year ago, I started using ed in place of vi This is a choice that I feel would benefit from some kind of explenation
downerending 6y ago
Didn't follow that, but invoking csh is indeed very, very sad.
pvtmert 6y ago
Actually most problems here already described well in Docker docs: - https://docs.docker.com/storage/bind-mounts/#configure-mount-consistency-for-macos https://docs.docker.com/storage/bind-mounts/#configure-mount... Due to it is being VM, contents are synced between host and VM. It takes time and damages IOPS greatly. Eg. Even running ~100MB Jar takes minutes... Side note that vim uses `swap` files `.$FILE.swp` when you write out, it removes the original file renames the temporary one to original one. So essentially inode or reference is different thats why it triggers sync with VM. Ed probably writes files in-place. Docker now uses inotify equalivent (fswatch?) on macOS, which is pretty much reliable than polling. Ofc if your app did not close file with exclusive read-write-lock, docker cannot do anything about it.
saagarjha 6y ago
Problems like these drive me absolutely mad, but I have the nasty tendency to waste huge amounts of time trying to track down the underlying issue. Sometimes this works and I can fix the problem; other times I figure it out, learn something new, and file a bug report or write a workaround with the new knowledge, but all too often I spend hours with little to show for it, give up far later than I should have, and have the papered-over issue in the back of my mind forevermore.
gbrown_ 6y ago
I'm sure the author tried some debugging but it's a bit sad not to see any in the post, instead their workaround is simply presented. I would assume the root of the difference in behavior comes down to calling fsync when editing files on such a mount. In lieu of having macOS's userland source I grepped through through copies of ed's source from GNU and OpenBSD. I don't see any calls to fsync. Whereas Vim by default does. Note most instances of vi are actually a link to Vim/a minimal version of Vim. Normally I'd test this with with some quick tracing but it appears I've not configured SIP to allow DTrace on my machine and I can't be bothered to reboot right now.

23 points by alexeiz 6y ago

We had something kind of similar for a large e-commerce project. Docker was loading the site super slow with a mapped drive so we had the brilliant idea to rsync the files from OSX into the container using a file watcher. This meant the page loads were faster , but it could take even longer for the watcher to diff the changes and sync them back up to the container. Pretty annoying. At some point, the container got optimized to not need the sync anymore for acceptable load times. Nobody is sure what we changed to help that out. ️

krick 6y ago

This is... beautiful. I wonder though, how different are the file edited with ed and the one edited with vi. Maybe ed fails to update the "date modified" under these conditions?

kjetijor 6y ago
ed might do truncate/write, and vim might do create/write/rename. New file, cache miss.
- alexlarsson 6y ago
  Yeah, this is obviously a replace vs in-place modify difference. Could replace the script with a simple copy-to-temp+rename-back.
  - cryptonector 6y ago
    Or not use ed(1).
    - alexlarsson 6y ago
      Personally I use magnets to flip bits around, ed(1) seems a bit to bloated for me.
      - diegoperini 6y ago
        Who needs magnets when you have cosmic rays?
- raverbashing 6y ago
  Yeah I feel the right way to solve this bug (in the compiler) would be to stat the file. Or maybe docker mounts it with -noatime (or something similar?)

kalium-xyz 6y ago

NixOS fixes the issue the author was using docker for. systems with explicit well managed state make development much less of a hell.

FeepingCreature 6y ago
Note: this depends crucially on your appraisal of the Nix language and ecosystem as "less hell".
- deleted 6y ago
  [deleted]
- kalium-xyz 6y ago
  Its an investment but there are no nasty surprises to it or security issues, just a lotta gotchas. Things get better by the day in terms of editor support and tooling integration so keep watch c:
  - vaibhavsagar 6y ago
    I love Nix and NixOS, but having "a lotta gotchas" doesn't square with "no nasty surprises".
    - kalium-xyz 6y ago
      Maybe im using the term wrong, I mean there is a steep learning curve but its nothing that will cause you serious issues like randomly deleting data or “how do I un-rm a file?”. The complexity is in using the tooling and not in the commands being idiotic and having dangerous side effects, if anything less so than other OSes

jmiskovic 6y ago

Off topic: Page starts with "snail text" logo and ends with quirky upside down link "What's snail text?". Sure, I'll bite. I press the link, and get subscription form without any explanation what I would be subscribing to. And then in bottom of page there's a second bait, "What's snail text?". What indeed. It is launching soon. And it has a twitter account. So that is known. Maybe one day we'll find out more.

croo 6y ago
I went to the twitter account then find the blog and checked over the blog post titles. I still don't have a clue about what snail text is. What indeed.
epse 6y ago
The first page of their blog says it's "the solution for busy people to sync calendars over sms"

ehnto 6y ago

> because I would not be thwarted by crappy software Oh mate, I don't think that's the issue here! This is like breaking your host OS and then making it pantomime it's own replacement so you can pretend you still have a working OS!

progre 6y ago

> One day about a year ago, I started using ed in place of vi This is a choice that I feel would benefit from some kind of explenation

downerending 6y ago

Didn't follow that, but invoking csh is indeed very, very sad.

pvtmert 6y ago

Actually most problems here already described well in Docker docs: - https://docs.docker.com/storage/bind-mounts/#configure-mount-consistency-for-macos https://docs.docker.com/storage/bind-mounts/#configure-mount... Due to it is being VM, contents are synced between host and VM. It takes time and damages IOPS greatly. Eg. Even running ~100MB Jar takes minutes... Side note that vim uses `swap` files `.$FILE.swp` when you write out, it removes the original file renames the temporary one to original one. So essentially inode or reference is different thats why it triggers sync with VM. Ed probably writes files in-place. Docker now uses inotify equalivent (fswatch?) on macOS, which is pretty much reliable than polling. Ofc if your app did not close file with exclusive read-write-lock, docker cannot do anything about it.

saagarjha 6y ago

Problems like these drive me absolutely mad, but I have the nasty tendency to waste huge amounts of time trying to track down the underlying issue. Sometimes this works and I can fix the problem; other times I figure it out, learn something new, and file a bug report or write a workaround with the new knowledge, but all too often I spend hours with little to show for it, give up far later than I should have, and have the papered-over issue in the back of my mind forevermore.

gbrown_ 6y ago

I'm sure the author tried some debugging but it's a bit sad not to see any in the post, instead their workaround is simply presented. I would assume the root of the difference in behavior comes down to calling fsync when editing files on such a mount. In lieu of having macOS's userland source I grepped through through copies of ed's source from GNU and OpenBSD. I don't see any calls to fsync. Whereas Vim by default does. Note most instances of vi are actually a link to Vim/a minimal version of Vim. Normally I'd test this with with some quick tracing but it appears I've not configured SIP to allow DTrace on my machine and I can't be bothered to reboot right now.