4 ms·

this is really not much different from ssh-ing to your GPU server and running ffmpeg. Very roundabout way to execute remote bash command on a server I dont mea

by slt2021 2y ago

this is really not much different from ssh-ing to your GPU server and running ffmpeg. Very roundabout way to execute remote bash command on a server I dont meant to discourage you, but it is possible to replace your entire repo with a simple bash alias: alias ffmpeg-over-ip='ssh myserver "ffmpeg \"\$@\""'

steelbrain 2y ago
This comment gave me flashbacks to another comment I read a while ago: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224 If your usecase is solved by an alias, that's really good! I am glad you can use an alias. My usecase required a bit more so I wrote this utility and am sharing it with my peers
- slt2021 2y ago
  Dropbox had a cutting edge file synchronization algorithm, they solved a problem of large file sync over unreliable network. There was a clear engineering IP they developed. (https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine https://dropbox.tech/infrastructure/rewriting-the-heart-of-o...) I looked over your source code and just saw a bash wrapper with webserver, so no significant IP. Any potential innovations: like possible distributed transcoding, sharding/partitioning transcoding pipeline to speed-up are missing. its just a bash wrapper, thats why I commented about bash alias. I don't mean to sound like a jerk, but I was honestly looking for some innovation about ffmpeg
  - steelbrain 2y ago
    This was not meant to offend. I appreciate you explaining your message further. There's no significant IP in this utility, it's something I wrote for a usecase and it works well for that usecase. I ran the server side on a windows machine, and I did not want to setup a full blown ssh server and expose it over the network for this usecase. Another thing was logging. The way logging is currently setup really hits the sweet spot for debugability for me. Lastly it's the rewrites. I've used the config to rewrite incoming codecs to something the machine supports. This is a purpose built utility that does one job and IMO does it fairly well. It's definitely not as complex as Dropbox but also not as simple as an ssh alias. I appreciate you sharing the alias code (not just the comment) so if some of our peers have usecases that could be solved by it, they are welcome to use that as well!
- KolmogorovComp 2y ago
  > My usecase required a bit more so I wrote this utility and am sharing it with my peers Can you expand on that?
  - steelbrain 2y ago
    For sure! One of the software I was working with was hardcoding what codecs it would use based on the operating system it was running on. The rewrites section of the configuration allows more than just file paths, I've used it to rewrite incoming codec requests
amelius 2y ago
I suppose this only works if you have some shared filesystem. Or does this work with piping too?
- slt2021 2y ago
  the original poster's project also requires shared filesystem. as for bash-ssh solution, you don't need shared FS, if you don't need intermediate results. you can use SCP to get the final result after transcoding has finished. something like: alias ffmpeg-over-ip='ssh myserver "ffmpeg \"\$@\" /tmp/output/"' alias download-results='scp myserver:/tmp/output/*.* .' ffmpeg-over-ip <args> && download-results my meta point being is, before engineering something with programming language, and handrolling webservers, with auth, and workers - just try to implement your system with bash scripts. Martin Klepmann created an entire database using just bash aliases in his book "Designing Data Intensive Applications"
  - whoopdedo 2y ago
    Even the temporary file is optional. Ffmpeg supports a number of network protocols. For instance, you could read from one port and write to another with ffmpeg -f webm -i tcp://[::]:55601?listen -c copy -f webm tcp://[::]:55602?listen There's also UDP, SFTP, and more convenient protocols such as SRT or the long-in-the-tooth RTMP. I expect it will eventually add WHIP/WHEP as well.
    - amelius 2y ago
      I experimented a bit with piping, but it turned out that mp4 doesn't always like to be streamed.
asveikau 2y ago
I would add screen or tmux to that because you may run a long job that you may want to get back to after a connection drop.

steelbrain 2y ago

This comment gave me flashbacks to another comment I read a while ago: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224 If your usecase is solved by an alias, that's really good! I am glad you can use an alias. My usecase required a bit more so I wrote this utility and am sharing it with my peers

slt2021 2y ago
Dropbox had a cutting edge file synchronization algorithm, they solved a problem of large file sync over unreliable network. There was a clear engineering IP they developed. (https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine https://dropbox.tech/infrastructure/rewriting-the-heart-of-o...) I looked over your source code and just saw a bash wrapper with webserver, so no significant IP. Any potential innovations: like possible distributed transcoding, sharding/partitioning transcoding pipeline to speed-up are missing. its just a bash wrapper, thats why I commented about bash alias. I don't mean to sound like a jerk, but I was honestly looking for some innovation about ffmpeg
- steelbrain 2y ago
  This was not meant to offend. I appreciate you explaining your message further. There's no significant IP in this utility, it's something I wrote for a usecase and it works well for that usecase. I ran the server side on a windows machine, and I did not want to setup a full blown ssh server and expose it over the network for this usecase. Another thing was logging. The way logging is currently setup really hits the sweet spot for debugability for me. Lastly it's the rewrites. I've used the config to rewrite incoming codecs to something the machine supports. This is a purpose built utility that does one job and IMO does it fairly well. It's definitely not as complex as Dropbox but also not as simple as an ssh alias. I appreciate you sharing the alias code (not just the comment) so if some of our peers have usecases that could be solved by it, they are welcome to use that as well!
KolmogorovComp 2y ago
> My usecase required a bit more so I wrote this utility and am sharing it with my peers Can you expand on that?
- steelbrain 2y ago
  For sure! One of the software I was working with was hardcoding what codecs it would use based on the operating system it was running on. The rewrites section of the configuration allows more than just file paths, I've used it to rewrite incoming codec requests

amelius 2y ago

I suppose this only works if you have some shared filesystem. Or does this work with piping too?

slt2021 2y ago
the original poster's project also requires shared filesystem. as for bash-ssh solution, you don't need shared FS, if you don't need intermediate results. you can use SCP to get the final result after transcoding has finished. something like: alias ffmpeg-over-ip='ssh myserver "ffmpeg \"\$@\" /tmp/output/"' alias download-results='scp myserver:/tmp/output/*.* .' ffmpeg-over-ip <args> && download-results my meta point being is, before engineering something with programming language, and handrolling webservers, with auth, and workers - just try to implement your system with bash scripts. Martin Klepmann created an entire database using just bash aliases in his book "Designing Data Intensive Applications"
- whoopdedo 2y ago
  Even the temporary file is optional. Ffmpeg supports a number of network protocols. For instance, you could read from one port and write to another with ffmpeg -f webm -i tcp://[::]:55601?listen -c copy -f webm tcp://[::]:55602?listen There's also UDP, SFTP, and more convenient protocols such as SRT or the long-in-the-tooth RTMP. I expect it will eventually add WHIP/WHEP as well.
  - amelius 2y ago
    I experimented a bit with piping, but it turned out that mp4 doesn't always like to be streamed.

asveikau 2y ago

I would add screen or tmux to that because you may run a long job that you may want to get back to after a connection drop.