4 ms·

Just curious on the motivation. Why not run a regular FTP server and have your application periodically look for new files to process? For horizontal scaling,

by winrid 6y ago

Just curious on the motivation. Why not run a regular FTP server and have your application periodically look for new files to process? For horizontal scaling, you just take a distributed lock on the file name.

meritt 6y ago
Because nodejs kubernetes modern cloud.
- dvfjsdhgfv 6y ago
  > Because nodejs kubernetes modern cloud. Not necessarily so. The history of FTP servers is ridden by bugs with practically no exceptions. At some point some folks decided they finally implement a bug-free implementation and even dared to call it "Very Secure FTPd." Needless to say, it turned out it has bugs, too. As most of these bugs were related to buffer overflows and similar issues, implementing a new FTP server in a safer language is not such a bad idea, and today's JavaScript is efficient enough to make it a reasonably well-working implementation. I pity the author though for the bugs they encounter and workarounds that will need to be implemented.
  - dominiek 6y ago
    Exactly my reasoning. See my comment above for more info. I'm making the maintenance of this less painful by doing a hacking/debugging session with manufacturers once a month where we hook up many devices and fix issues. After addressing most edge cases fewer are coming up now (despite a relentless stream of new cheaply manufactured devices)
  - tyingq 6y ago
    I agree, but I don't see why we would then assume that forking some ftp server library from npm would fare any better, security wise. I see a fairly alarming open issue: https://github.com/autovance/ftp-srv/issues/167 https://github.com/autovance/ftp-srv/issues/167
- dominiek 6y ago
  The main reason for using Node.js is because the rest of our stack is Node.js: https://bedrock.io https://bedrock.io We use MongoDB as persistence and have existing wrappers for dealing with Google Cloud Storage. Since it's an isolated service we could've used a different implementation language. In our case Node.js in our existing Kubernetes environment was the least amount of friction
admax88q 6y ago
That honestly sounds more complicated. FTP isn't that difficult of a protocol, especially if you only need to support one known client, you can take all sort of shortcuts. If you deploy an existing FTP server, and _then_ integrate with it at the filesystem level you now have two components, and your sysadmin requirements grow. Now you gotta administrate an FTP server that's probably written for classic UNIX single server usage, gotta handle filesystem permissions, gotta somehow hook up your distributed locks to the filesystem, sanitize filenames for your chosen filesystem. Honestly filesystems suck, there's so many gotchas from a security perspective, when all you really want is to pipe binary data in this side, and out the other side. I implemented an IRC bot in a few hours in javascript one day. Those classic IETF text based protocols are actually really fun and easy to implement, especially in a language that makes strings safe and easy (i.e. not C). I could easily see figuring out all the deployment concerns around integrating with an existing FTP server end up taking way longer than just integrating the subset needed for this use case.
- recursive 6y ago
  Active mode is pretty weird. Coordinating a single client across two ports sounds difficult to me, but I've never implemented it. If that's not a difficult protocol, then what is?
  - tuwtuwtuwtuw 6y ago
    Http3, IMAP, Caldav and MAPI? Cordinating a client across two ports sounds trivial compared to for example properly implementing client and server versions of IMAP search commands when no client or server follows the specification.
    - pixl97 6y ago
      Until you realize that the two ports commonly involve dealing with middleware that cant handle it properly.
      - tuwtuwtuwtuw 6y ago
dominiek 6y ago
I had considered this, but decided against this for a couple of reasons: - Scaling requirements are relatively low. Even though we're dealing with 10 thousands of devices, the amount of firware updates at a given time to those devices is minimal. Our main scaling challenges are around OCPP over websockets. Story for another day. - I have bad memories of ProFTPd etc buffer overflow exploits. - I wanted something simple that could bridge between FTP and our cloud persistence (MongoDB and Cloud Storage). - I found this Node.js library that I since then forked: https://github.com/autovance/ftp-srv https://github.com/autovance/ftp-srv - The great thing about this library is that it allows a quick implementation of a custom filesystem. - For Kubernetes pods the file system should really be treated as a /tmp - which we are doing. - When a charge station connects, the FTP username/password is a temporary generated set of tokens that is checked against our MongoDB. Essentially, I'm using FTP as a throwaway here. If you think through this you can imagine it would be quite a lift to accomplish this with an existing FTP server.
stjohnswarts 6y ago
They describe why, tons of edge cases from poorly implemented ftp clients, they needed full control over the protocol to handle weird edge cases because of multiple embedded clients.

by winrid 6y ago

meritt 6y ago

Because nodejs kubernetes modern cloud.

dvfjsdhgfv 6y ago
> Because nodejs kubernetes modern cloud. Not necessarily so. The history of FTP servers is ridden by bugs with practically no exceptions. At some point some folks decided they finally implement a bug-free implementation and even dared to call it "Very Secure FTPd." Needless to say, it turned out it has bugs, too. As most of these bugs were related to buffer overflows and similar issues, implementing a new FTP server in a safer language is not such a bad idea, and today's JavaScript is efficient enough to make it a reasonably well-working implementation. I pity the author though for the bugs they encounter and workarounds that will need to be implemented.
- dominiek 6y ago
  Exactly my reasoning. See my comment above for more info. I'm making the maintenance of this less painful by doing a hacking/debugging session with manufacturers once a month where we hook up many devices and fix issues. After addressing most edge cases fewer are coming up now (despite a relentless stream of new cheaply manufactured devices)
- tyingq 6y ago
  I agree, but I don't see why we would then assume that forking some ftp server library from npm would fare any better, security wise. I see a fairly alarming open issue: https://github.com/autovance/ftp-srv/issues/167 https://github.com/autovance/ftp-srv/issues/167
dominiek 6y ago
The main reason for using Node.js is because the rest of our stack is Node.js: https://bedrock.io https://bedrock.io We use MongoDB as persistence and have existing wrappers for dealing with Google Cloud Storage. Since it's an isolated service we could've used a different implementation language. In our case Node.js in our existing Kubernetes environment was the least amount of friction

admax88q 6y ago

That honestly sounds more complicated. FTP isn't that difficult of a protocol, especially if you only need to support one known client, you can take all sort of shortcuts. If you deploy an existing FTP server, and _then_ integrate with it at the filesystem level you now have two components, and your sysadmin requirements grow. Now you gotta administrate an FTP server that's probably written for classic UNIX single server usage, gotta handle filesystem permissions, gotta somehow hook up your distributed locks to the filesystem, sanitize filenames for your chosen filesystem. Honestly filesystems suck, there's so many gotchas from a security perspective, when all you really want is to pipe binary data in this side, and out the other side. I implemented an IRC bot in a few hours in javascript one day. Those classic IETF text based protocols are actually really fun and easy to implement, especially in a language that makes strings safe and easy (i.e. not C). I could easily see figuring out all the deployment concerns around integrating with an existing FTP server end up taking way longer than just integrating the subset needed for this use case.

recursive 6y ago
Active mode is pretty weird. Coordinating a single client across two ports sounds difficult to me, but I've never implemented it. If that's not a difficult protocol, then what is?
- tuwtuwtuwtuw 6y ago
  Http3, IMAP, Caldav and MAPI? Cordinating a client across two ports sounds trivial compared to for example properly implementing client and server versions of IMAP search commands when no client or server follows the specification.
  - pixl97 6y ago
    Until you realize that the two ports commonly involve dealing with middleware that cant handle it properly.
    - tuwtuwtuwtuw 6y ago

dominiek 6y ago

I had considered this, but decided against this for a couple of reasons: - Scaling requirements are relatively low. Even though we're dealing with 10 thousands of devices, the amount of firware updates at a given time to those devices is minimal. Our main scaling challenges are around OCPP over websockets. Story for another day. - I have bad memories of ProFTPd etc buffer overflow exploits. - I wanted something simple that could bridge between FTP and our cloud persistence (MongoDB and Cloud Storage). - I found this Node.js library that I since then forked: https://github.com/autovance/ftp-srv https://github.com/autovance/ftp-srv - The great thing about this library is that it allows a quick implementation of a custom filesystem. - For Kubernetes pods the file system should really be treated as a /tmp - which we are doing. - When a charge station connects, the FTP username/password is a temporary generated set of tokens that is checked against our MongoDB. Essentially, I'm using FTP as a throwaway here. If you think through this you can imagine it would be quite a lift to accomplish this with an existing FTP server.

stjohnswarts 6y ago

They describe why, tons of edge cases from poorly implemented ftp clients, they needed full control over the protocol to handle weird edge cases because of multiple embedded clients.