4 ms·

We have the same setup at our place. FTP communication isn't a "clusterfuck", it is the tried and tested communication method between any two large companies th

We have the same setup at our place. FTP communication isn't a "clusterfuck", it is the tried and tested communication method between any two large companies that aren't banks. To us, it is the only highly available and reliable one. Web services have downtimes and re-sends. Our partners practically never turn off their FTP machines. The only difference at our place is that we assume FTP works and do our tests on the file system. We don't test whether FTP works. We know it does.

skrebbel 10y ago
How did you deal with large files, dropped connections, and so on? I mean, sure, FTP the protocol works fine, and so does virtually all FTP software, but for communication it lacks an important feature: the ability to tell when a file completed uploading. When we started to upload reasonably large files on reasonably fast file systems this became problem. Our file watcher noticed that a new file arrived, but it's just started uploading. We tried polling the FTP server for connection status, but it turned out that the client supported and did at time use, resumed uploads on connection drops. We ended up having to parse each file just to tell whether it had finished uploading. The file fortunately was an enormous XML dump, so we could check completeness pretty cheaply. All of this work would've been avoided with a transactional protocol such as HTTP.
- tvjames 10y ago
  I've almost exclusively seen this handled using an upload with a temporary filename (usually convention based) and then renamed on completion. Watch then for the existence of the renamed files. Clean up temp files after a defined amount to time (30 days for example).
  - skrebbel 10y ago
    Ahyes, smart. We didn't control the client, unfortunately, so we couldn't do that. (we tried to communicate with the people who did but they turned out to be the supplier to the supplier to the customer of our customer, 5 countries away, and fixing anything at all was not high on their priority list :D)
    - NicoJuicy 10y ago
      Afterwards change the extension to .xmlok or .xmlfail (alt: zip it), no need for vendor adjustments
  - NicoJuicy 10y ago
    Did that to, even open sourced a client to remove folders depending extension and x days https://github.com/NicoJuicy/CLI-Cleanup-IO https://github.com/NicoJuicy/CLI-Cleanup-IO Ftp was because of legacy though, I would have preferred http post
- nurettin 10y ago
  Not sure what you mean by huge, but our partners frequently upload 100+ MB files in various XML, IATA and EDI formats. We also use an FTP client locally to read the file back. We don't trust the file system state when it is being controlled by a server. I'm guessing that the FTP server does the job of not showing half-finished files, since I've never heard of a half-finished file being read back from an FTP server in 15 years. I may have to validate this assumption, though.
- royjacobs 10y ago
  The benefit of FTP is that it allows you to resume transfers which is especially important with huge files. HTTP is not particularly nice in that regard. It can also be unclear where exactly the HTTP server is storing the intermediate data. If you're not careful it'll try to store the file in-memory before writing it to disk. Like tvjames mentions, you can rename the file once complete. Alternatively you can ask for a .md5 file to be uploaded as well and, if all else fails, you can simply poll for the "last changed" time of a file and assume that if it wasn't touched for 15 minutes it's probably done.
  - skrebbel 10y ago
    I think FTP the protocol would've been fine, and indeed is best suited for the job. The problems really stemmed from having to poll the filesystem because the FTP server was also not under our control. If you embed the FTP server in your app (say, with IKVM ), you can keep the whole intermediate file where you want (or even in RAM if it fits) and this entire problem shrinks 10x.
kodfodrasz 10y ago
I beg to differ. HTTP is far better for any usecase than FTP. I understand that sometimes one has to work because of constraint imposed from parties beyond the control of the project. This may force one to support FTP. But FTP is still a terrible protocol. Uploading files to a directory and processing those from a batch job is not a proper solution to an event driven solution. > To us, it is the only highly available and reliable one. As one who has had some experience with FTP I must say that you should consider reading after the meaning of HA and reliability. > Web services have downtimes and re-sends This is pure bullshit, pardon my French. How does FTP save you from a connection trouble? How does FTP have less downtime? FTP is a stateful protocol, cannot be simply load balanced and made HA. Also idempotency makes some HTTP operations safe to redo if connection troubles arouse. For really huge amounts of data there is BitTorrent.
- nurettin 10y ago
  I don't have a fr-en translator available so I will try to keep it simple for you. FTP does not require restarts or reloads, and lets you continue broken uploads. The only scaling is you can do here is vertical scaling. Sorry if that doesn't fit your world view where every service must be containerized and horizontally scaled. No matter what you do, some ignorant losers will keep on using their single server ftp and call it reliable because that is their experience.

skrebbel 10y ago

How did you deal with large files, dropped connections, and so on? I mean, sure, FTP the protocol works fine, and so does virtually all FTP software, but for communication it lacks an important feature: the ability to tell when a file completed uploading. When we started to upload reasonably large files on reasonably fast file systems this became problem. Our file watcher noticed that a new file arrived, but it's just started uploading. We tried polling the FTP server for connection status, but it turned out that the client supported and did at time use, resumed uploads on connection drops. We ended up having to parse each file just to tell whether it had finished uploading. The file fortunately was an enormous XML dump, so we could check completeness pretty cheaply. All of this work would've been avoided with a transactional protocol such as HTTP.

tvjames 10y ago
I've almost exclusively seen this handled using an upload with a temporary filename (usually convention based) and then renamed on completion. Watch then for the existence of the renamed files. Clean up temp files after a defined amount to time (30 days for example).
- skrebbel 10y ago
  Ahyes, smart. We didn't control the client, unfortunately, so we couldn't do that. (we tried to communicate with the people who did but they turned out to be the supplier to the supplier to the customer of our customer, 5 countries away, and fixing anything at all was not high on their priority list :D)
  - NicoJuicy 10y ago
    Afterwards change the extension to .xmlok or .xmlfail (alt: zip it), no need for vendor adjustments
- NicoJuicy 10y ago
  Did that to, even open sourced a client to remove folders depending extension and x days https://github.com/NicoJuicy/CLI-Cleanup-IO https://github.com/NicoJuicy/CLI-Cleanup-IO Ftp was because of legacy though, I would have preferred http post
nurettin 10y ago
Not sure what you mean by huge, but our partners frequently upload 100+ MB files in various XML, IATA and EDI formats. We also use an FTP client locally to read the file back. We don't trust the file system state when it is being controlled by a server. I'm guessing that the FTP server does the job of not showing half-finished files, since I've never heard of a half-finished file being read back from an FTP server in 15 years. I may have to validate this assumption, though.
royjacobs 10y ago
The benefit of FTP is that it allows you to resume transfers which is especially important with huge files. HTTP is not particularly nice in that regard. It can also be unclear where exactly the HTTP server is storing the intermediate data. If you're not careful it'll try to store the file in-memory before writing it to disk. Like tvjames mentions, you can rename the file once complete. Alternatively you can ask for a .md5 file to be uploaded as well and, if all else fails, you can simply poll for the "last changed" time of a file and assume that if it wasn't touched for 15 minutes it's probably done.
- skrebbel 10y ago
  I think FTP the protocol would've been fine, and indeed is best suited for the job. The problems really stemmed from having to poll the filesystem because the FTP server was also not under our control. If you embed the FTP server in your app (say, with IKVM ), you can keep the whole intermediate file where you want (or even in RAM if it fits) and this entire problem shrinks 10x.

kodfodrasz 10y ago

I beg to differ. HTTP is far better for any usecase than FTP. I understand that sometimes one has to work because of constraint imposed from parties beyond the control of the project. This may force one to support FTP. But FTP is still a terrible protocol. Uploading files to a directory and processing those from a batch job is not a proper solution to an event driven solution. > To us, it is the only highly available and reliable one. As one who has had some experience with FTP I must say that you should consider reading after the meaning of HA and reliability. > Web services have downtimes and re-sends This is pure bullshit, pardon my French. How does FTP save you from a connection trouble? How does FTP have less downtime? FTP is a stateful protocol, cannot be simply load balanced and made HA. Also idempotency makes some HTTP operations safe to redo if connection troubles arouse. For really huge amounts of data there is BitTorrent.

nurettin 10y ago
I don't have a fr-en translator available so I will try to keep it simple for you. FTP does not require restarts or reloads, and lets you continue broken uploads. The only scaling is you can do here is vertical scaling. Sorry if that doesn't fit your world view where every service must be containerized and horizontally scaled. No matter what you do, some ignorant losers will keep on using their single server ftp and call it reliable because that is their experience.