8 ms·
I had an integrator request this so I stood up a nodeJS server that only implements upload, not download. This way if they leaked their own password, a maliciou
by joshribakoff 9y ago
I had an integrator request this so I stood up a nodeJS server that only implements upload, not download. This way if they leaked their own password, a malicious actor is limited to forging data, and no real data can be leaked. Because it didn't work in FileZilla, they didn't want to use it. Worked at another company that shuffled data between big name gyms & health insurance companies, it also used CSV files sent over FTP in all directions, to my dismay. CSV isn't even a well defined format, and you get all kinds of impedance mismatches with different delimiter & escaping mechanisms, character encodings, BOM, etc. Other companies will just give you a SQL user & let you go to town mining their database directly. I don't understand whats so difficult about making an API, but sometimes it seems like no one wants to do it. You can't push back too much or they will just see you as a problem & decide not to integrate with you.
- pryelluw 9y agoUgh, reminds me of the time I had to integrate with a bunch of school software. They used csv over ftp (no auth). You just had to connect to the server and get the personal information of a district's students. Worse is you could google the URLs... I built a system that solved the problem but left shortly after it began to be integrated nationwide.
- matt_wulfeck 9y agowait.. are you saying that a district had public ftp with personal student information? Pretty sure that breaks the law.
- joshribakoff 9y agoEmail me at josh.ribakoff at Gmail I'd be interested in chatting. I think we worked at the same company. Or direct competitors
- pryelluw 9y agoSure, I'll do.
- tomc1985 9y agoThey already have an API, it's called FTP or SFTP or whatever.
- dcow 9y agoI don't know why everyone thinks they need an API to reimplement (poorly) basic unix functionality that's been around be for years.
- Spivak 9y agoEspecially when FTP and HTTP performance is basically comparable and will depend more on the network and storage than the protocol. Have we gotten to the point where anything not HTTP is considered old?
- digi_owl 9y agonever mind that most "API" are JSON (or some such) delivered over (S)HTTP anyways. But yes, anything but HTTP is "old". In large part because of corporate firewall rules that block anything but HTTP traffic (and more often than not filter even that traffic).
- joshribakoff 9y agoNo. I'm referring to an API to query data in realtime instead of batching the entire freaking database and then having out dated snapshot of the data
- EpicEng 9y ago>Because it didn't work in FileZilla, they didn't want to use it Sounds like they didn't need much if any of what you did. They wanted X and you gave them Y. Y would require them to change the way they worked. Did they need query capability? Seems like they didn't and SFTP would have suited them better. Honestly sounds to me like you went for an overly complex solution where something simple would have done just fine.
- microcolonel 9y agoI did a similar thing, but decided to have LIST/MLSD/NLST, but stub out GET/RETR/MFCT/MFF/MFMT to return nothing or static data.
- joshribakoff 9y agoWhat did you use to implement?
- microcolonel 9y agoWrote it myself because it was Elixir so why not. I'd be more careful doing this in C.
- posguy 9y agoACH needs to die. Its a very old system, and you have no clue if a transfer ever actually goes through. Only ACH rejects ever get sent back, and there isn't a definitive window in which you'll get that reject.
- colejohnson66 9y agoWait. ACH doesn’t even tell the sender if it succeeded??!
- ves 9y agoYeah there are no acks. It sucks.
- saarons 9y agoThe lack of acks is why most companies receiving money through ACH will "season" the transactions for a set amount of time. Most of the returns occur in the first 3 business days so most seasoning periods are just around that long. The reality is you can receive a reject up until (I believe) 60 days after the transaction date. I've seen a lot of companies be a little smarter here and reduce the seasoning times for repeat users using the same source bank account. It is a fascinating system to work with every day. Wire transfers are their own bundles of fun too.
- DaiPlusPlus 9y agoNow I finally know why my credit-union (which serves the high-technology industry exclusively) told me that they couldn't put a hold or otherwise segregate any cheques I deposit until they've cleared... because they have no way of knowing when/if they've cleared!
- wiredfool 9y agoIt's 60 days, + some possibility of up to 2 extra days at the end for Fed Reserve holidays, and then in cases of outright fraud, occasionally banks will ignore the rules and return stuff later than that.
- marktangotango 9y agoFunnily, you're complaining about csv formatting, guess you never worked with ACH formatting. It's cobol fixed length files. I implemented parsing/creation for this file format at two different companies. Hired on at one company, first day, my manager said, "hey, you have experience with ACH right? we have this project...". That's when you learn to start leaving stuff off your resume.
- nvahalik 9y agoNACHA cha cha cha!!
- labster 9y agoYou can't spell Atlach-Nacha without ACH.
- bct 9y agoYou can do a lot worse than ACH. It's hard to read, but it's simple and pretty well-defined. What _really_ sucks is one-off fixed-width formats that aren't well defined, or that change suddenly (oh, you thought that field would always be populated? lol no.)
- HillRat 9y agoYeah, ACH is surprisingly not-unpleasant, at least relative to nightmares like X12 EDI with its implicit looping constructs and billions of companion guides that supercede random parts of the base spec.
- sixdimensional 9y agoX12 EDI, been there earned the badge, both on the generating and receiving sides. 837, 834, 835 and others in health care - fun! Positional format, with situational meaning... really it is a fascinating format.
- 9y ago
- corndoge 9y agowhy use a huge codebase like node for such a simple security critical task
- jlgaddis 9y agoWhen all you have is a hammer...
- joshribakoff 9y agoHow else do you suggest you standup a sftp that cannot leak data even if the password is compromised? I'm all ears. Node is a runtime by the way and huge in what regard? The Linux codebase is also large.
- Karunamon 9y agoFirst thing that comes to my mind is a PAM plugin that generates a new root directory for the user when they log in, named something sane under a fixed hierarchy. A recursive walk can later pick them all up and clean up.
- kalleboo 9y agoCan't you just host a directory with -w- permissions? "Drop box" directories have been a staple of file sharing software for decades.
- Karrot_Kream 9y agoSeconding what kalleboo said, setting directory permissions has been a staple of file-sharing for decades. Setting up a node server to block read-access seems incredibly byzantine.
- joshribakoff 9y agoGood point but we also have to load balance this / have failover and provision users on the fly. I could have used messaging queues to provision PAM users on the whole cluster and this guy would have still complained because I used a non standard port. Not sure why that was a problem either but again we can't press back or question him. he would cancel the whole integration. Bottom line is our end user wanted real time data (presumably via an API)
- voltagex_ 9y agoWouldn't FileZilla have been happy with a fake blank directory response?
- joshribakoff 9y agoYeah I added that but the particular node sftp implementation I used from github just simply didn't work with FileZilla. Only the command line. Don't know why.
- teddyh 9y agoWhenever someone says “from github”, I mentally replace it with “from some random guy on the street” to give it its appropriate intuitive feeling.
- joshribakoff 9y agoSure I don't care who wrote it if it works and passes our code review and the license matches. Plenty of crap software comes out of big name companies and plenty of random people on the street can write good code.
- teddyh 9y agoIf you have no other alternative and can’t take the time to write it for yourself, and need something now, sure, go nuts and take something random off Github, PyPI, npm, or whatever. But code that does not have upstream development and support is dead code. A piece of code in production is like an Internet connection or electrical or water hookup – it must be, so to speak, “connected” upstream to whoever is providing continuous upgrades and security/bug fixes. Otherwise, it is (to continue stretching the analogy) like a mystery battery or water barrel – it could go bad in many ways at any time and you wouldn’t know it until your equipment starts to fail because of low voltage or voltage spikes, or you start to die of legionnaires’ disease. And unlike a battery or water, there are only cursory, no good and thorough, ways to test their quality, and no way to test their age as measured by the way they interact with the ever-changing outside world.
- kccqzy 9y agoReminds me of a time when I had to implement a parser for a certain kind of file to transfer money using an "API" provided by a financial institution in Asia. Yes we are using CSV too, also sent over SFTP. The file format is, unsurprisingly, ill-defined.