4 ms·

Based on own experience, there are a lot of systems out there that will only injest certain kinds of data via an FTP drop off. They won’t change or update their

Based on own experience, there are a lot of systems out there that will only injest certain kinds of data via an FTP drop off. They won’t change or update their systems, but everyone who wants better visibility to the data they’re dropping into the ftp system will have to write something to cover that. So, you’d probably need to build both halves of the FTP chain, but each would still need to deliver the value add if only half of the new system was being used.

memn0nis 7y ago
Interesting... would you mind giving me an example of the type of data you are referring to? Or perhaps an example use case / real world example?
- jvagner 7y ago
  Real world example: uploading product, inventory and pricing data to ChannelAdvisor for processing and then republishing out to multiple e-commerce platforms (Amazon, Jet, Rakuten, etc). CA is a big complex powerful system, and my team was uploading FTP files every few minutes. We had the same kind of issues that E2E email delivery and receipt testing would be important for. In our case, we needed to know more about how many of those files were being read, injested, etc. There were dependent processes that could only be triggered if the data had made it into the system. We relied on process scheduling intervals to get close, but knowing more about the state of the ftp pickup itself could’ve improved yields to what we were trying to do.
  - phonon 7y ago
    https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html ?
  - throwGuardian 7y ago
    What does it take for CA to modernise their intake? As a customer, you should really them to change, rather than settle for a band aid around their drawbacks.
    - jvagner 7y ago
      Big public SaaS company... they're not going to change on anything but their own roadmap schedule. They have competitors, and we looked at them, but we had at least two requirements that the newer, smaller, fresher competitors couldn't provide, and we couldn't do without. I'm not involved in this project anymore, so it's not a current conundrum for me, but we had this issue a few quarters ago.
  - zomglings 7y ago
    Do you mean you needed to know the state of processing on the ChannelAdvisor side? Or would you have been able to get this information from FTP access logs? Or could you have done it simply by checking if files existed? I am building something like this for data scientists (currently being used in finance) and wonder if it wouldn't also have served your team well.

memn0nis 7y ago

Interesting... would you mind giving me an example of the type of data you are referring to? Or perhaps an example use case / real world example?

jvagner 7y ago
Real world example: uploading product, inventory and pricing data to ChannelAdvisor for processing and then republishing out to multiple e-commerce platforms (Amazon, Jet, Rakuten, etc). CA is a big complex powerful system, and my team was uploading FTP files every few minutes. We had the same kind of issues that E2E email delivery and receipt testing would be important for. In our case, we needed to know more about how many of those files were being read, injested, etc. There were dependent processes that could only be triggered if the data had made it into the system. We relied on process scheduling intervals to get close, but knowing more about the state of the ftp pickup itself could’ve improved yields to what we were trying to do.
- phonon 7y ago
  https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html ?
- throwGuardian 7y ago
  What does it take for CA to modernise their intake? As a customer, you should really them to change, rather than settle for a band aid around their drawbacks.
  - jvagner 7y ago
    Big public SaaS company... they're not going to change on anything but their own roadmap schedule. They have competitors, and we looked at them, but we had at least two requirements that the newer, smaller, fresher competitors couldn't provide, and we couldn't do without. I'm not involved in this project anymore, so it's not a current conundrum for me, but we had this issue a few quarters ago.
- zomglings 7y ago
  Do you mean you needed to know the state of processing on the ChannelAdvisor side? Or would you have been able to get this information from FTP access logs? Or could you have done it simply by checking if files existed? I am building something like this for data scientists (currently being used in finance) and wonder if it wouldn't also have served your team well.