3 ms·

Something to simplify FTP. When you partner with a business in b2b most data is delivered through sftp channels. Make a tool that vastly simplifies that. Be i

Something to simplify FTP. When you partner with a business in b2b most data is delivered through sftp channels. Make a tool that vastly simplifies that. Be it onboarding, offboarding, alerting, emailing, the works.

memn0nis 7y ago
How is this different than file sharing (e.g., Dropbox)?
- jvagner 7y ago
  Based on own experience, there are a lot of systems out there that will only injest certain kinds of data via an FTP drop off. They won’t change or update their systems, but everyone who wants better visibility to the data they’re dropping into the ftp system will have to write something to cover that. So, you’d probably need to build both halves of the FTP chain, but each would still need to deliver the value add if only half of the new system was being used.
  - memn0nis 7y ago
    Interesting... would you mind giving me an example of the type of data you are referring to? Or perhaps an example use case / real world example?
    - jvagner 7y ago
      Real world example: uploading product, inventory and pricing data to ChannelAdvisor for processing and then republishing out to multiple e-commerce platforms (Amazon, Jet, Rakuten, etc). CA is a big complex powerful system, and my team was uploading FTP files every few minutes. We had the same kind of issues that E2E email delivery and receipt testing would be important for. In our case, we needed to know more about how many of those files were being read, injested, etc. There were dependent processes that could only be triggered if the data had made it into the system. We relied on process scheduling intervals to get close, but knowing more about the state of the ftp pickup itself could’ve improved yields to what we were trying to do.
      - phonon 7y ago
        https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html ?
      - throwGuardian 7y ago
        What does it take for CA to modernise their intake? As a customer, you should really them to change, rather than settle for a band aid around their drawbacks.
        jvagner 7y ago
        Big public SaaS company... they're not going to change on anything but their own roadmap schedule. They have competitors, and we looked at them, but we had at least two requirements that the newer, smaller, fresher competitors couldn't provide, and we couldn't do without. I'm not involved in this project anymore, so it's not a current conundrum for me, but we had this issue a few quarters ago.
      - zomglings 7y ago
        Do you mean you needed to know the state of processing on the ChannelAdvisor side? Or would you have been able to get this information from FTP access logs? Or could you have done it simply by checking if files existed? I am building something like this for data scientists (currently being used in finance) and wonder if it wouldn't also have served your team well.
masonhensley 7y ago
Yep, been persistently about 3-5 down from the top of my idea list for a while Some notes: * facilitation of non technical ops people (sales, client success, etc) * automates status reporting to stake holders (did we get the data yet?) * as part of status reporting, ability to mark data as QA’d / next steps * how to hand off data to internal team * on prem in your cluster (think regulated industries) * notification/dashboard of file arrivals (think did weekly file come from partner X?) Probably some big enterprise tool out there that does all this. Always wished there was an open source “sidecar” to a sftp server that would help lighten the load of random communication overhead of moving csv’s or other flat files back and forth between clients and keeping non technical people updated.

memn0nis 7y ago

How is this different than file sharing (e.g., Dropbox)?

jvagner 7y ago
Based on own experience, there are a lot of systems out there that will only injest certain kinds of data via an FTP drop off. They won’t change or update their systems, but everyone who wants better visibility to the data they’re dropping into the ftp system will have to write something to cover that. So, you’d probably need to build both halves of the FTP chain, but each would still need to deliver the value add if only half of the new system was being used.
- memn0nis 7y ago
  Interesting... would you mind giving me an example of the type of data you are referring to? Or perhaps an example use case / real world example?
  - jvagner 7y ago
    Real world example: uploading product, inventory and pricing data to ChannelAdvisor for processing and then republishing out to multiple e-commerce platforms (Amazon, Jet, Rakuten, etc). CA is a big complex powerful system, and my team was uploading FTP files every few minutes. We had the same kind of issues that E2E email delivery and receipt testing would be important for. In our case, we needed to know more about how many of those files were being read, injested, etc. There were dependent processes that could only be triggered if the data had made it into the system. We relied on process scheduling intervals to get close, but knowing more about the state of the ftp pickup itself could’ve improved yields to what we were trying to do.
    - phonon 7y ago
      https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html https://docs.pyfilesystem.org/en/latest/reference/ftpfs.html ?
    - throwGuardian 7y ago
      What does it take for CA to modernise their intake? As a customer, you should really them to change, rather than settle for a band aid around their drawbacks.
      - jvagner 7y ago
        Big public SaaS company... they're not going to change on anything but their own roadmap schedule. They have competitors, and we looked at them, but we had at least two requirements that the newer, smaller, fresher competitors couldn't provide, and we couldn't do without. I'm not involved in this project anymore, so it's not a current conundrum for me, but we had this issue a few quarters ago.
    - zomglings 7y ago
      Do you mean you needed to know the state of processing on the ChannelAdvisor side? Or would you have been able to get this information from FTP access logs? Or could you have done it simply by checking if files existed? I am building something like this for data scientists (currently being used in finance) and wonder if it wouldn't also have served your team well.

masonhensley 7y ago

Yep, been persistently about 3-5 down from the top of my idea list for a while Some notes: * facilitation of non technical ops people (sales, client success, etc) * automates status reporting to stake holders (did we get the data yet?) * as part of status reporting, ability to mark data as QA’d / next steps * how to hand off data to internal team * on prem in your cluster (think regulated industries) * notification/dashboard of file arrivals (think did weekly file come from partner X?) Probably some big enterprise tool out there that does all this. Always wished there was an open source “sidecar” to a sftp server that would help lighten the load of random communication overhead of moving csv’s or other flat files back and forth between clients and keeping non technical people updated.