3 ms·

> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale

> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. Much of US banking operates almost entirely on this premise and has done so forever. > error detection/reporting, business workflows, and/or role-based access control. I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. https://www.nacha.org/newrules https://www.nacha.org/newrules

KPGv2 1y ago
> Much of US banking operates almost entirely on this premise and has done so forever. I might be mistaken, but isn't banking famous for (1) the long hours (i.e., processes suck), (2) the drudgery of updating said Excel files (i.e., processes suck) (3) horribly expensive to access (i.e., processes suck) I have never once in my life as a corporate lawyer thought of banking as a model of operational efficiency.
SoftTalker 1y ago
Yep an early project of mine was building an import of account charges from an FTP file transmission. There was plenty of checking and validation, any records that didn't pass were rejected and the user got notified (I think via email) with the rejected records and the reasons. They could then correct them and resubmit. Granted the only real security was the FTP username and password, but it was all internal and at the time (1990s) that was good enough.
AdieuToLogic 1y ago
>> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. > Much of US banking operates almost entirely on this premise and has done so forever. This is a disingenuous statement as it relates to at least credit/debit/gift card transactions. Bank-to-bank and select high-volume Merchants communicate in certain circumstances with secure file transfers, especially for daily settlements. The vast majority of Merchants do not, and instead rely on secure web applications to authorize, capture, and settle their transactions. Perhaps other banking domains rely on Excel and similar to do business. I cannot speak to those business areas. > I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. And I'd recommend you take a look at different integration - online payment processing using Oribtal as described by Oracle. https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdirect_user_reference/SO04_16.htm https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdire...
frontfor 1y ago
> Much of US banking operates almost entirely on this premise and has done so forever. I've had the same epiphany when I worked for a fintech startup that interacts with financial institutions. Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes.
- AdieuToLogic 1y ago
  > Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support.
  - immibis 1y ago
    It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? More than a few systems do this internally, with some kind of central message queue.
    - AdieuToLogic 1y ago
      >>> Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. >> This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support. > It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. Problem is, there is no self-administration capability when a feed file is the sole transaction mechanism. This can work when the volume is relatively small, say a few dozen customers, or if the Merchant is known to be sophisticated enough. > You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? Not when there are capability requirements beyond batch processing, such as independent transaction research and/or manual corrections. Both are very common needs with Merchants lacking a dedicated system development department. EDIT: Consider a local bar or food truck which accepts credit/debit/gift card payments. Even if they use host capture and a canonical credit card terminal, the Merchant will certainly want to reconcile their receipts with the end-of-day settlement. This is not functionality "CSV/Excel" files and batch processing interfaces can provide.
least 1y ago
That the system works does not mean that individual components of it are advisable. It works because it must and that is with great development effort to keep these interfaces working.
Aurornis 1y ago
> Much of US banking operates almost entirely on this premise and has done so forever. Not literally. Banks have a lot of automations. The reasons they’re not real-time has more to do with various regulations and technicalities of recourse, not because it’s someone doing semi-manual processing of everything at each bank.
NewsaHackO 1y ago
To me, this line of argument is completely backwards. Are you arguing that when people didn't know better, they used flawed systems, or are you actually saying that having a unified method of interaction between users and systems though a web browser app is worse than dark IT excel sheet macros?
thesuitonym 1y ago
Years ago I was working for a company that was having trouble with their payroll processing vendor. As I was helping them out, I was astounded to find that the final step in payroll was to email a CSV file the bank for the ACH transfers. This was a fairly small company (~300 employees) and a national bank, so it strikes me as likely that this is how a lot of companies do business.

KPGv2 1y ago

> Much of US banking operates almost entirely on this premise and has done so forever. I might be mistaken, but isn't banking famous for (1) the long hours (i.e., processes suck), (2) the drudgery of updating said Excel files (i.e., processes suck) (3) horribly expensive to access (i.e., processes suck) I have never once in my life as a corporate lawyer thought of banking as a model of operational efficiency.

SoftTalker 1y ago

Yep an early project of mine was building an import of account charges from an FTP file transmission. There was plenty of checking and validation, any records that didn't pass were rejected and the user got notified (I think via email) with the rejected records and the reasons. They could then correct them and resubmit. Granted the only real security was the FTP username and password, but it was all internal and at the time (1990s) that was good enough.

AdieuToLogic 1y ago

>> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. > Much of US banking operates almost entirely on this premise and has done so forever. This is a disingenuous statement as it relates to at least credit/debit/gift card transactions. Bank-to-bank and select high-volume Merchants communicate in certain circumstances with secure file transfers, especially for daily settlements. The vast majority of Merchants do not, and instead rely on secure web applications to authorize, capture, and settle their transactions. Perhaps other banking domains rely on Excel and similar to do business. I cannot speak to those business areas. > I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. And I'd recommend you take a look at different integration - online payment processing using Oribtal as described by Oracle. https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdirect_user_reference/SO04_16.htm https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdire...

frontfor 1y ago

> Much of US banking operates almost entirely on this premise and has done so forever. I've had the same epiphany when I worked for a fintech startup that interacts with financial institutions. Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes.

AdieuToLogic 1y ago
> Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support.
- immibis 1y ago
  It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? More than a few systems do this internally, with some kind of central message queue.
  - AdieuToLogic 1y ago
    >>> Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. >> This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support. > It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. Problem is, there is no self-administration capability when a feed file is the sole transaction mechanism. This can work when the volume is relatively small, say a few dozen customers, or if the Merchant is known to be sophisticated enough. > You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? Not when there are capability requirements beyond batch processing, such as independent transaction research and/or manual corrections. Both are very common needs with Merchants lacking a dedicated system development department. EDIT: Consider a local bar or food truck which accepts credit/debit/gift card payments. Even if they use host capture and a canonical credit card terminal, the Merchant will certainly want to reconcile their receipts with the end-of-day settlement. This is not functionality "CSV/Excel" files and batch processing interfaces can provide.

least 1y ago

That the system works does not mean that individual components of it are advisable. It works because it must and that is with great development effort to keep these interfaces working.

Aurornis 1y ago

> Much of US banking operates almost entirely on this premise and has done so forever. Not literally. Banks have a lot of automations. The reasons they’re not real-time has more to do with various regulations and technicalities of recourse, not because it’s someone doing semi-manual processing of everything at each bank.

NewsaHackO 1y ago

To me, this line of argument is completely backwards. Are you arguing that when people didn't know better, they used flawed systems, or are you actually saying that having a unified method of interaction between users and systems though a web browser app is worse than dark IT excel sheet macros?

thesuitonym 1y ago

Years ago I was working for a company that was having trouble with their payroll processing vendor. As I was helping them out, I was astounded to find that the final step in payroll was to email a CSV file the bank for the ACH transfers. This was a fairly small company (~300 employees) and a national bank, so it strikes me as likely that this is how a lot of companies do business.