3 ms·

> I've discovered that when you start getting really cynical about the actual need for a web application - especially in B2B SaaS - you may become surprised at

> I've discovered that when you start getting really cynical about the actual need for a web application - especially in B2B SaaS - you may become surprised at how far you can take the business without touching a browser. Enabling customer self administration/configuration of a "B2B SaaS" system mandates some form of interaction. I would be surprised at how many would not expect "touching a browser" to do so. > A vast majority of the hours I've spent building web sites & applications has been devoted to administrative-style UI/UX wherein we are ultimately giving the admin a way to mutate fields in a database somewhere such that the application behaves to the customer's expectations. If there is no validation of information given nor system rules enforcement, then I would question the correctness of said sites/applications. > In many situations, it is clearly 100x faster/easier/less bullshit to send the business a template of the configuration (Excel files) and then load+merge their results directly into the same SQL tables. This approach obviates the value added by a system enforcing core software engineering concepts such as fitness of purpose, error detection/reporting, business workflows, and/or role-based access control. In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point.

bob1029 1y ago
> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. Much of US banking operates almost entirely on this premise and has done so forever. > error detection/reporting, business workflows, and/or role-based access control. I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. https://www.nacha.org/newrules https://www.nacha.org/newrules
- KPGv2 1y ago
  > Much of US banking operates almost entirely on this premise and has done so forever. I might be mistaken, but isn't banking famous for (1) the long hours (i.e., processes suck), (2) the drudgery of updating said Excel files (i.e., processes suck) (3) horribly expensive to access (i.e., processes suck) I have never once in my life as a corporate lawyer thought of banking as a model of operational efficiency.
- SoftTalker 1y ago
  Yep an early project of mine was building an import of account charges from an FTP file transmission. There was plenty of checking and validation, any records that didn't pass were rejected and the user got notified (I think via email) with the rejected records and the reasons. They could then correct them and resubmit. Granted the only real security was the FTP username and password, but it was all internal and at the time (1990s) that was good enough.
- AdieuToLogic 1y ago
  >> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. > Much of US banking operates almost entirely on this premise and has done so forever. This is a disingenuous statement as it relates to at least credit/debit/gift card transactions. Bank-to-bank and select high-volume Merchants communicate in certain circumstances with secure file transfers, especially for daily settlements. The vast majority of Merchants do not, and instead rely on secure web applications to authorize, capture, and settle their transactions. Perhaps other banking domains rely on Excel and similar to do business. I cannot speak to those business areas. > I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. And I'd recommend you take a look at different integration - online payment processing using Oribtal as described by Oracle. https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdirect_user_reference/SO04_16.htm https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdire...
- frontfor 1y ago
  > Much of US banking operates almost entirely on this premise and has done so forever. I've had the same epiphany when I worked for a fintech startup that interacts with financial institutions. Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes.
  - AdieuToLogic 1y ago
    > Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support.
    - immibis 1y ago
      It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? More than a few systems do this internally, with some kind of central message queue.
      - AdieuToLogic 1y ago
        >>> Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. >> This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support. > It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. Problem is, there is no self-administration capability when a feed file is the sole transaction mechanism. This can work when the volume is relatively small, say a few dozen customers, or if the Merchant is known to be sophisticated enough. > You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? Not when there are capability requirements beyond batch processing, such as independent transaction research and/or manual corrections. Both are very common needs with Merchants lacking a dedicated system development department. EDIT: Consider a local bar or food truck which accepts credit/debit/gift card payments. Even if they use host capture and a canonical credit card terminal, the Merchant will certainly want to reconcile their receipts with the end-of-day settlement. This is not functionality "CSV/Excel" files and batch processing interfaces can provide.
- least 1y ago
  That the system works does not mean that individual components of it are advisable. It works because it must and that is with great development effort to keep these interfaces working.
- Aurornis 1y ago
  > Much of US banking operates almost entirely on this premise and has done so forever. Not literally. Banks have a lot of automations. The reasons they’re not real-time has more to do with various regulations and technicalities of recourse, not because it’s someone doing semi-manual processing of everything at each bank.
- NewsaHackO 1y ago
  To me, this line of argument is completely backwards. Are you arguing that when people didn't know better, they used flawed systems, or are you actually saying that having a unified method of interaction between users and systems though a web browser app is worse than dark IT excel sheet macros?
- thesuitonym 1y ago
  Years ago I was working for a company that was having trouble with their payroll processing vendor. As I was helping them out, I was astounded to find that the final step in payroll was to email a CSV file the bank for the ACH transfers. This was a fairly small company (~300 employees) and a national bank, so it strikes me as likely that this is how a lot of companies do business.
zie 1y ago
Generally in the excel load process, one does sanity checks. However your database can do all the checking if you so wish. CHECK() gets you quite far. If you have PostgreSQL(PG) or some other advanced SQL database, you can even go forth with functions and views and triggers and everything else. We execute Python in our PG database and do testing through PGTAP.

bob1029 1y ago

> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. Much of US banking operates almost entirely on this premise and has done so forever. > error detection/reporting, business workflows, and/or role-based access control. I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. https://www.nacha.org/newrules https://www.nacha.org/newrules

KPGv2 1y ago
> Much of US banking operates almost entirely on this premise and has done so forever. I might be mistaken, but isn't banking famous for (1) the long hours (i.e., processes suck), (2) the drudgery of updating said Excel files (i.e., processes suck) (3) horribly expensive to access (i.e., processes suck) I have never once in my life as a corporate lawyer thought of banking as a model of operational efficiency.
SoftTalker 1y ago
Yep an early project of mine was building an import of account charges from an FTP file transmission. There was plenty of checking and validation, any records that didn't pass were rejected and the user got notified (I think via email) with the rejected records and the reasons. They could then correct them and resubmit. Granted the only real security was the FTP username and password, but it was all internal and at the time (1990s) that was good enough.
AdieuToLogic 1y ago
>> In short, while letting customers send Excel files "and then load+merge their results directly into the same SQL tables" might sound nice, this does not scale and will certainly result in a failure state at some point. > Much of US banking operates almost entirely on this premise and has done so forever. This is a disingenuous statement as it relates to at least credit/debit/gift card transactions. Bank-to-bank and select high-volume Merchants communicate in certain circumstances with secure file transfers, especially for daily settlements. The vast majority of Merchants do not, and instead rely on secure web applications to authorize, capture, and settle their transactions. Perhaps other banking domains rely on Excel and similar to do business. I cannot speak to those business areas. > I'd take a look at the Nacha (ACH) operating rules if you have any doubt that sophisticated business workflows can be built on top of flat files and asynchronous transmission. And I'd recommend you take a look at different integration - online payment processing using Oribtal as described by Oracle. https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdirect_user_reference/SO04_16.htm https://docs.oracle.com/cd/E69185_01/cwdirect/pdf/180/cwdire...
frontfor 1y ago
> Much of US banking operates almost entirely on this premise and has done so forever. I've had the same epiphany when I worked for a fintech startup that interacts with financial institutions. Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes.
- AdieuToLogic 1y ago
  > Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support.
  - immibis 1y ago
    It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? More than a few systems do this internally, with some kind of central message queue.
    - AdieuToLogic 1y ago
      >>> Having a website just isn't necessary for some of the day-to-day operations vs just sending CSV/Excel files back and forth for reconciliation, settlement, accounting purposes. >> This doesn't end well when there are hundreds of thousands of "reconciliation, settlement, accounting purposes" to support. > It kind of does. You still have all the same backend logic, but now it doesn't have to run concurrently with a user interface. Problem is, there is no self-administration capability when a feed file is the sole transaction mechanism. This can work when the volume is relatively small, say a few dozen customers, or if the Merchant is known to be sophisticated enough. > You can take a queue of operations, process operations off the queue and send their results back. Isn't that a pretty reasonable way for a system to behave? Not when there are capability requirements beyond batch processing, such as independent transaction research and/or manual corrections. Both are very common needs with Merchants lacking a dedicated system development department. EDIT: Consider a local bar or food truck which accepts credit/debit/gift card payments. Even if they use host capture and a canonical credit card terminal, the Merchant will certainly want to reconcile their receipts with the end-of-day settlement. This is not functionality "CSV/Excel" files and batch processing interfaces can provide.
least 1y ago
That the system works does not mean that individual components of it are advisable. It works because it must and that is with great development effort to keep these interfaces working.
Aurornis 1y ago
> Much of US banking operates almost entirely on this premise and has done so forever. Not literally. Banks have a lot of automations. The reasons they’re not real-time has more to do with various regulations and technicalities of recourse, not because it’s someone doing semi-manual processing of everything at each bank.
NewsaHackO 1y ago
To me, this line of argument is completely backwards. Are you arguing that when people didn't know better, they used flawed systems, or are you actually saying that having a unified method of interaction between users and systems though a web browser app is worse than dark IT excel sheet macros?
thesuitonym 1y ago
Years ago I was working for a company that was having trouble with their payroll processing vendor. As I was helping them out, I was astounded to find that the final step in payroll was to email a CSV file the bank for the ACH transfers. This was a fairly small company (~300 employees) and a national bank, so it strikes me as likely that this is how a lot of companies do business.

zie 1y ago

Generally in the excel load process, one does sanity checks. However your database can do all the checking if you so wish. CHECK() gets you quite far. If you have PostgreSQL(PG) or some other advanced SQL database, you can even go forth with functions and views and triggers and everything else. We execute Python in our PG database and do testing through PGTAP.