3 ms·
I would not consider it to be an optimal way to process data in volume. HTTP really wasn't built for things like this. You'll need to define in detail how those
by matt_s 5y ago
I would not consider it to be an optimal way to process data in volume. HTTP really wasn't built for things like this. You'll need to define in detail how those batches are operated on, what error conditions mean for a batch, etc. I've found using an API like this to be problematic.
If you want to stay in the HTTP request/response lifecycle like typical API's, your system could validate the entire batch first, which is time consuming, then respond with an appropriate response for your API - some 4xx error for invalid input if say 1 out of 1000 in the batch is invalid. If you accept the batch and begin processing immediately, then your response needs to include the status of each transaction. Either way, you will probably run into some limitation regarding timeouts in your stack (web server, load balancer, other network elements) and how large of a batch you can accept.
An idea to make this easier on your system would be accept the batch, respond with a unique job identifier. They can then check a different API endpoint for status and another endpoint to retrieve results that is paginated. This would allow you to background process large batches, responses can include status per item and you avoid running into time-outs.