4 ms·

Doing the work on the client can sometimes be the right choice, but I'll share a converse point. I was tasked with implementing a bulk record upload feature. Th

Doing the work on the client can sometimes be the right choice, but I'll share a converse point. I was tasked with implementing a bulk record upload feature. The UI was to accept up a CSV file with up to 1 million records. The API provided by the backend team accept max 1,000 records in JSON. So now I have to convert CSV, for which there's no official specification, to JSON, and then make (up to) 1,000 HTTP requests via HTTP 1.1 -- and a different back end team owns rate limiting for all ingress. I end up using a package called 'promise-limit' to ensure only N number of requests are open at a time; firing 1,000 simultaneous XMLHttpRequests is a bad idea. I end up fielding bug reports concerning the translation to JSON for edge cases. And collating the 1,000 network responses for any individual record that failed to persist was something we didn't get to building. So users were uploading 1M records, had some proportion fail, and had 0 visibility into what data the system had, and what data didn't make it. Eventually, we gutted all of that and the backend team added a file upload endpoint. But explaining why this shouldn't be a front end feature, to folks who primarily or exclusively do backend work, it's tough to communicate (1) all the pain points of dealing with the complexity in the front end and (2) the scalability issues. "Can't you just do it on the front end" is such a frequent retort I receive, for things we truthfully should do on the backend.

Chiron1991 6y ago
That's interesting. From my experience, the backend guys are usually more down-to-earth and often save the frontend guys from doing overcomplicated/bad stuff. Not wanting to implement a file upload endpoint for that task sounds suspiciously like some unmotivated guy that doesn't give a sheit about the app.
sitkack 6y ago
Your backend devs are shirking responsibility, the front end shouldn't prep data, it should handle input and do rendering.
josephg 6y ago
The argument to make to the backend team is a bit nuanced but it’s this: Generally each PUT/POST sent to the backend should have atomic semantics. If there’s some user action (like uploading a CSV) that should entirely succeed or entirely fail, you want to send it in a single HTTP request. If it matters that it happens exactly once (like submitting a forum post), then adding a nonce of some sort is a good idea too - just in case the request succeeds but the response gets dropped and the user retries the action. It’s often tempting to model URLs like database entries and send multiple requests to the server to model updating many records based on one action. That’s a mistake - have the client name the action, send one http request, and execute it in a single database transaction in the backend if you need to. This approach is way more reliable, debuggable and consistent. This sort of thing is absolutely the backend’s responsibility.
camgunz 6y ago
If you backend team doesn't hear "I might have to make 1000 requests" and immediately think "oh that probably times out" and "oh do we have to re-batch that data" and immediately make a file upload endpoint, they're committing malpractice. Good grief.
- oliv__ 6y ago
  "committing malpractice" is a very generous way of putting it.

Chiron1991 6y ago

That's interesting. From my experience, the backend guys are usually more down-to-earth and often save the frontend guys from doing overcomplicated/bad stuff. Not wanting to implement a file upload endpoint for that task sounds suspiciously like some unmotivated guy that doesn't give a sheit about the app.

sitkack 6y ago

Your backend devs are shirking responsibility, the front end shouldn't prep data, it should handle input and do rendering.

josephg 6y ago

The argument to make to the backend team is a bit nuanced but it’s this: Generally each PUT/POST sent to the backend should have atomic semantics. If there’s some user action (like uploading a CSV) that should entirely succeed or entirely fail, you want to send it in a single HTTP request. If it matters that it happens exactly once (like submitting a forum post), then adding a nonce of some sort is a good idea too - just in case the request succeeds but the response gets dropped and the user retries the action. It’s often tempting to model URLs like database entries and send multiple requests to the server to model updating many records based on one action. That’s a mistake - have the client name the action, send one http request, and execute it in a single database transaction in the backend if you need to. This approach is way more reliable, debuggable and consistent. This sort of thing is absolutely the backend’s responsibility.

camgunz 6y ago

If you backend team doesn't hear "I might have to make 1000 requests" and immediately think "oh that probably times out" and "oh do we have to re-batch that data" and immediately make a file upload endpoint, they're committing malpractice. Good grief.

oliv__ 6y ago
"committing malpractice" is a very generous way of putting it.