5 ms·
"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 su
by ConcernedCoder 8y ago
"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours."
TLDR;
Horrible architecture decisions like this can be very costly.
- noncoml 8y agoI wouldn’t call it architecture decision. It’s the equivalent of a bad SQL query.
- avargas 8y agoA bad sql query wouldn't do that. Look at their site, from the US, it calls cloud functions for a COP to USD conversion, on each place they render a currency (200+ requests just going to their homepage). I think it was built poorly, but that's just my opinion.
- whoisjuan 8y agoI agree with you. There are so many ways to make this infinitely more efficient. For starters, why are they re-calculating and re-rendering the value every time they get a new donation? Also, they could store those values in a more cold&cached-storage and just make the reads to update from Firestore. Don't use a freaking database that is charging you for every single read and write, to deal with mundane client-side renderings.
- Timshel 8y agoChanging the currency will re-trigger all those conversion calls ...
- dragonwriter 8y agoIt's more like a naive database design which doesn't consider the access pattern and therefore requires an inefficient query, which is definitely an architectural issue.
- diziet 8y agoIt's only 460k QPS, with 16k documents everything would be cached really well. Or a single instance of redis can serve that load of reads fairly easily.
- Xorlev 8y agoGood luck with that. A single Redis would not be able to serve that workload. Maybe a single machine but you're really pushing the limits there just with concurrent TCP connections. At that qps Redis has 2 microseconds per request. I agree it caches well but your proposed architecture is definitely not production quality.
- gcbirzan 8y agoThere is a difference between queries and connections. With pipelining, a redis machine can easily serve that. If they stored things in a better structure, like a list, they'd easily be able to get that. On my machine 5 concurrent requests, at 100 items, it can do ~6 500 000 items per second, at 300, 11 000 000 per second, and it kind of caps out at that. Even with 1 concurrent connection, at 600 items, you get 6M per second.
- derptron 8y agoRedislabs has hardware recommendations[1] for production environments. Shouldn't be hard for devs to look it up and implement it. [1]https://redislabs.com/redis-enterprise-documentation/administering/designing-production/hardware-requirements/ https://redislabs.com/redis-enterprise-documentation/adminis...
- thisismydevname 8y agoI don't think they seem aware of the architecture problem, besides they seem proud of having 40 billion request.
- kompiuter 8y agoLegit question here; what would be a good architecture for this case?
- minitech 8y agoSQL with no ORM. Harder to be unaware of what you’re querying when you have to write the queries. Not using attempted magic like Firebase would also fix the problem where the home page transfers 9 MB of data from Firebase on top of the 1 MB JavaScript, which appears to be… their entire database or something?? Accessible to the frontend??? Censored excerpt from that response: "email":{"stringValue":"soXXXXXXXsu@gmail.com"},"fechaDisponible":{"integerValue":"1527483600000"},"identification":{"nullValue":null},"key":{"stringValue":"1527560309061"},"listaNotificaciones":{"arrayValue":{"values":[{"stringValue":"alXXXXXXXXi@yahoo.com"},{"stringValue":"soXXXXXXXsu@gmail.com"},{"stringValue":"pXXXXX2@hotmail.com"},{"stringValue":"BeXXXXXXXXXXXXXXez@hotmail.com"},{"stringValue":"pXXXXXX0@gmail.com"},{"stringValue":"trXXXXXXXXXXro@gmail.com"},{"stringValue":"joXXXXXXXXXXXXie@hotmail.com"},{"stringValue":"ivXXXXXXXXxd@gmail. Also appears to expose, for each campaign, the poster’s bank name and date of birth. And wastes a bunch of various resources making separate requests to a currency conversion service for each amount, as others have noted. And requests /null and /undefined. This might be the most irresponsible development I’ve ever seen.
- gandhium 8y agoExposing contents of DB without even any SQL injections... I think they have no idea about network monitoring. They _have_ an idea about console, since they're logging stuff there extensively.
- derptron 8y agoA caching layer