3 ms·

My guess when confronted with the number of lines of Python code for the front page was 5000 SLoC. I lowballed it intentionally because I was under the suspicio

My guess when confronted with the number of lines of Python code for the front page was 5000 SLoC. I lowballed it intentionally because I was under the suspicion that it might be lower than one might normally guess. (Don't we all take pride in keeping our SLoC count as low as we can for any given problem?) The video had some closing points, including "NoSQL sucks for reports", which was explained briefly. OP, if you could, I would like for you to elaborate a bit more on this point. I am not doubting that it's true for Google App Engine, just interested to hear about the particulars.

stickfigure 10y ago
I can speak to this, being a heavy GAE user and formerly a heavy MongoDB user. For a reporting system you want an ad-hoc query language, fast in-database aggregation, and joins. The GAE datastore has none of the three, and MongoDB lacks joins. These are not fatal flaws - the GAE datastore has other advantages like infinite scalability, built-in synchronous geographic replication, failover, zero maintenance, etc. But for analytics, we replicate a subset of data to Postgres. It's still cheaper to have developers write occasional replication code than to hire a DBA to maintain the database, and I never have to worry that an ill-conceived sql statement will create an incident. I've come to the conclusion that using the GAE datastore as primary datastore and replicating to specialized databases is a pretty good architecture for systems that require reliability and low-maintenance. MongoDB's aggregation framework is really painful because the query language is weird and very low-level - you have to do most query planning yourself. And without joins you hit the limits of the kinds of questions you can ask the system very quickly. FWIW, my guess was 10k lines of code.
wodenokoto 10y ago
In the article he writes that it sucks because you end up writing queries in python, which would have been better suited to be written in SQL. I didn't listen to the talk, so I don't know if he mentions it there as well, and you are looking for a more in-depth explanation.

stickfigure 10y ago

I can speak to this, being a heavy GAE user and formerly a heavy MongoDB user. For a reporting system you want an ad-hoc query language, fast in-database aggregation, and joins. The GAE datastore has none of the three, and MongoDB lacks joins. These are not fatal flaws - the GAE datastore has other advantages like infinite scalability, built-in synchronous geographic replication, failover, zero maintenance, etc. But for analytics, we replicate a subset of data to Postgres. It's still cheaper to have developers write occasional replication code than to hire a DBA to maintain the database, and I never have to worry that an ill-conceived sql statement will create an incident. I've come to the conclusion that using the GAE datastore as primary datastore and replicating to specialized databases is a pretty good architecture for systems that require reliability and low-maintenance. MongoDB's aggregation framework is really painful because the query language is weird and very low-level - you have to do most query planning yourself. And without joins you hit the limits of the kinds of questions you can ask the system very quickly. FWIW, my guess was 10k lines of code.

wodenokoto 10y ago

In the article he writes that it sucks because you end up writing queries in python, which would have been better suited to be written in SQL. I didn't listen to the talk, so I don't know if he mentions it there as well, and you are looking for a more in-depth explanation.