3 ms·

Just curious. Could you explain this part? I used mongodb because it was incredibly easy to create a local cache of the JSON data I was getting from the apis

Just curious. Could you explain this part? I used mongodb because it was incredibly easy to create a local cache of the JSON data I was getting from the apis I was accessing. If you are using the data from the various APIs to render static pages, why do you need the local cache of the JSON data? And if you've got a local cache of all the JSON data, why render static pages rather than serve dynamic pages that reference the DB of cached data?

willholloway 14y ago
Sure, I would be happy to explain. > If you are using the data from the various APIs to render static pages, why do you need the local cache of the JSON data? Because the data that I am displaying for each movie is not available from any single source. streamJoy is the abstraction layer tying together the disparate data sources. Mongo was the right choice because instead of having to map out postgres schema, I could just store the entire JSON response as a dict. What I have learned from all of this is that API's are not always reliable. The information changes, you don't always know what you are going to get back. Mongo just made this early process where I didn't know what I was going to get much more fault tolerant. And, no schema mapping. > And if you've got a local cache of all the JSON data, why render static pages rather than serve dynamic pages that reference the DB of cached data? Rendering static pages allow you to use S3 or nginx to serve your html. For a one man operation like I am running, S3 is manna from heaven for scaling the serving of html files. I'm using nginx right now only because I haven't taken the time to use S3, but S3 is the smarter choice here. My goal with this was to have the smallest possible dynamic server footprint as possible. The other reason for caching json data is I run analysis on the db items, even though I haven't published any of those features yet. By having my own copy of the data, I can run a process on 92000 items in 6 minutes instead of taking a day due to API rate limits.
- derefr 14y ago
  > For a one man operation like I am running, S3 is manna from heaven for scaling the serving of html files. Have you tried putting CloudFlare in front of it? Buttered jelly-roll manna.
espeed 14y ago
You can accomplish the same thing as flat files by using something like Amazon CloudFront (http://aws.amazon.com/cloudfront/ http://aws.amazon.com/cloudfront/) -- CloudFront caches your dynamically generated pages and serves the generated page from cache rather than hitting your server each time. See http://aws.amazon.com/cloudfront/dynamic-content/ http://aws.amazon.com/cloudfront/dynamic-content/

willholloway 14y ago

Sure, I would be happy to explain. > If you are using the data from the various APIs to render static pages, why do you need the local cache of the JSON data? Because the data that I am displaying for each movie is not available from any single source. streamJoy is the abstraction layer tying together the disparate data sources. Mongo was the right choice because instead of having to map out postgres schema, I could just store the entire JSON response as a dict. What I have learned from all of this is that API's are not always reliable. The information changes, you don't always know what you are going to get back. Mongo just made this early process where I didn't know what I was going to get much more fault tolerant. And, no schema mapping. > And if you've got a local cache of all the JSON data, why render static pages rather than serve dynamic pages that reference the DB of cached data? Rendering static pages allow you to use S3 or nginx to serve your html. For a one man operation like I am running, S3 is manna from heaven for scaling the serving of html files. I'm using nginx right now only because I haven't taken the time to use S3, but S3 is the smarter choice here. My goal with this was to have the smallest possible dynamic server footprint as possible. The other reason for caching json data is I run analysis on the db items, even though I haven't published any of those features yet. By having my own copy of the data, I can run a process on 92000 items in 6 minutes instead of taking a day due to API rate limits.

derefr 14y ago
> For a one man operation like I am running, S3 is manna from heaven for scaling the serving of html files. Have you tried putting CloudFlare in front of it? Buttered jelly-roll manna.

espeed 14y ago

You can accomplish the same thing as flat files by using something like Amazon CloudFront (http://aws.amazon.com/cloudfront/ http://aws.amazon.com/cloudfront/) -- CloudFront caches your dynamically generated pages and serves the generated page from cache rather than hitting your server each time. See http://aws.amazon.com/cloudfront/dynamic-content/ http://aws.amazon.com/cloudfront/dynamic-content/