4 ms·

Tangent: I saw the "A Design Challenge" that the job submission should should include a document that addresses the issue. As each instance starts, it boots it

Tangent: I saw the "A Design Challenge" that the job submission should should include a document that addresses the issue. As each instance starts, it boots its AMI. That image includes a snapshot of the checked-out eBook repository as of the date the image was created. Each gerbil, as it starts, updates this cache to get the latest version. This involves checking out perhaps 500Mb of data per instance, as new books will have been added and existing books updated since the instance was frozen. As a result, it will often take 30–45 minutes to start 10 instances. Am I off base here thinking that a fairly obvious solution to address the 30-45 minute cold start is to NOT pull the entire library of book templates from svn at startup? Rather, why not simply update just the template(s) currently stacked up in the job queue? So instead of a 500MB pull your looking at maybe 1 or 2 before you can start processing a job in the queue. I wonder if this "challenge" is an existing problem or something they've already addressed. Has relying only on "on demand" cache updating been tried already (seems like it would have been since it seems like an obvious possible solution) if so what was the result?

mquander 15y ago
Maybe this is too pessimistic, but perhaps the challenge is just serving to filter out people who don't know what version control, EC2, or caching are, or who can't form a mental model of the problem based on the short description provided.
- giberson 15y ago
  Perhaps the intent of my comment doesn't convey well. I'm not really trying to judge the quality of this challenge. Actually, I just thought it was interesting and was curious if my initial "obvious" solution would actually provide improvement. Ie, I was trying to ascertain if this was one of those the obvious idea is the worse idea scenarios. Obviously the best solution would be to keep updating the base instance image with new books so it could start up up to date as was suggested in another comment below. But ignoring that possibility what other options can we explore to increase performance? I just genuinely wanted to discuss the problem and find out if its nature was one that got more interesting the harder you looked at it.
yzhengyu 15y ago
Well, while I found the question strange, perhaps it is more of a method to gauge whether the applicant is familiar with tools like cvs/svn/git? A reasonably experienced team developer [perhaps a dev admin guy as well] will probably know how to issue a 'svn update [url]' to obtain specific files [or directory trees] and avoid pulling the entire 500MB repository just to ensure those files are updated.

mquander 15y ago

Maybe this is too pessimistic, but perhaps the challenge is just serving to filter out people who don't know what version control, EC2, or caching are, or who can't form a mental model of the problem based on the short description provided.

giberson 15y ago
Perhaps the intent of my comment doesn't convey well. I'm not really trying to judge the quality of this challenge. Actually, I just thought it was interesting and was curious if my initial "obvious" solution would actually provide improvement. Ie, I was trying to ascertain if this was one of those the obvious idea is the worse idea scenarios. Obviously the best solution would be to keep updating the base instance image with new books so it could start up up to date as was suggested in another comment below. But ignoring that possibility what other options can we explore to increase performance? I just genuinely wanted to discuss the problem and find out if its nature was one that got more interesting the harder you looked at it.

yzhengyu 15y ago

Well, while I found the question strange, perhaps it is more of a method to gauge whether the applicant is familiar with tools like cvs/svn/git? A reasonably experienced team developer [perhaps a dev admin guy as well] will probably know how to issue a 'svn update [url]' to obtain specific files [or directory trees] and avoid pulling the entire 500MB repository just to ensure those files are updated.