pskomoroch

9 ms·

31.
Building a Data Intensive Web App with Hadoop, Hive, & EC2(cloudera.com)
55 points by pskomoroch 17y ago|14 comments
32.
by pskomoroch 17y ago
Agreed. I see this type of graph all over the place lately, along with versions that have lines connecting the slices. If a story or insight doesn't immediately jump out from your graph, you are doing it wrong.
33.
The Geekdom of Crowds: Gov 2.0 battles Gov 1.0(washingtonmonthly.com)
4 points by pskomoroch 17y ago|0 comments
34.
by pskomoroch 17y ago
In that case, there is nothing really cloud specific here... you just need to build a standard web app or web service that accepts user input and runs some calculations in the background. Any Rails or Django tutorial will be a good start.
35.
by pskomoroch 17y ago
What do you mean by "billed to me"? Do you mean you pay for the costs of user's computations, or users pay you to compute an answer?
36.
by pskomoroch 17y ago
This feature is really underutilized. I render these bootscripts on the fly with ERB to bootstrap clusters on EC2 with a few config vars, the script then determines node type at boot from the security group then pulls full source code from
37.
by pskomoroch 17y ago
more projects here: http://stackoverflow.com/questions/495698/what-resources-are... and here: http://delicious.com/pskomoroch/ab
38.
by pskomoroch 17y ago
I did an analysis on this a while back that told the same story, there is a blog post that includes a nice visualization of the share of travel search categories across the companies mentioned in the article: http://www.juiceanalytics.com/
39.
Making Twitter Talk(henry.sztul.com)
2 points by pskomoroch 17y ago|0 comments
40.
by pskomoroch 18y ago
Distributed, Large-Scale Latent Semantic Analysis by Index Interpolation. Sebastiano Vigna http://tinyurl.com/d99779 [PDF]
41.
by pskomoroch 18y ago
So by this logic, they could raid Amazon's datacenters and seize everything because a few people were using EC2 to send spam?
42.
by pskomoroch 18y ago
reserved instances: http://aws.amazon.com/about-aws/whats-new/2009/03/12/amazon-...
43.
by pskomoroch 18y ago
I'm planning on going into more detail in my next post, walking through some of the Python streaming mapreduce code. Is there anything in particular people want to see?
44.
by pskomoroch 18y ago
Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic
45.
by pskomoroch 18y ago
Here is a direct link to the full code example from the article: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... You will get much better performance out of Hadoop if you do this in Java, but I thought python would be more
46.
by pskomoroch 18y ago
This is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines. Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/inp
47.
by pskomoroch 18y ago
The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... AWS just put a pointer to the code bucket on s3, I'll add a li
48.
Cloudera Launches Hadoop Distribution and Configuration Tools(cloudera.com)
1 points by pskomoroch 18y ago|0 comments
49.
by pskomoroch 18y ago
pete pete@datawrangling.com python, machine learning, statistics
50.
by pskomoroch 18y ago
I'd take problem solving ability and creativity over typing speed any day, but one area where typing speed seems to help is for live coding or instructing others...
51.
by pskomoroch 18y ago
There is a start at it here: http://www.scipy.org/scipy/scikits/wiki/MachineLearning Still some active development, but needs more people: http://projects.scipy.org/scipy/scikits/browser/trunk/learn/...
52.
by pskomoroch 18y ago
I was at the meetup and there were few bits about how Facebook uses R, Hadoop, and python for different levels of data analysis. The guys sitting behind me were talking during the whole presentation which was really annoying... please don'
53.
by pskomoroch 18y ago
http://twitter.com/peteskomoroch mostly tweets on python, machine learning, general data crunching topics.
54.
by pskomoroch 18y ago
I've added static tags to each link which should help a bit, If I can find a way to quickly cluster them together into logical groups, I'll so that as well.
55.
by pskomoroch 18y ago
Yes, I'll take a pass at sorting these into groups this afternoon or at least add more descriptive information. Some of the original link titles are a bit mysterious, like this one: "Voter registration data; or, HERE IS YOUR HOPE, YOU FOOL
56.
by pskomoroch 18y ago
Thanks to whoever submitted this. It is an old post, but I just updated it this week with 230 new dataset links I collected in the last year. The page has about 400 datasets listed now. It also includes a tagged json version for groups li
57.
by pskomoroch 18y ago
I was thinking of doing a basic Django + Selenium + Hudson continuous integration how-to blog post, but this blows me out of the water :)
58.
by pskomoroch 18y ago
Google open sourced one: http://google-opensource.blogspot.com/2008/09/cmockery-easie...
59.
by pskomoroch 18y ago
I have a really tough time with this, and often bounce back and forth between side projects. Erdos was supposedly really good at helping people pick research problems to work on which would have just the right difficulty given their capabil
60.
by pskomoroch 18y ago
My friend Tom is a python hacker and works there, I pointed out the thread... maybe he'll chime in.

More ›