jdrock

22 ms·

181.
by jdrock 17y ago
FB should be more difficult as well because they aren't just showing streaming information. They're making (supposedly) intelligent decisions on what information to show you. Twitter is just showing you a "dumb" stream.
182.
by jdrock 17y ago
We've found Java's regular expression capabilities to be a bit frustrating at times. Although they're easy to use, they can be very slow for certain types of regexes. Does anyone know of a "fast regex" class?
183.
Our launch experience: beta to DEMO to post-revenue(80legs.wordpress.com)
5 points by jdrock 17y ago|1 comments
184.
by jdrock 17y ago
Maybe this demo reel makes sense to their target customers, but it made very little sense to me. Not sure if it really tells a story to members of the general population. Also, I personally felt that the "eye candy" wasn't so impressive.
185.
by jdrock 17y ago
You're right that the data center and the cloud will be very similar, but our assertion at 80legs is that both are very poor choices. I'm not arguing that it's impossible to do crawling on the cloud. I'm saying it's near-impossible to do
186.
by jdrock 17y ago
So a few points (I assume you're talking about the Elastic IPs): 1. Yes, each instance can have its own IP, but by default, each account is limited to 5 IP addresses. 2. You can increase your limit, but my guess is that it's difficult to do
187.
by jdrock 17y ago
The $4/million actually is not based on compute time, it's based on data transfer in/out of AWS. If I included compute time, it would be higher. And it's the bandwidth aspect that makes web crawling not feasible on AWS. Yes, you have a fe
188.
by jdrock 17y ago
The crawl runs on a super-heterogenous network of computers from around the world. The JVM sandbox is the only thing we know. Java is the only language that has something like this. Yes, Java isn't as sexy anymore. But running your code
189.
by jdrock 17y ago
Because of the way 80legs handles crawls, users don't need to worry about loop detection. There are really two issues here: 1. For each user crawl, we only allow the same url once, so any simple loops that involve the same urls are elimina
190.
by jdrock 17y ago
We are looking at storing data, doing pre-done crawls, offering data feeds, etc. It's something on the horizon for us. Hopefully guys that write parselets will be interested in becoming developers for the upcoming 80legs App Store. They'l
191.
by jdrock 17y ago
We're not using EC2. AWS doesn't and can't scale to our size, actually :) Side note: AWS guys have asked if we use them. We said "No, we'd be losing money then and wouldn't be able to scale to our size with you."
192.
by jdrock 17y ago
While complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need severa
193.
by jdrock 17y ago
Just a quick heads up - we'll be looking into allowing JRuby and Jython fairly soon for when you create 80Apps. You'll also be able to mashup third-party apps and libraries into your own code. This is something we hope to have in 2-3 month
194.
by jdrock 17y ago
The cost savings are still there with reserved instances if you do the math. Actually, the cost is not the biggest issue with the cloud. If you're talking about large-scale crawling, AWS will not adequately scale. You can't get enough nod
195.
by jdrock 17y ago
I feel this is problem systemic to any company funded by a seed incubator. Solving really hard problems or developing truly breakthrough technology typically requires significant funding, which seed incubators don't have. Companies solvin
196.
by jdrock 17y ago
Maybe the population in the South will increase as a result?
197.
by jdrock 17y ago
If you're trying to put together a business plan, investors will want to hear your "total addressable/available market", which means what's the most $ your company, specifically, could make. TAM is size of your company's market * price of 1
198.
by jdrock 17y ago
I like these ideas.. ultimately you want people that are interested in becoming productive, value-adding members of society, whether that be genius tech founders or hard-working floor moppers. The challenge is finding the right mix of obj
199.
by jdrock 17y ago
Agreed on the CS majors thing.. sounds completely arbitrary that somehow CS majors are more valuable than other majors.
200.
by jdrock 17y ago
I would feel uncomfortable giving recognizable VC firms a competitive advantage in this way. If KP gets to select who gets into the US, chances are they will have a lead on getting to invest in those people. That said, I also don't like th
201.
by jdrock 17y ago
I would take this collection over a Kindle any day :)
202.
Neil Gaiman's Bookshelves(blog.shelfari.com)
1 points by jdrock 17y ago|1 comments
203.
by jdrock 17y ago
..or I could work hard, get my startup to exit, and travel as much as I want for the many years in life afterward. Why does travel have to be done when I'm in my 20s/30s/etc.?
204.
by jdrock 17y ago
My first thought was "Xbox Achievements for the Internet". This may be a good 1-sentence elevator pitch for certain audiences. Definitely think working with the partner sites to make the end-user experience as transparent as possible is th
205.
by jdrock 17y ago
At scale, this site/idea could theoretically "own" the reputation market. There's definite value in that.
206.
by jdrock 17y ago
It's his past/future self stealing his book report.
207.
by jdrock 17y ago
So most of the ideas mentioned here don't really consider the costs involved with making a search engine. New technologies/concepts/ideas are great and all, but unless you're building a very niche/vertical search engine, you're going to re
208.
by jdrock 17y ago
Let me clarify.. How would you know how to value votes for different queries? Certain pages are more relevant to certain queries more so than others, and likewise for votes. Your approach doesn't have a way of accounting for this. And im
209.
by jdrock 17y ago
This approach would be suitable for "mainstream" search results, but wouldn't be able to serve the long tail. Or to put it more humorously, you wouldn't be able to find the earrings because only you (and no one else) were interested in them
210.
by jdrock 17y ago
For what it's worth, we actually are helping new search engines scale their capabilities. ..and not just on crawling (or indexing, as most people confuse it), but also on applying their search technology. There are a ton of really cool sea

More ›