5 ms·

80legs Web-Scale Apps Competition

21 points by jdrock 17y ago

vicaya 17y ago
Instead of a thinly veiled advertisement, how about some information on the capability of the hardware: histograms of RAM size, hard drives, CPU units (equivalents of some known CPU) and network speed/latency for these "50,000" machines? Without a reasonable description of hardware, any ideas would be just hand waving.
- jdrock 17y ago
  The 50,000+ computers come from a distributed grid computing system that's similar to a commercial opt-in SETI@home. Because of that, we don't know exact specs on the computers, but here are the limitations that 80Apps have: http://80legs.pbworks.com/80Apps#LimitationsonYourCode http://80legs.pbworks.com/80Apps#LimitationsonYourCode
  - vicaya 17y ago
    Well, do you have something like this?: http://setiathome.berkeley.edu/top_hosts.php http://setiathome.berkeley.edu/top_hosts.php BTW, how do you ensure somebody is not submitting a malware (intentional or not) that can DDoS someone by crawling the same host at pretty much the same time?
    - jdrock 17y ago
      I know Plura uses IP information to prevent bandwidth overages, but I don't think they have a list of top hosts. 80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.
- dotBen 17y ago
  The problem is that 80legs works to a trojan/malware type model - the runtime is installed as a background service on apps like Digsby. It's fair to say that many people who are using these apps don't know that their machine is being given tasks to work on in the background. Aside from the unknown bandwidth/cpu/throughput unknowns IMHO there is an ethical question of using this software. From what I can see there is no opt-out other than not to install the software the runtime is bundled with. And that assumes you even know it's being installed.
  - jdrock 17y ago
    Actually all Digsby users are told up-front during the install process about Plura. And you can easily turn off Plura from Digsby.
    - dotBen 17y ago
      Hmmmm. I would respectfully disagree. I'm somewhat conflicted as I have interests in a rival product in the twitter app space but IMHO it's not clear to your "average Joe" user what is going on. Further more, from my own experience, once you tell someone who's using an app that has Plura running what it is doing, people tend to be quite concerned and certainly not wanting to have it on their system. JDRock: I also believe you work with/for 80legs but can't see any disclosure in your posts. Not wanting to be a d*ck but if you are going to say that it's obvious and easy to turn off (and de-karma me for my comment) I think you should state whether that's from a NPOV or whether you're actually involved in the product.
      - jdrock 17y ago
        I think it's fairly obvious from my posts that I work for 80legs. If you try out the latest Digsby installation, you'll see that they've changed the install process. So while I suppose I am biased, it's also a fairly objective thing to observe. Lifehacker even posted a follow-up article applauding Digsby for the change, if I remember correctly. What I've seen is that if people are told about Plura up-front and have the option to turn it off from the get-go, they have no qualms about it. Digsby didn't do an awesome job about this initially, but they've corrected it. Also, I wasn't the only that d-ked you.
jlees 17y ago
I'd love to enter this if it wasn't just Java. I understand why, but it's still an unfortunate hurdle!
- elcron 17y ago
  Where does it say java? Afaik you can use any language that compiles to the jvm so languages that compile to it (e.g clojure) are fine. Not as much choice, but better than just java.
  - jdrock 17y ago
    We're exploring the use of other languages that work in the JVM, but for now it's just Java. The problem with other languages is that they all make use of reflection to get themselves working, which Plura's security sandbox does not allow. If anyone knows a way around this, please share!

21 points by jdrock 17y ago

Instead of a thinly veiled advertisement, how about some information on the capability of the hardware: histograms of RAM size, hard drives, CPU units (equivalents of some known CPU) and network speed/latency for these "50,000" machines? Without a reasonable description of hardware, any ideas would be just hand waving.

jdrock 17y ago
The 50,000+ computers come from a distributed grid computing system that's similar to a commercial opt-in SETI@home. Because of that, we don't know exact specs on the computers, but here are the limitations that 80Apps have: http://80legs.pbworks.com/80Apps#LimitationsonYourCode http://80legs.pbworks.com/80Apps#LimitationsonYourCode
- vicaya 17y ago
  Well, do you have something like this?: http://setiathome.berkeley.edu/top_hosts.php http://setiathome.berkeley.edu/top_hosts.php BTW, how do you ensure somebody is not submitting a malware (intentional or not) that can DDoS someone by crawling the same host at pretty much the same time?
  - jdrock 17y ago
    I know Plura uses IP information to prevent bandwidth overages, but I don't think they have a list of top hosts. 80legs enforces throttles and rate limits on the sites that are crawled. We try to make a best guess at a rate limit for each site. Sometimes that best guess is too fast, so we take corrective action when we're notified by the webmaster. Users can't override the crawling rates.
dotBen 17y ago
The problem is that 80legs works to a trojan/malware type model - the runtime is installed as a background service on apps like Digsby. It's fair to say that many people who are using these apps don't know that their machine is being given tasks to work on in the background. Aside from the unknown bandwidth/cpu/throughput unknowns IMHO there is an ethical question of using this software. From what I can see there is no opt-out other than not to install the software the runtime is bundled with. And that assumes you even know it's being installed.
- jdrock 17y ago
  Actually all Digsby users are told up-front during the install process about Plura. And you can easily turn off Plura from Digsby.
  - dotBen 17y ago
    Hmmmm. I would respectfully disagree. I'm somewhat conflicted as I have interests in a rival product in the twitter app space but IMHO it's not clear to your "average Joe" user what is going on. Further more, from my own experience, once you tell someone who's using an app that has Plura running what it is doing, people tend to be quite concerned and certainly not wanting to have it on their system. JDRock: I also believe you work with/for 80legs but can't see any disclosure in your posts. Not wanting to be a d*ck but if you are going to say that it's obvious and easy to turn off (and de-karma me for my comment) I think you should state whether that's from a NPOV or whether you're actually involved in the product.
    - jdrock 17y ago
      I think it's fairly obvious from my posts that I work for 80legs. If you try out the latest Digsby installation, you'll see that they've changed the install process. So while I suppose I am biased, it's also a fairly objective thing to observe. Lifehacker even posted a follow-up article applauding Digsby for the change, if I remember correctly. What I've seen is that if people are told about Plura up-front and have the option to turn it off from the get-go, they have no qualms about it. Digsby didn't do an awesome job about this initially, but they've corrected it. Also, I wasn't the only that d-ked you.

jlees 17y ago

I'd love to enter this if it wasn't just Java. I understand why, but it's still an unfortunate hurdle!

elcron 17y ago
Where does it say java? Afaik you can use any language that compiles to the jvm so languages that compile to it (e.g clojure) are fine. Not as much choice, but better than just java.
- jdrock 17y ago
  We're exploring the use of other languages that work in the JVM, but for now it's just Java. The problem with other languages is that they all make use of reflection to get themselves working, which Plura's security sandbox does not allow. If anyone knows a way around this, please share!