4 ms·

He's describing something that was the biggest factor for me not deciding to go back to school. I came to Google and started working on search engine spam. I al

He's describing something that was the biggest factor for me not deciding to go back to school. I came to Google and started working on search engine spam. I also periodically read papers on "Adversarial Information Retrieval" as it is called in the literature. Only a handful of the papers I read had useful ideas, and I don't think there's anything that can really be done to improve the situation. There are certain things you can't really research without access to the real thing. Information Retrieval is one of them, and I'd be equally skeptical of parallel computing research unless the university has a 1000 node cluster to work with. This sucks for the world, because the research in these things that Google does doesn't really escape to advance the state of the art elsewhere.

kermit_de_fro 16y ago
Minor picky point, but universities have access to such computing power via time-sharing at national labs if it is not available from their own department.
- moultano 16y ago
  And I do think that a lot of good parallel computing research gets done. It's just a lot harder.
Lewisham 16y ago
Sadly, most academics realize that they're data-starved, but industry isn't able to/doesn't want to work with academia to help gain access to that data. Even if you do get access, you're NDA'd so hard that publishing useful papers is a challenge. It's a shame, because universities are essentially free labor, just on much longer timeframes and lower probability of immediate success. I understand why things are the way they are, but I'd love to find some way of reconciling this issue. It's already killed a research project of mine before it started.
- moultano 16y ago
  Google realizes this too, and there are a lot of people here that would like to help. There are a few huge issues that always come up. 1. Paranoia about giving competitors the edge they need. 2. Paranoia about giving spammers the edge they need. 3. Privacy. It's nigh impossible to anonymize user data enough to release and still have it in a useful form. The AOL logs debacle hurt everyone. One of my friends had a really interesting take on this. His belief essentially was that the industry advances when people switch companies and (legal or not) take their institutional knowledge with them. Unfortunately, academia is hard to transfer into.
  - price 16y ago
    <quote>His belief essentially was that the industry advances when people switch companies and (legal or not) take their institutional knowledge with them.</quote> This is why it's so important that in California the law protects employees leaving companies to work for other companies -- this is legal there. You can't take code, data, hardware designs, or other concrete intellectual property -- that's forbidden by the agreements every tech company requires employees to sign. But if you couldn't take institutional knowledge with you, you couldn't work at a new company at all. (How could an ex-Googler unlearn how web-scale systems are built of many disposable pieces?) And if an agreement purports to restrict you from switching jobs, then California law repudiates that restriction. Unfortunately some other states have no such law, and cheerfully enforce non-competes against engineers trying to switch companies. And guess what? The industry doesn't move as fast in Massachusetts as in California.
    - kevinpet 16y ago
      If I could point to one thing other states could do to improve their economy, it would be adopt the exact language of California's anti-non-compete and the "you own your own ideas, unless developed as part of your work" law. Not that this is going to single handedly grow a silicon valley in Montana overnight, but it has so little downside. The reason for the exact language is because if they use the exact language consciously, they effectively import the case law surrounding those laws.
- edge17 16y ago
  While I do agree, it's important to note that the state of collaboration between American universities and industry is far greater than any other country and has helped both grow. It's one of the problems in places like India, where there isn't much collaboration between private institutions and industry. Of course things could always be better, but it could also be worse.

kermit_de_fro 16y ago

Minor picky point, but universities have access to such computing power via time-sharing at national labs if it is not available from their own department.

moultano 16y ago
And I do think that a lot of good parallel computing research gets done. It's just a lot harder.

Lewisham 16y ago

Sadly, most academics realize that they're data-starved, but industry isn't able to/doesn't want to work with academia to help gain access to that data. Even if you do get access, you're NDA'd so hard that publishing useful papers is a challenge. It's a shame, because universities are essentially free labor, just on much longer timeframes and lower probability of immediate success. I understand why things are the way they are, but I'd love to find some way of reconciling this issue. It's already killed a research project of mine before it started.

moultano 16y ago
Google realizes this too, and there are a lot of people here that would like to help. There are a few huge issues that always come up. 1. Paranoia about giving competitors the edge they need. 2. Paranoia about giving spammers the edge they need. 3. Privacy. It's nigh impossible to anonymize user data enough to release and still have it in a useful form. The AOL logs debacle hurt everyone. One of my friends had a really interesting take on this. His belief essentially was that the industry advances when people switch companies and (legal or not) take their institutional knowledge with them. Unfortunately, academia is hard to transfer into.
- price 16y ago
  <quote>His belief essentially was that the industry advances when people switch companies and (legal or not) take their institutional knowledge with them.</quote> This is why it's so important that in California the law protects employees leaving companies to work for other companies -- this is legal there. You can't take code, data, hardware designs, or other concrete intellectual property -- that's forbidden by the agreements every tech company requires employees to sign. But if you couldn't take institutional knowledge with you, you couldn't work at a new company at all. (How could an ex-Googler unlearn how web-scale systems are built of many disposable pieces?) And if an agreement purports to restrict you from switching jobs, then California law repudiates that restriction. Unfortunately some other states have no such law, and cheerfully enforce non-competes against engineers trying to switch companies. And guess what? The industry doesn't move as fast in Massachusetts as in California.
  - kevinpet 16y ago
    If I could point to one thing other states could do to improve their economy, it would be adopt the exact language of California's anti-non-compete and the "you own your own ideas, unless developed as part of your work" law. Not that this is going to single handedly grow a silicon valley in Montana overnight, but it has so little downside. The reason for the exact language is because if they use the exact language consciously, they effectively import the case law surrounding those laws.
edge17 16y ago
While I do agree, it's important to note that the state of collaboration between American universities and industry is far greater than any other country and has helped both grow. It's one of the problems in places like India, where there isn't much collaboration between private institutions and industry. Of course things could always be better, but it could also be worse.