Okay. I hate to say this and I hate to put it this way but... WUT? How exactly do we define this kind of research in a way that doesn't encompass ridiculously

Okay. I hate to say this and I hate to put it this way but... WUT? How exactly do we define this kind of research in a way that doesn't encompass ridiculously broad swaths of computer science research? Here's one example: "require that self-improving software require human intervention to move forward on each iteration," You just made genetic algorithms illegal. Under this regime this software and similar things would be illegal to run: http://adam.ierymenko.name/nanopond.shtml http://adam.ierymenko.name/nanopond.shtml For those who don't know much about them: GAs (and GP, and computer-based "artificial life," etc.) are systems that can and do produce self-improving code. They do so through the execution of massive numbers of iterative generations, replicating evolution or evolution-like processes in silico. There is no meaningful way to define a "step" in these systems that would not either require that every generation be halted -- effectively making them impossible to run -- or be meaningless. "Require that the first SMI developed have as part of its operating rules that a) it can’t cause any direct or indirect harm to humanity (i.e. Asimov’s zeroeth law), b) it should detect other SMI being developed but take no action beyond detection, c) other than required for part b, have no effect on the world." I can't even begin to address how bizarre that is. Again, how would one do this? (a) prohibits any form of sentience, more or less. (b) is just unworkable... how would you detect SMI vs. say a human operating behind a "mechanical turk" interface? ... I could go on. This whole essay is just a total howler. It's the sort of thing that would garner a cockeyed eyebrow-raise and a chuckle if it weren't coming from the head of the world's largest and most successful technology accelerator. The "conspiracy nut" center of my brain (located slightly behind the prefrontal cortex I think) is wondering whether this could be some kind of weird ploy to guarantee a monopoly on AI research and other next-generation computing research by Silicon Valley venture capitalists and their orbiting ecosystems such as YC. These sorts of bizarre, onerous requirements would have the effect of doing this by imposing a huge tax on this kind of R&D that smaller operations would not be able to afford. It would effectively make non-VC-funded and/or non-government-funded AI R&D illegal. But I doubt that. I think this is just bizarre reasoning plain and simple. I also do not think we are anywhere near AGI. I didn't study CS formally. I'm one of those CS autodidacts that learned to code when he was four. So instead I studied biology. I did so out of an interest in AI, and on the hypothesis that the best way to understand intelligence was to study how nature did it. I concentrated my studies in genomics, evolution, ecology, and neuroscience -- the four scales and embodiments at which biological intelligence operates. Nobody that I am aware of in bio/neuro takes the Kurzweil-esque predictions of AGI around the corner seriously. These are people who study actual intelligent systems... the only actual intelligent systems we know about in the universe. I really think a lot of CS people suffer from a kind of Dunning-Krueger effect when it comes to biological systems. Study living systems as they are -- not some naive over-simplification of them -- and prepare to be humbled. You rapidly realize that AGI will require a jump in computer technology at least equivalent to the vacuum tube -> IC leap, not to mention a jump in our theoretical understanding of living intelligent systems. The former might happen if Moore's Law continues, but the latter is a much tougher nut to crack. We can barely do genetic engineering beyond "shotgun method" hacking and we think we're going to duplicate biological intelligence? It's like saying we're about to colonize Alpha Centauri because we just managed to throw a rock really high up in the air. Come to think of it... the predictions of AGI around the corner remind me very much of the "Star Trek" predictions of interstellar space flight in the 40s and 50s. People who really understood space flight didn't take these predictions seriously, but the rest of the culture took a while to catch up. Finally... yeah, there is a small possibility that some kind of AI could be dangerous to us... but come on. There's a ton of other existential threats that are blood-dripping certainties. Take fossil fuels for example. Our civilization is absolutely dependent on a finite energy source it is eating at an exponentially increasing rate. That's not some pie-in-the-sky theoretical risk. It's cold, hard death, an existential threat looming on the horizon with the absolute physical certainty of a planet-killer asteroid. It's something that makes me fear for my children. If you want to avoid existential threats, maybe YC should be deepening its portfolio in the energy sector. If that problem isn't solved, there ain't going to be any AI. The future will look more like "The Road" by Cormac McCarthy. Something is seriously wrong with a culture that discusses such unlikely scenarios as this while real risks barrel toward us.