Imagine a computer that wants to calculate π to as many digits as possible. That computer will see humans as being made of atoms which it could use to build more computers; and worse, since we would object to that and might try to stop it, we’d be a potential threat that it would be in the AI’s interest to eliminate
PhD Theoretical Computer Science, UC Berkeley | Researcher at OpenAI
Ideally, “each time you build a more powerful machine, it effectively models human values and does what humans would like,” says Christiano. But he worries that these machines may stray from human values as they surpass human intelligence.
PhD Mathematics, Oxford | Research Fellow, Future of Humanity Institute
One of the things that makes AI risk scary is that it’s one of the few that is genuinely an extinction risk if it were to go bad. With a lot of other risks, it’s actually surprisingly hard to get to an extinction risk.
Student of Math and CompSci, future AI Risk researcher, dabbling entrepreneur.
In strategy games, the most powerful abilities are those which let you take more actions per turn or have a wider array of possible actions to perform the one best suited to the situation at hand. AI is literally a machine producing ideas, which let you act faster (and thus perfom more actions) or execute different plans (and thus have more choices). This is a serious game imbalance, and is one we...See More