Why information security is critical to the safe development of AI systems | Nova DasSarma (2022)

_Originally released June 2022._ Today’s guest, the computer scientist and polymath Nova DasSarma, works on computer and information security for the AI company Anthropic with the security team. One of her jobs is to stop hackers exfiltrating Anthropic’s incredibly expensive intellectual property, as recently happened to Nvidia. As she explains, given models’ small size, the need to store such models on internet-connected servers, and the poor state of computer security in general, this is a serious challenge.

The worries aren’t purely commercial though. This problem looms especially large for the growing number of people who expect that in coming decades we’ll develop so-called artificial ‘general’ intelligence systems that can learn and apply a wide range of skills all at once, and thereby have a transformative effect on society.

If aligned with the goals of their owners, such general AI models could operate like a team of super-skilled assistants, going out and doing whatever wonderful (or malicious) things are asked of them. This might represent a huge leap forward for humanity, though the transition to a very different new economy and power structure would have to be handled delicately. If unaligned with the goals of their owners or humanity as a whole, such broadly capable models would naturally ‘go rogue,’ breaking their way into additional computer systems to grab more computing power — all the better to pursue their goals and make sure they can’t be shut off.

As Nova explains, in either case, we don’t want such models disseminated all over the world before we’ve confirmed they are deeply safe and law-abiding, and have figured out how to integrate them peacefully into society. In the first scenario, premature mass deployment would be risky and destabilising. In the second scenario, it could be catastrophic — perhaps even leading to human extinction if such general AI systems turn out to be able to self-improve rapidly rather than slowly, something we can only speculate on at this point. If highly capable general AI systems are coming in the next 10 or 20 years, Nova may be flying below the radar with one of the most important jobs in the world.

In this episode:
• Rob’s intro [00:00:00]
• Why computer security matters for AI safety [00:06:03]
• State of the art in information security [00:15:45]
• The hack of Nvidia [00:25:14]
• The most secure systems that exist [00:34:51]
• Formal verification [00:46:26]
• How organisations can protect against hacks [00:52:42]
• Is ML making security better or worse? [00:56:34]
• Motivated 14-year-old hackers [00:59:32]
• Disincentivising actors from attacking in the first place [01:04:12]
• Hofvarpnir Studios [01:11:04]
• Capabilities vs safety [01:18:10]
• Interesting design choices with big ML models [01:27:08]
• Nova’s work and how she got into it [01:43:45]
• Anthropic and career advice [02:04:16]
• $600M Ethereum hack [02:17:01]
• Personal computer security advice [02:21:30]
• LastPass [02:29:27]
• Stuxnet [02:36:31]

----

The 80,000 Hours Podcast features unusually in-depth conversations about the world’s most pressing problems and what you can do to solve them.

Learn more, read the summary and find the full transcript on the 80,000 Hours website:
https://80000hours.org/podcast/episodes/nova-dassarma-information-security-and-ai-systems/

Видео Why information security is critical to the safe development of AI systems | Nova DasSarma (2022) канала 80,000 Hours