Evaluating what generative AI systems know about cybersecurity

Tim Finin

September 7, 2023

1950202 bytes

Tags: ai, cca, cci, chatgpt, cybersecurity, llm

The public release of OpenAI's ChatGPT system eight months ago signaled an inflection point for AI technology and its applications. While these AI systems have well-known shortcomings, they have the potential to help in many ways. After describing the technology, I will report on a recent evaluation of OpenAI's ChatGPT and Google's Bard ability to solve cybersecurity problems using two datasets designed to test students' knowledge: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCA results will be compared with those from a recent evaluation of 193 students from seven colleges and universities. Spoiler: one of the AI systems performed surprisingly well.

Presentation made at the INCS-CoE webinar on The Growing Role of AI in Cybersecurity, 2023/09/07

Counter: 100 downloads

Access Control: Public

OWL Tweet