Evaluating what generative AI systems know about cybersecurity

Tim Finin

September 7, 2023

1950202 bytes

ai, cca, cci, chatgpt, cybersecurity, llm

The public release of OpenAI's ChatGPT system eight months ago signaled an inflection point for AI technology and its applications. While these AI systems have well-known shortcomings, they have the potential to help in many ways. After describing the technology, I will report on a recent evaluation of OpenAI's ChatGPT and Google's Bard ability to solve cybersecurity problems using two datasets designed to test students' knowledge: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCA results will be compared with those from a recent evaluation of 193 students from seven colleges and universities. Spoiler: one of the AI systems performed surprisingly well.

Presentation made at the INCS-CoE webinar on The Growing Role of AI in Cybersecurity, 2023/09/07



OWL Tweet