Presentation

What generative AI systems know about cybersecurity

Tim Finin and Alan Sherman

September 7, 2023

1950202 bytes

PDF Document - Need a reader? Get one here

ai, cca, cci, chatgpt, cybersecurity, llm

The public release of OpenAI's ChatGPT system eight months ago signaled an inflection point for AI technology and its applications. While these AI systems have well-known shortcomings, they have the potential to help in many ways. After describing the technology, I will report on a recent evaluation of OpenAI's ChatGPT and Google's Bard ability to solve cybersecurity problems using two datasets designed to test students' knowledge: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCA results will be compared with those from a recent evaluation of 193 students from seven colleges and universities. Spoiler: one of the AI systems performed surprisingly well.

Presentation made at the INCS-CoE webinar on The Growing Role of AI in Cybersecurity, 2023/09/07

110 downloads

Public

OWL Tweet