What generative AI systems know about cybersecurity

Tim Finin

October 12, 2023

1903225 bytes

PDF Document - Need a reader? Get one here

ai, chatgpt, cybersecurity, llm

The public release of OpenA's ChatGPT system in November 2022 signaled an inflection point for AI technology and its applications. While these AI systems have well-known shortcomings, they have the potential to help in many ways. After describing the technology, I report on a recent evaluation of OpenAI's ChatGPT and Google's Bard ability to solve cybersecurity problems using two datasets designed to test students' knowledge: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCA results are compared with those from a recent evaluation of 193 students from seven colleges and universities. Spoiler: one of the AI systems performed surprisingly well.

Presentation for an online meeting of Maryland Computer Science teachers


OWL Tweet