Joyce Mahon, a Computer Science PhD student at University College Dublin supervised by Dr. Brett Becker and Dr. Brian Mac Namee has put ChatGPT to the test: the Leaving Certificate Computer Science (LCCS) – Higher Level no less! 

Can ChatGPT Plus successfully pass the Higher LCCS examination?

In short: yes it can, answering nearly all questions correctly.  Computer Science at Leaving Certificate level consists of two assessment components: an end of course examination (70%) and coursework assessment (30%). We used the 2021 LCCS Higher-level examination which comprised of 16 questions in three sections (A, B and C), not all of which are mandatory. These questions are broken down into a total of 62 sub-questions. We extracted the text from the 62 sub-questions in the 2021 LCCS exam and presented these to ChatGPT. No extra information was provided and each sub-question was presented in a fresh chat session. We then corrected the answers received from ChatGPT using the official Leaving Certificate marking scheme. As we expected, the majority of these  (55/62) were answered correctly.   

There is an element of choice in this examination. If only ChatGPT’s best answers are counted, it would receive very close to (or possibly exactly) 100 percent, depending on the marker and their interpretation of the marking criteria. Of course, this means that ChatGPT would receive a H1 – the highest grade possible (in this particular exam – higher level, 2021).  As the examination comprises 70% of the overall grade for this subject, a grade of H3 could potentially be achieved even before considering the Leaving Certificate coursework component worth 30% (see table below).

ChatGPT provides incorrect answers to some questions. Should we discard it as an educational tool?

Generative AI models such as ChatGPT are not infallible. Does this mean that they should be avoided? We are taught early on to not believe everything that we read. Perhaps today’s students should also be taught to not believe everything a large language model provides. Similarly, in important situations, people should always double check things. Carpenters measure twice so they cut (successfully) once. Students in traditional exams are encouraged to double-check their answers. The same applies when using ChatGPT. If anything, this is a good lesson for students to learn. 

Besides, the performance of LLMs on certain tasks (such as the one here) is impressive, and they are constantly improving. In our experiment many solutions contained more detail than the marking scheme prescribes. These models also have other benefits. For instance they can provide students (and educators) with unlimited examples, alternative approaches, and help with “blank page syndrome”. These models can even mark their own answers, although we have not tested this rigorously yet. 

Does the fact that ChatGPT can ace the LCCS diminish the integrity of the exam? 

In a word: no.  The Leaving Certificate is an invigilated exam, and was invigilated long before Generative AI, and for good reason. The advent of this technology is just another (albeit fast and large) step in the same direction humans have been going in for ages. In addition, the LCCS features 30% for coursework assessment which is overseen by their teacher. While ChatGPT could be used for this, the nature of this project work is much more involved than the questions found on a paper exam. It is likely that technology is a little bit off from being able to do as well on this coursework as it does on the paper exam. 

Is ChatGPT only useful for LCCS?

No, ChatGPT has been shown to be capable of generating  acceptable answers for multiple exam types in various subjects.  Recently ChatGPT passed a Higher Level English Leaving Certificate exam with ease (77%) and OpenAI has simulated test runs of various professional and academic exams.

ChatGPT provides a rapid response and a personalized learning experience where students can actively engage with material. It is adaptable and accessible, and can provide customized feedback, real time clarification and multilingual support. Its versatility as an educational tool is evident across diverse educational settings.

What Will the Future of Coding and Programming Exams Look Like?

Generative AI technology promises to change the future of programming but exactly how remains to be seen. Given that 30% of the LCCS requires students to write and manipulate code directly, this not only could change the nature of programming examinations like the LCCS but fundamentally change how people program computers in general and therefore how programming is taught and learned. Codex, a Generative AI in the GPT family superseded by GhatGPT has been shown in research studies to perform above the 75th percentile of real university students in programming courses (see here and here).

Programming can be mentally challenging, and progress – especially when learning – can seem slow. Generative AI can mitigate these challenges. In fact professional programmers using GitHub Copilot – powered by ChatGPT – cite speed, programmer satisfaction, and the conservation of mental energy amongst the biggest reasons they use it.

The nature of programming may also change from programmers directly entering line after line of code character by character on a keyboard, to entering prompts into a Generative AI model like ChatGPT. Devising prompts that get the desired results is known as prompt engineering and involves typing program descriptions in natural language like English rather than typing code in a programming language. Prompt Engineering is emerging as a new profession with some salaries in the six-figure range,  and Prompt Engineering is already being taught to students.  Will prompt engineering become the new programming? At some point in the future will prompt engineering be a large part of learning programming and therefore a large part of programming exams? At this point, only time will tell.

About the Authors

Joyce Mahon is a Computer Science PhD student at UCD and part of the Science Foundation Ireland Centre for Research Training in Machine Learning (www.ml-labs.ie) supervised by Dr. Brett Becker and Dr. Brian Mac Namee. 

Dr. Brian Mac Namee is an Associate Professor at the UCD School of Computer Science, director of the Science Foundation Ireland Centre for Research Training in Machine Learning (www.ml-labs.ie) and co-author of the textbook “Fundamentals of Machine Learning for Predictive Data Analytics” (https://machinelearningbook.com/). 

Dr. Brett Becker (www.brettbecker.com) is an Assistant Professor in the UCD School of Computer Science and Dr. Keith Quille (www.keithquille.com) is a Senior Lecturer at Technological University Dublin (www.brettbecker.com, www.keithquille.com). They are the authors of the textbook “Computer Science for Leaving Certificate” (https://goldenkey.ie/computer-science-for-leaving-cert/). 

Notes

The model used in this work was ChatGPT Plus (GPT-4: May 3, 2023 version). 

I’m thrilled to be a co-leader of Working Group 4 at ITiCSE 2023 this year: Transformed by Transformers: Navigating the AI Coding Revolution for CS Education.

Really looking forward to how this develops! Here’s a screenshot of the Brady Bunch:

We have been meeting weekly for over a month and amazingly, despite the timezone differences spanning several continents from North America to Australasia, we have 14-16 (of 16) on the call every week – amazing.

Abstract:
The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code, creating novel coding assignments, interpreting programming error messages, and more. However, these technologies continue to evolve at an astonishing rate leaving educators little time to adapt. This working group seeks to document the state-of-the-art for code generation LLMs, detail current opportunities and challenges related to their use, and present actionable approaches to integrating them into computing curricula.

This week I am co-organizing Dagstuhl Seminar 22442 – Toward Scientific Evidence Standards in Empirical Computer Science. We’re thrilled that 22 people made it from as far away as Australia, Canada, and the US along with colleagues from Finland, Germany, Ireland, Scotland and Switzerland. This post will be updated in several months when the report for this seminar is published. 

Organizers

Brett A. Becker (University College Dublin, IE) [dblp]
Christopher D. Hundhausen (Oregon State University – Corvallis, US) [dblp]
Ciera Jaspan (Google – Mountain View, US) [dblp]
Andreas Stefik (University of Nevada – Las Vegas, US) [dblp]
Thomas Zimmermann (Microsoft Corporation – Redmond, US)

Participants

• Andrew Begel (Carnegie Mellon University – Pittsburgh, US) [dblp]
• Michelle Craig (University of Toronto, CA) [dblp]
• Andrew Duchowski (Clemson University, US) [dblp]
• Neil Ernst (University of Victoria, CA)
• Arto Hellas (Helsinki University of Technology, FI) [dblp]
• Timothy Kluthe (University of Nevada – Las Vegas, US)
• Juho Leinonen (Aalto University, FI)
• Joseph Maguire (University of Glasgow, GB)
• Monica McGill (CSEdResearch.org – Peoria, US)
• Brad Myers (Carnegie Mellon University – Pittsburgh, US) [dblp]
• Andrew Petersen (University of Toronto Mississauga, CA) [dblp]
• Mauro Pezzè (University of Lugano, CH) [dblp]
• Paul Ralph (Dalhousie University – Halifax, CA)
• Kate Sanders (Rhode Island College – Providence, US) [dblp]
• Claudia Szabo (University of Adelaide, AU) [dblp]
• Jan Vahrenhold (Universität Münster, DE) [dblp]
• Titus Winters (Google – New York, US)
• Aman Yadav (Michigan State Universit – East Lansing, US) [dblp]

I am super thrilled to announce that I have been named one of the five inaugural Irish National Forum Teaching and Learning Research Fellowships! I was shortlisted from an initial field of 54 along with eight top-notch Irish T&L researchers. It’s super cool that two of these were UCD Colleagues (Geraldine O’Neill – who was also one of the five awardees – and Tara Cusack) and two others were AISHE colleagues (Bernadette Brereton and Ronan Bree).

The five fellows and information on their projects can be found on the National Forum website. My fellowship research is titled:

Teaching and Learning for the Next Era of Digital Innovation

This research aims to help teaching and learning in Ireland plan for the next digital era. It will explore how disciplines and departments are advancing towards emerging digital technologies with an eye to the future world into which graduates will enter. The project will inform the professional development of those who teach so that they might be better positioned to foster skills, knowledge and competencies to deal with realities that are hard to imagine from today’s perspective as technologies such as big data, the internet of things and artificial intelligence revolutionise all disciplines.

Given the ever-increasing prevalence of technology in modern life, there is a corresponding increase in the likelihood of digital devices being pertinent to a criminal investigation or civil litigation. As a direct consequence, the number of investigations requiring digital forensic expertise is resulting in huge digital evidence backlogs being encountered by law enforcement agencies throughout the world. It can be anticipated that the number of cases requiring digital forensic analysis will greatly increase in the future. It is also likely that each case will require the analysis of an increasing number of devices including computers, smartphones, tablets, cloud-based services, Internet of Things devices, wearables, etc. The variety of new digital evidence sources poses new and challenging problems for the digital investigator from an identification, acquisition, storage and analysis perspective. This paper explores the current challenges contributing to the backlog in digital forensics from a technical standpoint and outlines a number of future research topics that could greatly contribute to a more efficient digital forensic process. Keywords: Digital Evidence Backlog, Digital Forensic Challenges, Future Research Topics

Lillis, D., Becker, B., O’Sullivan, T., Scanlon, M. “Current Challenges and Future Research Areas for Digital Forensic Investigation” The 11th ADSFL Conference on Digital Forensics, Security and Law (CDFSL 2016), Daytona Beach, Florida, USA, May 2016.

The fifth Irish Supercomputer List (December 2015) was released today.

There are three new installations for the fifth list for a total of 28,  with two from Trinity College Dublin (#9 and 10) and one from University College Dublin (#11). The combined number of CPU cores on the current list is over 75,000. The list also shows that Ireland is following a global trend of harnessing coprocessor technologies for HPC with over one fifth of the machines on the list making use of accelerators including GPU and Xeon Phi, with a combined total of over 106,000 cores. The total performance of all machines is 678 Tflops/s.

Press clippings:

• InsideHPC: Irish Supercomputer List Reflects HPC Growth
• Silicon Republic: Trinity and UCD add to Irish supercomputer list

Volume 7, Number 2 of the All Ireland Journal of Teaching and Learning in Higher Education was published today. The journal is published by the All-Ireland Society for Higher Education.

The table of contents is below, and the issue can be accessed here.

Editorial