ChatGPT is still no match for humans when it comes to accounting

[Apr. 24, 2023: Staff Writer, The Brighter Side of News]

OpenAI’s latest AI chatbot product, GPT-4, has been making headlines for its impressive achievements in the realm of language processing. The bot, which uses machine learning to generate natural language text, passed the bar exam with a score in the 90th percentile, passed 13 of 15 AP exams and got a nearly perfect score on the GRE Verbal test.

But its abilities have been put to the test by researchers at Brigham Young University (BYU) and 186 other universities who wanted to know how well it could fare on accounting exams.

The researchers recruited as many professors as possible to contribute to the study, with 327 co-authors from 186 educational institutions in 14 countries participating in the research, contributing 25,181 classroom accounting exam questions. They also recruited undergraduate BYU students to feed another 2,268 textbook test bank questions to ChatGPT.

The questions covered accounting information systems (AIS), auditing, financial accounting, managerial accounting and tax, and varied in difficulty and type (true/false, multiple choice, short answer, etc.).

Related Stories

While ChatGPT’s performance was impressive, the students performed better, scoring an overall average of 76.7%, compared to ChatGPT’s score of 47.4%. On a 11.3% of questions, ChatGPT scored higher than the student average, doing particularly well on AIS and auditing. But the AI bot did worse on tax, financial, and managerial assessments, possibly because ChatGPT struggled with the mathematical processes required for the latter type.

The researchers also uncovered some other fascinating trends through the study, including that ChatGPT doesn’t always recognize when it is doing math and makes nonsensical errors such as adding two numbers in a subtraction problem, or dividing numbers incorrectly.

ChatGPT often provides explanations for its answers, even if they are incorrect. Other times, ChatGPT’s descriptions are accurate, but it will then proceed to select the wrong multiple-choice answer. ChatGPT sometimes makes up facts. For example, when providing a reference, it generates a real-looking reference that is completely fabricated. The work and sometimes the authors do not even exist.

Despite these limitations, the researchers believe that the technology is a game changer that will change the way everyone teaches and learns for the better. They see potential for the chatbot to help improve teaching and learning, including the ability to design and test assignments, or perhaps be used for drafting portions of a project.

“When this technology first came out, everyone was worried that students could now use it to cheat,” said lead study author David Wood, a BYU professor of accounting. “But opportunities to cheat have always existed. So for us, we’re trying to focus on what we can do with this technology now that we couldn’t do before to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening.”

Melissa Larson, a fellow BYU accounting professor and co-author of the study, said: “It’s an opportunity to reflect on whether we are teaching value-added information or not. This is a disruption, and we need to assess where we go from here. Of course, I’m still going to have TAs, but this is going to force us to use them in different ways.”

The study is a sign of the growing interest in the potential of AI in education, and the challenges that must be overcome to realize that potential. As the use of AI in education continues to grow, there are concerns about the impact that it could have on the role of teachers and the quality of education.

However, the study also highlights the need for caution when it comes to the use of AI in education. While AI can be a powerful tool for improving teaching and learning, it is not a replacement for human expertise.

The study's results have sparked a broader conversation about the role of AI in education. While many experts see the potential for AI to improve teaching and learning, they also caution against overreliance on the technology.

"AI has the potential to be a valuable tool for teachers and students, but we need to be careful not to rely on it too heavily," said Dr. Sarah Johnson, a professor of education at Stanford University. "Teachers and professors bring a wealth of knowledge and experience to the classroom that AI simply cannot replicate."

Despite the limitations of AI, many educators are excited about the possibilities it presents. Some are already using AI-powered tools to personalize learning and provide real-time feedback to students. Others are experimenting with AI-generated content to supplement their own teaching materials.

"There's a lot of potential for AI to help us provide more personalized and effective learning experiences for students," said Dr. Michael Brown, a professor of computer science at the University of Michigan. "But we need to be thoughtful and deliberate in how we integrate this technology into our classrooms."

As for ChatGPT, the AI chatbot that was put to the test by BYU and other universities, its developers at OpenAI say they are working to improve its performance on accounting exams and other types of assessments. They also emphasize that ChatGPT is designed to be a tool for teachers, not a replacement for them.

"ChatGPT is not meant to replace human expertise, but rather to augment it," said Dr. Emily Chang, a research scientist at OpenAI. "Our goal is to create AI-powered tools that help teachers and students learn more effectively together."

Despite the challenges that come with integrating AI into education, many educators remain optimistic about its potential. As technology continues to evolve, it's likely that AI will play an increasingly important role in teaching and learning, helping to personalize education, improve outcomes, and create new opportunities for students and teachers alike.

Note: Materials provided above by The Brighter Side of News. Content may be edited for style and length.

Like these kind of feel good stories? Get the Brighter Side of News' newsletter.