Exploring AI-generated output through assessment in a university setting: a case study

Authors

DOI:

https://doi.org/10.47408/jldhe.vi38.1386

Keywords:

assessment, artificial intelligence, higher education, AI literacy

Abstract

Generative AI and large language models have a significant impact on education and business, creating a pressing need for universities to act and support students in their development and understanding of these new technologies. While several emerging studies have investigated the use and perceptions of AI by students, few focus on how educators can actively engage students in developing their critical understanding. We discuss an innovative assessment approach that exposes students to AI using an AI-generated output, focusing them on evaluating output quality, and then use a survey to examine its success. The assessment intervention allowed them to explore and develop their understanding of the risks and limitations of AI.

Author Biographies

Richard Baylis, Swansea University

Richard M. Baylis is an Associate Professor in Accounting and Head of Accounting and Finance at the School of Management, Swansea University. His research interests include public sector accounting, audit and assurance, public value, and accounting education.

Lukas Helikum, Singapore University of Social Sciences

Lukas J. Helikum is a Senior Lecturer in Accountancy at the School of Business, Singapore University of Social Sciences. His research interests include judgment and decision-making in accounting, with an emphasis on financial and non-financial reporting, as well as accounting education.

Sarah Jones, Swansea University

Sarah Jones is Associate Dean, Education of the Faculty of Humanities and Social Sciences and Professor in Accounting at the School of Management, Swansea University. Her research interests include promoting sustainability education, professional accreditation, and the application of technology in higher education.

References

Ansari, A.N., Ahmad, S. and Bhutta, S.M. (2023) ‘Mapping the global evidence around the use of ChatGPT in higher education: a systematic scoping review’, Education and Information Technologies, 29, pp.11281–11321. Available at: https://doi.org/10.1007/s10639-023-12223-4

Awwad, E. (2024) ‘Why higher education should embrace AI now to advance learning’, Forbes 12 January. Available at: https://www.forbes.com/sites/forbesbusinesscouncil/2024/01/12/why-higher-education-should-embrace-ai-now-to-advance-learning/ (Accessed: 28 August 2025).

Ballantine, J., Boyce, G. and Stoner, G. (2024) ‘A critical review of AI in accounting education: threat and opportunity’, Critical Perspectives on Accounting, 99, p.102711. Available at: https://doi.org/10.1016/j.cpa.2024.102711

Bourne, E. and Bell, A. (2024) ‘Student perceptions of transferable skills learned during a simulated placement’, Innovations in Education and Teaching International, 62(2), pp.642–656. Available at: https://doi.org/10.1080/14703297.2024.2323989

Cotton, D.R.E., Cotton, P. A. and Shipway, J.R. (2023) ‘Chatting and cheating: ensuring academic integrity in the era of ChatGPT’, Innovations in Education and Teaching International, 61(2), pp.228–239. https://doi.org/10.1080/14703297.2023.2190148

European University Association (2023) Artificial intelligence tools and their responsible use in higher education learning and teaching. Available at: https://www.eua.eu/publications/positions/artificial-intelligence-tools-and-their-responsible-use-in-higher-education-learning-and-teaching.html (Accessed: 28 August 2025).

Faguy, A. (2023) ‘New York City Public Schools reverses ChatGPT ban’, Forbes, 18 May. Available at: https://www.forbes.com/sites/anafaguy/2023/05/18/new-york-city-public-schools-reverses-chatgpt-ban/ (Accessed: 28 August 2025).

Farrokhnia, M., Banihashem, S.K., Noroozi, O. and Wals, A. (2023) ‘A SWOT analysis of ChatGPT: implications for educational practice and research’, Innovations in Education and Teaching International, 61(3), pp.460–474. Available at: https://doi.org/10.1080/14703297.2023.2195846

Freeman, J. (2024) ‘Provide or punish? Students’ views on generative AI in higher education’, Higher Education Policy Institute (HEPI). Available at: https://www.hepi.ac.uk/2024/02/01/provide-or-punish-students-views-on-generative-ai-in-higher-education/ (Accessed: 28 August 2025).

Freeman-Wong, J., Munguia, D. and Mohr, J.J. (2023) ‘Building a strategy to harness ChatGPT in education’, California Management Review Insights, 65(4) Available at: https://cmr.berkeley.edu/2023/08/building-a-strategy-to-harness-chatgpt-in-education/ (Accessed: 9 September 2025).

Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R., Salvatori, T., Lukasiewicz, T., Petersen, P. and Berner, J. (2023) ‘Mathematical capabilities of ChatGPT’, in A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt and S. Levine (eds.) Advances in neural information processing systems. Vol. 36. Curran Associates, Inc, pp. 27699–27744.

Hack, K. and Knight, C. (2023) Higher education in the era of AI. Available at: https://www.advance-he.ac.uk/news-and-views/higher-education-era-ai (Accessed: 28 August 2025).

Heaven, W.D. (2023) ‘ChatGPT is going to change education, not destroy it’, MIT Technology Review, 6 April. Available at: https://www.technologyreview.com/2023/04/06/1071059/chatgpt-change-not-destroy-education-openai/ (Accessed: 28 August 2025).

Hendra, R. and Hill, A. (2019) ‘Rethinking response rates: new evidence of little relationship between survey response rates and nonresponse bias’, Evaluation Review, 43(5), pp.307–330. Available at: https://doi.org/10.1177/0193841X18807719

Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S.B., Santos, O.C., Rodrigo, M.T., et al. (2022) ‘Ethics of AI in education: towards a community-wide framework,’ International Journal of Artificial Intelligence in Education, 32(3), pp.504–526. Available at: https://doi.org/10.1007/s40593-021-00239-1

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., et al. (2023) ‘ChatGPT for good? On opportunities and challenges of large language models for education’, Learning and Individual Differences, 103, p.102274. Available at: https://doi.org/10.1016/j.lindif.2023.102274

Kisfalvi, V. and Oliver, D. (2015) ‘Creating and maintaining a safe space in experiential learning’, Journal of Management Education, 39(6), pp.713–740. Available at: https://doi.org/10.1177/1052562915574724

Kreber, C. (2001) ‘Learning experientially through case studies? A conceptual analysis’, Teaching in Higher Education, 6(2), pp.217–228. Available at: https://doi.org/10.1080/13562510120045203

Li, L., Ma, Z., Fan, L., Lee, S., Yu, H. and Hemphill, L. (2023) ‘ChatGPT in education: a discourse analysis of worries and concerns on social media’, Education and Information Technologies, 29, pp.10729–10762. Available at: https://doi.org/10.1007/s10639-023-12256-9

Moorhouse, B.L., Yeo, M. A. and Wan, Y. (2023) ‘Generative AI tools and assessment: guidelines of the world’s top-ranking universities’, Computers and Education Open, 5, p.100151. Available at: https://doi.org/10.1016/j.caeo.2023.100151

Neuert, C., Meitinger, K., Behr, D. and Schonlau, M. (2021) ‘Editorial: the use of open-ended questions in surveys’, Methods, Data, Analyses, 15(1), pp.3–6. Available at: https://mda.gesis.org/index.php/mda/article/view/366/280 (Accessed: 9 September 2025).

OpenAI (2022) ChatGPT-3.5. Available at: https://openai.com/index/chatgpt/ (Accessed: 28 August 2025).

Paisey, C. and Paisey, N. J. (2007) ‘Balancing the vocational and academic dimensions of accounting education: the case for a core curriculum’, Journal of Vocational Education & Training, 59(1), pp.89–105. Available at: https://doi.org/10.1080/13636820601145705

Preece, K. (2023) AI-lluminating the path to adapting assessment. Available at: https://www.advance-he.ac.uk/news-and-views/ai-lluminating-path-adapting-assessment (Accessed: 28 August 2025).

Ratten, V. and Jones, P. (2023) ‘Generative artificial intelligence (ChatGPT): implications for management educators’, The International Journal of Management Education, 21(3), p.100857. Available at: https://doi.org/10.1016/j.ijme.2023.100857

Stone, B.W. (2024) ‘Generative AI in higher education: uncertain students, ambiguous use cases, and mercenary perspectives’, Teaching of Psychology, 52(3), pp. 347-356. Available at: https://doi.org/10.1177/00986283241305398

Veine, S., Anderson, M.K., Andersen, N.H., Espenes, T.C., Søyland, T.B., Wallin, P. and Reams, J. (2020) ‘Reflection as a core student learning activity in higher education – insights from nearly two decades of academic development’, International Journal for Academic Development, 25(2), pp.147–161. Available at: https://doi.org/10.1080/1360144X.2019.1659797

Webb, S. (2023) AI and higher education: is it time to rethink teaching and assessment? Available at: https://teltales.port.ac.uk/2023/03/13/ai-and-higher-education-is-it-time-to-rethink-teaching-and-assessment/ (Accessed: 28 August 2025).

Willig, C., Rogers, W.S., Terry, G. and Hayfield, N. (2017) ‘Thematic Analysis’, in The SAGE handbook of qualitative research in Psychology. London: SAGE, pp. 17–36). Available at: https://methods.sagepub.com/hnbk/edvol/the-sage-handbook-of-qualitative-research-in-psychology-second-edition/chpt/2-thematic-analysis

Yiğit, S., Berşe, S., Dirgar, E. and Güner, S.G. (2024) ‘Views of health sciences undergraduates on ChatGPT, an artificial intelligence-powered language model: a qualitative study’, Innovations in Education and Teaching International, 62(4), pp.1258–1272. Available at: https://doi.org/10.1080/14703297.2024.2391044

Downloads

Published

11-12-2025

How to Cite

Baylis, R., Helikum, L., & Jones, S. (2025). Exploring AI-generated output through assessment in a university setting: a case study. Journal of Learning Development in Higher Education, (38). https://doi.org/10.47408/jldhe.vi38.1386

Issue

No. 38 (2025)

Section

Papers

License

Copyright (c) 2025 Journal of Learning Development in Higher Education

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).