Impact assessment of academic support provided by tertiary learning advisors: sharing an endeavour

Presentation abstract In New Zealand higher education (HE), there is a lack of consistent ways of collecting evidence of the impact made by academic literacy support from Tertiary Learning Advisors (TLAs) on students’ academic performance, retention, and success. TLAs in New Zealand and Australia are primarily involved in providing learning support to students in post-secondary education to encourage development of their academic literacy and essential study skills. They are professional educators who advise students on issues related to academic writing and other academic skills, such as time management or exam preparation, to facilitate achievement of students’ goals of tertiary study (Griffith University, 2021). While it may be recognised that provision of learning support is desirable for a meaningful and successful HE experience for many students, hard ev idence that learning support makes a difference to student retention and academic performance is difficult to find (Acheson, 2006, as cited in Breen and Prothero, 2015). This presentation sought to share an attempt to address this issue by investigating the impact of embedded academic literacy support provided by TLAs to three cohorts of students enrolled in undergraduate social work and early childhood education programs at my ITP (Institute of Technology and Polytechnic) in Auckland, New Zealand. Existing research in Australia, Canada, and the United Kingdom suggests that support that embeds academic literacy development in disciplines, rather than academic support that i s generic and/or provided through foundation courses, represents a best practice model (Glew et al., 2019).


Malik
Impact assessment of academic support provided by tertiary learning advisors: sharing an endeavour Journal of Learning Development in Higher Education, Special Issue 29: October 2023 2 The study was conducted from an empirical perspective of social science research, and uses a pre-post test research design, as previously used by Sebolai and Dzansi (2015) in a South African context, to measure the impact of embedded support on students' academic literacy.The support comprised up to seven workshops of up to 2 hours each, run by TLAs as part of the regular class timetable and delivered over the first eight weeks of the semester for the three cohorts, on discipline-specific study skills and relevant aspects of academic literacy.The students and the subject lecturer who organised the embedded academic literacy support were informed of the intent of study and their consent to participate based on their own volition, was sought.A test of up to 25 objective items -presented in multichoice, gap-filling, and matching questions that aimed to assess various aspects of academic literacy covered in the workshops -was administered in each case before and after the targeted intervention.The difference between pre-test and posttest mean/average was calculated and investigated for statistical significance.The results showed that there was a difference in the academic literacy levels of the students as a result of the targeted embedded support.
By sharing this attempt of impact assessment of embedded learning support, discussion/reflection was encouraged among the delegates on the feasibility and/or credibility of the pre-test and post-test research design.Such research then has the potential to allow gathering of evidence which satisfies the metrics-driven requirements for justification of funding for academic support services in tertiary education.
Keywords: impact assessment, academic support, learning advising.

Community response
The community recognised that the research was in early stages and this type of research presents challenges, because, as a member commented: "quantitative evidence of the impact of Learning Development practice is hard to find".While it may be difficult to  However, the researcher expressed that the resulting percentage size of the improvement was 'disappointing', with group 1 showing only an 8% improvement between the pre-test and post-test.As a member of the community commented, there are a multitude of pedagogical and contextual factors which can impact student learning which makes it difficult to firmly conclude that it was the academic literacy support which led to this improvement.It was agreed amongst members of the community that, considering the small sample size, the data could be analysed for statistical significance to give further insight.Further to this, one member suggested that the longer term academic achievement of the participants could be assessed to see whether this was affected by the intervention.
It was suggested that this may generate more significant data, and potentially offer a slightly more long-term view on the impacts of Learning Development support, than the view offered by a pre-/post-intervention test.
The research influenced members in that it "shows how we could all be undertaking small projects like this", as one member commented.Though the researcher stated their disappointment at the small amount of data that was generated, this conversely could encourage otheAr fledgling researchers to pursue their own research on a small scale.If similar quantitative studies were undertaken in this area of practice, it was pointed out, a meta-analysis could be conducted which could then provide more evidence of the positive impact of academic literacy support.One member commented "we all know we have" this impact, though we may find it difficult to prove to those above us in the HE hierarchy.This reflects an ongoing struggle that members of Learning Development teams have when justifying their work and directly influencing senior management.

Next steps and additional questions
From the response of the community, it seems that this research offers glimmers of potential and could influence similar studies in comparative settings, or might also prompt research examining outcomes for students when support is provided in a different format.
Therefore the following questions could be considered by the conference presenter: • In what ways might one-to-one provision impact student academic performance?
• What structure of embedded academic literacy provision within disciplines offers effective support for student success?Is provision that is 'little and often' preferable to longer and less regular provision embedded in discipline-specific teaching?
• Related to the methodology of the study, could a qualitative approach offer insight into the confidence gained by students who took part in the embedded workshops?

Author's reflection
It was heartening to find an attentive audience who could empathise with the challenge faced by learning advisors/learning developers of establishing, with credible quantitative evidence, the positive impact made by academic support.There were some very pertinent queries from the attending members related to this topic, such as, what my experience had been of finding relevant literature on quantitative studies of impact assessment of learning support, how I would ascertain that the academic support provided in my workshops resulted in improved assignment grades for students, and if I would consider getting qualitative evidence of impact assessment, which may be more feasible and credible.I am mindful of the significance of qualitative evidence that may come in the form of feedback, often unsolicited, from students about how the academic support made a positive difference to their assignment grades and their confidence in academic writing skills.I believe a robust mechanism of impact assessment of academic support needs to include both quantitative and qualitative evidence of the difference made.Such a system will be beneficial, both to maintain best practice of learning advising/learning development and to ensure its professional sustainability in current tertiary education environment.
establish a direct link between Learning Development input and test scores, the community did recognise that the data showed positive results when comparing preintervention and post-intervention performance.As shown below, each of the three groups involved in the study demonstrated improved performance on the post-intervention test up Malik Impact assessment of academic support provided by tertiary learning advisors: sharing an endeavour Journal of Learning Development in Higher Education, Special Issue 29: October 2023 3to 17%.This was, as one member commented: "encouraging evidence of Learning Development impact".

Figure 1 .
Figure 1.Average performance of three groups in semester 1, 2023.

I
was aware that my research was in very initial stages; it was based on a very small data set; and my practice setting of vocational education in New Zealand was quite different from the HE setting in the United Kingdom.All this had made me feel uncertain and anxious prior to and during my talk.However, the interest and judicious affirmation of the topic by the community members made me feel quite gratefully relieved.I was also delighted to see a very succinct and creative representation of the essence of my talk posted on Twitter later by one of the community members.(Bartram, 2023) Therefore, I feel encouraged to continue gathering evidence in similar or different format with cohorts in other programmes in future, if possible.I intend to conduct the statistical analysis of the difference identified in pre-post-test performance of the three groups in semester 1, 2023.I'm also keen to explore ways of examining uptake by students from these groups, of the support offered through 1-1 appointments in the rest of 2023, and the relationships (if any) between these and student retention/academic achievement.