Turning Data into Insight: Analyzing Chat Transcripts to Revitalize Virtual Reference Services
Session Format
In-person Full Paper Presentation
Conference Tracks
Outreach, Services, and Programs
Short Description
As academic libraries transition to tiered reference services, public service points are relying more heavily on the work of student assistants. This presentation reports a study of analyzing reference chat transcripts data to assess chat reference service quality and performances of graduate student assistants, and how chat transcripts data were used to implement new procedures, revise chat reference training, and retool supervision strategies for online reference services.
Long Description
As academic libraries transition to tiered reference services, public service points are relying more heavily on the work of student assistants. In a library where chat services are monitored by a team of graduate student assistants, there is a concern about the quality of service being provided. Chat transcript data offer a valuable lens through which to access the content and assess the quality of online reference transactions. This presentation reports a study of analyzing reference chat transcripts data to evaluate chat reference service quality and performances of graduate student assistants in an academic library. Anonymized chat transcripts were analyzed because they provided an objective and non-intrusive source of data to observe and evaluate students' performances. A rubric was designed to help us break down chat performance into measurable skill sets such as assessing patron information need and quality of information resources offered. It also provided a framework to rate these skills on a scale from poor to excellent. Over the course of the project, we analyzed over 750 chat transcripts from the work of eight graduate assistants from two semesters. We classified transcripts by patron question type and analyzed both overall and individual chat performances. Ultimately, we used the findings of our data analysis as insight to revise chat reference training, implement new procedures, and retool our supervision strategies for online reference services.
Learning Objectives
At the conclusion of this presentation, participants will be able to:
Learn about the development and implementation of chat assessment rubrics for appraising chat reference services.
Gain a better understanding of the interdependence of library data and library services.
Use and apply learned materials in assessing chat services at their own institutions.
Turning Data into Insight: Analyzing Chat Transcripts to Revitalize Virtual Reference Services
As academic libraries transition to tiered reference services, public service points are relying more heavily on the work of student assistants. In a library where chat services are monitored by a team of graduate student assistants, there is a concern about the quality of service being provided. Chat transcript data offer a valuable lens through which to access the content and assess the quality of online reference transactions. This presentation reports a study of analyzing reference chat transcripts data to evaluate chat reference service quality and performances of graduate student assistants in an academic library. Anonymized chat transcripts were analyzed because they provided an objective and non-intrusive source of data to observe and evaluate students' performances. A rubric was designed to help us break down chat performance into measurable skill sets such as assessing patron information need and quality of information resources offered. It also provided a framework to rate these skills on a scale from poor to excellent. Over the course of the project, we analyzed over 750 chat transcripts from the work of eight graduate assistants from two semesters. We classified transcripts by patron question type and analyzed both overall and individual chat performances. Ultimately, we used the findings of our data analysis as insight to revise chat reference training, implement new procedures, and retool our supervision strategies for online reference services.