VR Learning Platform - Efficacy Study Results February 2019
Sama Learning Deploys VR for General Chemistry at Scale
Sama Learning is a platform for the creation, iteration and distribution of STEM-based, VR courseware at the University level. The initial product is a library of VR lessons which incorporate exposition, active learning, problem-based inquiry and assessment in a virtual environment. These lessons are accompanied by Sama Learning’s analytics engine that track student interactions with the content, and performance on the integrated assessments, in order to correlate a student’s pathway through the content and their learning outcome. Efficacy-based content is core to Sama Learning’s methodology. In order to validate our effectiveness, Sama has established relationships at three Universities, in collaboration with Google providing Lenovo Mirage Solo hardware, to validate learning outcomes, overall efficacy and ability to deploy the coursework at a scale to support the general chemistry students in a university/college setting.
Sama’s pilot included three Universities: a California State University campus (CalSt), a private liberal arts college (LA), and an R1 institution (R1) and covered 796 students. The pilots were designed to evaluate a) deploying VR at scale and b) to quantitatively evaluate the learning outcome improvement gained from incorporating the VR lesson as an adjunct to the existing instructional strategies. Each University integrated the same lesson on Atomic Orbital Theory to their curriculum although each used a different implementation strategy.
At CalSt the VR lesson was assigned as an out-of-class task and students were given points for its completion. Groups of 25 students went through the course concurrently. Over the course of 3 days 414 students utilized the lesson, which constituted 99% of the General Chemistry student population. The learning outcome improvement was evaluated by a pre/post testing scheme which collected quantitative assessment of students’ mastery as well as qualitative data on their experience.
At R1 the lesson was integrated into the lab course which accompanies General Chemistry instruction and delivered to all 305 students enrolled in the course, with 36 students in the VR lesson concurrently. A mixture of qualitative and quantitative data was collected in a post survey to evaluate the student’s perceptions of the utility of the VR experience.
At LA a control trial was run which consisted of 77 student volunteers. Of those 77 students, 33 were given the VR lesson, in groups of 5-12 students, outside of class with the rest receiving the standard instruction only. Learning outcome improvement was evaluated via a quiz and in class final data evaluated across the experimental and control groups.
A brief demonstration of the user experience can be seen here.
General Findings:
The primary question being evaluated during the program was: What is the magnitude of learning outcome improvement shown by students with the incorporation of Sama Learning’s VR lesson. In both the pre/post testing and control trial, an average of a full letter grade improvement was observed. This magnitude of learning outcome improvement is significant in that the improvement of a letter grade in STEM classes is correlated with increased probability of students completing a STEM degree. Additionally, there was no gender dependency on the improvement observed, which was the only socioeconomic data collected. These results indicate that the use of Sama Learning’s VR lessons can significantly improve student learning outcomes across institutions.
Research Results from CalSt:
Average improvement from pre/post testing - 1 full standard deviation (pre avg = 4.02 ± 2.08, post avg = 6.12 ± 2.2)
Average individual improvement was 198% of pre-test score
72% of students showed some improvement
No gender dependence
Poorest performers on the pre-test showed largest improvement (277% average
individual improvement for students who "failed" the pre-quiz, vs the average individual
improvement of 198%).
The percentage of poor performers (F-D) who showed any improvement is 98%, vs. 70%
of the students who showed avg performance (C-B students) on the pre-quiz, and 53%
of students who demonstrated mastery (B-A students)
67% of students either agreed or strongly agreed that VR as an adjunct can improve
learning. Interestingly this is roughly the magnitude of students who showed some improvement. Further analysis is required to understand if there is a statistically significant correlation here.
Research Results from the LA Study:
The primary question being evaluated during the LA pilot was: What is the magnitude of learning outcome improvement shown by students with the incorporation of Sama Learning’s VR lesson. The results from the control trial are consistent with the pre/post testing scheme. In the LA study both groups, experimental and control, were given an identical quiz. The average score of the experimental group was a letter grade higher than the control group. Additionally, there was no gender dependency on the improvement observed, which was the only socioeconomic data collected. .
LA Control Study Results
Survey Results from R1:
100% of students responded yes when asked, “Adding VR to this course would help me learn
more effectively (y/n)”
92% of students responded positively when asked, “If given access to the VR lessons outside of
class time they would use it”
Observations
The pilot was also intended to evaluate our ability to integrate VR into instruction across an entire departments general course scale (typically 150-400 students for each department at each school), and the frequency with which VR should be utilized in the classroom given the current pedagogy. Using the standalone Mirage Solo headsets we were able to test multiple scenarios across 752 students with relative ease. This required a 5 minute tutorial on the use of the hardware prior to starting the lesson. The average time a student spent in the lesson was 43 minutes, with the longest time spent in the lesson being 120 minutes. A debrief with instructors led to the conclusion that 30-minute lessons were an ideal target for future courses. This is driven by two factors, the time available for a lesson of this type and the length of attention/engagement of the students. Maintaining adequate headset and controller charge at this scale did provide challenges which was mitigated by ensuring that headsets were charged immediately before and after use. For the highest volume of students 2, 1-hour workshops were followed by a 1-hour charging block. The most common complaint from the students was the weight of the headset during extended use. At LA when students were asked for the desired frequency of using VR as an accompaniment to their existing instruction, 66% respondents desired to have VR either once/week or once/month. This is in line with Sama Learning’s product design with VR lessons being delivered to students 2-3 times/month.
On-going Work
There is continued analysis of the collected data. The learning outcome improvement will further be evaluated by the relative performance of the experimental and control groups at LA against their final exam data. In addition to the learning outcome assessment, data was collected to evaluate the impact Sama Learning’s lessons had on a student’s growth vs fixed mindset (developed by Carol Dweck). Lastly, detailed analytics data was collected during the student experience in VR. This data included specific events which were tracked (i.e time to advance, number of problems correct, interaction events), as well as the frame-by-frame location of the headset, remote, pointer and button presses. This data is currently being analyzed to determine correlations between what the student was doing in the lesson and their learning outcome improvement. The results of this type of analysis will be used to develop adaptive learning algorithms to guide students to their optimal learning outcome, as well as understand what components in the VR lesson are most impactful to inform future developments. It is likely that larger student populations will be needed in order to develop adaptive learning models with high enough confidence.
Sama Learning is currently developing additional lessons to accompany University General Chemistry, with additionally planned lessons in Physics and Biology. These lessons will address the largest challenges in instruction currently. The success of this pilot has led to ongoing partnerships with the three pilot institutions with all three agreeing to adopt VR experiences into their curriculum. CalSt is going so far as to include the addition of a space for VR instruction in the plans for a new Chemistry building to be opened in 2020. Sama will expand our research to include the impact of multiple exposures to VR courses impact the students learning and self-assessment of their capabilities. Internal data evaluation will be focused on understanding what components of the VR lesson lead to this large improvement in learning outcomes.
Additional information can be provided by sending a request to barb@samalearning.net or chris@samalearning.net