John A. Tures, LaGrange College
Simulations have been frequently mocked in the popular media as a poor substitute for the real thing. But practice tests may provide the ability for students to properly prepare for the real exam. In this article, I examine evidence from several classes of mine, which test whether such exam simulations help students on actual tests, what percentage of students show improvement, and whether those who take practice tests outperform those who choose not to try them. I conclude with ideas about how best to use simulated tests in an online format, beyond the pandemic era.
Introduction: Aliens Movie & “Simulated” Jumps
In the 1980s movie “Aliens,” female protagonist Ellen Ripley asks the by-the-books young officer Gorman “How many drops is this for you, Lieutenant?” In the middle of a rapid descent onto a hostile planet, Gorman replies “Thirty-eight…simulated.” Tough space marine Vasquez snaps “How many ‘combat’ drops?” Gorman answers “Uh, two….including this one.” The others in the platoon react with dread at this admission of inexperience.
Flying from a mother ship to a relatively barren planet in a hostile environment may sound “Alien” to most educators. But so too was the environment of COVID-19 and the need for an overnight transition from in-person instruction and printed exams to a virtual world. I thankfully had taught summer online classes for several years. I knew what Zoom was (having utilized the technology for interviews for articles), and knew how to employ challenging open-book essay- based exams.
Providing online tests in a multiple-choice format that could be graded was another matter.
Distributing practice exams in a way that could generate quantitative results was out of the question, as I simply struggled to simply provide multiple choice questions for our college’s LMS system. I did attempt to utilize a “Kahoot” system, yet found myself with low involvement in online practice tests (40% participation). I also had a great difficulty in generating data that would enable me to see what was working, as students would sign in under unfamiliar usernames.
Help from a math professor who also knew more about the virtual environment assisted me in completing the Spring 2020 semester with online exams that students could take, and prepared me for the Fall 2020 semester, ready to use our campus LMS system. Moreover, I learned from the Spring 2020 term that unlike other practice exams I had experimented with, data could be downloaded into a usable format. But would students adopt the simulation? Would it help them in their grades for the real online tests?
Putting Online Practice Exams Into Practice
I administered seven exams overall to three of my 2020 classes: government, political economy and law. The class average on the practice tests was 57.81 while it was 82.8957 on the seven actual exams, an improvement of 25.087 points (the t-statistic was 6.58, with a p-value that was significant at the .0001 level).
This was not just a matter of only a few students showing improvement, as you can see in Figure 1. More than 80 percent of the class recorded better grades on the actual test, over the simulated exam, while the rest maintained the same level or declined in their test scores. One of the decliners was a student who got a perfect score on the practice test, and dropped only a few points on the actual exam, still receiving an “A.”
Not all students opted to take the practice test in my government class, a course filled with both majors and non-majors. I was able to examine the performance on the exam between those who had taken a practice test, and those who had not. The class average on the actual exam was 88.5% for those who took a practice test beforehand, and only 73.66% for those on the actual exam who chose to skip the simulation, a gap of nearly 15 percentage points, as you can see in Figure 2. A t-test produced a t-value of 8.46, significant at the .001 level. Granted, it was for a small sample size, since my political economy and law classes, full of experienced political science majors, know to take the practice tests, from experience, to do better on the actual test.
Lessons For Examinations Beyond The Pandemic Era
During the 2021 academic year, I continued the process, even as campus restrictions began to loosen a bit, for several reasons. I have employed practice exams in my courses since I was a graduate student instructor, even while teaching in other countries. But I found that online exams provide several advantages that transcend even offsetting the concerns about the coronavirus crisis and the need to go virtual.
First, online exams demonstrated themselves to be a useful tool in student instruction. Those taking the class outperformed their counterparts on the real exams if they had taken a simulated test, and often by several points. Sometimes, the students tried the exam multiple times to improve their scores, without even a suggestion from me to do so. By the end of the semester, fewer and fewer students eschewed the practice quiz; in some classes, every student took the practice test.
Second, such virtual simulations, when done through an LMS, provide the instructor plenty of information. Long Before COVID-19, I had been giving printed out practice tests to students. But these were printed out. I would read out the answer key, or email it to students, so they could study from it. Now, thanks to our LMS, I was able to gather data on which students were taking the practice exam, and who weren’t. I could see how students were doing, which questions they were getting right and wrong, and how they were improving.
Third, I noticed that students were not just showing improvement on other exams. Every year, our political science senior seminar students would take a nationally-normed exam. They used to perform below average on these tests, compared to our other college seniors and their peer group of other universities. But since we started getting practice exams to students, they’ve shown an ability to learn how to prepare better for such exams. They now perform much better in comparison to their fellow seniors, in our college or other similar schools.
There are some tips I would offer for instructors willing to employ these methods in their class, in-person and virtual. First of all, let students take the practice exam several times, so they can get used to preparing for the actual test better. Second, prepare to check on the practice test results before you give out the real deal, so you can be ready to help those who need it, based upon their performance. Third, gather the data so you can run such analyses, to see if your students are doing better on the practice and actual exams, the percentage of the class showing improvement, and whether those who take the practice test do better on the actual exam than those who choose not to do the practice test. It lets students. Showing this data to students lets them see the statistical value of these preparations. Finally, make sure you write the practice exam early enough so the students have time to utilize it before the real exam is given out.
Though virtual simulations were mocked in the “Aliens” movie, my research has shown that they can play a valuable role in a student’s education. They’ll never replace the real thing, but practice can make your pupils pretty good, ready for the non-virtual experience.
Published since 2005, The Political Science Educator is the newsletter of the Political Science Education Section of the American Political Science Association. All issues of the The Political Science Educator can be viewed on APSA Connects Civic Education page.
Editors: Colin Brown (Northeastern University), Matt Evans (Northwest Arkansas Community College)
APSA Educate has republished The Political Science Educator since 2021. Any questions or corrections to how the newsletter appears on Educate should be addressed to firstname.lastname@example.org.
Educate’s Political Science Educator digital collection.