2009-2010 GCAT Assessment
Dr. Scott Tonidandel
Students that responded to some portion of the GCAT survey were separated into one of three groups: used GCAT materials, did not use GCAT materials (control), and don’t know. Students who belonged to the ‘don’t know’ group received this label because students with the same professor did not consistently identify with either of the two groups above (GCAT or control) and the faculty member failed to respond to the post- survey. Therefore this data was removed from further analysis. Displayed below is a table detailing the participation of student respondents to the pre- and post- GCAT surveys. The fourth column (“Both N”) indicates the number of students in each group who completed both the pre- and post- surveys and who also took the surveys more than one week apart.
- Pre- GCAT Assessment
The following demographic information is representative of the 576 students that used GCAT materials and completed the pre- GCAT survey. Participating GCAT students reported attending 49 colleges and universities. The majority of the students are pursuing a degree in biology (72.2%), and an additional 12.0% are completing pre-medical coursework. The majority of the participants were seniors (64.2%), followed by juniors (24.7%). For 71.2% of students, the class was a requirement of their major. Basic demographic information is provided in the table below.*
Gender (%) School Year (%)
Male 39.6 Freshman 0.0
Female 59.7 Sophomore 6.4
Race/Ethnicity (%) Senior 64.2
American Indian/Alaskan Native 0.5 Other 4.3
Black/African American 2.8 Academic Major (%)
Caucasian/White 70.5 Biology 72.2
Hispanic/Latino 7.3 Chemistry 9.2
Multi Racial 1.0 Education 0.3
Other 0.3 Math/Computer Sci. 0.7
Overall GPA (%) Pre-medicine 12.0
3.50-4.00 45.3 Psychology 0.2
3.00-3.49 39.2 Non-science 0.7
2.50-2.99 13.2 Other 3.8
* Not every demographic item’s percentages add up to 100% due to students who chose not to respond to some items or who selected multiple options on the same item.
- Post- GCAT Assessment
GCAT Laboratory Experience
GCAT Activity (%)
Make your own probe 60.6
Able to get the chips scanned 83.3
Obtain useable data from the chips 65.5
Software Used (%)
MAGIC Tool 86.6
GCAT Activity Effectiveness
The 568 GCAT students who participated in the post-survey also rated the effectiveness of each of the following activities on a 7-point scale where 1 = not effective at all , 4 = moderately effective and 7 = highly effective . Students who rated an activity “not applicable” were excluded from calculations of mean scores, which caused the sample size for each activity to be less than 568.
Students assigned an average effectiveness value of 5.51 ( SD = 0.10) to all of the GCAT activities. Mean scores on individual activities ranged from 5.23 to 5.66, which demonstrates that students did not judge any activity to be drastically more or less effective than others. Additionally, all of the average ratings are above 4.0 on the 7-point scale, indicating that students judged all of the activities to be more than moderately effective. All activities should remain in the GCAT curriculum.
Eleven knowledge questions were presented in identical forms on the pre- and post- GCAT surveys. Students were instructed to answer without the use of notes or friends, and questions presented hypothetical scenarios pertaining to gene expression and microarray experimentation techniques. The following analysis only includes the responses of the 462 students who participated in both pre- and post- GCAT knowledge tests, and who also took the surveys more than one week apart. These 462 students represent 45 different classes. Correct response rates for each item, students’ knowledge gains, and effect sizes are found in the table on the following page.
On the pre- survey knowledge test, almost all correct response rates for each question were below 50%. The mean number of test items that students got correct before GCAT was 3.65 ( SD = 2.12). Item 5 was particularly difficult for student participants; only 4.9% of students answered this item correctly on the pre-program survey. There was improvement in knowledge scores after the GCAT program; the mean correct number across all the test items after GCAT was 5.38 ( SD = 2.27). Correct responses for each item increased on average by 15.62%. Questions 1 and 4 showed particularly large gains of improvement, 37.3% and 30.9% respectively. Knowledge gain and final performance were lowest on item 5 (6.3% increase from pre-, 11.2% correct at post- assessment); subject matter for this question relates to gene expression ratios using a graph. Future GCAT faculty and students should devote more time to this area. Furthermore, fewer than half of the student participants were able to answer items 2, 3, 5, 6, and 9 correctly after the GCAT program, indicating other areas of microarray experimentation and gene expression where improvements could be made in student knowledge. A paired samples t-test indicates that statistically significant gains were observed from pre- to post- assessment regarding the number of correct answers to knowledge questions ( t (470) = 15.53, p < 0.001, d = 0.79).
*All differences were statistically significant.
In the control group, students (representing three different classes) completed both pre- and post- GCAT assessments. Lectures and reading assignments in the control classes were congruent with other classes who used GCAT materials, but the control class did not conduct laboratory experiments. Pre - and post- assessment scores on the knowledge test were examined in order to verify the effectiveness of the GCAT program. The following table compares the mean number of test items that students got correct on the pre- and post- assessments and the amount of change experienced between these two testing times.
The GCAT group had higher pre- and post- assessment means than the control group and improved approximately 8 times as much as the control group. In order to determine whether the GCAT group improved significantly more than the control group, a mixed 2x2 analysis of variance was conducted, with time (pre- and post-) being the within-subjects factor and group (GCAT or control) as the between-subjects factor. The ANOVA showed a significant main effect of time, F (499) = 17.785, p < .001. This result indicates that there was significant change in knowledge test scores from pre- to post- assessment collapsing across group. A significant Time x Group interaction was also obtained, F (499) = 11.197, p = 0.001, d = 0.72. The rate of improvement from pre- to post- assessment significantly differed between the GCAT group and the control group. As displayed in the following graph, both groups improved over time, but the rate of improvement for the GCAT group was significantly greater than that of the control group over the course of the semester. Given the wide variety of activities that different classes may have engaged in and the fact that not all of the topics included in the knowledge test would be covered in individual classes, this result is extremely promising for the GCAT program.
Both the 472 GCAT students and the 30 control students rated how interested they were in genomics, life sciences, math/computer science, and research on a 10-point scale in the pre- and post- GCAT surveys, where 1 = not interested at all and 10 = extremely interested. Displayed below is a table with the average interest score for each area on the pre- and post- assessments.
Four 2x2 mixed ANOVAs were performed in order to identify any statistically significant differences in interest between the GCAT group and the control group. For genomics and math/computer science, the results showed no statistically significant difference between the GCAT group and the control group in terms of change in interest from pre- to post- assessment and showed no significant difference from pre- to post- collapsing across groups. Interest in life sciences showed both a significant interaction and a significant effect of time across groups. A significant Time x Group interaction of life sciences was obtained, F (488) = 11.369, p = 0.001, d = 0.70, demonstrating that while the control group’s interest in life sciences decreased, the GCAT group remained about the same. The effect across time, F(488) = 10.700, p = 0.001, showed that on a whole the interest in life sciences decreased from pre- to post- test. Also, a significant interaction was obtained for interest in research, F(493) = 5.402, p = 0.020, d = 0.24, which indicates that the control group’s rate of interest change was significantly less than that of the GCAT group.
55 faculty members responded to some part of the post- GCAT survey, 52 of which reporting that they used GCAT materials and the remaining 3 being in the control group. 8 professors responded to the survey twice due to their participation in both semesters of the 2009-2010 school year. All professors who participated in the survey twice used GCAT both semesters. Therefore, there were 63 total responses to the post- GCAT survey, 60 of which used GCAT and the other 3 reported being in the control group.
Of the 63 classes using GCAT materials, 60.3% of the teachers reported having fewer than 10 students use microarrays. The average number of microarrays used was 10.88 ( SD = 10.70). The average number of students who obtained useable data was 6.00 ( SD = 7.69).
Selection of GCAT Activities, Time Spent on GCAT Activities, and Assessment of Students’ Knowledge
Faculty members were asked to indicate which activities students participated in using GCAT materials and how many hours were allocated to each activity. They were also asked about the methods used to assess students’ knowledge of genomic course material. Only a few teachers responded to questions in this section. In response to a question about which activities the students did while using GCAT materials, 2 teachers responded “analyze data from a public domain source”, 1 responded “analyze their own data”, and 2 chose “other activities”. The teacher whose students analyzed their own data reported spending 6 hours on that activity, while the teachers whose students analyzed data from a public domain source reported spending 3 and 8 hours on that activity.
Teachers were also asked about the methods they used to assess students knowledge and understanding of genomics course material. Two teachers reported using a term paper or lab report. Two other teachers reported using informal feedback. One teacher used poster presentations and another teacher used tests.
Funding and Implementation
Of the 49 responses to the survey questions on funding, 38 reported receiving departmental funding in order to utilize GCAT materials. Six faculty members were supported by institutional funds, three reported receiving extramural funds, and the remaining two indicated that they received no funding to use the GCAT materials. Based on 48 responses, the average amount of funding received was $4,400.00 ( SD = $14,644.77). Of the professors who responded to the implementation question, 75% of the professors did not feel that their implementation of GCAT materials was limited by computer resources (3 professors did not respond at all).
Professors’ Evaluation of GCAT
After the GCAT program, professors rated their agreement with the following statements on a 5-point scale, where 1 = strongly disagree and 5 = strongly agree . Overall, the GCAT program was rated very favorably. Almost 70% of the faculty respondents strongly agreed (score of 5) with the statement “Overall, I had a positive experience using GCAT”.
There were more students this year who took both the pre- and post- surveys and who used GCAT than there were last year (472 this year and 355 last year). However, this year’s GCAT program had much fewer control groups than last year. Last year, there were 183 control students who took both the pre- and post- test, but there were only 30 this year. Efforts to further increase the sample size of the control group will allow the comparison with the GCAT group to be made even more easily. In the GCAT group, there are also still many students who do not complete both pre- and post- surveys, meaning that many students’ data could not be analyzed. This year, around 400 students’ data could not be used because of the students’ failure to complete both pre- and post- assessments or because they did not do so with an appropriate time period between the two (e.g. took the surveys less than one week apart). Therefore, continued efforts should be made to ensure participation by all students throughout the GCAT survey process.
Furthermore, a large amount of data (180 students) was discarded because students with the same professor were unable to consistently identify with one group (either GCAT or control) and their teacher failed to respond to the post faculty survey. Any gains we can make in improving faculty responses would also dramatically improve our sample size. Additionally, faculty members should be reminded to instruct their students that the pre- assessment should be taken before the administration of the GCAT materials and that the post- assessment should be taken after the completion of the course. There were still some students that were completing both of the surveys within hours of each other and had to be removed from analysis.
Faculty Comments 2009-2010
The microarrays that I used in 2009-10 were performed last summer. Because of this, the students did not perform either the pre nor post assessment for GCAT.
Some of the protocols online could be updated. There could be a 'helpful hints' section on the GCAT protocol page.