Weber State University
|
|
First Year ExperienceResults of Assessment 2006-2007 (submitted 08/01/07)
FIRST YEAR EXPERIENCE PROGRAM First Year Experience (FYE) standards of excellence were selected in advance of data collection. Most answer choices were formatted in a 5-point Likert scale ranging from least favorable rating (1) to most favorable rating (5), and achievement of FYE’s standard of excellence was defined as a mean rating of at least a 4.0. FYE reported out on all pre/post data and significant year-to-year changes of 0.5 in either direction. Additional data can be found in the appendices. 2006-2007 RESPONSES TO 2005-2006 ASSESSMENT FINDINGS 2006-2007
Assessment Year Goal 1:
For the one module that is only
evaluated within the overall course evaluation
(and with only one question),
implement a module-specific evaluation to be completed immediately after
the module. Career Services Module The FYE Coordinator, along with the FYE instructors, met with the Career Service Module liaison to discuss how to prepare students for the module and how to develop follow-up assignments. The discussion also included ways to modify modules to meet specific needs of the class, including the suggestion that EDUC 1105 instructors and their assigned module presenters meet well in advance of the module about their incorporation of the module learning outcomes into their respective course section. In Fall 2006, 43% (115) of the 265 students enrolled in FYE completed the Career Services Module evaluation [see Table 1]. Taking attendance is not required of instructors, so we cannot report a true response rate that is based on the number of students in attendance at the module. FYE’s standard of excellence was met in 8 of the 9 evaluated areas. The highest means were for the following topics: “awareness of career development class to further explore careers and majors” (4.42), “usefulness of exercises in understanding career exploration” (4.31), and “gaining a better understanding of services available through the Career Services Center” (4.30). The only rating that did not meet FYE’s standard of excellence was “gaining a better understanding of interests and knowing at least 3 to 5 careers of interest” (3.95). Our Career Services Module liaison has been informed of these findings so he can work with his team of counselors toward continuous improvement of the module.
“n” was 115 In Spring 2007, 74% (52) of the 70 students enrolled in FYE completed the Career Services Module evaluation. Once again, being that taking attendance is not required of instructors, a true response rate based on attendance at the module can not be reported. The standard of excellence was met in all evaluated areas. The highest means were on the following topics: “usefulness of exercises in understanding career exploration” (4.65), “work preferences regarding data, people, things, or ideas” (4.44), and “knowing that there is career information available on-line and how to access this information” (4.44). Data on Fall and Spring were kept separate because there is at least one significant difference between Fall and Spring FYE students: a larger percentage of Fall students are first-time freshmen.
“n” was 52 2006-2007
Assessment Year Goal 2: Investigate
the relatively low attendance numbers for the AAC Advising Module. Through last year’s assessment, we learned that attendance at the Advising Module was lower than we had hoped. We polled those instructors who consistently take roll. We concluded that there was no significant difference between Advising Module attendance and attendance during other typical class meetings. Although the FYE Department will recommend to its instructors that they take daily roll and reward exemplary attendance, we will probably not further pursue the attendance issue at this time. 2006-2007
Assessment Year Goal 3: To solicit
more detailed feedback on the areas in which FYE is not being evaluated highly. As an incentive to encourage focus group attendance, a full lunch and a prize drawing were advertised to the 100 former students who were invited to participate. Although all 100 were invited by phone call, only 13 RSVP’d. Reminder letters were sent and reminder calls were made to the 13, yet only four attended the focus group. In April, the focus group was conducted by AAC personnel affiliated with, yet not employed by, the FYE Department. For the purpose of obtaining more detailed information on areas in which FYE barely met or did not meet the standard of excellence, focus group participants were asked to complete a pre-focus group evaluation instrument that included all of the Fall 2006 student evaluation question responses that had a mean rating of 4.2 or lower [see Appendix Table A1]. Of the four participants, two students reported highly positive FYE course experiences, one student who indicated she is a high achiever reported that the course was not very beneficial to her, and one student was extremely critical of her instructor. Three of the four students’ opinions about their FYE experiences differed significantly so there was no pattern in their responses. The Department acknowledges that valid generalizations cannot be made from such a small sample size. Regardless of the small number of participants, noteworthy feedback was received including the suggestions that (1) more in-depth library training, such as how to use the copy center and researching tools, should be implemented because new freshmen are unaware of these resources; (2) FYE instructors should be mandated to administer their tests in different forms (e.g., Chi-tester and in class); (3) more instruction on Testing Center expectations (e.g. picture I.D requirement) should be shared; (4) students should be informed of all of the tutoring center locations and be taken to those locations; and (5) the service learning requirement should be made more helpful by connecting it with students’ major/career interests. The Coordinator will make sure this feedback reaches the most appropriate people so related program improvements can be considered. 2006-2007
Assessment Goal 4:
Improve FYE cluster faculty evaluations
of FYE. Although the FYE Coordinator did not schedule Fall 2006 or Spring 2007 appointments, faculty evaluations of the FYE department indicate that FYE met its standard of excellence in all three evaluated areas; this was an improvement from last year [see Table 3]. When asked what the most positive aspect of teaching in an FYE cluster has been, faculty listed seeing students mature academically, enjoying the interaction with students, and observing the diversity among students. When asked in what specific ways FYE can provide better support for FYE cluster faculty, one faculty member suggested encouraging increased discussion between faculty and FYE adjuncts. FYE plans to take this cluster faculty member’s advice. FYE is pleased about improvements in the cluster faculty evaluations of FYE.
* “n” was 6 For Spring (during which we offer few FYE clusters), only one completed cluster faculty evaluation was received, so (for confidentiality reasons) the findings were not reported in this document. As mentioned above, the FYE Coordinator will schedule appointments with FYE cluster faculty to increase the amount of feedback (e.g., the number of evaluations) received from them. OTHER EVALUATIONS OF COURSES AND SERVICES In addition to the aforementioned 2006-2007 assessment-related goals set as a result of 2005-2006 assessment report findings, the following data pertains to other areas continuously evaluated by the Department. ADJUNCT TRAINING To promote the professional development of FYE adjuncts, FYE sponsored semester Kick-offs and near-monthly adjunct meetings. To best plan future trainings, these learning opportunities were evaluated based on quality and content of the presentations. As decided before the start of the assessment year, evaluation results were only reported if five or more members of the target audience (adjuncts) were in attendance. The adjunct training schedule [see Appendix B1] consisted of two semester Kick-offs. Overall, the feedback on adjunct trainings surpassed the Department standard of excellence and attendance was deemed acceptable. In addition to the semester Kick-offs, FYE conducted six adjunct meetings/trainings [see Appendix B2]. Evaluations of each adjunct meeting met or exceeded FYE’s standard of excellence in terms of content and quality. Future training topics will continue to be based on FYE adjunct, student, and cluster faculty evaluations of the FYE programs and its services. STUDENT EVALUATIONS OF “INTRODUCTION TO THE UNIVERSITY” For Fall 2006, student evaluations of “Introduction to the University” instruction indicated that FYE met or exceeded its standard of excellence in 11 of the 14 evaluated areas [see Appendix C1]. FYE is pleased to report that all questions concerning instructor performance met the Department’s standard of excellence. When asked which course topics students thought would most help them be more successful students, the most frequent responses were time management (54), note taking (33), study skills (31), test taking (27), campus resources (22), academic advisement (12), and stress management (12). When asked what one change they recommend for future EDUC 1105 classes, the most frequent responses were to use more activities (22), assign less busywork (19), and improve the service learning module (9). Instructors will be provided with this feedback and improvements to the service learning project will be considered; feedback from this year resulted in a decrease in the service hours requirement from 10 hours to 5 hours. In Fall 2007, adjunct training will include a workshop on teaching annotation strategies (a note taking/study skills/test taking skills technique). The areas in which the standard of excellence was not met regarded course content; these means barely missed meeting the Department standard of excellence. Below-standard responses were to the following statements: “I have an increased ability to identify a major and a career of interest” (3.9), “I have an increased ability to employ effective academic skills such as note taking, textbook reading, test taking, and time management” (3.9), and “I feel this course was worthwhile overall” (3.8). These three areas were also rated relatively low last year, so they obviously need further inquiry and subsequent action. Also, regarding the major/career interest statement, this item will probably be eliminated from the course instrument now that a question seeking similar information is asked in the Career Services post-module evaluation. Note that the students rated this ability higher when it was evaluated right after the module was administered. The FYE Coordinator will consult the Career Services Module liaison to inquire about how FYE instructors might be able to help students increase their ability to identify a major and career of interest other than continuing to encourage them to meet with their academic advisor and career services counselor. Regarding the perception of how worthwhile the course is, the Department acknowledges that many students can’t possibly know how valuable the course is until they have had the opportunity to utilize what they learned in it during subsequent semesters; this is one reason why the focus group was conducted with prior students. FYE is planning to schedule another focus group made up of FYE alumni who are juniors and seniors with more experience in college post-FYE. For Spring 2007, FYE met or exceeded its standard of excellence in 13 of the 14 evaluated areas [see Appendix Table C2]. Once again, all questions concerning instructor performance met the Department’s standard of excellence. The one question that did not meet the Department’s standard of excellence was if students “had an increased ability to identify a major and career of interest” (3.9); this finding was discussed above. When asked which course topics students thought would most help them be more successful students, responses included time management (21), study skills (12), note taking (12), and campus resources (12). When asked what one change they would recommend for future EDUC 1105 classes, students indicated they would recommend more in-class activities (4), less busywork (4), and fewer essays (4). These findings are similar to others discussed above. Advising Module For First Year Experience’s “Introduction to the University” course, Academic Advisement Center advisors conduct two-day modules for each course section. These are an extended version of AAC’s General Studies Session. Data on Fall and Spring were once again kept separate. In Fall 2006, 67% (178) of the 265 FYE students completed both a pre-module and post-module evaluation. Taking attendance is not required of instructors, so we cannot report a true response rate that is based on the number of students in attendance at the module. Also, students’ evaluations were only considered if they attended both module dates and completed both the pre-module and post-module portions of the evaluation. This response rate was slightly higher than last year’s (62%). The FYE Coordinator will work with FYE adjuncts to try to facilitate increased attendance. FYE met or surpassed the Department standard of excellence in all areas evaluated in the Advising Module. The highest pre-session/post-session increases were in “how to navigate the AAC website” (1.96 rating increase), “how to declare a major” (1.57 increase), “how to access web-based information on majors” (1.45 increase), and “how to officially drop/withdraw from a class” (1.25 increase). These findings were consistent with the Fall 2005 findings.
Notes: Only feedback from those who completed both the pre-session and post-session parts of the tool was included. In Spring 2007, 72 students were registered in FYE. Of those, 58% (42) completed both the pre-session and post-session evaluation. Post-session means in all areas met the Department standard of excellence. The highest pre-session/post-session rating increases were in “how to navigate the AAC website” (2.03 rating increase), “how to declare a major” (1.77 rating increase), and “how to access web-based information on majors” (1.71 rating increase). The Department is pleased with these findings.
Notes: Only feedback from those who completed both the pre-session and post-session parts of the tool was included. Library Module For FYE’s “Introduction to the University” course, the Stewart Library conducts a one-day module. The FYE Coordinator, along with the FYE instructors, met with the Library Module liaison to discuss how to prepare students for the module and how to develop follow-up assignments. The discussion also included ways to modify modules to meet specific needs of the class, including the suggestion that EDUC 1105 instructors discuss the upcoming module with presenters well in advance of the module presentation date. In Fall [see Appendix Table D1], the Library Module surpassed the FYE Department standard of excellence in 5 of the 6 evaluated areas. The most frequent student comment was that the Library Module was very helpful (16). This year, at our request, the Library revised this instrument to conform to the 5-point scale we utilize in our other evaluations. Unfortunately, only 31 evaluations were recorded during Fall because the Library Module evaluations are completed online and the computer network malfunctioned during the administration of the evaluation. The one area in which the standard of excellence was not met was students’ indication that they felt “more comfortable asking for help from library staff” (3.97). The FYE Coordinator will work with the Library Module liaison to improve this area. In Spring [see Appendix Table D2], the Library Module met or surpassed the FYE Department standard of excellence in 4 of the 6 evaluated areas. The areas in which the standard of excellence were not met were students’ indication that they felt “more comfortable asking for help from library staff” (3.83) and “being better able to use Academic Search Premier to find full text articles” (3.83). Unfortunately, only 6 students submitted an evaluation. The low response rate has been addressed by asking the Library Module liaison and FYE instructors to ensure that students complete the evaluation before they leave the module. In an effort to further increase the number of evaluation respondents, the FYE Department will administer a paper-and-pencil instrument rather than an online instrument starting Fall 2007. The FYE Coordinator will work with the Library Module liaison to improve the areas which did not meet the standard of excellence. 2006-2007 Mentor Evaluation of FYE “Mentor Leadership Seminar” and “Introduction to the University” Instruction Fall mentor evaluations of FYE instruction indicated that FYE met or exceeded its standard of excellence in all 20 evaluated areas [see Appendix Table E1]. The areas with year-to-year differences of more than 0.5 are reported in Table 6.
* “n” was 8 When asked which topics in “Mentor Leadership Seminar” would help them be most successful as mentors, respondents mentioned mentor-mentor discussions (2), readings (2), mentor-student communications (2), service learning, and team work. When asked what one change they recommend for future “Mentor Leadership Seminar” classes, respondents suggested more of a variety of reading topics (2) and more information about the WSU campus. When asked, “Which of your roles in ‘Introduction to the University’ helped you mentor most effectively?” mentors listed getting to know students (4), teaching a class, and giving presentations. When asked what one change they recommend in the responsibilities required of a peer mentor, they suggested that they should promote better understanding of materials being taught in class while making sure teachers do not use mentors as substitute teachers. This feedback has been provided to the FYE Coordinator and FYE instructors to guide future program improvements. In response to this feedback, the Spring course was not taught around multiple themes rather than just one. Spring mentor evaluations of FYE instruction also indicated that FYE met or exceeded its standard of excellence in all 20 evaluated areas [see Appendix Table E2]. The areas with the greatest year-to-year differences of 0.5 or greater are reported in Table 7.
* “n” was 5 When asked which topics in “Mentor Leadership Seminar” would help them be most successful as mentors, respondents mentioned group discussions (2), presenting at a conference, presentation skills, resume building, and leadership skills. When asked what one change they recommend for future “Mentor Leadership Seminar” classes, respondents suggested improving service learning and focusing on long term leadership. When asked which of their roles in “Introduction to the University” helped them mentor most effectively, three respondents mentioned building relationships with students. When asked for one recommended change to peer mentor responsibilities, mentors suggested clarifying 5-minute presentation requirements and having someone other than a peer mentor conduct the campus tour. This information has been provided to the FYE Coordinator and Seminar instructor to be considered when planning future seminars. Feedback from the mentors has led to significant revisions to the “Mentor Leadership Seminar.” One of the foci of the course has become mentors’ professional development. In Spring, they were afforded the opportunity to co-present with their instructor at the state advising conference, of which two availed themselves. Also, they were coached concerning resume preparation and encouraged to meet with a Career Services counselor for further assistance. Mentors’ advice concerning the “Introduction to the University” service learning requirement contributed to project revisions. 2006-2007 ADJUNCT Evaluations of FYE Department Fall 2006 FYE adjuncts’ evaluations of the FYE Department indicate that FYE met or exceeded its standard of excellence in all areas [see Appendix Table F1]. The area with a year-to-year difference of more than 0.5 was “attributed to my students’ knowledge of WSU departments and services” (4.2 mean, -0.7 difference). Most of the means were lower than in the previous year and the Department suspects that one explanation for these low results is indirectly a result of the MAPP policy. A larger percentage of academically at-risk (e.g., under-prepared) students are taking the course, which may be more of a challenge to the instructors on a variety of fronts. The FYE Coordinator will poll the instructors on whether or not they felt this was the case. Further action will be taken subsequent to receipt of their feedback. When asked what the most positive aspect of teaching “Introduction to the University” has been, the most frequent responses were getting to know students (6), seeing a positive change in students (5) and observing the diversity of students (2). In Spring, FYE adjuncts’ evaluations of the FYE Department indicate that FYE met or exceeded its standard of excellence in all evaluated areas [see Appendix Table F2]. The areas with a positive or negative year-to-year difference of 0.5 or more were those that taught students to “demonstrate effective communication and interpersonal skills” (4.0 mean, -0.5 difference) and those that “instilled in students a sense of belonging to the WSU community” (4.6 mean, +0.6 difference). When asked what the most positive aspect of teaching “Introduction to the University” had been, respondents included watching students gain confidence (2), getting to know students (2), and seeing students apply skills they learned in FYE. When asked what specific ways FYE could better support future “Introduction to the University” teachers, adjuncts suggested focusing on the needs of developmental students, more academic freedom, and ensuring the service learning projects are more organized. Serving first-year students’ developmental needs will be one of the foci of the majority of next year’s adjunct trainings. To provide another avenue for feedback, the FYE Coordinator evaluated instructors through scheduled class visitations. She rated instructors using an abbreviated version of the student evaluation instrument. Mean evaluation scores for Fall (4.8, n=11) and Spring (4.8, n=6) exceeded FYE’s standard of excellence. ASSESSMENT CONCLUSIONS In summary, the most significant FYE changes resulting from what was learned in the 2006-2007 assessment are/will be the following: Transition to a pre/post evaluation model for all modules, affecting only the Library and Career Services modules. The same respective evaluation questions (with some minor wording revisions approved by them) would be asked. Transition to having FYE administer all the module evaluations, also affecting the Library and Career Services modules.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||