• Evaluating the Overall Effectiveness of Professional Development Courses: A Pre- and Post-Graduation Study

    by Thomas Hadley, Linda Bates-Parker, Katrina Jordan, Velta Kelly-Foster, and Julia Montier-Ball
    NACE Journal, September 2010

    Many career centers teach for-credit career development classes—learn how one university career center determined the effectiveness of its courses.


    Annually, the Career Development Center (CDC) at the University of Cincinnati (UC) teaches a one-credit Professional Development II (PDII) course to more than 970 students. The PDII course is mandatory for graduation from the colleges of business and engineering, and a liberal arts version is taught as an elective for the college of arts and sciences. These are the three academic cohorts surveyed in this study.

    The study was developed on the premise that the requirements of good course assessment usually present a challenge to those who teach. Also, it was hypothesized that the variation in students’ characteristics, environmental influences, work/family conditions, demographics, and so forth, might affect their potential for learning, retention, and use.

    Objectives of the survey were to:

    • Determine if the program was meeting course objectives;
    • Discern usefulness of course design, components, and instruction;
    • Measure learning outcomes over time;
    • Assess content relevancy;
    • Develop best practices for the profession;
    • Strengthen the case for course validity in times of budget cuts in higher education; and
    • Serve students’ (customers) needs and expectations.

    Although valuable, this type of course assessment was difficult, as participants were elusive and only a modicum of relevant data was accessible. But the authors took on the research challenge because they wanted to know how and what students were learning, were curious about the Gen-X and Millennial generational mindsets, and must adapt to meet students’ needs.

    Telling the Story

    The business and engineering surveys were identical, but a different pre-survey instrument was used for the arts and sciences cohort (the less career-focused group). This was because, although it was the smallest group in the study, it mirrors the diversity of students, majors, and nontraditional students in the college.

    The customized arts and sciences survey also took into account that these students were balancing school, work, and life, with little time for career planning.

    Thus, the PDII instructors on the research team, who had experience with these students, helped shape survey questions that could determine the factors that could affect student learning outcomes, i.e., class participation, understanding course concepts, quality of homework assignments, and post-grad application of job-search strategies.

    Phase I pre-survey data from the initial PDII course evaluations completed by students 2007-2008 are shown in Figures 1 through 3. A total of 662 students responded—which included 236 engineering students, 304 business students, and 50 arts and sciences students.

    The study was conducted in three phases over two years, asking respondents to evaluate each course component. The findings are based on survey participants reporting that they had:

    • Significant learning outcomes and were empowered to use them (Phase I survey).
    • Retained what they learned from the course through graduation (Phase II survey).
    • Used what they learned in conducting their job searches after graduation (Phase III survey).

    montier figure 1Phase I: First survey (pre-grad), a paper version given at the end of each quarter. To assess student learning outcomes from the PDII course over time, students were surveyed at the end of each quarter for one academic year. CDC’s 2007-2008 pre-survey course evaluation was a paper survey given in class by instructors; thus all students taking the PDII course from the three cohorts completed the survey.

    Phase II: Second survey (postgrad), an online version given three to nine months after the first survey. Next, the team measured how much of what students had learned in class was retained within three to nine months after taking the course, which for seniors would be after graduation. There were 116 responses; however, there were 374 comments written in—on everything from what they did not learn to advice on how the class could be improved—providing more evidence of how engaged students can be when given an opportunity to evaluate a process.

    Phase III: Third survey (post-grad), a phone version given nine to 18 months after the first survey. Finally, the research examined which aspects of the course were used and when by recent grads during their job search. It was determined how long it took them to find a job because, by this time, most had graduated, had received at least one job offer, and had secured employment. An e-mail was sent to 116 respondents to obtain accurate phone numbers; a total of 49 responded and took the third survey.

    montier figure 2Phase IV: Data analysis, organization, and presentation of findings. Throughout the study, categorical titles were used to cross-reference key findings in all phases. In Phase IV, all the findings, including progress reports and raw data, were compiled and organized for easy reference in a customized data base.

    Key Findings

    To process the findings, the attitudes and characteristics of the respondent populations, a blend of Gen X-ers and Millennials, must be considered. Although a decade apart, both generations share an obsession with technology and an affinity for mass-media marketing.
    Taking into account that students are well-integrated into a tech-savvy pop culture allows the filtering out of any career development precepts of previous cultures. It is hoped that the findings will help other career educators adapt their career/professional development course design to meet the needs of teaching their junior and senior students.

    What Students Learned

    The results from the first pre-graduation survey showed that a high percentage of students learned how to write targeted resumes and cover letters and could field behavioral questions during an interview. Some variations between engineering and business students were evident, possibly because co-op is mandatory only in the College of Engineering. But the respondents also confirmed learning newer processes, such as identifying transferable skills that are relevant to employers’ needs and identifying/articulating strengths and weaknesses.

    Students also uncovered attributes they were unaware of through self-assessments relating to personal qualities, values, interests, and personality type. The collected data from each college cohort was closely examined and revealed the soon-to-be-graduates were assessing what they had learned from taking the PDII courses and what they were actually using in the job search. A total of 68 percent reported having found a job within three months.

    Job-Search Strategies

    The top three job-search strategies covered in the PDII course that recent grads said were most helpful in obtaining full-time employment after graduation were:

    • Internet job searches on sites recommended by CDC instructors (64 percent)
    • Networking and/or informational interviews (61 percent)
    • Job-search strategies discussed in class and individual advising sessions (53.8 percent).

     Many of the skills taught in PDII and used later were difficult for Millennials and Gen-Xers to learn. In some cases, these skills, once learned, were ranked the highest:

    • Researching companies to prepare for interviews—77 percent
    • Expanding your network and making new contacts—75 percent
    • Articulating transferable skills and strengths in interviews—64 percent
    • Answering behavioral questions in interviews, giving examples—62 percent
    • Following up on job leads promptly/sending thank-you notes—62 percent
    • Doing advanced training programs, internships, and/or volunteer work to gain more career-related experience—50 percent.

    Gen-Xers and Millennials were able to gain confidence to successfully embark on the job search by using the PDII course components. The post-grad survey found that 59.8 percent had started their job search one to six months prior to graduation using what they learned in PDII classes. A few still reported being undecided.

    Efficacy Assessment

    Combined responses from all three cohorts indicated that 34 percent of respondents “feel much better prepared to launch a job search today than others I know that did not take PDII,” while 46 percent said “I feel moderately/somewhat better prepared to launch a job search today than others I know that did not take PDII,” and 19 percent responded “I feel as prepared to launch a job search today as others I know that did not take PDII.”

    About the Job Search

    Recent grads said they had retained a great deal of information over time (three to nine months) and were able to relate which job-search strategies covered in PDII course components, activities, and assignments were most helpful to them in obtaining full-time employment after graduation.

    Many students followed up with instructors after taking the 10-week PDII course and received additional career advising. To help recent grads further improve their skills, CDC held a job-search boot camp where employers helped to critique interviews and resumes.

    Best Practices

    montier figure 3 1. Create, state, and evaluate student learning outcomes (SLOs) to strengthen your case for teaching career or professional development courses. For the career center to remain viable in the wake of budget cuts, it is important to define what the center does, be prepared to explain how the work adds value to student learning, and promote the courses taught with confidence, because these courses are an essential contribution to the senior year experience. To get started, career development educators can take any career development curriculum offered and state the goal and course objectives; then ask what a student should know and/or be able to do after taking the course.

    For example, UC’s PDII course should have three components in order to measure its effectiveness:

    • The Learning Goal: Students should learn skills needed to transition successfully from college to career. (This is the foundation upon which the course is taught.)
    • The Course Objective: To help students identify transferable skills relevant to employer needs and write targeted resumes. (This pinpoints what the CDC expects them to know.)
    • The Learning Outcome: Write a targeted resume using the chronological format, bulleted action verbs, and keywords from a detailed job description. (This is what students should be able to do.)

    2. Career development educators should design relevant courses with the flexibility to either fill in the gaps or go beyond what may have previously been taught in other courses. The bar must be set higher for the more sophisticated Millennial minds of current engineering and business students (the two more career-focused control groups in the study). Some write-in comments included complaints about not learning new information in the course, due to having taken a similar co-op prep course in sophomore year. One student noted that, “The curriculum of the course needs to be revised. Students who have participated in co-op already know everything that is taught in PDII.”

    But what this student hasn’t considered is that obtaining a co-op job does not require them to engage in a full-scale job search in a competitive job market. With that in mind, course design needs to be adaptable, incorporating current industry and economic trends to teach an advanced agenda to add to the basics.

    Some students may naively only have a “Plan A” regarding the job search. Career services should challenge them by discussing what “Plans B and C” might look like, in the event that the co-op job does not transition into full-time employment, and/or the economy does not turn around quickly.

    One idea is to offer a variety of guest speakers with real professional networking opportunities. This was by far the highest rated experiential exponent of PDII by all students. Involving employers in discussions or mock interviews in class requires students to participate.

    Educators are aware that students only know what they have experienced, and CDC staff has discussed possibly making the engineering and business course syllabus more challenging. But in analyzing the data, they realized students felt as a result of taking PDII they had experienced significant learning outcomes in every area that would help them transition successfully from college to career.

    3. Design career development courses to meet students “where they are.” Career educators should shape professional/career development courses to meet the needs of their student populations derived from regular course assessments, student evaluations, current course design research, and career adviser and employer focus groups to receive feedback on possible gaps in student learning. Some aspects to consider when designing courses include generational characteristics and means of communicating; type of major (career-focused vs. non-career-focused) and its particular experiential learning requirements; and class status, preferably juniors and seniors, but career-mature sophomores could also benefit.

    The 1,297 write-in comments may be the most interesting part of the Phase II findings. Students reported that they felt taking the course before senior year was optimal, as that would allow for one to two years in which to integrate learning outcomes and build efficacy in the application of job-search strategies. Following are comments from two students on this topic:

    • “Offer it earlier and make it more robust. Also, consider offering specialized courses based on career path. For instance, my job search was significantly different because I was searching for a job in a creative field. Techniques for such a job search were not covered in this course.”
    • “It’s not that I didn’t use the ‘components’of this program, it’s that by senior year I basically knew them all. Honestly, this class needs to happen freshman or sophomore year, especially for students participating in the co-op program. I had already learned most of these things.”

    Additionally, a variation of the PDII syllabus was created for arts and sciences students, which includes some assessments and assignments to better meet their needs. During the last four years, this syllabus has undergone several critiques by the CDC’s curriculum expert. After these revisions, students report they appreciate the  opportunity to use such tools as the “values assessment” that can link their top rated values to certain occupational information.

    Conclusion

    Professional development courses should be designed to add value to student learning, with specific goals, objectives, and desired outcomes. Courses should be adaptable to ensure they contain new information rather than the same content as other classes offered. Finally, career services should use various methods to evaluate and update the classes, including reviewing course assessments, student evaluations, course design research, and focus group feedback.



    Copyright 2013 by the National Association of Colleges and Employers. All rights reserved.


Evaluating the Overall Effectiveness of Professional Development Courses: A Pre- and Post-Graduation Study