Description:

During a phone call with a technology provider, the conversation turned to the platform’s claim of strong student engagement. The representatives on the call touted the account activation rate as proof of its system’s strong student engagement. While the activation rates were impressive, I wondered, if a student logs onto a platform once, does that indicate engagement? This question prompted me to consider Carnegie Mellon University’s career operations and how we measured student engagement. On our marketing materials, we touted 9,000+ student appointments annually. The office created a nice infographic showcasing that one-third of the student body attended our on-campus career fairs the previous year. And we bragged about our activation rates on our career platform. I am proud of these numbers and I believe they speak well to the work we are trying to accomplish. But I realized these individual data points did not tell a story of engagement, just that our office was busy. I also realized these data points overlooked many students. Our team wondered: How could we account for the students who never scheduled an appointment because they had time conflicts due to a job or athletic commitment…but who were heavy users of our online educational content? We asked: How we account for the student who is dealing with severe anxiety and will never attend a crowded career fair…but who is a super-user of our online resources? We considered: What does it mean for a student to be engaged with the Career and Professional Development (CPDC)? And finally, we asked: Are we okay if students are not engaged with the CPDC? These questions led me to pull together a small team to further explore these questions. Our first task was to identify activities that we deemed as high-impact and requiring students to initiate a significant effort. These activities would then be compiled into a rubric that would generate a holistic engagement score. We considered what a first-year student might be expected to do versus a senior. We examined how to structure the rubric to reward ongoing effort instead of a flurry of activity during a single semester. Finally, we assigned weights to the different activities by assigning effort thresholds and limits to the number of credits one could receive in a particular activity category. Right away, our astronomical activation rates were doomed as an indicator of student engagement. Logging onto a website is not a high-impact activity nor a significant effort. But applying for at least three employment opportunities did meet the high-impact and student-initiated criteria for inclusion in the rubric. Over the past three semesters, we continued to iterate and refine the rubric and now have well tested engagement scorecard. As I met with campus leadership and shared this initiative, the application of this information evolved from our initial plans. Institutional Research and Analysis assisted us by supplying user demographics of our low-engaged students to learn who constituted this particular cohort. Conversely, knowing the student groups who make up our high-engagement cohort helped us program more efficiently. Academic leadership collaborated with the CPDC in outreaching to students who may be under-engaged on campus. Senior leadership is excited about the CPDC initiative as it may help identify additional high-impact educational practices that promote student satisfaction, retention, and successful outcomes. In today’s data-driven decision making environment, university career centers need to rethink how we measure student engagement. Singular metrics such as appointments, interviews, and overall program attendance make for a nice infographic, but they are only individual data points that do not allow leadership to draw any significant inferences. We need to create systems that allow for a holistic view of how students engage with a multi-faceted career services operation. In this session, I aim to show attendees CMU’s early efforts in changing the way we measure student engagement and how we act on this information.

Audience:

Career Services

Level:

Intermediate

Track:

Data Analytics & Decision Making

Type:

Traditional

Main Speaker:

Kevin Monahan, Carnegie Mellon University

Additional Speakers:

Doug Stouch, Carnegie Mellon University