Spotlight for Career Services Professionals, February 15, 2012
by Kevin Gaw 
The other day, I was assisting a career center team with one of their assessment projects and the topic of scaling came up. We had a really fun and important conversation about the design of the scales (OK, call me an assessment nut!). We wrestled with questions, such as:
- What specifically are we trying to measure?
- What would be the appropriate scale to use?
- What about the intervals on the scale?
- What about the anchor points, and how are they defined?
- Does the question match the scale?
- Will the scale deliver interpretable information?
The team recognized how important it is to answer these questions, all in advance to launching the project and collecting the data on the student learning outcome (SLO). Thus, the topic of this column: the importance of scaling.
The scale(s) used for an SLO project will directly influence how data are reported—this is why it is important to get it dialed in. The scale essentially standardizes into units the multiple levels of the construct being measured. Careful scale construction allows you to make important interpretations from the data. Poorly designed scales can jeopardize your efforts. Following are some of my ideas on the topic of scaling.
Let’s say we want to measure “career confidence.” Our hypothetical SLO is that we are measuring the career confidence of students after engaging in career counseling and learning how to make career decisions and plans. (Note: We have defined career confidence as being confident of one’s career decisions and career plans). We could develop a self-report rating scale that might look like:
Using the scale below, rate your level of career confidence as of today.
0 = I don’t feel confident with my career decisions and/or career plans. I feel lost and I need help deciding what to do with my major and my career.
1 = I feel slightly confident of my career decisions and/or career plans. I feel confused between several options and I need help in deciding what to do with my major and my career.
2 = I feel confident of my career decisions and career plans. I know what I want to do, but I have questions about how to make it all happen. I could use some help.
3 = I feel very confident of my career decisions and career plans. I know what I want to do and how to make it all happen. I may need assistance with some skills to help me out.
4 = I feel extremely confident of my career decisions and career plans. I know what I want to do and how to make it all happen. At this time, I do not feel like I need any assistance.
Notice that at each level there is sufficient description that defines (operationalizes) that level. This is done to help the rater (in this case, a student who has participated in career counseling) answer the question. Also, notice that the scale starts at 0: this is done with the assumption that when one feels no confidence, that state should be measured as zero confidence. Frequently, scale developers start scales at “1,” but mathematically, the scale actually starts at zero because the construct of interest theoretically has also a “zero point.”
Any value above 0 and below 1 is still interpretable. This allows one to interpret the data more accurately because the construct is anchored at zero, “no confidence.” What happens if you run this scale in a pre-post design, assessing career confidence when students initially arrive for their first appointment of career counseling and then again after three sessions. The change scores would then reflect student learning (they feel more confident with their career decisions and plans).
Scales that have intervals that are close for some levels, but further apart for some other levels introduce interpretation “noise.” Additionally, the mid-point of the scale ought to represent a theoretical mid-point of the construct. For example, the scale on the left, below, has three potential issues:
- It starts at “1” and therefore may not account for the construct’s actual zero state.
- The second and third levels (“not very important” and “somewhat important”) are measuring virtually the same thing.
- The mid-point of the scale (#3) conceptually is very close to #2 but very far from #4.
These issues make interpreting data with this scale a potential challenge. The scale on the right attempts to resolve these concerns. Small differences in scale development can yield big interpretative gains.
A possible scale, but…
|
This scale could be alternatively written as:
|
1 = Not important
2 = Not very important
3 = Somewhat important
4 = Very Important
5 = Extremely important
|
0 = Not important
1 = Somewhat important
2 = Important
3 = Very important
4 = Extremely important
|
How about if we want to measure “appropriate professional attire” at a career fair? Since we are measuring a student learning outcome, we are observing the students with whom we’ve worked. Let’s assume we have run a two-session “Dress for Success” workshop, complete with a fashion consultant and a practice event. We know the student participants and as they check in for the actual career fair, we rate their professional attire. Notice the deliberate levels of the scale below, and the attempt to create “equal” intervals along the scale.
Rate each Dress for Success Workshop participant’s career fair attire, using the scale below.
0 = PJs, slippers, shorts, T-shirt with political statements, messy hair, etc.
1 = Jeans or pants with holes, sneakers, T-shirts, shorts, club wear, etc.
2 = Jeans, dress shirt, casual sweater, loafers, etc.
3 = Business casual—button-down shirt and dress pants, blouse and skirt, dress shoes
4 = Coat and tie, pant suit, professional skirt and jacket, dress shoes
One last quick point: Using scales without definitions to operationalize each level runs the risk of missing out on interpretable data. For example, researchers frequently use seven-point scales because of the variability in the data. Variation is great, for sure. But what often happens is that only the upper and lower ends of the scale are defined, and the rater (student or observer) has to figure for themselves what the other levels might mean. A rating of “4” for you may mean something entirely different than my rating of “4,” and therefore the data is not the same. Even with a mid-point definition, the other non-defined values have sufficient variability in meaning that the data will be difficult to adequately interpret.
Using the scale below, rate your level of career happiness.
0 = Not happy at all
6 = Extremely happy
To remedy this, define each level carefully. Test out the levels. Have colleagues try the scale and then run the data: Does it do what you want? Refine the scale and try it again. There’s nothing wrong in letting your rater know what each level means; this is why rubrics are such good assessment tools! More on the latter, later.
Send in your SLO! If selected, it will be featured and discussed in a future “Measuring SLOs” column. Send your suggestions to kgaw@gsu.edu and kgray@naceweb.org.