Qualitative Coding for Continuous Improvement
Part I: How to Conduct Qualitative Coding
Learner feedback is the foundation of iterative design. But how do you conquer a mountain of survey data to find actionable suggestions?
The tool: Qualitative Coding
The purpose: Summarize feedback to find themes and patterns
Use case: Building interactive feedback into an instructional program starts before the program even launches. As an instructional designer, you need to be clear with your SMEs and Stakeholders:
What are we trying to accomplish with this piece of learning? What business problem are we trying to solve?
The answers to those questions should inform how you design an evaluation of your content. If you decide to go with a post-completion survey, you may end up with a lot of data to sort through.
After the learning has launched and the feedback starts to roll in, your SMEs and Stakeholders may ask you:
What feedback did we receive?
You could give them access to all the information and let them draw their own conclusions. But often, the loudest voices are not necessarily the majority of voices, and you may end up making costly and time-consuming design changes to address the requests of a very small number of people.
Qualitative coding is used to summarize your data, and give your stakeholders clear, actionable feedback for continuous improvement of your learning product.
Note - This also works great when beta-testing new learning!
Instructions:
Download and clean your survey data.
Remove any duplicates.
Establish your framework: What types of feedback are you looking for? These were likely established when you set up your survey.
If not, you are probably looking for some version of a Kirkpatrick Level 1 and 2 Evaluation - Did learners like it? Did they meet the learning objectives?
Go through each piece of feedback and, in the column next to the information, summarize the suggestion in one or two key words.
Example: Next to “I think the quizzes were too difficult” might code it as “Quiz Difficulty”
Example: Next to “I couldn’t find the link in the module” might code it as “link”
After each piece of data is assigned a code, use the ‘sort and filter’ feature in excel to select similar codes.
Example: “Graphics too red” and “Graphics too blue” might combine as “graphics color”
Ask yourself: are there similar codes? For example, is the request for adjusting the quiz question related to the request for clarifying information on slide 2?
Combine any codes you think might be reflecting the same theme.
In your next review, find the most significant codes - likely, either the codes that most clearly answer your research question, or those that have the highest number of instances.
Start counting your codes. It may be helpful to make a new tab in your spreadsheet and write out the code, a summary of what this code means, and a count-if formula to note any instances of the code.
Next, write your report. Some stakeholders will want to see all the data, but most are happy with statistics like:
“90% of our learners said the scenario in the module was helpful.”
“20% of learners would like to see a video version of the written content.”
“5% of learners said the black font was difficult to read.”
From this data, you can start to develop recommendations for continuous improvement.
The Pros:
Using qualitative coding means you have evidence-based suggestions for improvement of your designs. This means future actions are driven by data, rather than anecdotes.
Using qualitative coding helps you uncover themes and trends, which give you a more complete picture of your learners expectations. The approach is systematic, rather than haphazard.
The Cons:
This process can take time, depending on how much data you have to code. Coding in-depth interviews takes much longer than coding a few sentences that answer a specific question. Designing surveys to be as targeted as possible (without asking leading questions!) can help speed up the process.
Ideally, you should triangulate your data with another person by asking them to perform the coding process separately (double-blind qualitative coding), and then compare notes. Having up to three people who are familiar with the subject matter code and analyze data is a best practice when conducting qualitative research, but may not be practical when applied at your workplace.
As with all survey data, you need to acknowledge how representative your sample is compared to the overall dataset. For example, I have used this tool to evaluate an e-learning that had over 18,000 unique visitors, but only 1,700 data points. This is less than 10% of all users. Have a conversation with your stakeholders about what constitutes a ‘good’ response rate for your specific use case.
Additionally, you should acknowledge that if a post-completion survey is made an optional task for learners, only those who have strong, and often negative, feedback will feel compelled to share it. Take strong feedback ‘with a grain of salt.’
Finally, effective survey design takes a lot of up-front communication with stakeholders.
Establishing the goals and purpose of the learning through a systematical analysis is a key to developing good metrics of success.
The takeaways
Qualitative Coding is just one way to summarize qualitative data - others exist, and it is up to you, as an Instructional Designer, to use the best evaluation tool for your situation.
You can use this free excel template with your own data.
What are some ways you have analyzed qualitative data - let me know in the comments below!
