Going Beyond Big Data: Why Edtech and Higher Ed Efficacy Needs Qualitative Research


Organizations often question why qualitative research is critical to product success. Market research is one of O’Donnell Learn’s (ODL) cornerstone service offerings, so we like to answer that question.


Here’s the simple answer: Talking directly to customers will answer the why behind their behavior.


The data and analytics capabilities of your platform or product may provide answers as to how your customer is using the product, but the data will not always explain why users are not using certain features, why they are timing out before they meet assignment goals, or why they never use the product at all.


Some years ago a client at a large company asked ODL to survey hundreds of students at 20 higher ed institutions around the use of a new interactive ebook product, which was feature-rich. Before we agreed to manage the survey, we urged the client to let us conduct qualitative research—a few in-depth student interviews, or a couple of virtual focus groups with students at both two- and four-year schools. As with many product research projects, time and budget was one of the reasons the client wanted to bypass the qualitative activities and launch right into the quantitative research. However, time and budget weren't the only reason; they also wanted to skip the qualitative research phase because they believed that quantitative research was more valid as it provided real data to compel further product investment.   


To conduct a successful quantitative research survey right out of the gate, you need to ask the right questions. Unless you understand the context that your customers are in, you can’t always ask the right questions. This was the situation with our client and their product. Nonetheless, we forged ahead with a set of questions around the use of the features, and specifically a tool that the client thought would be very compelling for the students.


The first survey was multiple choice, and the results were all over the map and hard to interpret. After reviewing the responses to the questions, we took a look at the open-ended responses. It turned out that the majority of the students had never used an ebook before and were struggling with the switch from print to digital, and most were not even aware of the feature and its functions. The takeaway: A little bit of investment in qualitative research upfront is always useful, no matter how much you think you know your customer, it is often hard to disinvest yourself of your assumptions.


In another more recent ODL engagement, a client was launching its first competency-based education (CBE) curriculum and included qualitative research in its product development and pre-launch plan. The curriculum was self-paced online CBE curriculum so that the students would be tackling competencies at different times.  To address the asynchronous nature of the students’ experience, we provided an online research environment that allowed for self-paced feedback as well as asynchronous discussion. We learned a lot about how students approach self-paced curriculum, even how the differences in prior knowledge and workflow approach affected their ability to accelerate through the competencies. We also saw how marketing influenced their expectations for the curriculum and how that, in turn, impacted their satisfaction level. This research helped the client tweak many aspects of the curriculum, not just the structure and content of the curriculum, but also the messaging, onboarding and coaching components.


A very tangible example of the value of qualitative research is found in the recent NPR story, How to Fix a Graduation Rate of 1 in 10? Ask the Dropouts. While institutions and product developers are working to help students meet learning outcomes, engage in class, and graduate, their efforts to foster student success are often bogged down in the quantitative, the assessment, the big data. As this article details, sometimes it is most efficient to speak with the learner to find out what they think is affecting their performance, and why are they dropping out.


While there is data to show who graduates and who leaves, in the case of SJSU, as with many schools, no exit interviews are ever conducted when a student leaves. There’s no complete clarity as to why the student leaves, and therefore no clear understanding as to why the graduation rates are so poor. SJSU Professor Marcos Pizzaro decided some qualitative research was needed and acquired a grant to interview students about why they left before graduating. The information they gathered was very personal, but also very addressable.


As a result, the university is providing an additional 500 classes, improving advisory services, and conducting social events to engage students.


There’s no doubt - the present is a very exciting time for edtech, and there’s a lot to learn from the many new products and online resources that have become available, and the data they create. Nonetheless, direct and candid feedback and insight cannot be a nice-to-have, it’s essential to meeting your audience exactly where they are. We’d love to hear about your experiences with product development and the types of research you’ve found to be most informative. Please share your story!

6 views