Going Beyond Big Data: Why Edtech and Higher Ed Efficacy Needs Qualitative Research

Published by Carrie O'Donnell
on Sep 13, 2016

Organizations often question why qualitative research is critical to product success. Market research is one of O’Donnell Learn’s (ODL) cornerstone service offerings, so we like to answer that question.

Here’s the simple answer: Talking directly to customers will answer the why behind their behavior.

The data and analytics capabilities of your platform or product may provide answers as to how your customer is using the product, but the data will not always explain why users are not using certain features, why they are timing out before they meet assignment goals, or why they never use the product at all.

Some years ago a client at a large company asked ODL to survey hundreds of students at 20 higher ed institutions around the use of a new interactive ebook product, which was feature-rich. Before we agreed to manage the survey, we urged the client to let us conduct qualitative research—a few in-depth student interviews, or a couple of virtual focus groups with students at both two- and four-year schools. As with many product research projects, time and budget was one of the reasons the client wanted to bypass the qualitative activities and launch right into the quantitative research. However, time and budget weren’t the only reason; they also wanted to skip the qualitative research phase because they believed that quantitative research was more valid as it provided real data to compel further product investment.   

To conduct a successful quantitative research survey right out of the gate, you need to ask the right questions. Unless you understand the context that your customers are in, you can’t always ask the right questions. This was the situation with our client and their product. Nonetheless, we forged ahead with a set of questions around the use of the features, and specifically a tool that the client thought would be very compelling for the students.

The first survey was multiple choice, and the results were all over the map and hard to interpret. After reviewing the responses to the questions, we took a look at the open-ended responses. It turned out that the majority of the students had never used an ebook before and were struggling with the switch from print to digital, and most were not even aware of the feature and its functions. The takeaway: A little bit of investment in qualitative research upfront is always useful, no matter how much you think you know your customer, it is often hard to disinvest yourself of your assumptions.

In another more recent ODL engagement, a client was launching its first competency-based education (CBE) curriculum and included qualitative research in its product development and pre-launch plan. The curriculum was self-paced online CBE curriculum so that the students would be tackling competencies at different times.  To address the asynchronous nature of the students’ experience, we provided an online research environment that allowed for self-paced feedback as well as asynchronous discussion. We learned a lot about how students approach self-paced curriculum, even how the differences in prior knowledge and workflow approach affected their ability to accelerate through the competencies. We also saw how marketing influenced their expectations for the curriculum and how that, in turn, impacted their satisfaction level. This research helped the client tweak many aspects of the curriculum, not just the structure and content of the curriculum, but also the messaging, onboarding and coaching components.

A very tangible example of the value of qualitative research is found in the recent NPR story, How to Fix a Graduation Rate of 1 in 10? Ask the Dropouts. While institutions and product developers are working to help students meet learning outcomes, engage in class, and graduate, their efforts to foster student success are often bogged down in the quantitative, the assessment, the big data. As this article details, sometimes it is most efficient to speak with the learner to find out what they think is affecting their performance, and why are they dropping out.

While there is data to show who graduates and who leaves, in the case of SJSU, as with many schools, no exit interviews are ever conducted when a student leaves. There’s no complete clarity as to why the student leaves, and therefore no clear understanding as to why the graduation rates are so poor. SJSU Professor Marcos Pizzaro decided some qualitative research was needed and acquired a grant to interview students about why they left before graduating. The information they gathered was very personal, but also very addressable.

As a result, the university is providing an additional 500 classes, improving advisory services, and conducting social events to engage students.

There’s no doubt – the present is a very exciting time for edtech, and there’s a lot to learn from the many new products and online resources that have become available, and the data they create. Nonetheless, direct and candid feedback and insight cannot be a nice-to-have, it’s essential to meeting your audience exactly where they are. We’d love to hear about your experiences with product development and the types of research you’ve found to be most informative. Please share your story!


FALL 2020: Collecting Learner Feedback and Applying the Results

Published By Brett Christie, PhD
on Oct 15, 2020

Recently, O’Donnell Learn CEO, Carrie O’Donnell and I began hosting short webinars to address critical issues higher education faculty are currently facing. Our aim is to provide concise, actionable resources to help faculty create a more successful learning experience, particularly in the context of this semester’s challenges.

Challenges in Equity and Inclusion for Online Learning: An Interview with Dr. Brett Christie

Published By Joana Jebsen
on Sep 03, 2020

At O’Donnell Learn, we’ve always believed the path to great learning includes every learner and provides alternative pathways for different learners. When higher education shifted to emergency remote learning in March, significant gaps in equity and inclusion became evident. As institutions seek to resolve these gaps for online learning, we reached out to Dr. Brett […]

Introducing Media Lab: Aligning Learning Design and Media, All Under One Roof

Published By Carrie O'Donnell
on Sep 01, 2020

We recently made an exciting announcement – the launch of Media Lab! Media Lab was conceived out of a desire to disrupt the current media development model and offer best-in-class learning experiences at any scale. By bringing learning design and learning media production under one roof, our Media Lab is poised to help drive this […]