Going Beyond Big Data: Why Edtech and Higher Ed Efficacy Needs Qualitative Research

Published by Carrie O'Donnell
on Sep 13, 2016

Organizations often question why qualitative research is critical to product success. Market research is one of O’Donnell Learn’s (ODL) cornerstone service offerings, so we like to answer that question.

Here’s the simple answer: Talking directly to customers will answer the why behind their behavior.

The data and analytics capabilities of your platform or product may provide answers as to how your customer is using the product, but the data will not always explain why users are not using certain features, why they are timing out before they meet assignment goals, or why they never use the product at all.

Some years ago a client at a large company asked ODL to survey hundreds of students at 20 higher ed institutions around the use of a new interactive ebook product, which was feature-rich. Before we agreed to manage the survey, we urged the client to let us conduct qualitative research—a few in-depth student interviews, or a couple of virtual focus groups with students at both two- and four-year schools. As with many product research projects, time and budget was one of the reasons the client wanted to bypass the qualitative activities and launch right into the quantitative research. However, time and budget weren’t the only reason; they also wanted to skip the qualitative research phase because they believed that quantitative research was more valid as it provided real data to compel further product investment.   

To conduct a successful quantitative research survey right out of the gate, you need to ask the right questions. Unless you understand the context that your customers are in, you can’t always ask the right questions. This was the situation with our client and their product. Nonetheless, we forged ahead with a set of questions around the use of the features, and specifically a tool that the client thought would be very compelling for the students.

The first survey was multiple choice, and the results were all over the map and hard to interpret. After reviewing the responses to the questions, we took a look at the open-ended responses. It turned out that the majority of the students had never used an ebook before and were struggling with the switch from print to digital, and most were not even aware of the feature and its functions. The takeaway: A little bit of investment in qualitative research upfront is always useful, no matter how much you think you know your customer, it is often hard to disinvest yourself of your assumptions.

In another more recent ODL engagement, a client was launching its first competency-based education (CBE) curriculum and included qualitative research in its product development and pre-launch plan. The curriculum was self-paced online CBE curriculum so that the students would be tackling competencies at different times.  To address the asynchronous nature of the students’ experience, we provided an online research environment that allowed for self-paced feedback as well as asynchronous discussion. We learned a lot about how students approach self-paced curriculum, even how the differences in prior knowledge and workflow approach affected their ability to accelerate through the competencies. We also saw how marketing influenced their expectations for the curriculum and how that, in turn, impacted their satisfaction level. This research helped the client tweak many aspects of the curriculum, not just the structure and content of the curriculum, but also the messaging, onboarding and coaching components.

A very tangible example of the value of qualitative research is found in the recent NPR story, How to Fix a Graduation Rate of 1 in 10? Ask the Dropouts. While institutions and product developers are working to help students meet learning outcomes, engage in class, and graduate, their efforts to foster student success are often bogged down in the quantitative, the assessment, the big data. As this article details, sometimes it is most efficient to speak with the learner to find out what they think is affecting their performance, and why are they dropping out.

While there is data to show who graduates and who leaves, in the case of SJSU, as with many schools, no exit interviews are ever conducted when a student leaves. There’s no complete clarity as to why the student leaves, and therefore no clear understanding as to why the graduation rates are so poor. SJSU Professor Marcos Pizzaro decided some qualitative research was needed and acquired a grant to interview students about why they left before graduating. The information they gathered was very personal, but also very addressable.

As a result, the university is providing an additional 500 classes, improving advisory services, and conducting social events to engage students.

There’s no doubt – the present is a very exciting time for edtech, and there’s a lot to learn from the many new products and online resources that have become available, and the data they create. Nonetheless, direct and candid feedback and insight cannot be a nice-to-have, it’s essential to meeting your audience exactly where they are. We’d love to hear about your experiences with product development and the types of research you’ve found to be most informative. Please share your story!

Leave a Reply

Your email address will not be published. Required fields are marked *


Resiliency in the Now Normal: Spending for Sustainability and Scale is Key

Published By Carrie O'Donnell
on Apr 22, 2021

Earlier in April, Matt Reed proposed the best use of the $12B included in President Biden’s infrastructure legislation for updating infrastructure in community colleges, would be “ways that situate colleges to be more resilient in future economic headwinds.”  For those of you unfamiliar with Reed, not only does he write the “Confessions of a Community College Dean” blog on Inside Higher Ed, nearly 18 years of his career has been in community college leadership positions. Dr. Joshua Kim, director of online programs and strategy at Dartmouth College, fondly refers to him as “Dean Dad.” In fact, Kim penned a response in support of the infrastructure spending recommendations Reed made in his post and offered an additional recommendation of his own: learning designers. 

Learn While Doing: Course Innovations in Real Time

Published By Brett Christie, PhD
on Mar 30, 2021

We recently learned there are approximately 20,000 learning designers in the US compared to over 1,500,000 faculty creating online courses. Additionally, a study we conducted last summer with 475 higher ed faculty revealed: Nearly half were simply mirroring their face-to-face instruction, Only 22% were designing their courses differently for online, More than 40% had never taught online or had only taught online for one to two terms. But here’s an even more startling fact: faculty were spending nearly 49 hours prepping an online course for the first time. Converting an existing course for online? Twenty-three hours.