Going Beyond Big Data: Why Edtech and Higher Ed Efficacy Needs Qualitative Research – O'Donnell Learn

Going Beyond Big Data: Why Edtech and Higher Ed Efficacy Needs Qualitative Research

Published by Carrie O'Donnell
on Sep 13, 2016

Organizations often question why qualitative research is critical to product success. Market research is one of O’Donnell Learn’s (ODL) cornerstone service offerings, so we like to answer that question.

Here’s the simple answer: Talking directly to customers will answer the why behind their behavior.

The data and analytics capabilities of your platform or product may provide answers as to how your customer is using the product, but the data will not always explain why users are not using certain features, why they are timing out before they meet assignment goals, or why they never use the product at all.

Some years ago a client at a large company asked ODL to survey hundreds of students at 20 higher ed institutions around the use of a new interactive ebook product, which was feature-rich. Before we agreed to manage the survey, we urged the client to let us conduct qualitative research—a few in-depth student interviews, or a couple of virtual focus groups with students at both two- and four-year schools. As with many product research projects, time and budget was one of the reasons the client wanted to bypass the qualitative activities and launch right into the quantitative research. However, time and budget weren’t the only reason; they also wanted to skip the qualitative research phase because they believed that quantitative research was more valid as it provided real data to compel further product investment.   

To conduct a successful quantitative research survey right out of the gate, you need to ask the right questions. Unless you understand the context that your customers are in, you can’t always ask the right questions. This was the situation with our client and their product. Nonetheless, we forged ahead with a set of questions around the use of the features, and specifically a tool that the client thought would be very compelling for the students.

The first survey was multiple choice, and the results were all over the map and hard to interpret. After reviewing the responses to the questions, we took a look at the open-ended responses. It turned out that the majority of the students had never used an ebook before and were struggling with the switch from print to digital, and most were not even aware of the feature and its functions. The takeaway: A little bit of investment in qualitative research upfront is always useful, no matter how much you think you know your customer, it is often hard to disinvest yourself of your assumptions.

In another more recent ODL engagement, a client was launching its first competency-based education (CBE) curriculum and included qualitative research in its product development and pre-launch plan. The curriculum was self-paced online CBE curriculum so that the students would be tackling competencies at different times.  To address the asynchronous nature of the students’ experience, we provided an online research environment that allowed for self-paced feedback as well as asynchronous discussion. We learned a lot about how students approach self-paced curriculum, even how the differences in prior knowledge and workflow approach affected their ability to accelerate through the competencies. We also saw how marketing influenced their expectations for the curriculum and how that, in turn, impacted their satisfaction level. This research helped the client tweak many aspects of the curriculum, not just the structure and content of the curriculum, but also the messaging, onboarding and coaching components.

A very tangible example of the value of qualitative research is found in the recent NPR story, How to Fix a Graduation Rate of 1 in 10? Ask the Dropouts. While institutions and product developers are working to help students meet learning outcomes, engage in class, and graduate, their efforts to foster student success are often bogged down in the quantitative, the assessment, the big data. As this article details, sometimes it is most efficient to speak with the learner to find out what they think is affecting their performance, and why are they dropping out.

While there is data to show who graduates and who leaves, in the case of SJSU, as with many schools, no exit interviews are ever conducted when a student leaves. There’s no complete clarity as to why the student leaves, and therefore no clear understanding as to why the graduation rates are so poor. SJSU Professor Marcos Pizzaro decided some qualitative research was needed and acquired a grant to interview students about why they left before graduating. The information they gathered was very personal, but also very addressable.

As a result, the university is providing an additional 500 classes, improving advisory services, and conducting social events to engage students.

There’s no doubt – the present is a very exciting time for edtech, and there’s a lot to learn from the many new products and online resources that have become available, and the data they create. Nonetheless, direct and candid feedback and insight cannot be a nice-to-have, it’s essential to meeting your audience exactly where they are. We’d love to hear about your experiences with product development and the types of research you’ve found to be most informative. Please share your story!


Purposeful Learning DesignTM – Why People are at the Center of Everything We Do

Published By Carrie O'Donnell
on Jan 15, 2021

I’m often asked why our company is so good at collaborating with  faculty. The first time I heard this question, I didn’t realize clients found this unique to O’Donnell Learn. We’ve always been great partners  because people are at the heart of everything we do. In fact, our entire design philosophy - Purposeful Learning DesignTM - is anchored around people.  It starts with this truth: learning is for people. As such, learning design should be grounded in empathy and it should promote success for both learners and faculty. Purposeful Learning Design is the philosophy we embrace to ensure we never lose sight of this truth. It is comprised of six key considerations.

2020 Reflective: Breathe Deep and Stay Agile

Published By Joana Jebsen
on Dec 22, 2020

While end-of-year survey results are still to come for how enrolled students are evaluating their 2020 college experience, I queried two of our staff, both students, about their overall experience and considered how these compared to what I was seeing with O’Donnell Learn clients. Kellie, our graphic design intern, just completed her Associates Degree in Graphic design. Prior to 2020, all of her courses met on campus. Cathryn, a Learning Design Associate is currently enrolled in an online Master of Education program in eLearning and Instructional Design. As this program was conceived as online delivery, it serves as a benchmark to compare against courses that were forced by COVID to switch delivery modes.

Evolving Our Project Management to Advance Client Results

Published By Craig Leonard
on Dec 16, 2020

If you were to talk to anyone in our organization, you’d quickly discover that we are all truly passionate about learning. So much so, that always learning is one of our core values. Learning from our successes, failures and each other is integral to how we operate and how we’ve grown into the agile company we are today.