How McGraw-Hill identifies at-risk students

For the past several years, McGraw-Hill Education’s user research department reported an issue with the company’s Assessment and Learning in Knowledge Spaces (ALEKS) offering. The web-based, artificially intelligent assessment and learning system was great at assessing students and creating customized learning paths based on their individual needs, but instructors were deluged with data.

“In our quarterly instructor survey, we found, again and again over a two-year period, that instructors love our reports, which are highly visual and really robust, but it’s a lot to process,” says Lori Anderson, vice president of technical product management at McGraw-Hill Education.

In the K12 environment, when teachers aren’t instructing in the classroom, they’re creating lesson plans or monitoring student activities. They were struggling to carve out time to take deep dives into every student’s ALEKS assessments. In the higher education market, Anderson says, instructors may not have as many duties outside the classroom, but they often have hundreds of students.

“They were saying, ‘We love your reports but they’re too time-consuming. Just tell me which students are at risk of failing. Tell me which students are doing weird stuff, like maybe cheating. Put my eyeballs on the data I should care about in all of these reports.”

Those user reports led Eric Cosyne, director of applied research at ALEKS, to take the first steps toward the development of Insights, a companion to ALEKS intended to provide instructors with the information they need to identify how and where students are struggling and put them back on the right path. Insights has earned McGraw-Hill Education a 2019 Digital Edge 50 Award for digital innovation.

Interacting with ALEKS

Originally developed at UC Irvine in 1994 with a grant from the National Science Foundation, ALEKS was acquired by McGraw-Hill Education in 2013. ALEKS specializes in quantitative disciplines like math and chemistry for which the ALEKS team maps out the content of a course in a knowledge structure. ALEKS performs an initial assessment of a student’s knowledge, determining where they’re strong and weak, and then maps out a learning path customized for that individual.

McGraw-Hill Education

“We can identify precisely what students know, don’t know, and – the real kicker – what they’re most ready to learn in a course area,” Anderson says.

With that data, and using machine learning on billions of data points from past students who have interacted with ALEKS, the AI is able to make inferences about what the student is most ready to learn next. ALEKS also periodically reassesses students to check for knowledge retention, as students often forget previously learned material.

“It’s really about data and about using those knowledge structures to draw inferences among the topics that makes ALEKS unique,” Anderson says.

The reporting features within ALEKS are intended to help instructors by showing them how students are progressing within a course, what material the student already knows and what material the student is struggling with. McGraw-Hill’s user research showed that instructors were having trouble coping with that enormous amount of data. The instructors needed to know what they were looking for, had to spend time searching for issues student by student, and had to be well-trained on the reporting within the platform to make sense of it all.

“Once we had Eric [Cosyne]’s algorithm for processing the data to meet the customers’ needs, we had a story-mapping session,” Anderson explains. That story-mapping session was used to create a blue-sky concept for Insights, which was then whittled down into a minimum viable product (MVP).

Meeting customer needs

The initial version of Insights, which the company intends to deploy to a limited group of users in the next several weeks, focuses on four key areas:

• Topics failed. Content that students have attempted multiple times in ALEKS without success.
• Learning decreased. Students who show a significant drop in successful learning despite continuous time spent in the system.
• Unusual learning. Students who show a significant spike in learning compared to previous learning; a sign that they might not be doing their own work.
• Students whose time spent in ALEKS varies, showing bursts of activity followed by long periods of inactivity, or time spent in ALEKS with no sign of productive activity.

“Insights is a summary of all the data in reports that we have, summarized in a way that’s easily digestible to the instructor and bucketed in the four categories that instructors continually tell us that they really care about,” Anderson says.

Insights uses those four categories to determine if a student is veering off course and sends an email alert to their instructors. Instructors can then send a message to students directly from the Insights page or drill down into the full ALEKS report to better understand the issue and determine appropriate actions.

The initial blue-sky concept of Insights included text alerts – a feature requested by higher education instructors – but instructor preferences for format and frequency of alerts were all over the map. The team opted to carve text alerts off from the MVP and leave them for a later iteration to get Insights to market faster.

Getting sponsor support

Anderson says the Insights project garnered immediate support within the ALEKS team, where its utility was clear, but the project still needed support within the larger organization. To get on the roadmap at McGraw-Hill Education, all technology projects must be sponsored by business units.

“It’s a direct partnership between our technology teams and our business partners,” Anderson explains.

To get that sponsorship, Anderson, who also manages the UX team at ALEKS, had a UX designer create low-fidelity mockups and prototypes of Insights to share with business partners. “I start with the UX, so people can rally around a visual representation of this abstract thing,” she says. “You have to convert the abstract to something concrete.”

In the end, that not only helped rally support, it also helped them focus the MVP into something the team could build within a six-month timeframe.

The Higher Education business unit, the original sponsor for Insights, “immediately saw the benefits of surfacing intuitive, action-oriented insights to busy instructors who have large classes with many students,” Anderson says. “They saw it as a competitive advantage and opportunity to better meet their customers’ needs.”

The McGraw-Hill School group, which focuses on the K12 market, also signed on to sponsor Insights in an effort to better help teachers identify and help struggling students.

“The biggest challenge to date, I think, was getting that MVP line drawn correctly. With 15+ stakeholders in the room, that’s a lot of opinions and knowledge. There were a lot of opinions in the room and some people are closer to the customers than others.”