Automatically Learning Topics and Difficulty Levels of Problems in Online Judge Systems

Wayne Xin Zhao, Wenhui Zhang, Yulan He, Xing Xie, Ji-Rong Wen

Research output: Contribution to journalArticle

Abstract

Online Judge (OJ) systems have been widely used in many areas, including programming, mathematical problems
solving, and job interviews. Unlike other online learning systems, such as Massive Open Online Course,
most OJ systems are designed for self-directed learning without the intervention of teachers. Also, in most
OJ systems, problems are simply listed in volumes and there is no clear organization of them by topics or
difficulty levels. As such, problems in the same volume are mixed in terms of topics or difficulty levels. By analyzing
large-scale users’ learning traces, we observe that there are two major learning modes (or patterns).
Users either practice problems in a sequential manner from the same volume regardless of their topics or
they attempt problems about the same topic, which may spread across multiple volumes. Our observation
is consistent with the findings in classic educational psychology. Based on our observation, we propose a
novel two-mode Markov topic model to automatically detect the topics of online problems by jointly characterizing
the two learning modes. For further predicting the difficulty level of online problems, we propose
a competition-based expertise model using the learned topic information. Extensive experiments on three
large OJ datasets have demonstrated the effectiveness of our approach in three different tasks, including skill
topic extraction, expertise competition prediction and problem recommendation.
Original languageEnglish
Article number27
Number of pages33
JournalACM Transactions on Information Systems
Volume36
Issue number3
DOIs
Publication statusPublished - 13 Mar 2018

Fingerprint

Online systems
Mathematical programming
Learning systems
Experiments

Bibliographical note

© ACM, 2018. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Information Systems, VOL# 36, ISS# 3, (13 Mar 2018)

Keywords

  • Topic models
  • expertise learning
  • online judge systems

Cite this

Zhao, Wayne Xin ; Zhang, Wenhui ; He, Yulan ; Xie, Xing ; Wen, Ji-Rong. / Automatically Learning Topics and Difficulty Levels of Problems in Online Judge Systems. In: ACM Transactions on Information Systems. 2018 ; Vol. 36, No. 3.
@article{3ede1e9f1ea94ec0a74a2eb56a6f67e2,
title = "Automatically Learning Topics and Difficulty Levels of Problems in Online Judge Systems",
abstract = "Online Judge (OJ) systems have been widely used in many areas, including programming, mathematical problemssolving, and job interviews. Unlike other online learning systems, such as Massive Open Online Course,most OJ systems are designed for self-directed learning without the intervention of teachers. Also, in mostOJ systems, problems are simply listed in volumes and there is no clear organization of them by topics ordifficulty levels. As such, problems in the same volume are mixed in terms of topics or difficulty levels. By analyzinglarge-scale users’ learning traces, we observe that there are two major learning modes (or patterns).Users either practice problems in a sequential manner from the same volume regardless of their topics orthey attempt problems about the same topic, which may spread across multiple volumes. Our observationis consistent with the findings in classic educational psychology. Based on our observation, we propose anovel two-mode Markov topic model to automatically detect the topics of online problems by jointly characterizingthe two learning modes. For further predicting the difficulty level of online problems, we proposea competition-based expertise model using the learned topic information. Extensive experiments on threelarge OJ datasets have demonstrated the effectiveness of our approach in three different tasks, including skilltopic extraction, expertise competition prediction and problem recommendation.",
keywords = "Topic models, expertise learning, online judge systems",
author = "Zhao, {Wayne Xin} and Wenhui Zhang and Yulan He and Xing Xie and Ji-Rong Wen",
note = "{\circledC} ACM, 2018. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Information Systems, VOL# 36, ISS# 3, (13 Mar 2018)",
year = "2018",
month = "3",
day = "13",
doi = "10.1145/3158670",
language = "English",
volume = "36",
journal = "ACM Transactions on Information Systems",
issn = "1046-8188",
publisher = "ACM",
number = "3",

}

Automatically Learning Topics and Difficulty Levels of Problems in Online Judge Systems. / Zhao, Wayne Xin; Zhang, Wenhui; He, Yulan; Xie, Xing; Wen, Ji-Rong.

In: ACM Transactions on Information Systems, Vol. 36, No. 3, 27, 13.03.2018.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Automatically Learning Topics and Difficulty Levels of Problems in Online Judge Systems

AU - Zhao, Wayne Xin

AU - Zhang, Wenhui

AU - He, Yulan

AU - Xie, Xing

AU - Wen, Ji-Rong

N1 - © ACM, 2018. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Information Systems, VOL# 36, ISS# 3, (13 Mar 2018)

PY - 2018/3/13

Y1 - 2018/3/13

N2 - Online Judge (OJ) systems have been widely used in many areas, including programming, mathematical problemssolving, and job interviews. Unlike other online learning systems, such as Massive Open Online Course,most OJ systems are designed for self-directed learning without the intervention of teachers. Also, in mostOJ systems, problems are simply listed in volumes and there is no clear organization of them by topics ordifficulty levels. As such, problems in the same volume are mixed in terms of topics or difficulty levels. By analyzinglarge-scale users’ learning traces, we observe that there are two major learning modes (or patterns).Users either practice problems in a sequential manner from the same volume regardless of their topics orthey attempt problems about the same topic, which may spread across multiple volumes. Our observationis consistent with the findings in classic educational psychology. Based on our observation, we propose anovel two-mode Markov topic model to automatically detect the topics of online problems by jointly characterizingthe two learning modes. For further predicting the difficulty level of online problems, we proposea competition-based expertise model using the learned topic information. Extensive experiments on threelarge OJ datasets have demonstrated the effectiveness of our approach in three different tasks, including skilltopic extraction, expertise competition prediction and problem recommendation.

AB - Online Judge (OJ) systems have been widely used in many areas, including programming, mathematical problemssolving, and job interviews. Unlike other online learning systems, such as Massive Open Online Course,most OJ systems are designed for self-directed learning without the intervention of teachers. Also, in mostOJ systems, problems are simply listed in volumes and there is no clear organization of them by topics ordifficulty levels. As such, problems in the same volume are mixed in terms of topics or difficulty levels. By analyzinglarge-scale users’ learning traces, we observe that there are two major learning modes (or patterns).Users either practice problems in a sequential manner from the same volume regardless of their topics orthey attempt problems about the same topic, which may spread across multiple volumes. Our observationis consistent with the findings in classic educational psychology. Based on our observation, we propose anovel two-mode Markov topic model to automatically detect the topics of online problems by jointly characterizingthe two learning modes. For further predicting the difficulty level of online problems, we proposea competition-based expertise model using the learned topic information. Extensive experiments on threelarge OJ datasets have demonstrated the effectiveness of our approach in three different tasks, including skilltopic extraction, expertise competition prediction and problem recommendation.

KW - Topic models

KW - expertise learning

KW - online judge systems

UR - https://dl.acm.org/citation.cfm?doid=3146384.3158670

U2 - 10.1145/3158670

DO - 10.1145/3158670

M3 - Article

VL - 36

JO - ACM Transactions on Information Systems

JF - ACM Transactions on Information Systems

SN - 1046-8188

IS - 3

M1 - 27

ER -