Decision Trees and Random Forests

The literature on classification algorithms has highlighted various techniques to deal with classification problems, e.g. k-nearest neighbours; hierarchical-based models, decision trees (DTs), and random forests (RFs); probability-based models, including naive Bayes classification and logistic regression classification; support vector machines; layered models. This chapter introduces DT and RF classifiers by giving the basic theory of the DT diagnosis tool, its data structure, the ensemble model that combines DTs into a decision Forest model, and their applications in machine fault diagnosis. The algorithms most commonly used by DTs to make splitting decisions are described. These include univariate splitting criteria, e.g. Gini index, information gain, distance measure, and orthogonal criterion; and multivariate splitting criteria, e.g. greedy learning, linear programming, linear discriminant analysis, perceptron learning, and hill-climbing search. The chapter also presents DT pruning methods including error-complexity pruning, minimum-error pruning, and critical-value pruning.