Lesson 13: Decision Trees and Random Forests

Concept and intuition

Decision Trees and Random Forests is a core topic in machine learning because it shapes how we frame the problem, choose tools, and judge results. Tree-based methods are powerful because they handle nonlinear relationships well and often require less feature scaling or manual transformation than linear models.

When learning tree-based models that split data into rules and combine multiple trees for stronger performance, do not focus only on formulas. The more important habit is to ask what the model is trying to learn, what assumptions it makes, and what could go wrong when the data is noisy, incomplete, or biased.

How it fits into a workflow

In a real project, tree-based models that split data into rules and combine multiple trees for stronger performance sits inside a larger workflow: define the problem, prepare data, choose features, train a model, evaluate it carefully, and improve the system over time. Strong machine learning practice is iterative rather than one-shot.

This means you should connect tree-based models that split data into rules and combine multiple trees for stronger performance to practical questions such as: What data is available? How will predictions be used? Which errors are most costly? How will the system be monitored after deployment? Those questions matter as much as model accuracy.

Common mistakes and practical advice

A common beginner mistake is to treat tree-based models that split data into rules and combine multiple trees for stronger performance as a purely technical task. In practice, success depends on data quality, evaluation design, and the clarity of the business goal. Even a sophisticated model can fail if the data pipeline is weak or the target is poorly defined.

As you read the code example in this lesson, pay attention to how the inputs are shaped, how training and prediction are separated, and how the output is interpreted. Good coding habits make machine learning work more reliable, explainable, and easier to improve.

Three practical examples

Comparing a decision tree and a random forest

This code example focuses on clarity rather than production scale. Read the comments, then study the notes below to understand why each step matters.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

data = load_wine()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, random_state=42
)

tree = DecisionTreeClassifier(random_state=42)
forest = RandomForestClassifier(n_estimators=200, random_state=42)

tree.fit(X_train, y_train)
forest.fit(X_train, y_train)

print("Tree accuracy:", accuracy_score(y_test, tree.predict(X_test)))
print("Forest accuracy:", accuracy_score(y_test, forest.predict(X_test)))

Code walkthrough

• A decision tree learns a sequence of feature-based splits.
• A random forest trains many trees and combines their predictions.
• The forest often performs better because averaging reduces sensitivity to individual tree quirks.
• Tree methods can model nonlinear interactions that simple linear models may miss.

Summary and key takeaways

• Decision trees are intuitive but can overfit if left unconstrained.
• Random forests improve stability by combining many trees.
• Tree-based methods are strong all-purpose tools for tabular data.
• Interpretability and performance must be balanced when choosing between one tree and many trees.

Exercises

• Why might a single decision tree overfit?
• What advantage does a random forest gain from multiple trees?
• Change the number of estimators in the random forest and observe the result.
• In what kind of project would a single decision tree still be useful?

Continue your learning