Lesson 14: k-Nearest Neighbors

Concept and intuition

k-Nearest Neighbors is a core topic in machine learning because it shapes how we frame the problem, choose tools, and judge results. k-Nearest Neighbors is useful for intuition because it shows that prediction can come directly from similarity rather than a heavily parameterized model.

When learning how instance-based learning predicts using the closest examples in feature space, do not focus only on formulas. The more important habit is to ask what the model is trying to learn, what assumptions it makes, and what could go wrong when the data is noisy, incomplete, or biased.

How it fits into a workflow

In a real project, how instance-based learning predicts using the closest examples in feature space sits inside a larger workflow: define the problem, prepare data, choose features, train a model, evaluate it carefully, and improve the system over time. Strong machine learning practice is iterative rather than one-shot.

This means you should connect how instance-based learning predicts using the closest examples in feature space to practical questions such as: What data is available? How will predictions be used? Which errors are most costly? How will the system be monitored after deployment? Those questions matter as much as model accuracy.

Common mistakes and practical advice

A common beginner mistake is to treat how instance-based learning predicts using the closest examples in feature space as a purely technical task. In practice, success depends on data quality, evaluation design, and the clarity of the business goal. Even a sophisticated model can fail if the data pipeline is weak or the target is poorly defined.

As you read the code example in this lesson, pay attention to how the inputs are shaped, how training and prediction are separated, and how the output is interpreted. Good coding habits make machine learning work more reliable, explainable, and easier to improve.

Three practical examples

kNN classification with scaled features

This code example focuses on clarity rather than production scale. Read the comments, then study the notes below to understand why each step matters.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("knn", KNeighborsClassifier(n_neighbors=5))
])

model.fit(X_train, y_train)
print(model.predict(X_test[:5]))

Code walkthrough

• kNN does not learn a global formula in the same way as linear models.
• It predicts by looking at nearby points in the feature space.
• Scaling is important because distance-based methods are sensitive to feature magnitude.
• The choice of `n_neighbors` affects smoothness and sensitivity to noise.

Summary and key takeaways

• kNN is easy to understand and useful for teaching similarity-based prediction.
• Distance matters, so scaling usually matters too.
• The value of `k` controls how local or how averaged the decision becomes.
• kNN can be slow on large datasets because it compares new samples to stored training points.

Exercises

• Why is feature scaling important for kNN?
• What might happen if `k=1` on noisy data?
• Change `n_neighbors` to 3 and 9 and compare predictions.
• Name one reason kNN may be less practical on very large datasets.

Continue your learning