K-means clustering in Python with Scikit-Learn
K-means clustering is one of the most popular unsupervised machine learning algorithms used to partition data into distinct groups. Whether you’re segmenting customers, organizing images, or analyzing patterns in large datasets, k means clustering python provides an efficient and intuitive solution. In this comprehensive guide, we’ll explore how to implement kmeans python using scikit-learn, understand the underlying algorithm, and apply it to real-world scenarios.
1. Understanding k means clustering in machine learning
What is k-means clustering?
K-means clustering is an unsupervised learning algorithm that partitions a dataset into k distinct, non-overlapping clusters. The algorithm works by identifying centroids—the center points of each cluster—and assigning each data point to the nearest centroid. The goal is to minimize the within-cluster variance, ensuring that data points within the same cluster are as similar as possible.
Unlike supervised learning algorithms that require labeled data, k-means discovers hidden patterns automatically, making it invaluable for exploratory data analysis and pattern recognition tasks.
How the k means algorithm works
The k means algorithm follows an iterative process that alternates between two main steps:
- Assignment step: Each data point is assigned to the nearest centroid based on Euclidean distance
- Update step: The centroids are recalculated as the mean of all points assigned to each cluster
This process continues until convergence—when the assignments no longer change or the centroids stabilize within a specified tolerance.
Mathematically, the algorithm minimizes the objective function:
$$J = \sum_{i=1}^{k} \sum_{x \in C_i} ||x – \mu_i||^2$$
where \(C_i\) represents the i-th cluster, \(\mu_i\) is the centroid of cluster \(C_i\), and \(||x – \mu_i||^2\) is the squared Euclidean distance between point \(x\) and centroid \(\mu_i\).
Key characteristics of kmean
Advantages:
- Computationally efficient and scales well to large datasets
- Simple to understand and implement
- Works well when clusters are spherical and similar in size
- Guaranteed to converge to a local optimum
Limitations:
- Requires specifying the number of clusters k in advance
- Sensitive to initial centroid placement
- Assumes clusters are convex and isotropic
- Struggles with clusters of varying sizes and densities
2. Setting up your environment for kmeans sklearn
Installing required libraries
Before implementing sklearn kmeans, you need to set up your Python environment with the necessary libraries:
# Install required packages
pip install numpy pandas matplotlib scikit-learn seaborn
Importing essential modules
Let’s import all the modules we’ll need for our k means clustering python implementation:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score, davies_bouldin_score
# Set random seed for reproducibility
np.random.seed(42)
# Configure plotting style
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_palette("husl")
3. Basic implementation of kmeans python
Creating a sample dataset
Let’s start with a simple example using synthetic data to understand how sklearn clustering works:
# Generate sample data with 4 distinct clusters
X, y_true = make_blobs(n_samples=300, centers=4,
cluster_std=0.60, random_state=42)
# Visualize the data
plt.figure(figsize=(10, 6))
plt.scatter(X[:, 0], X[:, 1], s=50, alpha=0.6)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.title('Sample Dataset for K-Means Clustering')
plt.show()
Implementing basic k means clustering in machine learning
Here’s how to implement a simple kmeans sklearn model:
# Initialize KMeans with 4 clusters
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
# Fit the model to the data
kmeans.fit(X)
# Get cluster labels and centroids
labels = kmeans.labels_
centroids = kmeans.cluster_centers_
# Visualize the results
plt.figure(figsize=(10, 6))
plt.scatter(X[:, 0], X[:, 1], c=labels, s=50, alpha=0.6, cmap='viridis')
plt.scatter(centroids[:, 0], centroids[:, 1], c='red', s=200,
alpha=0.8, marker='X', edgecolors='black', linewidths=2)
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.title('K-Means Clustering Results')
plt.colorbar(label='Cluster')
plt.show()
Understanding KMeans parameters
The sklearn kmeans implementation offers several important parameters:
- n_clusters: The number of clusters to form (default: 8)
- init: Method for initialization (‘k-means++’, ‘random’, or array)
- n_init: Number of times the algorithm runs with different centroid seeds
- max_iter: Maximum number of iterations for a single run (default: 300)
- tol: Tolerance for convergence (default: 1e-4)
- random_state: Seed for reproducibility
# Example with custom parameters
kmeans_custom = KMeans(
n_clusters=4,
init='k-means++', # Smart initialization
n_init=10, # Run 10 times with different seeds
max_iter=300, # Maximum iterations
tol=1e-4, # Convergence tolerance
random_state=42
)
kmeans_custom.fit(X)
4. Advanced techniques for python clustering
Determining the optimal number of clusters
One of the biggest challenges in k means clustering python is choosing the right value for k. Here are three popular methods:
Elbow method
The elbow method plots the within-cluster sum of squares (inertia) against different values of k:
# Calculate inertia for different k values
inertia_values = []
k_range = range(2, 11)
for k in k_range:
kmeans_temp = KMeans(n_clusters=k, random_state=42, n_init=10)
kmeans_temp.fit(X)
inertia_values.append(kmeans_temp.inertia_)
# Plot the elbow curve
plt.figure(figsize=(10, 6))
plt.plot(k_range, inertia_values, marker='o', linewidth=2, markersize=8)
plt.xlabel('Number of Clusters (k)')
plt.ylabel('Inertia (Within-Cluster Sum of Squares)')
plt.title('Elbow Method for Optimal k')
plt.grid(True, alpha=0.3)
plt.show()
The “elbow” point where the rate of decrease sharply changes indicates the optimal k.
Silhouette analysis
The silhouette score measures how similar a point is to its own cluster compared to other clusters:
# Calculate silhouette scores
silhouette_scores = []
for k in range(2, 11):
kmeans_temp = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = kmeans_temp.fit_predict(X)
score = silhouette_score(X, labels)
silhouette_scores.append(score)
# Plot silhouette scores
plt.figure(figsize=(10, 6))
plt.plot(range(2, 11), silhouette_scores, marker='o', linewidth=2, markersize=8)
plt.xlabel('Number of Clusters (k)')
plt.ylabel('Silhouette Score')
plt.title('Silhouette Analysis for Optimal k')
plt.grid(True, alpha=0.3)
plt.show()
print(f"Best k based on silhouette score: {silhouette_scores.index(max(silhouette_scores)) + 2}")
Silhouette scores range from -1 to 1, where values closer to 1 indicate better-defined clusters.
Davies-Bouldin index
This metric measures the average similarity between each cluster and its most similar cluster:
# Calculate Davies-Bouldin scores (lower is better)
db_scores = []
for k in range(2, 11):
kmeans_temp = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = kmeans_temp.fit_predict(X)
score = davies_bouldin_score(X, labels)
db_scores.append(score)
# Plot Davies-Bouldin scores
plt.figure(figsize=(10, 6))
plt.plot(range(2, 11), db_scores, marker='o', linewidth=2, markersize=8)
plt.xlabel('Number of Clusters (k)')
plt.ylabel('Davies-Bouldin Index')
plt.title('Davies-Bouldin Index for Optimal k (Lower is Better)')
plt.grid(True, alpha=0.3)
plt.show()
Feature scaling and preprocessing
Feature scaling is crucial for k means algorithm performance since it relies on distance calculations:
# Original data without scaling
X_original, _ = make_blobs(n_samples=300, centers=4,
cluster_std=[1.0, 2.5, 0.5, 1.5],
random_state=42)
# Apply StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_original)
# Compare clustering results
fig, axes = plt.subplots(1, 2, figsize=(15, 6))
# Without scaling
kmeans_unscaled = KMeans(n_clusters=4, random_state=42, n_init=10)
labels_unscaled = kmeans_unscaled.fit_predict(X_original)
axes[0].scatter(X_original[:, 0], X_original[:, 1],
c=labels_unscaled, s=50, alpha=0.6, cmap='viridis')
axes[0].set_title('K-Means without Scaling')
axes[0].set_xlabel('Feature 1')
axes[0].set_ylabel('Feature 2')
# With scaling
kmeans_scaled = KMeans(n_clusters=4, random_state=42, n_init=10)
labels_scaled = kmeans_scaled.fit_predict(X_scaled)
axes[1].scatter(X_scaled[:, 0], X_scaled[:, 1],
c=labels_scaled, s=50, alpha=0.6, cmap='viridis')
axes[1].set_title('K-Means with Scaling')
axes[1].set_xlabel('Feature 1 (scaled)')
axes[1].set_ylabel('Feature 2 (scaled)')
plt.tight_layout()
plt.show()
Handling outliers and noise
Outliers can significantly affect kmeans sklearn results. Here’s how to detect and handle them:
# Add outliers to dataset
X_with_outliers = np.vstack([X, np.random.uniform(-10, 10, (20, 2))])
# Fit k-means
kmeans_outliers = KMeans(n_clusters=4, random_state=42, n_init=10)
labels_outliers = kmeans_outliers.fit_predict(X_with_outliers)
# Calculate distances to nearest centroid
distances = np.min(kmeans_outliers.transform(X_with_outliers), axis=1)
# Identify outliers (points far from their centroids)
threshold = np.percentile(distances, 95)
outlier_mask = distances > threshold
# Visualize
plt.figure(figsize=(10, 6))
plt.scatter(X_with_outliers[~outlier_mask, 0],
X_with_outliers[~outlier_mask, 1],
c=labels_outliers[~outlier_mask], s=50, alpha=0.6, cmap='viridis')
plt.scatter(X_with_outliers[outlier_mask, 0],
X_with_outliers[outlier_mask, 1],
c='red', s=100, alpha=0.8, marker='x', label='Outliers')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.title('K-Means with Outlier Detection')
plt.legend()
plt.show()
5. Real-world applications of kmeans python
Customer segmentation example
Let’s implement a practical customer segmentation scenario using k means clustering in machine learning:
# Create sample customer data
np.random.seed(42)
n_customers = 500
# Generate features: annual income and spending score
data = pd.DataFrame({
'Annual_Income': np.random.normal(60, 20, n_customers),
'Spending_Score': np.random.normal(50, 25, n_customers),
'Age': np.random.normal(40, 15, n_customers)
})
# Clean negative values
data = data.clip(lower=0)
# Prepare features for clustering
features = data[['Annual_Income', 'Spending_Score']].values
# Scale the features
scaler = StandardScaler()
features_scaled = scaler.fit_transform(features)
# Determine optimal k using elbow method
inertias = []
for k in range(2, 10):
km = KMeans(n_clusters=k, random_state=42, n_init=10)
km.fit(features_scaled)
inertias.append(km.inertia_)
# Apply K-Means with optimal k=4
optimal_k = 4
kmeans_customer = KMeans(n_clusters=optimal_k, random_state=42, n_init=10)
data['Cluster'] = kmeans_customer.fit_predict(features_scaled)
# Visualize customer segments
plt.figure(figsize=(12, 7))
scatter = plt.scatter(data['Annual_Income'], data['Spending_Score'],
c=data['Cluster'], s=100, alpha=0.6, cmap='viridis')
plt.xlabel('Annual Income (k$)')
plt.ylabel('Spending Score (1-100)')
plt.title('Customer Segmentation using K-Means Clustering')
plt.colorbar(scatter, label='Segment')
plt.grid(True, alpha=0.3)
plt.show()
# Analyze segments
segment_analysis = data.groupby('Cluster').agg({
'Annual_Income': 'mean',
'Spending_Score': 'mean',
'Age': 'mean'
}).round(2)
print("\nSegment Characteristics:")
print(segment_analysis)
Image color quantization
K-means can reduce the number of colors in an image, a technique called color quantization:
# Create a simple synthetic image
from sklearn.utils import shuffle
def create_sample_image():
"""Create a sample gradient image"""
x = np.linspace(0, 1, 100)
y = np.linspace(0, 1, 100)
X_img, Y_img = np.meshgrid(x, y)
# Create RGB channels
R = (X_img * 255).astype(int)
G = (Y_img * 255).astype(int)
B = ((X_img + Y_img) / 2 * 255).astype(int)
image = np.stack([R, G, B], axis=2)
return image
# Generate sample image
image = create_sample_image()
h, w, c = image.shape
# Reshape image to 2D array of pixels
image_array = image.reshape(h * w, c)
# Sample pixels for faster processing
image_sample = shuffle(image_array, random_state=42)[:1000]
# Apply K-Means for color quantization
n_colors = 8
kmeans_colors = KMeans(n_clusters=n_colors, random_state=42, n_init=10)
kmeans_colors.fit(image_sample)
# Get cluster centers (quantized colors)
colors = kmeans_colors.cluster_centers_.astype(int)
# Predict labels for all pixels
labels = kmeans_colors.predict(image_array)
# Recreate image with quantized colors
quantized_image = colors[labels].reshape(h, w, c)
# Display results
fig, axes = plt.subplots(1, 2, figsize=(14, 6))
axes[0].imshow(image)
axes[0].set_title('Original Image')
axes[0].axis('off')
axes[1].imshow(quantized_image)
axes[1].set_title(f'Quantized Image ({n_colors} colors)')
axes[1].axis('off')
plt.tight_layout()
plt.show()
Document clustering
Text documents can be clustered based on their content using k means clustering python:
from sklearn.feature_extraction.text import TfidfVectorizer
# Sample documents
documents = [
"Machine learning is a subset of artificial intelligence",
"Deep learning uses neural networks with multiple layers",
"Python is a popular programming language for data science",
"Natural language processing helps computers understand human language",
"Supervised learning requires labeled training data",
"Java and C++ are object-oriented programming languages",
"Unsupervised learning finds patterns in unlabeled data",
"JavaScript is commonly used for web development",
"Computer vision enables machines to interpret visual information",
"SQL is used for database management and queries"
]
# Convert documents to TF-IDF vectors
vectorizer = TfidfVectorizer(stop_words='english', max_features=50)
doc_vectors = vectorizer.fit_transform(documents)
# Apply K-Means clustering
n_clusters = 3
kmeans_docs = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
doc_clusters = kmeans_docs.fit_predict(doc_vectors)
# Display results
print("Document Clustering Results:\n")
for cluster in range(n_clusters):
print(f"Cluster {cluster}:")
cluster_docs = [doc for doc, label in zip(documents, doc_clusters) if label == cluster]
for doc in cluster_docs:
print(f" - {doc}")
print()
# Get top terms per cluster
order_centroids = kmeans_docs.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names_out()
print("Top terms per cluster:")
for i in range(n_clusters):
print(f"Cluster {i}:", end=' ')
top_terms = [terms[ind] for ind in order_centroids[i, :5]]
print(", ".join(top_terms))
6. Optimizing and troubleshooting kmeans sklearn
Improving convergence speed
Several techniques can speed up the k means algorithm:
# Mini-Batch K-Means for large datasets
from sklearn.cluster import MiniBatchKMeans
# Generate large dataset
X_large, _ = make_blobs(n_samples=10000, centers=5, random_state=42)
# Standard K-Means
import time
start = time.time()
kmeans_standard = KMeans(n_clusters=5, random_state=42, n_init=10)
kmeans_standard.fit(X_large)
time_standard = time.time() - start
# Mini-Batch K-Means
start = time.time()
kmeans_minibatch = MiniBatchKMeans(n_clusters=5, random_state=42,
batch_size=100, n_init=10)
kmeans_minibatch.fit(X_large)
time_minibatch = time.time() - start
print(f"Standard K-Means time: {time_standard:.4f} seconds")
print(f"Mini-Batch K-Means time: {time_minibatch:.4f} seconds")
print(f"Speedup: {time_standard/time_minibatch:.2f}x")
# Compare inertia
print(f"\nStandard K-Means inertia: {kmeans_standard.inertia_:.2f}")
print(f"Mini-Batch K-Means inertia: {kmeans_minibatch.inertia_:.2f}")
Dealing with initialization sensitivity
The k-means++ initialization helps avoid poor local optima:
# Compare random vs k-means++ initialization
fig, axes = plt.subplots(1, 2, figsize=(15, 6))
# Random initialization
kmeans_random = KMeans(n_clusters=4, init='random', n_init=1, random_state=42)
labels_random = kmeans_random.fit_predict(X)
axes[0].scatter(X[:, 0], X[:, 1], c=labels_random, s=50, alpha=0.6, cmap='viridis')
axes[0].scatter(kmeans_random.cluster_centers_[:, 0],
kmeans_random.cluster_centers_[:, 1],
c='red', s=200, alpha=0.8, marker='X', edgecolors='black')
axes[0].set_title(f'Random Init (Inertia: {kmeans_random.inertia_:.2f})')
# K-means++ initialization
kmeans_pp = KMeans(n_clusters=4, init='k-means++', n_init=1, random_state=42)
labels_pp = kmeans_pp.fit_predict(X)
axes[1].scatter(X[:, 0], X[:, 1], c=labels_pp, s=50, alpha=0.6, cmap='viridis')
axes[1].scatter(kmeans_pp.cluster_centers_[:, 0],
kmeans_pp.cluster_centers_[:, 1],
c='red', s=200, alpha=0.8, marker='X', edgecolors='black')
axes[1].set_title(f'K-means++ Init (Inertia: {kmeans_pp.inertia_:.2f})')
plt.tight_layout()
plt.show()
Evaluating cluster quality
Beyond the mean python calculation, here are comprehensive evaluation metrics:
from sklearn.metrics import calinski_harabasz_score
# Fit K-Means
kmeans_eval = KMeans(n_clusters=4, random_state=42, n_init=10)
labels_eval = kmeans_eval.fit_predict(X)
# Calculate multiple metrics
silhouette = silhouette_score(X, labels_eval)
calinski = calinski_harabasz_score(X, labels_eval)
davies = davies_bouldin_score(X, labels_eval)
print("Cluster Quality Metrics:")
print(f"Silhouette Score: {silhouette:.4f} (higher is better, range: -1 to 1)")
print(f"Calinski-Harabasz Score: {calinski:.4f} (higher is better)")
print(f"Davies-Bouldin Index: {davies:.4f} (lower is better)")
print(f"Inertia: {kmeans_eval.inertia_:.4f} (lower is better)")
# Calculate per-cluster statistics
for i in range(4):
cluster_points = X[labels_eval == i]
cluster_size = len(cluster_points)
cluster_variance = np.var(cluster_points, axis=0).mean()
print(f"\nCluster {i}:")
print(f" Size: {cluster_size} points")
print(f" Variance: {cluster_variance:.4f}")
Common pitfalls and solutions
Problem 1: Empty clusters
# If a cluster becomes empty during iteration
# Solution: Use n_init parameter to run multiple times
kmeans_stable = KMeans(n_clusters=10, n_init=20, random_state=42)
kmeans_stable.fit(X)
print(f"All clusters populated: {all(np.bincount(kmeans_stable.labels_) > 0)}")
Problem 2: Scale sensitivity
# Always scale features with different ranges
from sklearn.preprocessing import MinMaxScaler, RobustScaler
# Compare different scalers
X_mixed = np.column_stack([
np.random.normal(0, 1, 300), # Feature 1: small range
np.random.normal(0, 100, 300) # Feature 2: large range
])
scalers = {
'StandardScaler': StandardScaler(),
'MinMaxScaler': MinMaxScaler(),
'RobustScaler': RobustScaler()
}
for name, scaler in scalers.items():
X_scaled = scaler.fit_transform(X_mixed)
kmeans_temp = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans_temp.fit(X_scaled)
print(f"{name} - Inertia: {kmeans_temp.inertia_:.4f}")
7. Conclusion
K-means clustering python remains one of the most practical and widely-used algorithms in machine learning and data science. Through this guide, we’ve explored everything from basic sklearn kmeans implementation to advanced techniques for optimizing cluster quality. The algorithm’s simplicity, efficiency, and versatility make it an essential tool for anyone working with unsupervised learning tasks.
Whether you’re segmenting customers, organizing images, or discovering patterns in complex datasets, mastering kmeans sklearn gives you a powerful analytical capability. Remember to always preprocess your data appropriately, choose the optimal number of clusters using methods like the elbow technique or silhouette analysis, and validate your results with proper evaluation metrics. With these skills in your toolkit, you’re well-equipped to tackle real-world clustering challenges and extract meaningful insights from unlabeled data.
