t-SNE

t-SNE (t-Distributed Stochastic Neighbor Embedding) is a non-linear dimensionality reduction technique built specifically for visualization — it excels at revealing cluster structure in 2D/3D that PCA's linear projections often miss entirely.

Why PCA Isn't Always Enough

PCA can only capture linear structure — the concentric-circles or crescent-moon shapes that broke linear SVM boundaries are exactly the kind of non-linear structure PCA also struggles to represent meaningfully in a low-dimensional projection. t-SNE instead focuses on preserving local neighborhood relationships — points that were close together in the original high-dimensional space stay close together in the 2D embedding, regardless of the overall shape of that structure.

The Core Idea, Conceptually

The "t-distributed" part refers to using a heavy-tailed distribution in the low-dimensional space, which helps prevent points from being crushed together in the middle of the plot — a genuine practical improvement over earlier, related techniques.

Python Implementation

from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt

data = load_digits()
X, y = data.data, data.target

tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_embedded = tsne.fit_transform(X)

plt.scatter(X_embedded[:,0], X_embedded[:,1], c=y, cmap="tab10", alpha=0.6)
plt.colorbar(label="Digit")
plt.show()
# t-SNE typically reveals tight, well-separated clusters per digit --
# often noticeably clearer than a 2D PCA projection of the same data

Perplexity — The Key Hyperparameter

Perplexity roughly controls how many neighbors each point considers when building its local probability distribution — think of it as a soft target for the "effective number of nearest neighbors." Typical values range from 5 to 50; too low can fragment genuine clusters into noise, too high can blur distinct clusters together.

Critical Limitations to Know

• Distances between clusters aren't meaningful. t-SNE preserves local neighborhoods well but distorts global distances — two clusters appearing far apart in the plot doesn't necessarily mean they're proportionally that different in the original space.
• Cluster sizes aren't meaningful either. A visually large cluster in a t-SNE plot doesn't necessarily correspond to more actual variance or spread in the original data.
• Non-deterministic and slow. Results can vary between runs (unless random_state is fixed), and t-SNE doesn't scale well to very large datasets.
• Not meant for general preprocessing. Unlike PCA, t-SNE has no simple transform() for new, unseen points — it's built for one-time visualization, not as a reusable feature transformation.

Practical Use Cases

• Visually exploring whether a high-dimensional dataset has natural cluster structure, before committing to a clustering algorithm
• Sanity-checking a classification problem's difficulty — classes that overlap heavily in a t-SNE plot are a hint the problem may be genuinely hard

Common Mistakes

• Interpreting distances or cluster sizes between groups in a t-SNE plot as quantitatively meaningful — only local neighborhood relationships are reliably preserved.
• Using t-SNE as a general-purpose feature reduction step before a downstream model — it's designed for visualization, not as a reusable transform (PCA or UMAP are far more appropriate for that).
• Not trying multiple perplexity values — a single run's plot can look misleadingly clean or messy depending on this one setting.

Interview Relevance

Q: "Why shouldn't you trust the distance between two clusters in a t-SNE plot?" t-SNE optimizes specifically for preserving local neighborhoods, not global distances — it can and does distort the relative spacing between distant groups to achieve a good local layout, so visual gaps between clusters don't reliably reflect how different those groups actually are in the original feature space.

Practice Question

You run t-SNE twice on the same dataset with different random seeds and get visually different plots. Does this mean t-SNE is broken? Explain what's actually happening.

Related ML Notes

• PCA
• UMAP
• Exploratory Data Analysis