Tensor Operations

The practical reference for the tensor manipulation operations used throughout every code example in this hub — indexing, reshaping, and combining tensors.

Indexing and Slicing

x = torch.arange(12).reshape(3, 4)
print(x[0])          # first row
print(x[:, 1])        # second column, across all rows
print(x[1:3, 0:2])    # a sub-block
print(x[-1])          # last row

Reshaping — view() vs reshape()

x = torch.arange(12)
print(x.view(3, 4))       # reinterprets the SAME underlying memory -- fails if not contiguous
print(x.reshape(3, 4))     # like view(), but falls back to copying if the data isn't contiguous
print(x.unsqueeze(0).shape)   # (1, 12) -- adds a dimension of size 1
print(x.squeeze().shape)       # removes dimensions of size 1

.view() requires the tensor's data to be contiguous in memory (e.g. after certain operations like .transpose(), it may not be) — .reshape() is the safer general-purpose choice, since it copies data automatically when needed.

Combining Tensors

a = torch.zeros(2, 3)
b = torch.ones(2, 3)

torch.cat([a, b], dim=0)     # (4, 3) -- concatenate ALONG an existing dimension
torch.stack([a, b], dim=0)    # (2, 2, 3) -- combine along a NEW dimension

The distinction between cat and stack is one of the most common sources of shape-mismatch confusion: cat joins tensors along an already-existing dimension (the result has the same number of dimensions as the inputs); stack creates a brand new dimension (the result has one more dimension than the inputs).

Element-Wise vs Matrix Operations

a = torch.tensor([[1., 2.], [3., 4.]])
b = torch.tensor([[5., 6.], [7., 8.]])

print(a * b)        # element-wise multiplication
print(a @ b)         # matrix multiplication -- see Matrix Multiplication
print(a.T)            # transpose

Common Mistakes

• Using .view() on a non-contiguous tensor (e.g. right after .transpose()) and hitting a runtime error — call .contiguous() first, or just use .reshape() instead.
• Confusing cat and stack when combining a batch of individually-processed tensors — using the wrong one produces a shape one dimension off from what's expected, a very common bug.

Interview Relevance

Q: "What's the difference between torch.cat and torch.stack?" cat concatenates tensors along an existing dimension, so the result has the same number of dimensions as the inputs. stack combines tensors along a brand-new dimension, so the result has one more dimension than the inputs — used when you want to preserve each input as a distinct "slice" rather than merging them along an existing axis.

Practice Question

You have 5 tensors, each of shape (3, 4), representing 5 separate samples. How would you combine them into one tensor of shape (5, 3, 4)?

Related DL Notes

• PyTorch Tensors
• Matrix Multiplication
• Broadcasting (PyTorch)