Intro to PyTorch

Created

!nvidia-smi prints out details about the GPU.

Tensors

Tensor is essentially an array.

tensor and Tensor different.

A tensor has a datatype that can be specified with dtype.

  • NumPy and PyTorch have different default data types.
  • print(data.dtype) to print it out.
  • float32 is better for efficiency

torch.zeros torch.ones make a tensor of all zeros / all ones.

torch.arange(1,10) does a range from [1 to 10).

rr + 2 is a broadcast operation. applies to every single one.

a.matmul(b) (product) and a @ b (dot product or matrix multiplication)

shape .shape to access the shape

reshape via rr.view. it "folds".

  • it doesn't take extra memory
  • make sure the data is not being copied with any other variation.

Devices and NumPy

Convert between tensor and NumPy:

  • e.g., plotting

data.device

data = torch.arange(1, 7, device='cuda')

  • specifies what to run on

data = data.to('cuda')

data.cpu().numpy()

  • copies data from gpu to cpu and then it can be converted to numpy.

The numpy shares the same piece of memory as the original tensor.

Convert a scalar back to a Python value with .item().

Autograd

When you specify requires_grad=True when you define a Tensor. Then, given a bunch of inputs, each one has a property called grad.

y.backward() will not clear out the previously stored gradient. It will accumulate. Then you must x.grad = None.

calculate the gradient of 100 items. Then you can gradient accumulate. calculate sequentially and pretend you are doing it in order.

Models

A neural network is just a bunch of layers chained together.

better to define models as a child of the nn.Module class.

# Pass our input through our model
model(input) # model.forward(input)

same thing

Once you define module, you're done with forward propagation

import torch.optim as optim has a bunch of optimization methods

Linked from Machine Learning: Deep Learning