Tensors from Scratch #1: Tensors and their Shapes

27 Jul, 2024

This is a series of blog posts that will let you understand what exactly are Tensors, and how you can too implement them from scratch in C. I will be going from basic definitions, illustrations, and explanations to the actual code implementation in this series.

Don't worry, I'm here to help :)

Before jumping into the implementation of tensors, we first need to understand what are tensors and how can we define them.

Tensors

What the hell do I mean when I say the word tensor? Let's start from the beginning.

Imagine a number say,

x = 33

This number $x$ is just a value and it is "dimensionless" i.e. we cannot attach any arrows to this number in the sense that tells us what this number is made up of. We call numbers like this Scalars. The dimensionality of a scalar is $0$ .

Note: in order to not confuse ourselves, I will refer to "dimensionality" as rank

Now, imagine a list of numbers say,

y = [33, 2, 5, 10, . . .]

This list of numbers is made up of values that go from $33$ , $2$ , $5$ , and so on. This list of numbers has a "dimension" i.e. we can attach an arrow $\to$ to this list that starts from the first value until it reaches the end of the list. We call lists of numbers like this Vectors. The dimensionality of a vector is $1$ .

Now, imagine a list of list of numbers say,

z = [\begin{matrix} 33 & 12 & 13 & \dots & 42 \\ 40 & 7 & 2 & \dots & 56 \\ ⋮ \\ 29 & 4 & 60 & \dots & 80 \end{matrix}]

This list of list of numbers is made up of $n$ lists (each containing some values). This list of list of numbers has "dimensions" as you might have guessed i.e. we can attach exactly two arrows: an arrow $\to$ for the values in each list inside, and an arrow $↓$ for each list that makes up the big list. We call a list of list of numbers like this Matrices. The dimensionality of a matrix is $2$ .

As you can see the dimension number constantly increases starting from $0$ to $n$ and the "mathematical object" formed using dimension $n$ is different from an object formed using dimension $n - 1$ or $n + 1$ . Scalars are different than vectors, vectors are different than matrices, and so on...

It would be good if we had a general term to refer to "mathematical objects" like above of dimension $n$ including $0$ . This term is called, a Tensor.

Scalars are 0-dimensional Tensors i.e. rank-0 Tensors.
Vectors are 1-dimensional Tensors i.e. rank-1 Tensors
Matrices are 2-dimensional Tensors i.e. rank-2 Tensors.

In short,

N-dimensional lists (or arrays) are n-dimensional Tensors.

A thing to note here is that for a given tensor, the values that make up the tensor will be of the same data type i.e. the type of one value cannot be different from the type of another value within the same tensor.

Shapes

In the previous section we defined what a tensor is and determined the ranks of tensors. However, there is one more thing intrinsic to tensors that can give us a lot more information about them. This thing is called the shape of a tensor.

Shape is defined as the number of elements present in each dimension of the tensor.

We write the shape of a tensor as a list of numbers telling us how many elements are present in each dimension. Here, the term "elements" can either mean numbers (values) or list of values or even a list of list of values depending on the rank of the tensor in question. Let's go through some examples.

Scalar

We know that scalars are rank 0 tensors. They are just numbers and do not have any dimension. So for the scalar:

x = 33

The shape will be written as:

shape = ()

Yes, it is empty since scalars do not have any dimension.

Vector

We know that vectors are rank 1 tensors. A vector is a list of numbers of length $n$ along one single dimension. So for the vector:

x = [x_{0}, x_{1}, x_{2}, . . ., x_{n - 1}]

The shape will be written as:

shape = (n)

There are $n$ numbers (elements) along the one and only dimension of a vector hence the shape contains a single entry which is the number $n$ for a vector.

Matrix

We know that matrices are rank 2 tensors. A matrix is a list of list of numbers where the inner list can have $n$ elements (numbers) and the outer list can have $m$ elements (lists). So for the matrix:

x = [\begin{matrix} x_{0, 0} & x_{0, 1} & x_{0, 2} & \dots & x_{0, n - 1} \\ x_{1, 0} & x_{1, 1} & x_{1, 2} & \dots & x_{1, n - 1} \\ ⋮ \\ x_{m - 1, 0} & x_{m - 1, 1} & x_{m - 1, 2} & \dots & x_{m - 1, n - 1} \end{matrix}]

The shape will be written as:

shape = (m, n)

Along the columns (dimension = 1) there are $n$ elements which are numbers and along the rows (dimension = 0) there are $m$ elements which are lists of numbers.

Given a shape, we go from left to right. The leftmost number then tells us how many elements are present in the first dimension. The second leftmost number tells us how many elements are present in the second dimension, and so on.

Sometimes, people use the word axis in place of dimension. Thus, for the above case axis 0 is dimension 0 with $m$ elements, axis 1 is dimension 1 with $n$ elements.

In general, for a rank-N tensor, the shape can be written as:

axis = (0, 1, . . ., n - 1)

shape = (s_{0}, s_{1}, . . ., s_{n - 1})

The illustration below will clear out any doubts about shapes of tensors. Visually, we can imagine that in order to go from dimensionality $n$ to $n + 1$ we essentially perform a copy onto the next available dimension.

shapes of tensors

Information from shape

For this section, let me tell you how much information we can get from the shape $S$ of a tensor $T$ in a question-answer format for easier understanding.

What is the rank of $T$ ?

Answer: Easy, the length of the shape $S$ gives us the rank/dimension of the tensor $T$ . For a scalar, there are $0$ entries in the shape so rank becomes $0$ . For a vector, there is $1$ entry in the shape as written above so rank is $1$ . In general, the length of shape $S$ gives us the dimensionality (rank) of the tensor $T$ .

l e n (S) = r a n k (T)

How many numbers or values does $T$ contain i.e. what is the total "size" of it?

Answer: When we multiply all the entries of shape $S$ we get how many values does $T$ contain. This number is also called the total size of a tensor. A scalar with empty shape $()$ means that it contains only $1$ value which is correct. A vector with shape $(n)$ means that it contains $n \times 1$ values. A matrix with shape $(m, n)$ means that it contains $m \times n$ values. In general case:

totalsize (T) = \prod_{i = 0}^{n - 1} s_{i} = s_{0} \times s_{1} \times . . . \times s_{n - 1}

Summary

In short, tensor is an umbrella term for "mathematical objects" or "containers" that have a rank/dimensionality associated with them.

Scalars are rank-0 tensors, vectors are rank-1 tensors, matrices are rank-2 tensors and so on.
One important property of a tensor is its shape. Shape is defined as: the number of elements present in each axis/dimension of the tensor.
Shape is written as entries for each dimension between parenthesis $($ and $)$ .
From the shape we can determine: the rank/dimensionality of the tensor by shape's length, and the total number of values a tensor contains by multiplying all entries of shape.

We're not done...

This part was an overview of what tensors and their shapes are from a high level. In the next part, we will focus on the type of operations that we can perform on tensors. I will explain how shapes play a role in those operations, and we will tackle something called "shape broadcasting" there. See you soon :)

#C #tensors