Consolidating Fundamental Concepts

What is covariance? It was the first thing I asked myself when solving the exam of module one of the last course of the specialization of mathematics for machine learning. Consequently, I was forced to first interpret what variance and standard deviation mean.

The standard deviation allows us to know how dispersed the data are with respect to the mean. Taking the rule of thumb (rule 68-95-99.7) of the normal distribution as a reference, 68% of the data are within one standard deviation of the mean, 95% are within two standard deviations of the mean, and 99.7% of the data are within 3 standard deviations. In other words, this measure allows us to know the dispersion of our data.

The variance represents these differences from the mean, squared. In other words, it measures this dispersion with a higher and only positive value (not to go into details).

What is covariance?

Having written this brief introduction, we begin. The covariance allows us to calculate the variation with respect to the mean (just like the variance) but with two random variables instead of one, allowing us to know if there is a dependence between the two variables. The first thing I thought when I read this was "this is not really necessary, since I have been using linear correlation coefficients or regression lines".

But the covariance is still a fundamental component for the calculation of both, and that is why the correlation coefficients tell us what percentage of the variations in a dependent variable is explained by the variation in the independent variable. How do we interpret the covariance? In case the high values of any of the variables correspond to the high values of the other, we will have a positive value in the covariance (the same applies to small values).

In case high values of a variable correspond to small values, the covariance will tend to be negative. Therefore, the sign determines the trend in the linear relationship of the variables.

The covariance is determined as follows:

$$Cov(X,Y)=\frac{\sum_1^n (x_i-\bar x)(y_i-\bar y)}{n}$$

In case of having more than 2 variables we can make a covariance matrix, determined by:

$$\sum=\begin{bmatrix} Var(X_1) & Cov(X_1, X_2) & ... & Cov(X_1,X_n) \\ Cov(X_2,X_1)&Var(X_2)&...&Cov(X_2,X_n) \\ \vdots & \vdots & \ddots & \vdots \\ Cov(X_n,X_1)&Cov(X_n,X_2)&...&Var(X_n) \end{bmatrix}$$

Important properties

1. The covariance of two equal variables is the same as the variance of the same variable.

Two variables are independent if their covariance is 0, i.e. they are not correlated.

Just wonderful to be able to understand a little more what is behind correlation and linear regression analysis. If I feel like it, I will continue to publish this kind of interesting "curiosities".