In a previous post, I showed one way to arrive at the correlation coefficient but it doesn’t really convince me of its need. What follows is a derivation that will show you a more direct meaning for the correlation coefficent and how it emerges as a quantity of interest when we look at the following optimization problem.

Given two random variables a very natural question is to ask how similar these two are. This is a loaded question because we need to 1) state in what way they can be similar and 2) state how certain we are about their similarity.

Let’s say we will 1) only look for a linear relationship and 2) say they are close based on the square error. That is, we want to find coefficients such that

I won’t go through the derivation as it only uses the techniques here. When optimized you will find the following.

Let’s see the error it generates

And there is your correlation coefficient. We now see that are linearly dependent when the and, in general, the correlation coefficient implies the strength of linear dependence between random variables.