In this quick article we will learn what is mean absolute deviation and standard deviation let’s say you have a history test scores here I have six data points where I have shown the scores out of hundred and the average score here is 70. and when you plot this on a chart you will see that 70 is an average this yellow line and these are the individual data points for example you have 75 here you have 65 here and so on and you can see that all these data points are near the average data point so what we’re trying to learn is how far apart the individual data points are from the average or how spread out the data points are with respect to average.
You can have a different data set let’s say a score in the mathematics test and those scores might look like this again the average is same but when you plot them on a chart you will find that data points are quite spread out they are quite far away from average data points and in statistics and data science it helps if you can figure out how far apart individual data points are from the average data point or how spread out these data points are and we need a metric or a number to represent this concept here my data set is simple I can easily say that in the right hand side the data points are far apart but when.
You have millions of data points and if you want to come up with a single number or a metric which can tell you this then that will be super useful so pause this video for a moment and think about what that metric could be well as a busy very basic approach common sense approach would tell you that why don’t you take the difference of individual score with respect to average so let’s do that 75 how far away it is from average well my average is 70. so this guy is five points away similarly 72 is two points away 60 point is 68 is 2.7 and so on and if you take a mean of this number that will be 3.16 and for the second case you do the same thing and when you take a mean you get 23.
This is interesting look this number is telling you that for the history test my mean is 3.16 but for mathematic days my mean is 23 which means here I have a higher number and that implies that this particular data set the mathematic test data set is wide spread out you know with respect to the average number so this is interesting now this mean number is nothing but mean absolute deviation you can also call it mad all right so m80 could be quite a significant or a useful metric that can represent how spreader spread your data points are but there could be a scenario where using mean absolute deviation might not be enough let’s take this particular example here see here mean absolute deviation is 3.33 on left and right hand side but if you look at the chart you can see that on a right hand side although I have four points right at the average.
I have one data point which is far away see this is far away on the left hand side all my data points are between 65 to 75 range but here my data points are in 63 to 83 range so my range is widespread so here the data distribution is kind of spread out and when you look at mad it tells you it’s the same number 3.33 so in this case mad is not very useful so we need to come up with something else again pause the video can you come up with some other concept which can represent this data distribution accurately well if you give it a little zee you might come up with this idea that how about we again we keep the same data set same absolute everything but we add a new column and we take a square of these individual numbers so here see look at this column this column is nothing but the square of this second column so 5 square is 25 2 square is 4 2 square is 4 5 square is 25 and so on and then you take an average of this column okay.
I took the average of this column which is 12.66 then I took the square root of that average why did I take square root well because you squared the numbers here individually therefore you want to do a square root okay so here I come up with number 3.55 on the right hand side I do the same mathematics and come up with average and square root of average but this time I came up with 6.02 so see here the number is 3.55 here the number is 6.02 so higher the number means the data points are spread out more they are more widely spread out this square root is called standard deviation standard deviation is nothing but you take individual data points so here this this particular symbol is called sigma and sigma is your standard deviation and standard deviation is nothing but you take individual data point which is x.
I minus mu is average then you take square of individual data points then this symbol means you sum it up and then you take the average and then you take the square root see this is not complicated friends we already did this math if you look at this is exactly what we did so we took the difference first okay so that difference is x I minus u so that is this particular column then we took a square of that okay so that is the square then sum and then divided by n is number of data points which is an average so that is this average and then we took the square root which is this square root and hence you got 3.55 versus 6.02 so standard deviation is something you will use a lot in data science machine learning statistics these are these are the very basic concepts that you need to know when you’re learning statistics or data science you might have heard about this term l1 and l2 norm if you’re doing machine learning let’s say linear regression you might have done a ridge regression and lasso regression so those are l1 and l2 l1 usually refers to mean absolute division and l2 refers to the standard deviation.
So we’ll be using this standard deviation in future videos as well my next video is going to be most likely on normal distribution so I hope this video gave you some basic understanding of standard deviation and mean absolute deviation.