{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measures of spread"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "import scipy.stats\n",
    "\n",
    "food = pd.read_pickle(\"../data/processed/food\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw in the measures of central tendency section that the mean can be a poor representation of data if the data is skewed, and that we should therefore be careful when someone presents us with a mean (or average) without any further information.\n",
    "\n",
    "One of the types of 'further information' that can help us is a measure of spread of the data around the mean value.\n",
    "We usually use the *variance* and the *standard deviation* to quantify measure of spread.\n",
    "Both are easy to calculate, and even easier to convert between each other.\n",
    "We also sometimes see the range and inter--quartile range.\n",
    "These are simpler to calculate, but less useful.\n",
    "\n",
    "## Range\n",
    "\n",
    "Simply the largest value, minus the smallest value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food.P344pr.max() - food.P344pr.min()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the full spread of the data set so doesn't tell us anything very useful in this case, and the range is very susceptible to outliers.\n",
    "Therefore the inter--quartile range is more common.\n",
    "\n",
    "## Inter--quartile range\n",
    "\n",
    "As we saw above the range is very susceptible to outliers, so the inter--quartile range is effectively a 'trimmed' range by taking off the very lowest and the very highest values.\n",
    "In fact, we take off the top 25% and the bottom 25% and specify this range.\n",
    "This has the disadvantage of only describing the middle 50% of our data, but this is usually outweighed by the fact that this is less susceptible to outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food.P344pr.quantile(0.75) - food.P344pr.quantile(0.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's common to plot a **boxplot** to depict the range and interquartile range.\n",
    "\n",
    "The figure below is a boxplot of income:\n",
    "\n",
    "- the mean is the green horizontal bar\n",
    "- the interquartile range (representing 50% of the incomes) is the box\n",
    "- the full range are the stalks (whiskers) at the very top and bottom of the figure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food.boxplot(column = \"P344pr\")\n",
    "plt.xlabel(\"\")\n",
    "plt.ylabel(\"Income (£)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The range and inter--quartile range are useful when beginning to understand and explore your data, but are not usually reported.\n",
    "Instead the variance and standard deviation are more commonly reported, because they are more useful.\n",
    "\n",
    "## Variance\n",
    "\n",
    "To calculate the variance:\n",
    "\n",
    "1. subtract the mean from each score\n",
    "1. square the result\n",
    "1. sum the results to produce one value\n",
    "1. divide by $n - 1$ (number of observations minus one)\n",
    "\n",
    "$$\n",
    "\\frac{\\Sigma (x - \\bar{x}) ^ 2}{n-1}\n",
    "$$\n",
    "\n",
    "The numerator (top half of the equation) is sometimes referred to as the 'sum of squares'.\n",
    "\n",
    "The denominator (bottom half of the equation) uses $n - 1$ rather than simply the number of observations because we have reduced our degrees of freedom in assuming that the sample mean is the same as the population mean.\n",
    "This is known as [Bessel's correction](https://en.wikipedia.org/wiki/Bessel%27s_correction).\n",
    "\n",
    "To calculate the variance of the population we must assume that the population mean is the same as the sample mean that we have observed.\n",
    "As we now effectively 'know' the population mean it means we can vary all but one of our observations, while the final observation is determined to make sure the mean remains constant.\n",
    "\n",
    "For example if our sample mean is 100 we assume our population mean is also 100.\n",
    "If we have two observations these might be 110 and 90 (mean 100).\n",
    "If we change the 110 value to 120, the 90 value *must* change to 80 to ensure the sample mean (and therefore the population mean) remains 100, so there is only one degree of freedom.\n",
    "We would therefore use $2 - 1$ as the denominator in our variance calculation.\n",
    "\n",
    "The variance of our income data is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food[\"P344pr\"].var()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard deviation\n",
    "\n",
    "As you've probably noticed the variance is not in the units of the original data.\n",
    "This is where the standard deviation comes in.\n",
    "In fact the unit of the variance is the *square* of the unit of the original data.\n",
    "The standard deviation is therefore a measure of spread in the units of the original data, and is calculated simply by square rooting the variance:\n",
    "\n",
    "$$\n",
    "\\sqrt{\\frac{\\Sigma (x - \\bar{x}) ^ 2}{n-1}}\n",
    "$$\n",
    "\n",
    "The standard deviation of the income is therefore:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food.P344pr.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard deviation is a measure of how far the data points are on average from the mean.\n",
    "A small standard deviation means the mean fairly accurately represents the data; a large standard deviation means the mean does not represent the data well.\n",
    "\n",
    "In the case of our example the standard deviation of the income data is quite large compared to the mean, suggesting a lot of variability even in the trimmed income data. The histograms support this, and suggest the mean might not be as good as the median in summarising this data.\n",
    "\n",
    "In fact we can estimate that about 95% of cases fall between (I'll get on to why I'm multiplying by 1.96 later):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food.P344pr.mean() - 1.96 * food.P344pr.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food.P344pr.mean() + 1.96 * food.P344pr.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is most of the data set, so the variability of the data suggest the mean may not be a useful summary of the data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}