CMSC320_Project03

In this part of this project you will practice and experiment with linear regression using data from gapminder.org. We recommend spending a little time looking at material there, it is quite an informative site.

We will use a subset of data provided by gapminder provided by Jennifer Bryan described in its github page

Get the data from: https://github.com/jennybc/gapminder/blob/master/data-raw/08_gap-every-five-years.tsv

(A mirror of that data is available in the project repository, in the mirror directory.)

import pandas as pd
data = pd.read_csv("gap.tsv", sep='\t')
data.head()

    country	 continent	year	lifeExp	pop	        gdpPercap
0	Afghanistan	Asia	1952	28.801	8425333	    779.445314
1	Afghanistan	Asia	1957	30.332	9240934	    820.853030
2	Afghanistan	Asia	1962	31.997	10267083	853.100710
3	Afghanistan	Asia	1967	34.020	11537966	836.197138
4	Afghanistan	Asia	1972	36.088	13079460	739.981106

For this exercise you will explore how life expectancy has changed over 50 years across the world, and how economic measures like gross domestic product (GDP) are related to it.

Exercise 1: Make a scatter plot of life expectancy across time.

Question 1: Is there a general trend (e.g., increasing or decreasing) for life expectancy across time? Is this trend linear? (answering this qualitatively from the plot, you will do a statistical analysis of this question shortly)

A slightly different way of making the same plot is looking at the distribution of life expectancy across countries as it changes over time:

    fig, ax = plt.subplots()

    ax.violinplot(life_exp_per_year,years,widths=4,showmeans=True)
    ax.set_xlabel("Year")
    ax.set_ylabel("Life Expectancy")
    ax.set_title("Violin Plot Example")
    fig.savefig("violin.png")

This type of plot is called a violin plot, and it displays the distribution of the variable in the y-axis for each value of the variable in the x-axis. (It is okay to use other plotting libraries and tools to create this plot, and others throughout the assignment.) Note that in order for the example code above, you will have to wrangle the data using Pandas. The way I did it was to create an array for each year, storing the Life Expectancy values for that year, then collected those arrays into a list: life_exp_per_year. It’s not the only way of doing it, but it’s how I first thought of it.

Question 2: How would you describe the distribution of life expectancy across countries for individual years? Is it skewed, or not? Unimodal or not? Symmetric around it’s center?

Based on the Violin plot you made, consider the following questions.

Question 3: Suppose I fit a linear regression model of life expectancy vs. year (treating it as a continuous variable), and test for a relationship between year and life expectancy, will you reject the null hypothesis of no relationship? (do this without fitting the model yet. I am testing your intuition.)

Question 4: What would a violin plot of residuals from the linear model in Question 3 vs. year look like? (Again, don’t do the analysis yet, answer this intuitively)

Question 5: According to the assumptions of the linear regression model, what should that violin plot look like? That is, consider the assumptions the linear regression model you used assumes (e.g., about noise, about input distributions, etc); do you think everything is okay?

Exercise 2: Fit a linear regression model using, e.g., the LinearRegression function from Scikit-Learn or the closed-form solution, for life expectancy vs. year (as a continuous variable). There is no need to plot anything here, but please print the fitted model out in a readable format.

Question 6: On average, by how much does life expectancy increase every year around the world?

Question 7: Do you reject the null hypothesis of no relationship between year and life expectancy? Why?

Exercise 3: Make a violin plot of residuals vs. year for the linear model from Exercise 2.

Question 8: Does the plot of Exercise 3 match your expectations (as you answered Question 4)?

Exercise 4: Make a boxplot (or violin plot) of model residuals vs. continent.

Question 9: Is there a dependence between model residual and continent? If so, what would that suggest when performing a regression analysis of life expectancy across time?

Exercise 5: As in the Moneyball project, make a scatter plot of life expectancy vs. year, grouped by continent, and add a regression line. The result here can be given as either one scatter plot per continent, each with its own regression line, or a single plot with each continent’s points plotted in a different color, and one regression line per continent’s points. The former is probably easier to code up.

Question 10: Based on this plot, should your regression model include an interaction term for continent and year? Why?

Exercise 6: Fit a linear regression model for life expectancy including a term for an interaction between continent and year. Print out the model in a readable format, e.g., print the coefficients of the model (no need to plot). Hint: adding interaction terms is a form of feature engineering, like we discussed in class (think about, e.g., using (a subset of) polynomial features here).

Question 11: Are all parameters in the model significantly (in the p-value sense) different from zero? If not, which are not significantly different from zero? Other libraries (statsmodels or patsy may help you solve this problem)

Question 12: On average, by how much does life expectancy increase each year for each continent? (Provide code to answer this question by extracting relevant estimates from model fit)

Exercise 7: Make a residuals vs. year violin plot for the interaction model. Comment on how well it matches assumptions of the linear regression model.