Variance Of Parameter In Linear Regression

Introduction

Linear regression is a fundamental technique in statistics and econometrics used to model the relationship between a dependent variable and one or more independent variables. The linear regression model is widely used in various fields, including economics, finance, and social sciences. In this article, we will discuss the variance of the parameter estimate in linear regression, specifically the variance of the slope coefficient, $\hat{\beta_1}$. We will reconcile two apparently contrasted results about the variance of $\hat{\beta_1}$ found in two econometric books.

The Linear Regression Model

The linear regression model is given by:

$$y = \beta_0 + \beta_1x + \epsilon$$

where $y$ is the dependent variable, $x$ is the independent variable, $\beta_0$ is the intercept, $\beta_1$ is the slope coefficient, and $\epsilon$ is the error term.

Variance of the Slope Coefficient

The variance of the slope coefficient, $\hat{\beta_1}$, is a crucial concept in linear regression. It measures the amount of variation in the estimated slope coefficient. The variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

where $\sigma^2$ is the variance of the error term, $x_i$ is the $i^{th}$ observation of the independent variable, $\bar{x}$ is the mean of the independent variable, and $n$ is the sample size.

Contrasted Results

We found two apparently contrasted results about the variance of $\hat{\beta_1}$ in two econometric books. The first result states that the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

This result is widely accepted and is a standard result in linear regression.

The second result states that the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

where $R^2$ is the coefficient of determination.

Reconciliation of Contrasted Results

At first glance, the two results appear to be in conflict. However, upon closer inspection, we can see that the second result is a special case of the first result. The second result assumes that the independent variable is not correlated with the error term, which is a common assumption in linear regression. In this case, the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

However, if the independent variable is correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

This is the standard result in linear regression.

Conclusion

In conclusion, the variance of the slope coefficient, $\hat{\beta_1}$, in linear regression is a crucial concept that measures the amount of variation in the estimated slope coefficient. We reconciled two apparently contrasted results about the variance of $\hat{\beta_1}$ found in two econometric books. The second result is a special case of the first result, assuming that the independent variable is not correlated with the error term. However, if the independent variable is correlated with the error term, then the variance of $\hat{\beta_1}$ is given by the standard result in linear regression.

References

• Greene, W. H. (2018). Econometric Analysis. 8th ed. Prentice Hall.
• Wooldridge, J. M. (2016). Introductory Econometrics: A Modern Approach. 5th ed. Cengage Learning.

Appendix

Derivation of the Variance of the Slope Coefficient

The variance of the slope coefficient, $\hat{\beta_1}$, can be derived using the following steps:

1. Write the linear regression model as:

$$y = \beta_0 + \beta_1x + \epsilon$$

2. Take the derivative of the log-likelihood function with respect to $\beta_1$:

$$\frac{\partial \ln L}{\partial \beta_1} = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i) x_i$$

3. Set the derivative equal to zero and solve for $\beta_1$:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n (y_i - \bar{y})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

4. Take the variance of $\hat{\beta_1}$:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

This is the standard result in linear regression.

Derivation of the Variance of the Slope Coefficient with Correlated Independent Variable and Error Term

If the independent variable is correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

where $R^2$ is the coefficient of determination.

This result can be derived using the following steps:

1. Write the linear regression model as:

$$y = \beta_0 + \beta_1x + \epsilon$$

2. Take the derivative of the log-likelihood function with respect to $\beta_1$:

$$\frac{\partial \ln L}{\partial \beta_1} = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i) x_i$$

3. Set the derivative equal to zero and solve for $\beta_1$:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n (y_i - \bar{y})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

4. Take the variance of $\hat{\beta_1}$:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

This result assumes that the independent variable is correlated with the error term.

Derivation of the Variance of the Slope Coefficient with Uncorrelated Independent Variable and Error Term

If the independent variable is not correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

This result can be derived using the following steps:

1. Write the linear regression model as:

$$y = \beta_0 + \beta_1x + \epsilon$$

2. Take the derivative of the log-likelihood function with respect to $\beta_1$:

$$\frac{\partial \ln L}{\partial \beta_1} = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i) x_i$$

3. Set the derivative equal to zero and solve for $\beta_1$:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n (y_i - \bar{y})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

4. Take the variance of $\hat{\beta_1}$:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

Q&A

Q: What is the variance of the slope coefficient in linear regression? A: The variance of the slope coefficient, $\hat{\beta_1}$, is a measure of the amount of variation in the estimated slope coefficient. It is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

Q: What is the difference between the variance of the slope coefficient with correlated and uncorrelated independent variable and error term? A: If the independent variable is correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

where $R^2$ is the coefficient of determination.

If the independent variable is not correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

Q: How do I determine if the independent variable is correlated with the error term? A: You can use the following methods to determine if the independent variable is correlated with the error term:

1. Correlation analysis: Calculate the correlation coefficient between the independent variable and the error term. If the correlation coefficient is significant, then the independent variable is correlated with the error term.
2. Regression analysis: Run a regression analysis with the independent variable as the dependent variable and the error term as the independent variable. If the regression coefficient is significant, then the independent variable is correlated with the error term.
3. Diagnostic tests: Use diagnostic tests such as the Breusch-Pagan test or the White test to determine if the independent variable is correlated with the error term.

Q: What is the implication of the variance of the slope coefficient on the interpretation of the results? A: The variance of the slope coefficient has a significant implication on the interpretation of the results. If the variance of the slope coefficient is large, then the estimated slope coefficient is less reliable and may not be statistically significant. On the other hand, if the variance of the slope coefficient is small, then the estimated slope coefficient is more reliable and may be statistically significant.

Q: How can I reduce the variance of the slope coefficient? A: You can reduce the variance of the slope coefficient by:

1. Increasing the sample size: A larger sample size will reduce the variance of the slope coefficient.
2. Improving the quality of the data: High-quality data will reduce the variance of the slope coefficient.
3. Using robust standard errors: Robust standard errors will reduce the variance of the slope coefficient.
4. Using a different regression model: A different regression model may reduce the variance of the slope coefficient.

Q: What are the limitations of the variance of the slope coefficient? A: The variance of the slope coefficient has several limitations:

1. Assumes linearity: The variance of the slope coefficient assumes a linear relationship between the independent variable and the dependent variable.
2. Assumes homoscedasticity: The variance of the slope coefficient assumes that the variance of the error term is constant across all levels of the independent variable.
3. Does not account for non-linear relationships: The variance of the slope coefficient does not account for non-linear relationships between the independent variable and the dependent variable.

Conclusion

In conclusion, the variance of the slope coefficient, $\hat{\beta_1}$, is a crucial concept in linear regression that measures the amount of variation in the estimated slope coefficient. We reconciled two apparently contrasted results about the variance of $\hat{\beta_1}$ found in two econometric books. The variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

If the independent variable is correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

where $R^2$ is the coefficient of determination.

We also discussed the implications of the variance of the slope coefficient on the interpretation of the results and how to reduce the variance of the slope coefficient.

References

• Greene, W. H. (2018). Econometric Analysis. 8th ed. Prentice Hall.
• Wooldridge, J. M. (2016). Introductory Econometrics: A Modern Approach. 5th ed. Cengage Learning.

Appendix

Derivation of the Variance of the Slope Coefficient

The variance of the slope coefficient, $\hat{\beta_1}$, can be derived using the following steps:

1. Write the linear regression model as:

$$y = \beta_0 + \beta_1x + \epsilon$$

2. Take the derivative of the log-likelihood function with respect to $\beta_1$:

$$\frac{\partial \ln L}{\partial \beta_1} = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i) x_i$$

3. Set the derivative equal to zero and solve for $\beta_1$:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n (y_i - \bar{y})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

4. Take the variance of $\hat{\beta_1}$:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

This is the standard result in linear regression.

Derivation of the Variance of the Slope Coefficient with Correlated Independent Variable and Error Term

If the independent variable is correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

where $R^2$ is the coefficient of determination.

This result can be derived using the following steps:

1. Write the linear regression model as:

$$y = \beta_0 + \beta_1x + \epsilon$$

2. Take the derivative of the log-likelihood function with respect to $\beta_1$:

$$\frac{\partial \ln L}{\partial \beta_1} = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i) x_i$$

3. Set the derivative equal to zero and solve for $\beta_1$:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n (y_i - \bar{y})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

4. Take the variance of $\hat{\beta_1}$:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \times \frac{1}{1 - R^2}$$

This result assumes that the independent variable is correlated with the error term.

Derivation of the Variance of the Slope Coefficient with Uncorrelated Independent Variable and Error Term

If the independent variable is not correlated with the error term, then the variance of $\hat{\beta_1}$ is given by:

$$Var(\hat{\beta_1}) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

This result can be derived using the following steps:

1. Write the linear regression model as:

$$y = \beta_0 + \beta_1x + \epsilon$$

2. Take the derivative of the log-likelihood function with respect to $\beta_1$:

$$\frac{\partial \ln L}{\partial \beta_1} = \sum_{i=1}^n (y_i - \beta_0 - \beta_1x_i) x_i$$

3. Set the derivative equal to zero and solve for $\beta_1$:

$$\hat{\beta_1} = \frac{\sum_{i=1}^n (y_i - \bar{y})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

4. Take the variance of $\hat{\beta_1}$:

Var(\hat{\beta