Simple and Multiple Regression Notes as per Syllabus

Regression Explained:

Regression analysis is a statistical technique used to model the relationship between a dependent variable (Y) and one or more independent variables (X). It helps us understand how changes in the independent variables can influence the dependent variable. Here's a breakdown:

• Dependent Variable (Y): This is the variable you're trying to predict or explain. It's often the outcome you're interested in.
• Independent Variable (X): These are the variables you believe might influence the dependent variable. You can have one or more independent variables in a regression model.

The goal of regression analysis is to find the best-fitting equation that minimizes the difference between the actual values of the dependent variable (Y) and the values predicted by the model.

Types of Regression:

There are several types of regression, but here are some of the most common:

1.    Simple Linear Regression: This is the most basic type of regression, involving only one independent variable (X) and one dependent variable (Y). It helps us estimate a straight line that best fits the data points.

2.    Multiple Linear Regression: This extends simple linear regression by incorporating two or more independent variables (X1, X2, ..., Xn) to predict the dependent variable (Y). It helps us model the combined influence of multiple factors.

3.    Logistic Regression: This is a special type of regression used for predicting binary outcomes (Yes/No, Pass/Fail). It estimates the probability of an event occurring based on the independent variables.

4.    Polynomial Regression: This uses polynomial terms (X^2, X^3, etc.) of the independent variables to model more complex, non-linear relationships between X and Y.

5.    Non-linear Regression: This encompasses various techniques for modeling non-linear relationships between X and Y, unlike the straight line in simple linear regression.

 

Properties of Correlation

Correlation refers to the statistical association between two variables. It describes the direction and strength of the linear relationship between them. However, correlation doesn't necessarily imply causation - one variable causing the other.

Here are some key properties of correlation:

• Direction: Correlation can be positive, negative, or zero.
• Positive correlation: As the value of one variable increases, the value of the other variable tends to increase as well.
• Negative correlation: As the value of one variable increases, the value of the other variable tends to decrease as well.
• Zero correlation: There is no linear relationship between the variables. Their values fluctuate independently.
• Strength: The correlation coefficient (discussed next) quantifies the strength of the relationship, ranging from -1 (perfect negative correlation) to +1 (perfect positive correlation). A value of 0 indicates no linear relationship.
• Linearity: Correlation measures the linear association between variables. Non-linear relationships might not be captured by correlation.

Correlation Coefficient (r)

The correlation coefficient (r) is a numerical measure that summarizes the strength and direction of the linear relationship between two variables. It ranges from -1 to +1.

• r = +1: Perfect positive correlation (values move together in the same direction).
• r = -1: Perfect negative correlation (values move together in opposite directions).
• r = 0: No linear correlation (no predictable relationship between the variables).

The closer the absolute value of r is to 1 (either positive or negative), the stronger the linear relationship. The closer r is to 0, the weaker the relationship.

Computation of Correlation Coefficient (Numerical Example)

Here's an example of calculating the correlation coefficient (r) by hand:

Scenario: A teacher wants to investigate the relationship between students' study hours (X) and their exam scores (Y). They collect data from 5 students:

Student	Study Hours (X)	Exam Score (Y)
1	4	60
2	6	75
3	8	88
4	5	70
5	3	55

Steps:

1. Calculate the Mean (Average) for X and Y:

• ΣX (sum of all X values) = 4 + 6 + 8 + 5 + 3 = 26
• Mean of X (X̄) = ΣX / n (number of students) = 26 / 5 = 5.2 hours
• ΣY (sum of all Y values) = 60 + 75 + 88 + 70 + 55 = 248
• Mean of Y (Ȳ) = ΣY / n = 248 / 5 = 49.6 points

2. Calculate the Deviation from the Mean (X - X̄) and (Y - Ȳ) for each student:

Student	Study Hours (X)	Exam Score (Y)	X - X̄	Y - Ȳ
1	4	60	-1.2	10.4
2	6	75	0.8	25.4
3	8	88	2.8	38.4
4	5	70	-0.2	20.4
5	3	55	-2.2	5.4

3. Calculate the Product of Deviations from the Mean (X - X̄) * (Y - Ȳ):

Student	X - X̄	Y - Ȳ	(X - X̄) * (Y - Ȳ)
1	-1.2	10.4	-12.48
2	0.8	25.4	20.32
3	2.8	38.4	107.52
4	-0.2	20.4	-4.08
5	-2.2	5.4	-11.88

4. Calculate the Sum of Products of Deviations from the Mean (Σ(X - X̄) * (Y - Ȳ)):     Σ(X - X̄) * (Y - Ȳ) = -12.48 + 20.32 + 107.52 - 4.08 - 11.88 = 89.36

5. Calculate the Variance of X (σ^2_x) and Variance of Y (σ^2_y):

• We won't calculate these variances for this hand-worked example, but they are typically required for most statistical software to compute the correlation coefficient. The formulas involve squaring the deviations from the mean and summing them, then dividing by n-1 (degrees of freedom).

6. Alternative Way to Find r (using the provided Σ(X - X̄) * (Y - Ȳ)):

While software typically uses variances, for a basic understanding, we can use the following formula with the calculated Σ(X - X̄) * (Y - Ȳ):

r = Σ(X - X̄) * (Y - Ȳ) / [ √(Σ(X - X̄)² * Σ(Y - Ȳ)²) ]

Note: You'll need to calculate Σ(X - X̄)² and Σ(Y - Ȳ)² as well (following steps similar to those for Σ(X - X̄) * (Y - Ȳ)) for this formula.

7. Interpretation:

Using statistical software or the alternative formula (step 6), you'll obtain the correlation coefficient (r). Analyze the result based on the properties of correlation mentioned earlier:

• A positive r value indicates a positive correlation between study hours and exam scores (as expected).
• The closer the absolute value of r is to 1, the stronger the positive linear relationship.

Remember: Correlation doesn't imply causation. Other factors might influence exam scores besides study hours. This example demonstrates the basic steps for calculating the correlation coefficient by hand.

Numerical Examples:

Simple Linear Regression Example:

Imagine you want to predict house prices (Y) based on their square footage (X). You collect data on several houses and perform a simple linear regression analysis. The resulting equation might look like:

Y = a + b * X

where:

• Y = Predicted house price
• a = Y-intercept (average price when square footage is 0 - likely not meaningful here)
• b = Slope (change in price per unit increase in square footage)
• X = Square footage of the house

Multiple Linear Regression Example:

Now, consider predicting house prices (Y) based on both square footage (X1) and the number of bedrooms (X2). You perform a multiple linear regression analysis. The resulting equation might be:

Y = a + b1 * X1 + b2 * X2

where:

• Y = Predicted house price
• a = Y-intercept
• b1 = Coefficient for square footage (impact on price)
• b2 = Coefficient for number of bedrooms (impact on price)
• X1 = Square footage of the house
• X2 = Number of bedrooms

These are simplified examples, and actual models may involve more complex calculations. However, they illustrate the core concept of regression analysis.

 

Feature                          Simple                 Multiple

 

Dependent Variable        One (Y)                 One (Y)

Independent Variables    One (X)                 Multiple (X1, X2, ..., Xn)

Model                              Straight Line          Hyperplane

Use Case Analysis         Impact of a 1         Impact of Multiple

Variable on Y        Variables control others

 

1. Numerical Example for Simple Regression

A civil engineering student is studying the relationship between the curing time of concrete (in days) and the compressive strength of the concrete (in megapascals, MPa). The student collects data from several concrete samples and wants to build a linear regression model to predict the compressive strength based on the curing time. The following data is given:

| Curing Time (days) | Compressive Strength (MPa)          |

|---------------------------|------------------------------------------------|

| 3                             | 20                                                     |

| 7                             | 35                                                     |

| 14                           | 50                                                     |

| 21                           | 65                                                     |

| 28                           | 80                                                     |

 

 Solution: 

 

1.  Calculate the regression coefficient (b1): 

   - We'll use the formula: \(b_1 = \frac{\sum{(X_i - \bar{X})(Y_i - \bar{Y})}}{\sum{(X_i - \bar{X})^2}}\)

   - First, calculate the means:

     - \(\bar{X} = \frac{3 + 7 + 14 + 21 + 28}{5} = 14\)

     - \(\bar{Y} = \frac{20 + 35 + 50 + 65 + 80}{5} = 50\)

   - Next, compute the numerator:

     - \(\sum{(X_i - \bar{X})(Y_i - \bar{Y})} = (3-14)(20-50) + (7-14)(35-50) + \ldots + (28-14)(80-50) = 1050\)

   - Compute the denominator:

     - \(\sum{(X_i - \bar{X})^2} = (3-14)^2 + (7-14)^2 + \ldots + (28-14)^2 = 140\)

   - Finally, calculate \(b_1\):

     - \(b_1 = \frac{1050}{140} = 7.5\)

2.  Regression equation: 

   - The regression equation is given by: \(Y = b_0 + b_1X\)

   - We need to find \(b_0\):

     - \(\bar{Y} = b_0 + b_1\bar{X}\)

     - \(50 = b_0 + 7.5 \cdot 14\)

     - \(b_0 = 50 - 105 = -55\)

   - The regression equation becomes: \(Y = -55 + 7.5X\)

3.  Estimate the compressive strength at curing time of 10 days: 

   - Plug in \(X = 10\) into the regression equation:

     - \(Y = -55 + 7.5 \cdot 10 = 20\)

Therefore, when the curing time is 10 days, the estimated compressive strength is approximately 20 MPa. 🏗️🔍

2. Numerical Example for Multiple Regression.

Let's evaluate the multiple regression equation for the given dataset. We have the following data:

|  Y     |   X1 |  X2  |

|-------|------|------|

| 140   | 60   | 22   |

| 155   | 62   | 25   |

| 159   | 67   | 24   |

| 179   | 70   | 20   |

| 192   | 71   | 15   |

| 200   | 72   | 14   |

| 212   | 75   | 14   |

| 215   | 78   | 11   |

 

To perform multiple linear regression, we'll follow these steps:

1.  Calculate the Regression Sums: 

   - Calculate the sum of squares for X1 and X2:

     - Σ(X1^2) = Σ(X1)^2 / n = (60^2 + 62^2 + ... + 78^2) / 8 = 263.875

     - Σ(X2^2) = Σ(X2)^2 / n = (22^2 + 25^2 + ... + 11^2) / 8 = 194.875

   - Calculate the cross-products:

     - Σ(X1Y) = Σ(X1 * Y) - (Σ(X1) * Σ(Y)) / n = (60*140 + 62*155 + ... + 78*215) - (555 * 1,452) / 8 = 1,162.5

     - Σ(X2Y) = Σ(X2 * Y) - (Σ(X2) * Σ(Y)) / n = (22*140 + 25*155 + ... + 11*215) - (145 * 1,452) / 8 = -953.5

     - Σ(X1X2) = Σ(X1 * X2) - (Σ(X1) * Σ(X2)) / n = (60*22 + 62*25 + ... + 78*11) - (555 * 145) / 8 = -200.375

 

2.  Calculate the Regression Coefficients: 

   - The formula to calculate b1 (regression coefficient for X1) is:

     - b1 = [(Σ(X2^2) * Σ(X1Y)) - (Σ(X1X2) * Σ(X2Y))] / [(Σ(X1^2) * Σ(X2^2)) - (Σ(X1X2))^2]

     - b1 = [(194.875 * 1,162.5) - (-200.375 * -953.5)] / [(263.875 * 194.875) - (-200.375)^2]

     - b1 ≈ 3.148

   - The formula to calculate b2 (regression coefficient for X2) is:

     - b2 = [(Σ(X1^2) * Σ(X2Y)) - (Σ(X1X2) * Σ(X1Y))] / [(Σ(X1^2) * Σ(X2^2)) - (Σ(X1X2))^2]

     - b2 = [(263.875 * -953.5) - (-200.375 * 1,162.5)] / [(263.875 * 194.875) - (-200.375)^2]

     - b2 ≈ -1.656

3.  Calculate the Intercept (b0): 

   - The intercept (b0) can be calculated as:

     - b0 = Y - b1*X1 - b2*X2

     - b0 = 181.5 - 3.148*69.375 - (-1.656)*18.125

     - b0 ≈ -6.867

4.  Estimated Linear Regression Equation: 

   - The estimated linear regression equation is:

     - ŷ = b0 + b1*X1 + b2*X2

     - In our example, it is:

       - ŷ = -6.867 + 3.148*X1 - 1.656*X2

5.  Interpretation: 

   - b0 = -6.867: When both X1 and X2 are zero, the mean value for Y is approximately -6.867.

   - b1 = 3.148: A one-unit increase in X1 is associated with a 3.148-unit increase in Y, assuming X2 is held constant.

   - b2 = -1.656: A one-unit increase in X2 is associated with a 1.656-unit decrease

 

3. Numerical Example for Fitting a bivariate Model

Fitting a Bivariate Model in Simple Linear Regression (Civil Engineering Example)

This example demonstrates how to fit a simple linear regression model by hand for civil engineering graduate students.

Scenario: We want to explore the relationship between concrete compressive strength (Y) (in MPa) and water-cement ratio (X) for concrete mixtures. A civil engineering graduate student has collected data from several concrete cylinder tests:

Sample	Water-Cement Ratio (X)	Concrete Compressive Strength (Y)
1	0.40	32.5
2	0.45	28.7
3	0.50	25.1
4	0.55	21.8
5	0.60	19.2

Steps:

1.    Calculate the Mean (Average) for X and Y:

• ΣX (sum of all X values) = 0.40 + 0.45 + 0.50 + 0.55 + 0.60 = 2.50
• Mean of X (X̄) = ΣX / n (number of samples) = 2.50 / 5 = 0.50
• ΣY (sum of all Y values) = 32.5 + 28.7 + 25.1 + 21.8 + 19.2 = 127.3
• Mean of Y (Ȳ) = ΣY / n = 127.3 / 5 = 25.46

2.    Calculate the Deviation from the Mean (X - X̄) and (Y - Ȳ) for each sample:

 

 

Sample	Water-Cement Ratio (X)	Concrete Compressive Strength (Y)	X - X̄	Y - Ȳ
1	0.40	32.5	-0.10	7.04
2	0.45	28.7	-0.05	3.24
3	0.50	25.1	0.00	-0.36
4	0.55	21.8	0.05	-3.66
5	0.60	19.2	0.10	-6.26

3.    Calculate the Product of Deviations from the Mean (X - X̄) * (Y - Ȳ):

Sample	X - X̄	Y - Ȳ	(X - X̄) * (Y - Ȳ)
1	-0.10	7.04	-0.704
2	-0.05	3.24	-0.162
3	0.00	-0.36	0.000
4	0.05	-3.66	-0.183
5	0.10	-6.26	-0.626

4.    Calculate the Sum of Products of Deviations from the Mean (Σ(X - X̄) * (Y - Ȳ)):

Σ(X - X̄) * (Y - Ȳ) = -0.704 - 0.162 + 0.000 - 0.183 - 0.626 = -1.675

Note: To calculate the slope (b) of the regression line, we would need the variance of X (σ^2_x). However, for a hand calculation, it's more efficient to use software tools that can directly calculate all the necessary statistics.

This example demonstrates the initial steps of fitting a simple linear regression model by hand.

Standard error of estimate (SEE), also known as standard error of regression, is a statistical measure used in regression analysis to estimate the average difference between the predicted values from the regression model and the actual values of the dependent variable (Y).

In simpler terms, it tells you how much the actual data points tend to deviate from the fitted regression line. A lower standard error of estimate indicates a better fit for the model, meaning the predicted values are closer to the actual values on average.

Here's the formula for the standard error of estimate:

SEE = √(Σ(Y - Ŷ)² / (n - 2))

Where:

• Σ (sigma) represents the sum of
• Y represents the actual value of the dependent variable
• Ŷ (Y hat) represents the predicted value of the dependent variable based on the regression model
• n is the total number of data points
• (n - 2) is the degrees of freedom for the regression model (number of data points minus the number of estimated parameters - typically 2 for slope and intercept in simple linear regression)

Understanding the Formula:

• Σ(Y - Ŷ)²: This calculates the squared deviations between the actual values (Y) and the predicted values (Ŷ) for all data points. Squaring ensures positive values regardless of the direction of the difference.
• (n - 2): This term accounts for the degrees of freedom. Since the regression model estimates the slope and intercept (2 parameters), we subtract 2 from the total number of data points (n) to get a more accurate estimate of the variability around the regression line.
• √: This takes the square root of the sum of squared deviations, converting it from squared units back to the original units of the dependent variable (Y).

Interpretation:

The standard error of estimate is interpreted in the same units as the dependent variable (Y). For example, if your Y variable is measured in meters and your SEE is 2 meters, it suggests that on average, the actual data points deviate from the fitted line by 2 meters.

A lower SEE indicates a more precise model, meaning the predicted values are closer to the actual values. However, it's important to consider the SEE in relation to the scale of the data. A small SEE compared to the range of Y values might be more meaningful than a larger SEE compared to a smaller range.

4. Standard Error of Estimate (SEE) Example in Civil Engineering (Hand-worked)

Scenario: A civil engineering student is studying the relationship between the thickness (X) of concrete slabs (in cm) and their deflection (Y) under a specific load (in mm). They collect data from several concrete slab tests and want to calculate the SEE of a fitted simple linear regression model.

Data (Example):

Slab Thickness (X - cm)	Deflection (Y - mm)
10	5.2
12	4.8
15	4.1
18	3.5
20	3.2

Steps:

1. We'll assume the student has already fitted a simple linear regression model and obtained the predicted deflection values (Ŷ) for each slab thickness.

2. Calculate the Squared Deviations from the Regression Line (Y - Ŷ)² for each sample:

Sample	Slab Thickness (X)	Deflection (Y)	Predicted Deflection (Ŷ)	Y - Ŷ	(Y - Ŷ)²
1	10	5.2	(Let's assume Ŷ1 = 4.5)	0.7	0.49
2	12	4.8	(Let's assume Ŷ2 = 4.2)	0.6	0.36
3	15	4.1	(Let's assume Ŷ3 = 3.8)	0.3	0.09
4	18	3.5	(Let's assume Ŷ4 = 3.4)	0.1	0.01
5	20	3.2	(Let's assume Ŷ5 = 3.1)	0.1	0.01

Note: You'll need the actual predicted deflection values (Ŷ) from your regression analysis to perform this step.

3. Calculate the Sum of Squared Deviations from the Regression Line (Σ(Y - Ŷ)²):

Σ(Y - Ŷ)² = 0.49 + 0.36 + 0.09 + 0.01 + 0.01 = 0.96

4. Calculate the Degrees of Freedom (df):

df = n - 2 (where n is the number of data points)

df = 5 - 2 = 3 (since we estimated two parameters - slope and intercept - in simple linear regression)

5. Calculate the Standard Error of Estimate (SEE):

SEE = √(Σ(Y - Ŷ)² / df)

SEE = √(0.96 / 3) ≈ 0.56 mm (rounded to two decimal places)

Interpretation:

In this example, the standard error of estimate (SEE) is approximately 0.56 mm. This suggests that, on average, the actual deflection values (Y) deviate from the fitted regression line by about 0.56 millimeters.

Note:

• A lower SEE indicates a better fit for the model, meaning the predicted deflection values are closer to the actual values in this case.
• It's important to consider the SEE in relation to the range of deflection values in the data. Here, a 0.56 mm deviation might be significant if the typical deflection range is small.

This hand-worked example demonstrates the calculation of SEE for a simple linear regression model in civil engineering. Remember to replace the assumed predicted deflection values (Ŷ) with the actual values obtained from your model for a more accurate calculation.