Skip to content

Latest commit

 

History

History
156 lines (92 loc) · 4.94 KB

3. Logistic Regression.md

File metadata and controls

156 lines (92 loc) · 4.94 KB

Logistic Regression

Logistic regression is a statistical method used to model and predict the probability of a binary outcome (e.g., success/failure, yes/no) based on one or more independent variables. Unlike linear regression, logistic regression predicts probabilities, which are then used to classify outcomes.

Key Concepts

  1. Binary Dependent Variable:
    The outcome variable ($Y$) has two possible values, typically coded as 0 and 1.

    • Example: 1 = Approved, 0 = Not Approved.

  2. Independent Variables:
    These can be continuous, categorical, or a mix of both.

    • Example: Income, age, education level.

  3. Logistic Function (Sigmoid Function):
    Converts the linear combination of inputs into probabilities between 0 and 1.

    $$P(Y=1 \mid X) = \frac{1}{1 + e^{-(\beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_kX_k)}}$$

  4. Log-Odds Transformation: The logistic model estimates the log-odds of the probability:

    $$\text{Log-Odds} = \ln\left(\frac{P}{1-P}\right) = \beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_kX_k$$

Assumptions of Logistic Regression

  1. Binary Outcome: The dependent variable is binary.
  2. Independence of Observations: Each observation is independent.
  3. Linearity in the Log-Odds: The relationship between the independent variables and the log-odds is linear.
  4. No Multicollinearity: Independent variables are not highly correlated.

Types of Logistic Regression

  1. Binary Logistic Regression:
    Predicts a binary outcome (e.g., pass/fail).

  2. Multinomial Logistic Regression:
    Predicts outcomes with more than two categories without an order (e.g., types of transportation).

  3. Ordinal Logistic Regression:
    Predicts outcomes with ordered categories (e.g., customer satisfaction levels: low, medium, high).

Steps in Logistic Regression

Step 1: Define the Model

Specify the dependent and independent variables.

Step 2: Fit the Model

Estimate the coefficients ($\beta_0, \beta_1, \dots$) using maximum likelihood estimation (MLE).

Step 3: Interpret Coefficients

  • Positive Coefficient: Increases the log-odds of $Y = 1$.
  • Negative Coefficient: Decreases the log-odds of $Y = 1$.

Step 4: Predict Probabilities

Convert the log-odds to probabilities using the logistic function.

Step 5: Evaluate the Model

Assess model performance using accuracy, precision, recall, and $AUC-ROC$ (Area Under the Curve - Receiver Operating Characteristic).

Example

Problem:

A bank wants to predict whether a loan applicant will default ($Y = 1$) or not ($Y = 0$) based on income ($X_1$) and credit score ($X_2$).

Dataset:

Income ($X_1$) Credit Score ($X_2$) Default ($Y$)
40,000 600 1
50,000 650 0
60,000 700 0
70,000 750 0
80,000 800 1

Solution:

  1. Logistic Model:

    $$P(Y=1 \mid X_1, X_2) = \frac{1}{1 + e^{-(\beta_0 + \beta_1X_1 + \beta_2X_2)}}$$

  2. Estimated Coefficients:

    $$\beta_0 = -10$$

    $$\beta_1 = 0.0001$$

    $$\beta_2 = 0.02$$

  3. Prediction for New Applicant: Applicant with income $X_1 = 65,000$ and credit score $X_2 = 720$:

    $$\text{Log-Odds} = -10 + 0.0001(65,000) + 0.02(720) = -10 + 6.5 + 14.4 = 10.9$$

    $$P(Y=1) = \frac{1}{1 + e^{-10.9}} \approx 0.999$$

    The applicant is very likely to default.

Model Evaluation Metrics

  1. Confusion Matrix:

    • True Positive (TP): Correctly predicted defaults.
    • False Positive (FP): Incorrectly predicted defaults.
    • True Negative (TN): Correctly predicted non-defaults.
    • False Negative (FN): Incorrectly predicted non-defaults.

  2. Accuracy:

    $$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{Total Observations}}$$

  3. Precision:

    $$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

  4. Recall:

    $$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

  5. AUC-ROC: Measures the model's ability to distinguish between classes. A higher value indicates better performance.

Applications of Logistic Regression

  1. Healthcare: Predicting the likelihood of a disease based on patient attributes.
  2. Finance: Determining loan defaults or creditworthiness.
  3. Marketing: Classifying customers as likely or unlikely to purchase.

Conclusion

Logistic regression is a powerful and widely used statistical tool for binary classification problems. By understanding its principles, assumptions, and evaluation metrics, you can use logistic regression to make accurate predictions and informed decisions.

Next Steps: ANOVA and MANOVA