By Northhaven Analytics Data Science Team
Introduction: Why Model Complexity Matters in Data Science
In the quest to build the perfect machine learning model, data scientists often fear overfitting—where a model learns the noise instead of the signal. However, there is an equal and opposite danger that is often overlooked but equally destructive: underfitting.
Underfitting in machine learning is a scenario where a model cannot capture the underlying trend of the data. It occurs when a model is too simple to learn the complex patterns in the training data. While an overfit model hallucinates patterns that don’t exist, an underfit model is blind to the patterns that do exist. It suffers from high bias and low variance, resulting in a system that performs poorly on both training data and unseen data.
For any practitioner of ML, understanding underfitting and overfitting is the key to mastering the bias and variance tradeoff. A model that is likely underfitting will fail to generalize well to new data, rendering it useless for production.
In this comprehensive guide, we will explore what underfitting is, why underfitting occurs, and how to detect underfitting using learning curves. We will analyze the causes of underfitting, compare underfitting vs overfitting, and provide actionable strategies to address underfitting in machine learning.
What is Underfitting? Understanding the High Bias Problem
Underfitting happens when a machine learning model is unable to capture the relationship between the input and output variables accurately. It generates a high error rate on both the training set and the test data.
Defining the Underfit Model
An underfit model is like a student who didn’t study for the exam and tries to guess the answers based on a few simple rules. The learning model is too simple to represent the reality.
- Underfitting occurs when a model assumes a static or linear relationship in complex, non-linear data.
- Underfitting occurs when a machine learning algorithm is too rigid to adapt to the data’s nuances.
The Signal vs. The Noise: High Bias and Low Variance
To understand underfitting, we must look at the error components.
- High Bias: The model makes strong assumptions about the data (e.g., assuming data is linear when it is curved). This leads to systematic errors.
- Low Variance: The model is not sensitive to fluctuations in the training data. If you retrain it on a different slice of data, it will likely produce the same wrong answer.
A model with high bias and low variance is the hallmark of underfitting. In contrast, overfitting is characterized by low bias but high variance. The goal of data science is to find the balance between overfitting and underfitting.
Why Underfitting Occurs: Common Causes of Underfitting
Underfitting occurs due to specific structural or procedural failures in the modeling process. It is rarely a data issue; it is usually a model issue.
1. The Model is Too Simple
The most common cause is that the model complexity is insufficient. For example, using a simple linear regression model to predict the trajectory of a rocket (which is non-linear) will result in high error. The linear model is structurally incapable of bending to fit the curve. The model is unable to capture the geometry of the data points.
2. Excessive Regularization
Techniques used to prevent overfitting, such as L1 or L2 regularization, work by constraining the model. They prevent overfitting by penalizing large coefficients. However, if the penalty is too high, you might over-constrain the system. This lead to underfitting, as the model is forced to be too simple to fit the training data.
3. Insufficient Feature Engineering
If the input features do not contain enough information to predict the target, the model fails to learn. Underfitting often happens when critical variables are missing or when the relationships in the data are too complex for the provided raw features.
How to Detect Underfitting: Signs and Indicators
You cannot fix what you cannot measure. To address underfitting, you must first identify it.
1. High Training Error
The primary sign of underfitting is poor performance on the training data. If your model struggles to make accurate predictions even on the data it was trained on, it is underfitting.
- Overfitting: Performs well on training data but poorly on test data.
- Underfitting: Performs poorly on training data but poorly on test data as well.
2. Analyzing Learning Curves
Learning curves are the best diagnostic tool.
- In an underfitting scenario, both the training error and validation error are high and plateau quickly.
- The gap between the training curve and validation curve is small (indicating low variance), but the error level is unacceptable (high bias).
3. Visualizing the Fit
For a simple model, plotting the prediction line against the data points often reveals the issue. If the data forms a parabola and your machine learning algorithm draws a straight line, the model fails to capture the underlying patterns in the data.
Underfitting vs. Overfitting: The Battle for Generalization
Understanding underfitting requires juxtaposing it with its nemesis. Underfitting is the opposite of underfitting (overfitting).
- Overfitting: The model is too complex. It memorizes the training data.
- Underfitting: The model is too simple. It ignores the patterns in the training data.
Both result in a model that fails to generalize. When you train the model, you are navigating a spectrum. You want a model complex enough to capture the signal but simple enough to ignore the noise. Techniques designed to prevent overfitting can inadvertently cause underfitting if applied aggressively.
Strategic Solutions: How to Address Underfitting
Once you have identified that your model may underfitting in machine learning scenarios, you have several levers to pull.
1. Increase Model Complexity
If your linear regression model is failing, switch to a polynomial regression or a decision tree. If a small neural network is underfitting, add more layers. You need a complex model that has the capacity to learn the non-linear data. Using a different model architecture is often the quickest fix.
2. Feature Engineering and Data Augmentation
Give the model more information. Create interaction terms, polynomial features, or domain-specific ratios. This helps the learning algorithm see patterns in the data that were previously hidden.
3. Reduce Regularization
If you are using Ridge or Lasso regression, decrease the regularization parameter (lambda). Allow the model more freedom to fit the data samples. Stop trying to prevent overfitting so aggressively.
4. Increase Training Time
In deep learning or gradient boosting, underfitting is typically observed in the early epochs. Simply training longer might resolve the high training error as the model learns more granular details.
Practical Examples of Underfitting in Finance
Underfitting in machine learning models deployed in finance can be disastrous.
- Credit Scoring: A simple model that only looks at income might underfit because it ignores data points like payment history or debt-to-income ratio. It fails to distinguish good borrowers from bad ones (high bias).
- Algorithmic Trading: A linear model attempting to predict stock prices will fail because financial markets are inherently non-linear and chaotic. The model is unable to adapt to market volatility.
Conclusion: Achieving the Right Fit
Underfitting is a common pitfall, especially when data scientists are overly cautious about overfitting. However, a model that has underfitting or overfitting issues is equally useless.
To build robust ML models, you must monitor model accuracy and error rates diligently. You must ensure your model architecture matches the complexity of the problem. Whether you use machine learning for fraud detection or price forecasting, the goal is to detect underfitting early and correct it by adding complexity or features.
Underfitting occurs when the model is not powerful enough. Don’t let your AI be weak. Empower it with the right data and the right structure.
Ready to optimize your ML pipeline? Explore how Northhaven Analytics builds balanced, high-fidelity synthetic data to train models that neither underfit nor overfit.
