Today we’re going to talk about one of the most important yet often overlooked aspects of working with data: input normalization. Now, I know what you might be thinking “Who needs this fancy ‘normalization’ stuff? Just give me all the data and let me do my thing!” Well, hold your horses there, partner! Let’s begin exploring with why input normalization is crucial for any machine learning project and how to implement it in Python.
First what exactly is input normalization? In simple terms, it’s a process of transforming the values of our data inputs so that they have similar scales or ranges. This can help improve the performance of various algorithms by reducing the impact of outliers and making them more robust to noise. It also helps in preventing overfitting and underfitting issues during model training, which are common problems when dealing with real-world datasets.
Now that we’ve established what input normalization is, how to implement it in Python using the popular Scikit-Learn library. First, you need to install Scikit-Learn if you haven’t already done so:
# This line installs the Scikit-Learn library using the pip command
# The "install" command is used to install new packages or libraries
# "pip" is a package management system used to install and manage software packages written in Python
# "scikit-learn" is the name of the library being installed
pip install scikit-learn
Once that’s taken care of, let’s say we have a dataset with two features ‘age’ and ‘salary’. We want to normalize the ‘salary’ feature because it has a wide range of values (from $10k to $5 million) which can affect our model’s performance.
Here’s how you can do that:
# Import necessary libraries
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
# Load the dataset
df = pd.read_csv('data.csv')
# Extract the 'salary' feature and convert it to a NumPy array
salary = df['salary'].values # Changed 'Salary' to 'salary' to match the column name in the dataset
# Create an instance of the MinMaxScaler class from Scikit-Learn
scaler = MinMaxScaler() # Created an instance of the MinMaxScaler class to use for scaling the data
# Fit the scaler on our data (i.e., learn its parameters)
scaler.fit(salary) # Fit the scaler on the 'salary' feature to learn its parameters
# Transform our 'salary' feature using the learned parameters
salary_norm = scaler.transform(salary) # Transformed the 'salary' feature using the learned parameters and stored it in a new variable 'salary_norm'
That’s it! We now have a normalized version of our ‘Salary’ feature, which has been scaled to a range between 0 and 1 (assuming we didn’t set any custom min/max values). This can help improve the performance of various algorithms that are sensitive to input scaling.
But wait what if you want to normalize multiple features at once? Well, Scikit-Learn has got your back! You can use a pipeline object to chain together multiple preprocessing steps:
# Import necessary libraries
from sklearn.pipeline import Pipeline # Import Pipeline from sklearn.pipeline library
from sklearn.preprocessing import MinMaxScaler, StandardScaler # Import MinMaxScaler and StandardScaler from sklearn.preprocessing library
# Define the pipeline with two stages: MinMaxScaler for 'Salary' and StandardScaler for all other features
pipe = Pipeline([('scaler_sal', MinMaxScaler()), ('scaler_other', StandardScaler())]) # Create a pipeline object with two stages: scaler_sal and scaler_other
# Fit the pipeline on our data (i.e., learn its parameters)
pipe.fit(df[['Salary'] + df.drop(['Salary'], axis=1).columns]) # Fit the pipeline on the 'Salary' column and all other columns in the dataframe
# Transform our entire dataset using the learned parameters
X_norm = pipe.transform(df) # Transform the entire dataframe using the learned parameters from the pipeline
# The purpose of this script is to demonstrate how to use a pipeline object in Scikit-Learn to chain together multiple preprocessing steps, such as MinMaxScaler and StandardScaler. This can help improve the performance of various algorithms that are sensitive to input scaling. The pipeline object allows for easy application of the same preprocessing steps to multiple features at once.
And that’s it! We now have a normalized version of all features in our dataset, which can help improve the performance and robustness of various machine learning algorithms.