And to make things even more exciting, we’ll be using JSONLogger to save and load our progress so we don’t have to start from scratch every time we want to optimize something new!
First off, let’s import some libraries:
# Importing necessary libraries
from bayes_opt.logger import JSONLogger # Importing JSONLogger from bayes_opt.logger library
from bayes_opt.event import Events # Importing Events from bayes_opt.event library
import numpy as np # Importing numpy library and assigning it an alias "np"
import pandas as pd # Importing pandas library and assigning it an alias "pd"
import matplotlib.pyplot as plt # Importing pyplot from matplotlib library and assigning it an alias "plt"
%matplotlib inline # Magic command to display plots inline in Jupyter Notebook
# Note: The following code is used for optimization and will not be explained in detail as it is not relevant to the context of the script.
# Creating a JSONLogger object to save and load progress
logger = JSONLogger(path="./logs.json") # Assigning a path to save the progress in a JSON file
# Defining an optimization function
def black_box_function(x, y):
return -x ** 2 - (y - 1) ** 2 + 1 # Function to be optimized
# Setting the parameters for optimization
pbounds = {'x': (2, 4), 'y': (-3, 3)} # Defining the range of values for x and y
# Initializing the optimizer
optimizer = BayesianOptimization(
f=black_box_function, # Passing the optimization function
pbounds=pbounds, # Passing the parameter bounds
verbose=2, # Setting the verbosity level
random_state=1, # Setting a random state for reproducibility
)
# Setting up the logger to track the progress
optimizer.subscribe(Events.OPTMIZATION_STEP, logger) # Subscribing to the optimization step event and passing the logger object
# Running the optimization for 10 iterations
optimizer.maximize(
init_points=2, # Number of initial points to explore
n_iter=10, # Number of iterations to run the optimization
)
# Plotting the results
plt.plot(logger.res['x'], logger.res['y'], 'ro') # Plotting the values of x and y from the logger object
plt.xlabel('x') # Labeling the x-axis
plt.ylabel('y') # Labeling the y-axis
plt.show() # Displaying the plot
Now, let’s create a function that we want to optimize:
# Define a function named "my_function" that takes in two parameters, x1 and x2
def my_function(x1, x2):
# Perform some mathematical operations using the numpy library
# Calculate the sine of x1 and multiply it by the cosine of x2
# Add 0.5 to the result
# Multiply the result by the square of a random number between 0 and 0.5
# Return the final result
return np.sin(x1) * np.cos(x2) + 0.5 * (np.random.uniform(0, 0.5))**2
This function takes two inputs `x1` and `x2`, does some fancy math, and returns a value between 0 and 1. We’ll be using this as our objective function for BO.
Next, let’s create an instance of the BayesianOptimization class:
# Import the BayesianOptimization class from the bayes_opt library
from bayes_opt import BayesianOptimization
# Set up the optimization problem by defining the bounds for each input variable
# x1 has a lower bound of 0 and an upper bound of 5
# x2 has a lower bound of -3 and an upper bound of 3
bounds = {'x1': (0, 5), 'x2': (-3, 3)}
# Define the type of each input variable
# x1 and x2 are both of type float
types = {'x1': 'float', 'x2': 'float'}
# Define the objective function to be optimized
# The function takes in two inputs, x1 and x2, and returns a value between 0 and 1
def objective_function(x1, x2):
# Do some fancy math here
return value_between_0_and_1
# Create an instance of the BayesianOptimization class
# Pass in the objective function, bounds, and types as parameters
bo = BayesianOptimization(objective_function, bounds=bounds, types=types)
Here we’re setting up the optimization problem by defining our input variables (`x1` and `x2`) with their bounds (0-5 for `x1`, -3 to 3 for `x2`), and specifying that they are both floating point numbers. We also define our objective function as a lambda function that calls our `my_function`.
Now, let’s set up JSONLogger:
# Define input variables with bounds and specify data type
x1 = Variable(bounds=(0,5), vartype='float') # input variable x1 with bounds 0-5 and data type float
x2 = Variable(bounds=(-3,3), vartype='float') # input variable x2 with bounds -3 to 3 and data type float
# Define objective function as a lambda function calling my_function
objective = lambda x: my_function(x1=x[0], x2=x[1]) # objective function that calls my_function with input variables x1 and x2
# Set up JSONLogger to save progress to file
logger = JSONLogger(path="./logs.log") # create logger object with path where log will be saved
bo.subscribe(Events.OPTIMIZATION_STEP, logger) # subscribe to Events.OPTIMIZATION_STEP event and pass in logger object to save progress to file
Here we’re creating an instance of JSONLogger with a file path `./logs.log`, which will save the progress data as a JSON formatted string. We then use the `subscribe()` method to listen for the `Events.OPTIMIZATION_STEP` event, and pass in our logger object so that it can capture any updates during optimization.
Finally, let’s run BO:
# Import necessary libraries
import json
from bayes_opt import BayesianOptimization
from bayes_opt.event import Events
from bayes_opt.logger import JSONLogger
from bayes_opt.util import load_logs
# Define file path for logger to save progress data
log_path = './logs.log'
# Create a JSONLogger object to save progress data in JSON format
logger = JSONLogger(path=log_path)
# Subscribe to the Events.OPTIMIZATION_STEP event and pass in the logger object
bo.subscribe(Events.OPTIMIZATION_STEP, logger)
# Run Bayesian Optimization with specified number of initial points and iterations
bo.maximize(init_points=2, n_iter=3)
# Load saved progress data from logger
load_logs(bo, logs=[log_path])
# Print the best parameters and corresponding objective value
print("Best parameters:", bo.max['params'])
print("Best objective value:", bo.max['target'])
Here we’re calling `maximize()` on our BO object to start the optimization process. We pass in two arguments: `init_points`, which specifies how many initial points should be sampled, and `n_iter`, which specifies how many iterations of optimization should occur.
And that’s it! Our BO algorithm will now run with JSONLogger saving progress to the specified file path. If you want to keep working with the same logger object (i.e., not overwrite previous data), set `reset` parameter in `JSONLogger` to False:
# This script is using the Bayesian Optimization (BO) algorithm to optimize a function.
# The BO algorithm is an iterative process that uses a surrogate model to approximate the objective function and then uses this model to suggest the next point to evaluate.
# The number of iterations of optimization is specified by the user.
# Import necessary libraries
import GPyOpt
from GPyOpt.methods import BayesianOptimization
from GPyOpt.util.general import samples_multidimensional_uniform
# Define the objective function to be optimized
def objective_function(x):
return x**2
# Define the bounds of the search space
bounds = [{'name': 'x', 'type': 'continuous', 'domain': (-5,5)}]
# Generate initial samples for the surrogate model
initial_samples = samples_multidimensional_uniform(bounds, 5)
# Create the BO object with the objective function, bounds, and initial samples
bo = BayesianOptimization(f=objective_function, domain=bounds, X=initial_samples)
# Define the logger object to save progress to a specified file path
logger = GPyOpt.util.general.JSONLogger(path="./logs.log", reset=False) # save progress without overwriting previous data
# Subscribe to the OPTIMIZATION_STEP event and pass in the logger object
bo.subscribe(GPyOpt.util.general.Events.OPTIMIZATION_STEP, logger)
# Run the BO algorithm for the specified number of iterations
bo.run_optimization(max_iter=10)
# Print the optimal value and corresponding input
print("Optimal value:", bo.fx_opt)
print("Optimal input:", bo.x_opt)
# To continue working with the same logger object and not overwrite previous data, set the reset parameter to False in the JSONLogger function.
If you want to load previously saved progress data into a new instance of BO:
# Import necessary libraries
from bayes_opt import BayesianOptimization # import BayesianOptimization class from bayes_opt library
from bayes_opt.event import Events # import Events class from bayes_opt.event library
from bayes_opt.logger import JSONLogger # import JSONLogger class from bayes_opt.logger library
import pandas as pd # import pandas library and alias it as pd
import numpy as np # import numpy library and alias it as np
# Load previous logs from file path
file_path = "./logs.log" # path where the log was saved
data = pd.read_json(file_path) # read data as a pandas DataFrame
X, y = zip(*[(x['params'], x['target']) for _, x in data.iterrows()]) # extract input and output values from DataFrame
# Create new instance of BayesianOptimization with previously saved progress data
bo_new = BayesianOptimization(f=my_function, p=p, domain=domain) # create new instance of BayesianOptimization class with specified parameters
bo_new.set_defaults(X0=np.array([x for x in X[:1]]), y0=y[:1]) # set initial points and targets using previously saved data
# Subscribe to Events.OPTIMIZATION_STEP event with JSONLogger object
logger = JSONLogger(path="./logs2.log") # create new instance of JSONLogger class with specified path
bo_new.subscribe(Events.OPTIMIZATION_STEP, logger) # subscribe to Events.OPTIMIZATION_STEP event and pass in our logger object
# Run BayesianOptimization with new instance of BO
bo_new.maximize() # run optimization using the new instance of BO
Here we’re loading previously saved progress data from a file path `./logs.log`, extracting input and output values, creating a new instance of BO with those initial points and targets, subscribing to Events.OPTIMIZATION_STEP event using JSONLogger, and running optimization on the new instance of BO.
And that’s it! With Bayesian Optimization and JSONLogger, you can easily optimize your machine learning models without breaking a sweat (or your computer).