Here’s how it works:
1. You set up two servers, one as your primary Salt Master and one as your backup. Let’s call them “SaltMasterA” and “SaltMasterB”.
2. On each minion (the little worker bees that do the actual salt stuff), you tell them to connect to both Salt Masters using a special configuration file called “minion_id”. This tells the minions which servers they should talk to, in case one goes down.
3. When SaltMasterA is up and running, all of your minions will connect to it normally. But if something happens (like an alien invasion or a power outage), SaltMasterB will take over automatically!
4. To make this happen, you need to set up some failover logic in your salt configuration files. This tells the system which server should be used as the primary and which one is the backup. Here’s what it might look like:
# /etc/salt/minion_id on each minion
# This file specifies the master server(s) for each minion to connect to.
# The 'master' key is used to specify a list of master servers, in order of preference.
# If the first server is unavailable, the minion will try to connect to the next one in the list.
# In case of a failover event (e.g. alien invasion or power outage), SaltMasterB will automatically take over as the primary server.
# To set up failover logic, we need to specify which server is the primary and which one is the backup.
# This is done by adding annotations to the 'master' key, indicating the order of preference.
# Here is an example of how it might look like:
# /etc/salt/minion_id on each minion
master:
- SaltMasterA # This is the primary server
- SaltMasterB # This is the backup server
This tells your minions to connect to both Salt Masters, but they will prefer “SaltMasterA” by default. If that server goes down for some reason (like a zombie apocalypse), the minion will automatically switch over to “SaltMasterB”.
5. To make sure everything is working properly, you can test your failover configuration using the salt-call command:
bash
# This script is used to test the failover configuration for Salt minions.
# It will run the state.highstate command on each minion with the debug flag enabled.
# Run this on each minion
sudo salt '*' state.highstate -l debug
# The '*' wildcard will run the command on all minions.
# The state.highstate command will apply all available states on the minion.
# The -l debug flag will enable debug logging for more detailed information.
# This command will ensure that the minion is properly configured to failover to a backup SaltMaster server.
# It is important to test this configuration to ensure that the minion will switch over in case of a server failure.
# This can be done by manually shutting down the primary SaltMaster server and checking if the minion switches to the backup server.
# The salt-call command is used to run Salt commands locally on the minion.
# In this case, we are using it to test the failover configuration.
salt-call state.highstate -l debug
# The -l debug flag is used to enable debug logging for more detailed information.
# The state.highstate command will apply all available states on the minion.
# Since we are running this command locally on the minion, we do not need to use the '*' wildcard.
This will run a highstate (which basically means “apply all of our configurations”) and log any errors or warnings to your terminal. If everything is working properly, you should see output like this:
# This script runs a highstate on multiple minions and logs any errors or warnings to the terminal.
# Run highstate on minion1 and log any errors or warnings with debug level
salt 'minion1' state.highstate -l debug
# The -l flag sets the log level to debug, allowing for more detailed output.
# Run highstate on minion2 and log any errors or warnings with debug level
salt 'minion2' state.highstate -l debug
# The -l flag sets the log level to debug, allowing for more detailed output.
# Example output from salt-call on each minion
# The following lines are example output from the highstate execution on each minion.
# Execute highstate function on SaltMasterA
2019-08-30 15:47:46,352 [DEBUG ] salt.states: (SaltMasterA) Executing function: highstate
# The debug level output shows that the highstate function is being executed on SaltMasterA.
# Connect to minion1 via SaltMasterA
2019-08-30 15:47:46,352 [INFO ] salt.client: (SaltMasterA) Connected to minion1 via master 'SaltMasterA'
# The info level output shows that the script has successfully connected to minion1 through SaltMasterA.
# Execute highstate function on SaltMasterB
2019-08-30 15:47:46,352 [DEBUG ] salt.states: (SaltMasterB) Executing function: highstate
# The debug level output shows that the highstate function is being executed on SaltMasterB.
# Connect to minion2 via SaltMasterB
2019-08-30 15:47:46,352 [INFO ] salt.client: (SaltMasterB) Connected to minion2 via master 'SaltMasterB'
# The info level output shows that the script has successfully connected to minion2 through SaltMasterB.
As you can see, each minion is connecting to its preferred Salt Master (“SaltMasterA” for “minion1”, and “SaltMasterB” for “minion2”). But if something happens (like a zombie apocalypse), they will automatically switch over to the backup server!
That’s it, With this failback configuration in place, you can rest easy knowing that your salt master is always up and running, no matter what kind of crazy stuff goes down.