Using the Salt Stack Scheduler

Overview

SaltStack provides a built in scheduling system that allows you to run any functions you could run in a state file. Of particular interest in this article is using the scheduler to periodically run highstates on a Salt Minion and a couple of the management tools to list and remove those scheduled jobs.

Prerequisites

The first time I tried to get a scheduled job running I was tailing the /var/log/messages file and noted the following error coming from the minion:

Nov 27 09:13:50 hpc-node-01 salt-minion[1825883]: [ERROR   ] Missing python-croniter. Ignoring job hs.

A quick google search found the solution, install the “croniter” package using “pip”. I ran the following command AND restarted the salt-minion service:

# /opt/saltstack/salt/bin/pip install croniter
Collecting croniter
  Downloading croniter-5.0.1-py2.py3-none-any.whl.metadata (30 kB)
Requirement already satisfied: python-dateutil in /opt/saltstack/salt/lib/python3.10/site-packages (from croniter) (2.8.1)
Requirement already satisfied: pytz>2021.1 in /opt/saltstack/salt/lib/python3.10/site-packages (from croniter) (2022.1)
Requirement already satisfied: six>=1.5 in /opt/saltstack/salt/lib/python3.10/site-packages (from python-dateutil->croniter) (1.16.0)
Downloading croniter-5.0.1-py2.py3-none-any.whl (24 kB)
Installing collected packages: croniter
Successfully installed croniter-5.0.1
# systemctl restart salt-minion
#

Listing existing Jobs

With the correct packages installed on the salt minion I decided to list what jobs were already setup, the output confirmed that none were:

# salt 'hpc-node-01*' schedule.list
hpc-node-01.my-domain:
    schedule: {}
#

Adding a job

If I was to manually run a highstate I could run either of the two commands:

salt hpc-node-01* state.apply

OR

salt hpc-node-01* state.sls servers.hpc-node-01

The commands will execute the init.sls file located in /srv/salt/servers/hpc-node-01/, so all I need to schedule a job is to tell it the:

• module.function required
• location off the file
• cron style scheduling (for this example every 6 hours)

Initial testing via an SLS file failed, but using the CLI worked first go, here is the CLI command to add the job:

# salt 'hpc-node-01*' schedule.add cron-highstate function='state.sls' job_args='["servers.hpc-node-01"]' cron='0 0,6,12,18 * * *'

The Output:

hpc-node-01.my-domain:
    ----------
    changes:
        ----------
        hs:
            added
    comment:
        Added job: cron-highstate to schedule.
    result:
        True
#

Checking if the Job is present

Using the schedule.list command returns the following output confirming the job has been scheduled to run:

# salt 'hpc-node-01*' schedule.list
hpc-node-01.my-domain:
    schedule:
      cron-highstate:
        args:
        - servers.hpc-node-01
        cron: 0 0,6,12,18 * * *
        enabled: true
        function: state.sls
        jid_include: true
        maxrunning: 1
        name: cron-highstate
        saved: true
#

Removing the job can be achieved using the function schedule.delete <job-name>

#salt 'hpc-node-0*' schedule.delete cron-highstate

hpc-node-01.my-domain:
    ----------
    changes:
        ----------
        cron-highstate:
            removed
    comment:
        Deleted Job cron-highstate from schedule.
    result:
        True

Submitting a scheduler job via an SLS file

After some more experimentation with SLS files I have a simple format for the scheduler state file. The file contents is shown below, although its called “cron-highstate” it does what a highstate would do but just using a normal state file execution using state.sls:

#
# File is in /srv/salt/servers/hpc-node-01/scheduled-job.sls
#
cron-highstate:
  schedule.present:
    - function: state.sls
    - job_args:
      - 'servers.hpc-node-01'
    - cron: '30 6,12,18,0 * * *'

The output from executing the State file:

# salt 'hpc-node-01*' state.sls servers.hpc-node-01.scheduled-job
hpc-node-01.int.tri.edu.au:
----------
          ID: cron-highstate
    Function: schedule.present
      Result: True
     Comment: Adding new job cron-highstate to schedule
     Started: 14:29:52.905105
    Duration: 20.989 ms
     Changes:
              ----------
              cron-highstate:
                  added

Summary for hpc-node-01.int.tri.edu.au
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  20.989 ms
[root@salt hpc-node-01]#

-oOo-