Using the Salt Stack Scheduler

Overview

SaltStack provides a built in scheduling system that allows you to run any functions you could run in a state file. Of particular interest in this article is using the scheduler to periodically run highstates on a Salt Minion and a couple of the management tools to list and remove those scheduled jobs.

Prerequisites

The first time I tried to get a scheduled job running I was tailing the /var/log/messages file and noted the following error coming from the minion:

Nov 27 09:13:50 hpc-node-01 salt-minion[1825883]: [ERROR   ] Missing python-croniter. Ignoring job hs.

A quick google search found the solution, install the “croniter” package using “pip”. I ran the following command AND restarted the salt-minion service:

# /opt/saltstack/salt/bin/pip install croniter
Collecting croniter
  Downloading croniter-5.0.1-py2.py3-none-any.whl.metadata (30 kB)
Requirement already satisfied: python-dateutil in /opt/saltstack/salt/lib/python3.10/site-packages (from croniter) (2.8.1)
Requirement already satisfied: pytz>2021.1 in /opt/saltstack/salt/lib/python3.10/site-packages (from croniter) (2022.1)
Requirement already satisfied: six>=1.5 in /opt/saltstack/salt/lib/python3.10/site-packages (from python-dateutil->croniter) (1.16.0)
Downloading croniter-5.0.1-py2.py3-none-any.whl (24 kB)
Installing collected packages: croniter
Successfully installed croniter-5.0.1
# systemctl restart salt-minion
#

Listing existing Jobs

With the correct packages installed on the salt minion I decided to list what jobs were already setup, the output confirmed that none were:

# salt 'hpc-node-01*' schedule.list
hpc-node-01.my-domain:
    schedule: {}
#

Adding a job

If I was to manually run a highstate I could run either of the two commands:

salt hpc-node-01* state.apply

OR

salt hpc-node-01* state.sls servers.hpc-node-01

The commands will execute the init.sls file located in /srv/salt/servers/hpc-node-01/, so all I need to schedule a job is to tell it the:

• module.function required
• location off the file
• cron style scheduling (for this example every 6 hours)

Initial testing via an SLS file failed, but using the CLI worked first go, here is the CLI command to add the job:

# salt 'hpc-node-01*' schedule.add cron-highstate function='state.sls' job_args='["servers.hpc-node-01"]' cron='0 0,6,12,18 * * *'

The Output:

hpc-node-01.my-domain:
    ----------
    changes:
        ----------
        hs:
            added
    comment:
        Added job: cron-highstate to schedule.
    result:
        True
#

Checking if the Job is present

Using the schedule.list command returns the following output confirming the job has been scheduled to run:

# salt 'hpc-node-01*' schedule.list
hpc-node-01.my-domain:
    schedule:
      cron-highstate:
        args:
        - servers.hpc-node-01
        cron: 0 0,6,12,18 * * *
        enabled: true
        function: state.sls
        jid_include: true
        maxrunning: 1
        name: cron-highstate
        saved: true
#

Removing the job can be achieved using the function schedule.delete <job-name>

#salt 'hpc-node-0*' schedule.delete cron-highstate

hpc-node-01.my-domain:
    ----------
    changes:
        ----------
        cron-highstate:
            removed
    comment:
        Deleted Job cron-highstate from schedule.
    result:
        True

Submitting a scheduler job via an SLS file

After some more experimentation with SLS files I have a simple format for the scheduler state file. The file contents is shown below, although its called “cron-highstate” it does what a highstate would do but just using a normal state file execution using state.sls:

#
# File is in /srv/salt/servers/hpc-node-01/scheduled-job.sls
#
cron-highstate:
  schedule.present:
    - function: state.sls
    - job_args:
      - 'servers.hpc-node-01'
    - cron: '30 6,12,18,0 * * *'

The output from executing the State file:

# salt 'hpc-node-01*' state.sls servers.hpc-node-01.scheduled-job
hpc-node-01.int.tri.edu.au:
----------
          ID: cron-highstate
    Function: schedule.present
      Result: True
     Comment: Adding new job cron-highstate to schedule
     Started: 14:29:52.905105
    Duration: 20.989 ms
     Changes:
              ----------
              cron-highstate:
                  added

Summary for hpc-node-01.int.tri.edu.au
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  20.989 ms
[root@salt hpc-node-01]#

Alternatives to CRON format

The scheduler module support several different way to nominate the time or frequency to run the required job. If you were going to schedule something to run every hour, you could use the format shown below:

cron-highstate:
  schedule.present:
    - function: state.sls
    - job_args:
      - 'servers.hpc-node-01'
    - minutes: 60

Note: You can achieve the same time span using ‘seconds: 3600’ or ‘hours: 1’

If a job needs to run until a specific date, then the “until” option could be used.

cron-highstate:
  schedule.present:
    - function: state.sls
    - until: '2025-06-31 23:59:59'
    - job_args:
      - 'servers.hpc-node-01.billing-rollover'

There is also an ‘after’ option with the same date format and a ‘once’ option.

More information on the various formats is available at the Salt Project home page: https://docs.saltproject.io/salt/user-guide/en/latest/topics/scheduler.html

-oOo-