Randomizing CRON start times in Saltstack

I’ve recently started using scheduled jobs to initiate highstates after building a SaltStack Engine than can track the progress of the jobs and I did it that way to de-couple the highstate from anything else. However the HPC stack I manage has a lot of nodes that would have started Highstating at the same time and this would have initiated a lot of I/O on the Salt Master’s filesystem. To lighten the load on the disks, I remembered writing a randomizer some 10 years earlier for a clustered storage system I managed.

By using a random value for the minute (and even the hour if you need to) the values in the “cron” attribute will allow the jobs to spread out and can be pushed to multiple Salt Minion safely knowing they will run at different times.

For a random value between 1 and 59 I used: {{ range(1,59) | random }} in the code sample below.

highstate:
 schedule.present:
 - function: highstate
 - name: '/usr/local/bin/my-script.sh > /dev/null 2>&1'
 - cron: '{{ range(1,59) | random }} */6 * * *'

To view what is already scheduled use:

salt <node> schedule.list

Running this against hpc-node-04 returned:

[root@salt hpc-node-04]# salt 'hpc-node-04*' state.sls servers.hpc-node-04.scheduled-job

hpc-node-04.my.domain:
           ID: highstate
Function: schedule.present
  Result: True
 Comment: Modifying job highstate in schedule
 Started: 08:13:58.834834
Duration: 27.654 ms
 Changes:
          ----------
          highstate:
              ----------
              new:
                  ----------
                  cron:
                      19 */6 * * *
                  enabled:
                      True
                  function:
                      state.highstate
                  jid_include:
                      True
                  maxrunning:
                      1
                  name:
                      highstate
              old:
                  ----------
                  cron:
                      25 3,9,15,21 * * *
                  enabled:
                      True
                  function:
                      state.highstate
                  jid_include:
                      True
                  maxrunning:
                      1
                  name:
                      highstate
Summary for hpc-node-04.int.tri.edu.au
Succeeded: 1 (changed=1)
Failed: 0
Total states run: 1
Total run time: 27.654 ms
[root@salt hpc-node-04]#

-oOo-