Managing CRON via SaltStack

One of the key messages this guide aims to emphasize is “simplicity”. Hand in hand with “simplicity” is “Visibility”, if you can see something related to a State then you can get a better idea of the implications the State will have. One of the powerful features of the UNIX environment has been the CRON scheduling, running non-stop in the background of all Linux, BSD, HPUX, AIX environments.

Salt Stack provides a CRON module for managing CRONTAB entries very effectively. Like mot things in Salt Stack, managing the CRONTAB entries on a Salt Minon can be done in many ways:

1. A single file with all entries coded.
2. Pillar data with all entries coded as a dictionary/list structure.
3. A directory (hopefully called “cron”) with a single file per crontab entry.

Managing a single CRON entry

If you have a single CRON task (or maybe two) then a file called cron.sls with the basic code as shown below is ideal:

#
# CRON file configuration to cleanup /var/log/remote
# Retention 40 days, run at 23:57 every night.
#
Clean-up-var-log-remote-cron:
  cron.present:
    - name: '/usr/local/bin/log-cleanup.sh -r 40 -d /var/log/remote > /dev/null 2>&1'
    - user: root
    - minute: 57
    - hour: 23
    - daymonth: '*'
    - month: '*'
    - dayweek: '*'
    - identifier: 'CLEAN_UP_REMOTE_LOGS'
#
# End of File

The file layout provides high visibility and easy management of the task; however, it really doesn’t scale well when there are lots of entries!

Managing lots of CRON entries

If you have a dozen or more CRON entries to manage then you could use a Salt Pillar to hold the CRON data and a cron.sls State file that reads the Pillar data and applies it using Jinja templating. While an excellent use of templating and data management, the pillar concept also reduces immediate visibility of the data as you now need to look in multiple places. Here is the revised cron.sls file:

#
# Configure for multiple crontab entries
# See: /srv/pillar/hpc-metrics-01.sls for cron data.
#
{% for k,v in pillar.get('cron',{}).items() %}
{{ k }}:
  cron.present:
    - user: root
    - name: '{{ v['task'] }}'
    - minute: '{{ v['minute'] }}'
    - hour: '{{ v['hour']}}'
    - daymonth: '{{ v['dayofmonth'] }}'
    - month: '{{v['month']}}'
    - dayweek: '{{ v['dayofweek']}}'
    - identifier: '{{ v['identifier'] }}'
{% endfor %}
#
# End of File

And the pillar data looks like this:

server_name: hpc-metrics-01
fqdn: hpc-metrics-01.my.domain
webroot: /var/www/dashboard
cron:
  NEW_LOG:
    user: 'root'
    identifier: 'NEW_LOG'
    hour: '0'
    minute: '0'
    month: '*'
    dayofmonth: '*'
    dayofweek: '*'
    task: 'touch /data/self-heal/reactor-logs/reactor-`date "+%Y-%m-%d"`.log > /dev/null 2>&1'
… more entries here.

To reference the pillar data, you need to have an entry in the /srv/pillar/top.sls file. It’s formatted just like the entry in the /srv/salt/top.sls file that is used for “High Stating”.

Using loop indexes in a CRON

The following code snippet has been included to show a way to do something from A to Z in a loop, in this case, it was to call a shell script and pass in a directory tiebreaker, but it also uses the loops “index” value, in this case 1 to 7 to split the job over several hours to reduce system load on the Salt Minion:

{% for TB in 'ABCDEFG' %}
remove-empty-dirs-cron-{{TB}}-0:
  cron.present:
    - name: '/usr/local/bin/remove-empty-dirs.sh -t {{TB}} > /dev/null 2>&1'
    - user: root
    - minute: 0
    - hour: {{loop.index}}
    - daymonth: '*'
    - month: '*'
    - dayweek: '*'
    - identifier: 'REMOVE_EMPTY_DIRS_{{TB}}-0'
{% endfor %}

Managing CRON data using the “Super Simple” method

There is also another way to manage lots of CRON entries. If you consider that a single CRON entry is like a “role” and you keep the single role – single task idea, then a single State file that defines the CRON entry can be put into a directory dedicated to CRON entries. Then an “init.sls” file can be used to list the CRON entries like a shopping list.

This method allows for high visibility of what States are being applied, and you can also document and manage the removal of any CRON tasks. Keep in mind, once applied, a CRON entry remains forever until you explicitly remove it. The removal process is achieved using the “cron.absent” module. The following init.sls file located in /etc/salt/servers/hpc-metrics-01/cron/ defines a way to apply and remove CRON entries simply and with high visibility:

#
# Single file single CRON entry. Use cron.absent to remove old ones
#
include:
  - .cleanup-db-files
  - .cleanup-logs-dir
  - .cms-emails-student-list.sls
  - .cms-emails-new-inductions.sls
  - .cms-emails-exteneded-leave.sls
  - .cms-emails-ending-positions.sls
  - .cms-emails-document-reminders.sls
  - .cms-emails-departures.sls
  - .cms-accounts-departures.sls
#
# End of File

Removing entries

Removing entries from a cron file is done using the “identifier” field:

rm1:
  cron.absent:
    – identifier: PUT_IDENTIFIER_HERE
rm2:
  cron.absent:
    – identifier: PUT_NEXT_IDENTIFIER_HERE
#
# End of File

Crontab files

For Salt to manage the Salt Minion’s CRONTAB(s), it needs to use identifiers and structure the file in a clear way. Shown below is a CRONTAB managed entirely by SaltStack, you can still manually add entries. Best of all, your manually added entries will be retained when SaltStack makes changes.

# Lines below here are managed by Salt, do not edit
# SALT_CRON_IDENTIFIER:TRIM_REPO_BACKUP
27 23 * * * /usr/local/bin/log-cleanup.sh -r 7 -d /data/backups/repo  > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:TRIM_SALT_HOSTS_DATA
30 5 * * * /usr/local/bin/log-cleanup.sh -r 7 -d /data/salt/hosts  > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:SCRIPT-HOSTS-TO-JSON
0 1 * * * /usr/local/bin/minions-to-json-file.sh > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:SCRIPT-HIGHSTATE-PROD
15 */6 * * * /usr/local/bin/highstate-prod.sh > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:LOG_CLEANUP_highstate
58 23 * * * /usr/local/bin/log-cleanup.sh -r 7 -d /logs/highstate > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:LOG_CLEANUP_log-check
57 23 * * * /usr/local/bin/log-cleanup.sh -r 7 -d /logs/log-check > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:SCRIPT-COUNT-EVENTS
* * * * * /usr/local/bin/count-events.sh > /dev/null 2>&1
# SALT_CRON_IDENTIFIER:CLEAN_UP_SELF_HEAL
30 5 * * * /usr/local/bin/log-cleanup.sh -r 21 -d /logs/self-heal> /dev/null 2>&1

-oOo-