Managing LVM Storage

For most server environments you are going to have a disk for the OS and possibly a disk for additional data. And in most cases, you would define the layout of the disks in a kickstart file using Logical Volume Manager (LVM). LVM is a robust mechanism to achieve this and is fully supported in a kickstart file. This results in a system that after booting has all disks built and with mounted filesystems available for use.

However, there is no reason why you wouldn’t consider automating this via a state file and using the LVM module to manage additional disks, leaving the basic OS build to the Anaconda Kickstart process and additional disks to a system that can be changed over time and managed accordingly.

Automating the creation of Volume Groups (VG), the management of attached Physical Volumes (PV) and the crafting Logical Volumes (LV) using a State file is an ideal way to manage LVM based storage. It should be noted that almost all performance applications may need their storage to be crafted so that I/O issues are addressed, especially for applications like MariaDB and striping logical volumes across multiple disks provide enhanced performance if done correctly.

The SaltStack LVM Module can manage LVM Physical Volumes, craft Volume Groups and setup Logical Volumes as the basic example below shows:

Example 1 – NEW Volume Group with one disk

# vg_data.sls
#
# Configure a new VG called vg_Data, with the /dev/sdb as
# the first disk, then force LV creation.
#
vg_data:
  lvm.vg_present:
    - devices:
      - /dev/sdb
lv_tmp:
  lvm.lv_present:
    - vgname: vg_data
    - size: 250G
    - force: true
#
# End of file

The state file above comes from a live example with a path to the code being /srv/salt/servers/hpc-node-03/lvm/vg_data. The VG creation and LV management is contained on a per VG bases, but you could separate the VG part out and then have a file per LV. If there are a number of LV’s to create and manage then separating them out into directories makes for a more structured and manageable implementation.

The following is the output from the sample above:

Command to run:

salt hpc-node-03* state.sls servers.hpc-node-03.lvm.vg_data

Output:

hpc-node-03.my-domain.local:
----------
          ID: vg_data
    Function: lvm.vg_present
      Result: True
     Comment: Created Volume Group vg_data
     Started: 13:52:00.273400
    Duration: 304.769 ms
     Changes:
              ----------
              created:
                  ----------
                  Output from vgcreate:
                      Volume group "vg_data" successfully created
                  vg_data:
                      ----------
                      Actual Physical Volumes:
                          1
                      Allocated Physical Extents:
                          0
                      Current Logical Volumes:
                          0
                      Current Physical Volumes:
                          1
                      Free Physical Extents:
                          228920
                      Internal Volume Group Number:
                          -1
                      Maximum Logical Volume Size:
                          -1
                      Maximum Logical Volumes:
                          0
                      Maximum Physical Volumes:
                          0
                      Open Logical Volumes:
                          0
                      Physical Extent Size (kB):
                          4096
                      Total Physical Extents:
                          228920
                      UUID:
                          JotUIH-I9UL-wVp0-W26D-HGoX-yg69-EP77eb
                      Volume Group Access:
                          r/w
                      Volume Group Name:
                          vg_data
                      Volume Group Size (kB):
                          937656320
                      Volume Group Status:
                          772

Logged into the host, we can get additional information using the pvs, vgs and lvs command from the salt-minion’s CLI.

Output from “pvs”, “vgs” and “lvs” commands:

# pvs
  PV         VG      Fmt  Attr PSize    PFree
  /dev/sda3  vg00    lvm2 a--   882.21g  778.21g
  /dev/sdb   vg_data lvm2 a--  <894.22g <644.22g
#
# lvs
  LV      VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log…
  lv_data vg00    -wi-ao----  16.00g                                                 
  lv_root vg00    -wi-ao----  16.00g                                                 
  lv_sw   vg00    -wi-ao----  64.00g                                                 
  lv_var  vg00    -wi-ao----   8.00g                                                 
  lv_tmp  vg_data -wi-a----- 250.00g
#
# vgs
  VG      #PV #LV #SN Attr   VSize    VFree
  vg00      1   4   0 wz--n-  882.21g  778.21g
  vg_data   1   1   0 wz--n- <894.22g <644.22g

Example 2 multiple disks and stripes

In this example, three disks are defined as PV’s (as each is a single line operation with no options, there is no “:” after the function name):

/dev/sda:
  lvm.pv_present
/dev/sdb:
  lvm.pv_present
/dev/sdc:
  lvm.pv_present

Defining the VG and assigning disks can be implemented as shown below:

vg_data:
  lvm.vg_present:
    - devices:
      - /dev/sda
      - /dev/sdb
      - /dev/sdc

Defining an LV using the newly created VG is implemented as shown below, but now we stripe across the three physical disks in the VG, this spreads the I/O across three disk queues at the hardware level boosting performance:

lv_db_stripe:
  lvm.lv_present:
    - vgname: vg_data
    - size: 100G
- stripes: 3
- stripesize: 8K

Formatting & Mounting Storage

Using the previous example of creating LVM based storage, we can also use SaltStack to “mount” those Logical Volumes into the file system. However, there is a step in between, the LV is created but it can’t be mounted until it is formatted with your choice of file system.

Let’s assume you want to create a 4TB /scratch disk on a server. We already have our PV and VG from the previous example to work with. The following code shows how this is done and in the HPC implementation I have three nodes that are set up this way. An idea way to structure this is to create an LVM directory under the server, then a VG directory and the LV’s can be plain YAML based SLS files. You can then create an “init.sls” file at the VG directory level to create the VG, then invoke each LV creation script separately (just for clarity).

#
# /srv/salt/servers/hpc-node-03/LVM/vg_data/lv_scratch.sls
#
# Create 4TB lv_scratch using VG called vg_Data
# The module does not support 100%FREE size option that the CLI tool does.
# Then format the block dev with ext4 and mount /scratch
#
lv_scratch:
  lvm.lv_present:
    - vgname: vg_data
    - size: 4096G
    - force: true
  blockdev.formatted:
    - name: /dev/mapper/vg_data-lv_scratch
    - fs_type: ext4
/scratch:
  file.directory:
    - user: root
    - group: users
    - mode: '0775'
  mount.mounted:
    - device: /dev/mapper/vg_data-lv_scratch
    - fstype: ext4
    - persist: true
    - opts:
      - defaults
#
# End of file

The first time this is run the output from the run will show the formatting status:

          ID: lv_scratch
    Function: blockdev.formatted
        Name: /dev/mapper/vg_data-lv_scratch
      Result: True
     Comment: /dev/mapper/vg_data-lv_scratch has been formatted with ext4
     Started: 12:18:11.147788
    Duration: 914.016 ms
     Changes:
              ----------
              new:
                  ext4
              old:
                  ext4

Once the initial state run is completed, subsequent state runs will show:

Comment: /dev/mapper/vg_data-lv_scratch already formatted with ext4

As we used the persist option, the state run also created an entry in /etc/fstab for use:

# cat /etc/fstab|grep scratch
/dev/mapper/vg_data-lv_scratch    /scratch    ext4  defaults    0 0

If you are just mounting other types of disks rather than crafting LV’s etc then the salt.state.mount Module provides a number of functions to mount various types of devices and can manage the /etc/fstab file to so devices can be re-mounted automatically on a re-boot.

In the basic example shown below, SaltStack provides a simple mount function to create the mount point if needed, save the entry to /etc/fstab and mount the specified device.

/mnt/tmp-data:
  mount.mounted:
    - device: /dev/mapper/vg_data-lv_tmp
    - mkmnt: true
    - persist: true
  - opts:
    - defaults

If not specifically defined, the mount mounted function should be able to determine the file system type (fstype) option, but you can specify it just as you would in the /etc/fstab file.

Also to note is the device can be a UUID, but if you’re dynamically building a system from scratch you may not be able to work out the UUID easily.

Example 2 – Mounting an ISO

Mounting an ISO image can also be done via the mount module, the only difference being the options.

/mnt/ol94:
  mount.mounted:
    - device: /data/isos/OracleLinux-R9-U4-x86_64-dvd.iso
    - fstype: iso9660
    - opts: loop
    - dump: 0
    - pass_num: 0
    - persist: True
    - mkmnt: True

-oOo-