Autosave for IOCs#

Introduction#

Autosave saves the state of an IOC to disk and restores it on restarts. epics-containers manages autosave configuration in a prescriptive fashion that is discussed here.

The official autosave documentation in here.

Different facilities have used different approaches to configuring autosave. epics-containers has settled on using .req files. These files simply list the set of PVs to save with autosave. The naming convention for these files is the same as that which the AreaDetector modules have adopted. EPICS Db info tags were also considered but rejected because supplying overrides to them would be problematic.

Because of this, AreaDetector Support modules can use autosave in epics-containers without any additional configuration (unless you disagree with the default choice of saved PVs). Many other support modules have adopted the same approach and will just work.

Other support modules can have their autosave configuration specified in ibek-support or in individual IOC instances.

Configuring autosave for IOCs#

To use autosave, the generic IOC being used must have compiled in the autosave module. The following snippet in the Dockerfile will do this:

COPY ibek-support/autosave/ autosave
RUN ansible.sh autosave

To enable autosave in an IOC instance that uses the above generic IOC requires adding the following to the ioc.yaml instance configuration (our examples are for the motion IOC BL45P-MO-IOC-01):

  - type: autosave.Autosave
    P: BL47P-MO-IOC-01      # a prefix for all autosave monitoring PVs

Startup Script#

The startup script entries that handle autosave will look the similar for all IOC instances as follows.

First in pre init stage we have:

# Autosave pre iocInit
set_requestfile_path("/epics", "autosave")
set_savefile_path("/autosave")
save_restoreSet_status_prefix BL47P-MO-IOC-01
save_restoreSet_Debug 0
save_restoreSet_NumSeqFiles 3
save_restoreSet_SeqPeriodInSeconds 600
save_restoreSet_DatedBackupFiles 1
save_restoreSet_IncompleteSetsOk 1
set_pass0_restoreFile autosave_positions.sav
set_pass1_restoreFile autosave_settings.sav
asSetFilename $(PVLOGGING)/src/access.acf

save_restoreSet values can be adjusted with arguments in the yaml but the pass0 and pass1 save files are fixed for epics-containers. These files will be saved into /autosave inside the IOC instance’s container filesystem. Along side them will be the sequence files and dated backup files.

/autosave will be mounted into the namespace’s autosave Persistent Volume Claim with a subfolder named after the IOC instance. For our BL47P-MO-IOC-01 example we have:

  • namespace: p47-beamline

  • PVC name: p47-autosave-claim

  • subPath in the PVC: bl47p-mo-ioc-01

Hence a pod that mounts the root of the PVC p47-autosave-claim will be able to browse the autosave files of all IOC instances in the namespace.

After iocInit we have

# Autosave post iocInit
create_monitor_set autosave_positions.req, 5, ""
create_monitor_set autosave_settings.req, 30, ""

The two req files list the PVs and have fixed names in epics-containers:

  • autosave_positions.req: lists PVS to be restored at phase 0 before record processing starts (usually motor positions)

  • autosave_settings.req: lists PVS to be restored at phase 1 after record processing starts

These files are generated by ibek from the req files in the support modules included in the generic IOC but can be also overridden on an instance basis.

Sources of req files#

There are 3 places ibek sources autosave req files from:

  • The built support module may have req files in its Db database template folder alongside the Database templates

  • The ibek-support subfolder for the support module, these may override the above

  • The config folder for the ioc instance, these override both of the above

In all cases the filenames must follow a strict pattern to identify the template from which the referenced PVs are being sourced. (this is the Areadetector approach). Overriding is simply a matter of creating the same file name in ibek-support or instance config.

Pattern: <template_stem_name>_{positions|settings}.req

e.g.

  • template: basic_asyn_motor.template

  • phase 0 .req file: basic_asyn_motor_positions.req

  • phase 1 .req file: basic_asyn_motor_settings.req

Build time behaviour#

The initial gathering of the .req files is done at container build time. Each time ansible.sh is executed in the Dockerfile it builds the given support module, but also symlinks all .req files from ibek-support/<module> to the \epics\autosave folder in the container.

These files are provided by the creator of the ibek-support/<support_module> module recipe. If the upstream support module includes req files that match the AreaDetector naming convention then there is no need to provide any .req files here.

If the support module has .req files with a different naming convention then one way to support this is to copy those files into the ibek-support/<support_module> folder and rename them appropriately. This is in keeping with the epics-containers philosophy of not requiring changes to the upstream support modules in order to support them. However, if the support module maintainer is open to adopting the defacto standard for .req files then that is the preferred approach.

Adding .req files to ibek-support gives us a way to supply autosave information that has historically been done out of band. DLS have a specific use for this as discussed in Diamond Light Source Autosave Approach.

Runtime behaviour#

All of the .req files supplied by support modules include the same macros as the templates that their PVs come from. Hence these need to be substituted with values that the individual IOC instance is using before passing to autosave.

In addition the multiple .req files from multiple support modules need to be gathered into a single file for each of the autosave phases. These are to be called autosave_positions.sav.req and autosave_settings.req and passed to the create_monitor_set function as we saw above.

Both file gathering and substitution are handled by ibek in the start.sh script that all epics-containers IOC instances use. The command that performs this step is:

ibek runtime generate-autosave

This performs the following steps:

  • symlinks all .req files found in the support modules of the generic IOC to \epics\autosave

    • if the file exists then don’t overwrite - allowing the ibek-support to provide overrides

  • symlinks all .req files found in \epics\ioc\config to \epics\autosave

    • overwriting any existing files thus providing instance overrides

  • generates two substitution files

    • /epics/runtime/autosave_positions.subst

    • /epics/runtime/autosave_settings.subst

  • runs MSI over the above substitution files, with the path pointing to \epics\autosave

The substitution files are copies of /epics/runtime/ioc.subst except that the EPICS Db template file names are replaced with autosave req file names using the naming convention described in Sources of req files.

The MSI output is two files autosave_positions.sav.req and autosave_settings.req which are in turn passed to create_monitor_set in the startup script.z

Diamond Light Source Autosave Approach#

At DLS we generate our .req files from comments added to the template files. This has had the unfortunate side effect of making DLS forks of support modules deviate from upstream. This does not fit well with the epics-containers philosophy of using upstream support modules without modification.

For this reason we supply a tool to extract the comment information from the internal DLS fork of the support modules and generate the .req files.

This tool should be used when:

  • the upstream support module does not provide .req files in the AreaDetector style

  • the DLS fork of the support module has autosave comments in the template files

  • the module is DLS internal or is public but originated from DLS

The tool is called builder2ibek and can be installed from PyPi.

The following example was used to extract the autosave information from the pmac module:

cd /dls_sw/prod/R3.14.12.7/support/pmac/2-5-22
builder2ibek autosave db/* --out-folder /scratch/hgv27681/work/ioc-pmac/ibek-support/pmac