Convert a builder XML IOC instance to ibek YAML#

builder2ibek xml2yaml converts a DLS XMLbuilder IOC definition file (.xml) into an ibek ioc.yaml that can be consumed by ibek at runtime to generate st.cmd and database files.

Prerequisites#

In order to test the converted IOC you should be working inside a devcontainer.

There are two options:

builder2ibek devcontainer — the devcontainer shipped with this repository. It has the full ibek-support and ibek-support-dls submodule trees checked out, so every supported module’s YAML is available. Run builder2ibek with uv run:

uv run builder2ibek xml2yaml ...

Generic IOC devcontainer — the devcontainer for a specific Generic IOC (e.g. ioc-bl11i-rga). Only the ibek support YAML that was built into that container image is available, so schema validation can only validate entities that the target IOC already supports. Here builder2ibek is not installed in the repo but can be run directly with uvx:

uvx builder2ibek xml2yaml ...

In both cases the ibek support YAML for every module referenced in the XML must be present — either in the builder2ibek submodules or in the Generic IOC’s installed support tree.


1. Locate the builder XML#

DLS XMLbuilder IOC definitions are in the builder support module e.g.

/dls_sw/work/R3.14.12.7/support/BL11I-BUILDER/etc/makeIocs/BL11I-CS-IOC-09.xml

Example — BL11I-CS-IOC-09.xml:

<?xml version="1.0" ?>
<components arch="linux-x86_64">
    <devIocStats.devIocStatsHelper ioc="BL11I-CS-IOC-09" name="CSIOC2"/>
    <autosave.Autosave bl="True" iocName="BL11I-CS-IOC-09"
        name="BL11I-CS-IOC-09.AS" path="/dls_sw/i11/epics/autosave" skip_1="True"/>
    <pvlogging.PvLogging/>
    <asyn.AsynIP name="rgaPort" port="10.111.5.1:5025"/>
    <hidenRGA.hidenRGA_qga BUFFER_SIZE="100" P="BL11I-EA-RGA-01"
        PORT="rgaPort" Q="" name="ENV.RGA"/>
</components>

Each XML element maps to an ibek entity type: <module.Class attribute="value"/>.


2. Run xml2yaml#

uv run builder2ibek xml2yaml BL11I-CS-IOC-09.xml --yaml bl11i-cs-ioc-09.yaml

BLGui entities (auto_BLGui_status, auto_BLGui_device) are dropped unless you pass --blgui. Pass it only when the generic IOC binary includes the BLGui module; it then emits BLGui.status and BLGui.device. builder2ibek reconvert passes --blgui itself for each IOC whose existing ioc.yaml contains BLGui.* entities, so it regenerates them from the XML. BLGui entities added to ioc.yaml by hand are not kept. If that file cannot be parsed it warns and reconverts without --blgui. There is no option to force this on or off.

For the example above this produces bl11i-cs-ioc-09.yaml:

# yaml-language-server: $schema=/epics/ibek-defs/ioc.schema.json

ioc_name: "{{  _global.get_env('IOC_NAME') }}"

description: auto-generated by https://github.com/epics-containers/builder2ibek

entities:

  - type: epics.EpicsEnvSet
    name: EPICS_TZ
    value: GMT0BST

  - type: epics.EpicsEnvSet
    name: STREAM_PROTOCOL_PATH
    value: /epics/runtime/protocol/

  - type: devIocStats.iocAdminSoft
    IOC: '{{ ioc_name | upper }}'

  - type: autosave.Autosave
    P: 'BL11I-CS-IOC-09:'
    debug: false

  - type: pvlogging.PvLogging
    access_file: /epics/support/pvlogging/src/access.acf

  - type: asyn.AsynIP
    name: rgaPort
    port: 10.111.5.1:5025

  - type: hidenRGA.hidenRGA_qga
    BUFFER_SIZE: 100
    P: BL11I-EA-RGA-01
    PORT: rgaPort

Notice that asyn.AsynIP and hidenRGA.hidenRGA_qga remain as separate entities in the output, reflecting the original XML structure. The ibek entity models for both must be present in the ibek-support* submodules used by the Generic IOC.

Runtime patterns#

A module with a runtime pattern in ibek-runtime-streamdevice or ibek-runtime-support needs no compiled support in the image: its files are vendored into the IOC instance. xml2yaml checks every IOC for such modules, against each library’s latest release tag. A module with a pattern in both libraries is an error. A module that also has a generic model in ibek-support* (the module: of a support YAML there) uses its pattern only once src/builder2ibek/patterns.py lists it as aligned with that model; until then it stays build-time. It also stays build-time when an entity leaves a parameter to the generic model’s default and the pattern defaults it differently: builder2ibek drops a value equal to the generic default, so the pattern would fill in its own. That rule needs the ibek-support submodules checked out, as they are when builder2ibek runs from its repository with uv run; without them xml2yaml warns once and uses every pattern. When the --yaml path is services/<ioc>/config/ioc.yaml in a services repo, xml2yaml adds each pattern with ibek pattern add <library>:<module>@<tag> services/<ioc>, which writes the pattern files under config/, runtime-lock.yaml, ioc.schema.json and the ioc.yaml schema header.

A beamline module, one named BL<nn><x> such as BL21I (not BL15I-BUILDER or I19Server), is never compiled into a generic IOC, so it comes only from its pattern in ibek-runtime-support, whatever generic model ibek-support-dls has for it. xml2yaml keeps each of its entities the pattern has a model for and accepts. It leaves out of the ioc.yaml every other entity of the module, every entity of a beamline module with no pattern (such as BL20J), and every entity that refers to the id of one left out. Each is a warning that names the IOC, the entity type, its name, P and R, and the reason. A comment at the top of the ioc.yaml, headed DROPPED FOR MANUAL REVIEW, lists each with its reason and its YAML as builder2ibek converted it, commented out, so that it can be found in the services repo, added to the pattern, and restored.

The beamline modules of ibek-runtime-support are routed to it, and so is detectorPlugins, whose gdaPlugins model takes the builder objects adUtil.gdaPlugins and gdaPlugins.gdaPlugins. Such an IOC needs a generic IOC image that builds every entity type the pattern’s sub-entities create (ADCore and ffmpegServer) and takes every argument they pass; otherwise detectorPlugins is reported as an unresolved module. Its other patterns are not used: an IOC that uses one keeps its entities with a warning. A builder repo such as BL15I-BUILDER is never a pattern: its templates are to be migrated into the beamline module (BL15I), so every one of its entities is left out and listed for review in the same way, with one warning per entity type. A kept entity leaves out each parameter whose value is its pattern model’s default. An entity is taken to refer to a dropped one only through an object parameter of its pattern or generic model.

  • Pattern-only IOC: every other entity is one the ioc-streamdevice-runtime image of the latest published GitHub release of ioc-streamdevice builds, as its published schema lists. Its values.yaml image is set to that image, in place of the template’s REPLACE_WITH_IMAGE_URI or an older ioc-streamdevice-runtime tag. When that schema cannot be fetched there is no pattern-only check: xml2yaml warns, leaves values.yaml alone, and adds every pattern.

  • Any other IOC keeps the image its values.yaml names. A pattern whose entity types that image’s published schema already has is installed by the image and is not added, except a beamline module’s pattern, which is always added: beamline modules are deployed per instance. When that schema cannot be fetched (for example an image on a GitLab registry ibek has no token for) every pattern is added, with a warning. When values.yaml still holds REPLACE_WITH_IMAGE_URI, every pattern is added with a warning: set the generic IOC image and re-run xml2yaml to drop the patterns that image already installs.

  • --image <ref> names the generic IOC image to plan against in place of the one values.yaml names. Patterns whose entity types its published schema has are left out, as for any other IOC, and the IOC is never moved to ioc-streamdevice-runtime. A values.yaml that still holds REPLACE_WITH_IMAGE_URI gets the image; one that names another image is left alone, with a warning. With any other --yaml path the patterns left out are only listed. For a GitLab generic IOC that beamline2yaml knows (ioc-dlslinuxvac), which publishes no schema, the entity types are those of the modules its Dockerfile builds with ansible.sh and of the patterns it installs with ibek pattern install, each at the library release the install pins, read from GitLab at the tag given (GITLAB_TOKEN). A pattern reference that names no pattern of its release is a warning, and adds no types.

After the patterns, xml2yaml reports each module the IOC uses as one of:

  • pattern: added from its runtime pattern, a beamline module’s always;

  • build-time: compiled into the image. A module whose every entity type the image’s schema has is build-time, a pattern module included (a note says the image installs it). With no image schema to go on, a module with a generic model in ibek-support* is build-time; a note gives the reason when its pattern was not used;

  • unresolved: nothing can run it, a warning with the reason. That is a beamline module with no pattern taken (one never compiled into a generic IOC), and, with an image schema, any module the image does not build and no pattern takes; with none, a module with no generic model whose pattern rejects its entities or that has no pattern.

Two notes list the pattern and build-time modules, and each unresolved module is a warning. beamline2yaml shows the same report per IOC, planned on the entities xml2yaml writes against the generic IOC image it picks, under modules: and in its --json output. It blocks an IOC with an unresolved module, with no generic IOC image, or that targets another OS: the IOC is reported, but no folder is written and a dev-c7 legacy folder is kept.

For a mixed IOC on a generic IOC image hosted on a GitLab registry, builder2ibek and ibek pattern add can fetch the image’s published schema only when that GitLab server is listed in IBEK_GITLAB_HOSTS, a comma-separated list of host[:port] entries (a bare host means HTTPS on port 443; in a GitLab CI job ibek also adds CI_SERVER_HOST itself), with a token in GITLAB_TOKEN for a private project. Without it every pattern is added and the ioc.yaml keeps the default schema header.

A pattern already in runtime-lock.yaml is left at its locked version; move it with ibek pattern update. Any other --yaml path only lists the patterns and the image. BUILDER2IBEK_PATTERN_TAG, BUILDER2IBEK_SUPPORT_PATTERN_TAG and BUILDER2IBEK_RUNTIME_TAG fix the ibek-runtime-streamdevice tag, the ibek-runtime-support tag and the ioc-streamdevice-runtime tag in place of the latest. Only the placeholder or an older ioc-streamdevice-runtime tag on the image: line is replaced; a newer tag someone pinned is kept, with a note. Any problem adding patterns (no network, no git, a schema that cannot be fetched, an unwritable file) is a warning, and the ioc.yaml is written, with one exception: an IOC with beamline entities fails to convert, and nothing is written, when ibek-runtime-support cannot be indexed. Each library is indexed on its own, so ibek-runtime-support failing costs no other IOC its ibek-runtime-streamdevice patterns, nor the reverse. A beamline module with a pattern in both libraries is a warning, and an IOC that uses it fails.


3. Review and adjust#

Note

If your IOC uses the older vacuumValve, interlock, or temperature modules, xml2yaml will automatically translate many entities to their dlsPLC equivalents. Some cases (e.g. vacuumValveRead2, pump templates) require manual fixup. See the dlsPLC migration reference for the complete mapping table and argument transformation rules.

Auto-conversion is a best-effort translation; review the output for:

  • Commented-out elements: builder XML comments (<!-- ... -->) are skipped. If the original had alternative configurations in comments, choose the right one now.

  • Default values: some converters inject sensible defaults (e.g. autosave debug: false). Verify these are appropriate.

  • Module coverage: if a module has no converter, its entities keep the XML attributes of the original, except that name, gda_name, gda_desc and gda_curr_name are removed and the values “True” and “False” become booleans. For many entities this is all that is needed, as we try to make the xxx.ibek.support.yaml as compatible with the original XML as possible.

  • Schema validation: open the YAML in VSCode with the Red Hat YAML extension; the $schema line at the top will highlight any type mismatches.

Important

You will need to update the schema when you have made changes — see 6. Updating the schema.

For schema validation to work your ioc.yaml should have the following header:

# yaml-language-server: $schema=/epics/ibek-defs/ioc.schema.json

4. Add to the test suite#

If your IOC is using support modules that are not currently covered in the builder2ibek tests then it could be worth updating the tests as follows.

To prevent regressions, copy the XML into tests/samples/ and regenerate:

cp BL11I-CS-IOC-09.xml tests/samples/
cd tests/samples && ./make_samples.sh

make_samples.sh converts all XML files in the folder and updates the corresponding .yaml files. Review the diff, then commit both. Run ./update-schema first if you changed any support YAML; see The golden-file test model for how the baselines work.


5. Verify with db-compare#

Once you have converted an IOC, the definitive verification is to generate the EPICS database inside the devcontainer and compare it record-by-record against the original builder _expanded.db. See Verify a converted IOC using db-compare for the full workflow. In brief:

# inside the devcontainer
ibek dev instance /workspaces/my-project/iocs/my-ioc
ibek runtime generate2 /epics/ioc/config
uv run builder2ibek db-compare \
    /dls_sw/.../MY-IOC_expanded.db \
    /epics/runtime/ioc.db

6. Updating the schema#

If you have added new entity models to ibek-support* or changed existing models, regenerate the global schema that VSCode uses for validation.

In the builder2ibek devcontainer:

./update-schema # first time only - after that use ibek to update schema as below

In a generic IOC devcontainer:

ibek ioc generate-schema > /epics/ibek-defs/ioc.schema.json

Both commands update the ioc instance schema in /epics/ibek-defs/ioc.schema.json#

Relationship between XML elements and ibek entity types#

XML element prefix

ibek entity type prefix

notes

devIocStats.devIocStatsHelper

devIocStats.iocAdminSoft

renamed by converter

autosave.Autosave

autosave.Autosave

converter adjusts path handling

pvlogging.PvLogging

pvlogging.PvLogging

direct mapping

asyn.AsynIP

asyn.AsynIP

direct mapping

hidenRGA.hidenRGA_qga

hidenRGA.hidenRGA_qga

direct mapping

EPICS_BASE.EpicsEnvSet

epics.EpicsEnvSet

renamed by converter

The converter modules in src/builder2ibek/converters/ handle the non-trivial mappings. There is one file per support module that needs more than the default. See src/builder2ibek/converters/ for the full list.