The ask was small: add a vacuum IOC instance for I15. Getting there meant changing support metadata in three repositories, patching three upstream module sources at build time, cutting two releases, stopping three times for a human to merge, and running the whole thing twice — because the first version generated cleanly and would have been dead on arrival. This is the chain as it actually ran.
Nothing here is unusual on its own. The cost comes from the depth: a fix in a support model has to climb three tiers before an IOC can use it, and the tier below that isn't ours to change at all.
Numbered because the order was forced, not chosen. Each step either uncovered the next problem or was blocked until the previous one landed.
builder2ibek xml2yaml turned 338 XMLbuilder elements into 335
ibek entities. Every entity type round-tripped. This part worked exactly as
intended and took seconds.
ibek runtime generate2 — the command that turns ioc.yaml plus
the support metadata into the IOC's database and startup script — returned 144
validation errors. NX102_femto had no entity model at all;
NX102_readBool was missing three macros; NX102_temperature
demanded a desc that 122 of 123 entities don't set;
ether_ip only had EtherIPInit; mks937bGaugeEGU
demanded four inputs it should have derived.
All 144 errors were one shape: the entity model in a support YAML did not match the
database template it wraps. In three variants — a template with no model at all
(NX102_femto, NX102_readInt, NX102_PID, and all
but one of ether_ip's); a model missing macros the template accepts
(NX102_readBool had no bit, zsv or
osv); and a model demanding parameters XMLbuilder derives or leaves blank.
The fix is the same in all three: hand-edit the *.ibek.support.yaml to add
the missing model, add the missing parameter, or give the over-strict one a
default: so it becomes optional. Small, mechanical edits — but
ether_ip and mks937b live in the public GitHub repo while
dlsPLC and vacuumSpace live in the DLS GitLab one, so identical
work was split by ownership across two review and merge processes.
vacuumSpace was passing XMLbuilder short-names (GAUGE1,
PIRG13) straight into database macros instead of device PV names.
Every gauge, img and pirg link in every vacuum space group pointed at a PV that
could never connect.
BL15I-BUILDER is a beamline repo whose entities the converter
silently dropped. It turned out to have no compiled code, so it became a
vendored runtime pattern rather than a built module — which meant a new pattern
folder, a PR, and a v0.2.0 tag in a third support repository before
the IOC could reference it.
Bump two submodule pins, commit. Refresh the vendored copy that CI uses because
GitHub Actions can't reach DLS GitLab. Regenerate every sample. Commit again.
The order is forced: the refresh tool refuses unless the pin is already committed,
and two guards read git ls-tree HEAD, so staging isn't enough.
And it happens twice over, because the same two support repos are pinned by two
consumers: builder2ibek, which converts the XML, and
ioc-dlslinuxvac, which builds the container that runs the result. Both have
to land on the same metadata. If they drift, the IOC is converted against one version of
the entity models and executed against another — and the symptom is a generate failure
in the cluster, not a diff anyone reviewed.
git push origin main rejected by the pre-receive hook. Push a branch,
open a merge request, switch to a browser, merge, come back. The automated chain
stops dead here and waits for a person.
Point ioc-dlslinuxvac at the two new submodule commits.
Straightforward — and, on its own, not enough.
ibek-support repos are submodules inside it. So
entity-model edits, module build fixes, conversion and generate2 can all be
exercised in a single shell against the real EPICS build. Publishing an image and driving
CI in two repositories is the last step, not a step per iteration — otherwise every one
of these fifteen steps would have cost a release cycle.
The image was two template versions behind, and 4.6.3 carries the only two things
that make a vendored pattern work at all: ibek runtime place-files in
start.sh, and the config folder on msi's include path.
ioc-template#83.
The same protected-branch stop as step 07. Merge the MR, then push tag
2026.8.1 to trigger the image build and the schema publish.
Building that image is what started everything below.
The generated IOC references mks937b and digitelSpc.
Neither was in the Dockerfile — six lines to add, one COPY and one
ansible.sh per module.
None were bugs in our code. All four are properties of the DLS module sources
that only bite outside the DLS build environment — and each was fixed
declaratively in that module's *.install.yml rather than by forking
the module, which is the pattern worth keeping.
| Module | Symptom | Cause and fix |
|---|---|---|
| digitelSpc | Device 'stream' refers to unknown record type 'scalcout' |
The Makefile lists stream.dbd before calcSupport.dbd.
Our StreamDevice is built with calc, so its stream.dbd declares a
scalcout device and dbdExpand needs the record type defined first. The DLS
production build never hit it. Fixed by reordering the two. |
| digitelSpc | No rule to make target … stExample-gui |
The repo ships only stExample-gui.eds; the script itself is generated
by the etc/makeIocs build, which containers disable. The SCRIPTS=
line is commented out. |
| digitelSpc mks937b |
ccppc: No such file or directory |
Both leave CROSS_COMPILER_TARGET_ARCHS = vxWorks-ppc604_long
uncommented, so make goes on to cross-build for a toolchain the container has never had.
Every sibling vacuum module leaves that line commented; these two are the outliers. |
| digitelSpc | example IOCs and EDM screens built | The etc and opi directories are commented out of the
directory lists, matching what mks937a/b already do. |
The real check on a converted IOC is builder2ibek db-compare against the
original expanded database from /dls_sw/prod. That needs msi
and the compiled support modules, which exist only in the Generic IOC's image — and,
while the converter itself was being rewritten, a working checkout of
builder2ibek and its test suite, which sits in a container with no EPICS
build.
.subst; the other expanded it with msi and compared. Four
unrelated build defects had to be fixed before the verification that mattered could
run at all.
1360 extra records, none missing, and every vacuum space linking at a
:GAUGEG / :IMGG / :IONPG / :PIRGG /
:VALVEG prefix instead of a real device. The space and
space_b entity models instantiated all five group templates
unconditionally. XMLbuilder does not — it counts the devices of each type and picks
one of three shapes.
| Devices of a type | What XMLbuilder creates |
|---|---|
| 0 | a dummy at <device>:XXXG; the space links to it |
| 1 | nothing — the space links straight at the device |
| >1 | a group at <device>:XXXG; the space links to it |
The fix went into the converter, not the instance: it now expands space /
space_b into the entities builder would have created — per-component
delays, the 8-slot padding, the gap-closing that 37 real spaces across
/dls_sw depend on — and emits a plain template. The support YAML lost 570
lines and gained 16. Groups went from 50 to 7, matching the original exactly, and all
50 space→device links match. Because the fix was in the converter, the instance was
reconverted rather than patched.
The third protected-branch stop: merge the module fixes, cut an image that contains
them, then set that tag in the instance's values.yaml and deploy.
Fifteen steps and eight repositories after the request.
Ranked by how much of the chain each one lengthens. The conversion itself isn't on the list — that part is fine.
| Friction | What it costs | Weight |
|---|---|---|
| A converted IOC can be wrong in a way nothing checks | Generation, validation and CI all passed on an IOC whose every vacuum space pointed at PVs that were never created. Only a comparison against XMLbuilder's own output found it, and that comparison is manual. | High |
| Entity-model gaps only surface at generate time | Every new IOC is a fresh discovery exercise. 15 dlsPLC templates still have no model, waiting for whoever needs them first. | High |
| Protected branches mid-chain | Three full stops that no amount of automation can cross. All are small merges of changes already reviewed in-session. | High |
| Failures that are silent until runtime | Missing place-files, ibek below its floor, IOC_VERSION unset, spaces linked at group prefixes — all build green and fail in the cluster. |
High |
| No single container can verify an IOC | The converter and the EPICS build live in different images, so the verification loop spans two containers and has to be driven across them by hand. | High |
| Support metadata lives in three places | Public repo, DLS repo, plus a vendored copy so CI can run without DLS network. Each needs its own commit, and a guard exists to catch them drifting. | Medium |
| The image's module list is hand-maintained | Nothing reconciles it against what the IOCs using that image reference, so a missing module is indistinguishable from a working image until an IOC needs it. | Medium |
| Module sources encode their original build environment | vxWorks cross-targets, EDM screen builds, dbd orderings that only work against one particular StreamDevice build. Each is a two-line patch, and each is found by a failed build, one at a time. | Medium |
| Template drift is invisible until you look | Nothing told us the image was two versions behind, or that the gap mattered. It surfaced from a passing remark about a pin. | Medium |
| Sample regeneration couples unrelated modules | A vacuum fix reruns every sample IOC, minutes at a time, and an unrelated upstream change can turn the suite red. | Medium |
| Upstream lands changes unvalidated against converted IOCs | A four-line addition to positioner made a required parameter no XMLbuilder IOC could supply, breaking six samples before it was fixed upstream. |
Medium |
| Release tags outlive their contents | An instance pinned to a tag cut before the modules it needs looks perfectly correct in review and fails in the cluster. | Medium |
The database the IOC loads was compared record by record against the one XMLbuilder produces for the same beamline.
DESC the original left
blank and 9 carry a fuller one, both on mks937a/b relay setpoints and neither related to spaces;
the remaining 2 are the space interlock enable records, which differ by a handful of states and
labels because the module is pinned at 4-9ec1 where the original was built at
4-10.
The IOC starts. It has not yet been run against the real hardware at I15 — but with parity at this level, connecting it to the beamline is expected to be uneventful.