Edit and debug an application with hotfix#

In this tutorial you will open BlueAPI’s persistent source checkout in VS Code, make an edit, and restart BlueAPI to try it. In the same VS Code window, you will then use breakpoints to inspect the application: Launch starts it under the debugger, while Attach connects to the running process. Finally, you will undo the tutorial edit and return BlueAPI to normal operation.

Complete setup first and arrange an interruption window with the application owner: stopping the server or pausing at a breakpoint interrupts its service. The example uses BlueAPI on P47, with pod p47-blueapi-0 and container blueapi. Substitute your own pod, container and source repository when working on another application.

Check the checkout and lifecycle wiring#

On your workstation:

module load ec/p47
podbench status

If this is the first time the workload has been used for hotfix, follow Prepare a workload for hotfix first. It walks you through generating and deploying the chart changes, replacing the pod, and initializing its persistent source checkout. If someone has already prepared the workload, use the status output to see how far they got.

Look for p47-blueapi-0 in the POD column and blueapi in TARGET. After the chart changes have been deployed and the pod replaced, but before the checkout is initialized, its row looks like this:

POD             TARGET   SEAT  HOTFIX
p47-blueapi-0   blueapi  —     ready for init, normal

ready for init means you still need the Initialize an empty claim step in the preparation guide. Once you have completed that step, run podbench status again. You are looking for:

POD             TARGET   SEAT  HOTFIX
p47-blueapi-0   blueapi  —     initialized, normal

initialized means the persistent checkout has been created; normal means the application is running under the lifecycle supervisor. The SEAT column can be empty at this point—you will create or reconnect to a seat when you open VS Code.

If the row instead contains legacy wiring, follow the preparation guide to update the supervisor and roll out a replacement pod. An existing initialized checkout survives that rollout and does not need initializing again.

On later visits, initialized, normal means you can go straight to opening VS Code. Rerunning hotfix enable regenerates the local chart files and reports unchanged when they already match; it does not deploy changes or restart the pod. Rerunning hotfix init on an initialized claim is refused, preserving the existing checkout.

Open the checkout in VS Code#

On your workstation, run:

podbench ide vscode p47-blueapi-0

VS Code connects to a debug seat beside the application. Opening it leaves BlueAPI running. Open a terminal in that VS Code window and inspect the persistent checkout:

cd /podbench/app
git status --short
git diff

If both Git commands produce no output, the checkout has no tracked changes or untracked files. Otherwise, review the changes before adding your own: this checkout may contain someone else’s work and may be at a different revision from the container image. Keep those existing edits throughout the tutorial.

The commands below say whether to run them in this seat terminal or on your workstation. Keep both available.

Edit and run normally#

Open /podbench/app/src/blueapi/service/main.py. Find health_probe and add a logging line just before its return:

def health_probe() -> HealthProbeResponse:
    """If able to serve this, server is live and ready for requests."""
    LOGGER.info("Podbench hotfix tutorial: health probe reached")
    return HealthProbeResponse(status=Health.OK)

Save the file. In the seat terminal, restart BlueAPI to load the edited source:

podbench restart

This stops the application child, reruns its full startup command, and waits for health checks. It keeps the pod and your VS Code connection in place. The spinner may remain visible while BlueAPI initializes devices; wait for the command to finish before continuing.

Now request the health endpoint from the seat terminal:

curl -fsS http://127.0.0.1:8000/healthz

You should receive a successful health response. On your workstation, inspect the application log:

kubectl logs p47-blueapi-0 -c blueapi --tail=40

Look for a line containing:

Podbench hotfix tutorial: health probe reached

It may appear several times because Kubernetes also calls the health endpoint. Seeing this message confirms that BlueAPI is executing your edited checkout. If it is missing, check that you saved the file and that the restart succeeded before moving on.

This Python source edit needs no reinstall. If you change dependencies as well, follow restart and recover to reinstall them.

Launch under the debugger#

Use Launch when you want to debug startup or rerun the application after an edit. First, stop the running application from the seat terminal:

podbench stop

The command should finish with the application stopped. To check, run podbench status on your workstation. The HOTFIX column should now contain:

initialized, stopped, HELD

HELD means Podbench is keeping the hold-aware liveness check satisfied so Kubernetes does not restart the container while you debug. Readiness checks continue, so the pod may show as unready while the application is stopped.

In VS Code:

  1. Open Run and Debug and choose the Podbench: Launch — … entry whose command is the BlueAPI server. The text after the dash depends on how your deployment starts BlueAPI.

  2. Set a breakpoint on the logging line you added. To investigate startup, you can also set one on a line that runs while the server starts.

  3. Press F5 to launch. If Podbench reports that the application is already running, run podbench stop in the seat terminal and try again.

Launch runs the selected server command inside the application container, using its captured interpreter, environment and working directory. It runs that command directly; podbench start reruns the full parent startup script.

Once the server is listening, request the health endpoint from the seat terminal:

curl -fsS http://127.0.0.1:8000/healthz

VS Code should pause on your logging line. A Kubernetes health check may reach the breakpoint before your request does. Inspect Call Stack and Variables, then press F5 to continue. A request waiting on the breakpoint will remain pending until you resume execution.

To try an edit during this session, change the log message and save the file, then use VS Code’s Restart debugging control. It launches the server again with your saved changes; the workspace and seat stay connected.

When finished, choose Stop Debugging. Ending a Launch session leaves the application stopped and held, including if the launched process exits or its connection is lost. In the seat terminal, restore normal execution:

podbench start

Wait for startup to complete. Workstation podbench status should again show initialized, normal, with no HELD marker.

You can also run Podbench: Stop and Podbench: Start through VS Code’s Tasks: Run Task command. If the terminal commands are missing, see troubleshooting to update the seat’s Podbench package.

Attach to the running process#

Use Attach to inspect BlueAPI while it is already running. Wait for the previous section’s podbench start to complete before attaching.

The restarted server has a new PID. Each time you start a Podbench: Attach session, its preparation task finds the current process by matching the executable and command arguments in the same application container. It uses that process’s PID, so you can reuse the configuration after Stop/Start.

In VS Code:

  1. In Run and Debug, choose Podbench: Attach — … for the BlueAPI server.

  2. Keep a breakpoint on your logging line and press F5 to attach.

  3. Run the health request from the seat terminal:

    curl -fsS http://127.0.0.1:8000/healthz

Inspect the paused process as before, then continue and disconnect the debugger. Ending an Attach session leaves the server running and releases that session’s probe hold. The injected Python debugger may remain loaded until the next application restart.

If Podbench cannot find a matching process, confirm the application is running. If the executable or command arguments have changed, rerun podbench ide vscode from your workstation to regenerate the configurations. Reconnect after a pod replacement too. If several processes match, resolve the ambiguity before retrying; see troubleshooting.

Undo the tutorial change#

End any debugging session, remove the logging line you added, and save the file. Review your diff in the seat terminal to confirm that you have kept any edits that were there before the tutorial:

git diff

If BlueAPI is running, load the restored source with podbench restart in the seat terminal. If Launch left it stopped, use podbench start instead. Wait for the command to finish, then check from your workstation:

podbench status
kubectl get pod p47-blueapi-0

The HOTFIX column should show initialized, normal without HELD. In the pod listing, check that all containers are ready. Starting and stopping the application child should not increase Kubernetes’ container restart count.

You can now close VS Code. The checkout stays on the persistent claim, ready for your next session; you do not need to initialize it again.

Apply the same loop to an IOC#

The same Stop → Launch → Start and Attach workflows apply to a native IOC. For the P47 PMAC IOC, first follow the IOC example in Prepare a workload for hotfix. It uses pod bl47p-mo-ioc-01-0, container bl47p-mo-ioc-01, and the ioc-pmac source repository.

Then open its seat from your workstation:

podbench ide vscode bl47p-mo-ioc-01-0 --target bl47p-mo-ioc-01

Choose the generated Podbench: Attach — … entry for the IOC executable to inspect the running IOC. To debug startup, run podbench stop in the seat and choose its Podbench: Launch — … entry. This launches the executable under GDB inside the application container. When you end Launch, run podbench start to return to normal operation.

For this IOC, Start reruns the startup script, using /podbench/app/ioc/start.sh when present. Launch runs the selected executable directly, as in the BlueAPI example.

Native source changes also need a build: follow the IOC’s build procedure and ensure the launched executable is the rebuilt binary. Cloning ioc-pmac into the checkout alone does not replace the support binaries supplied by the image.