Running epilot integrations: monitoring, errors, maintenance

epilot integration maintenance starts the moment an interface goes live. From then on it has to run reliably, make failures visible and keep up with every change in epilot or in the systems around it. This article covers what running an interface actually involves and who can take that on.

I am Marcel R. G. Berger, a software developer and architect with more than 20 years of experience. I work with epilot as an external consultant and developer for apps and customisations. This piece is for IT leads at grid operators, municipal utilities and energy suppliers who run, or are planning, their own interfaces around epilot.

What happens to an epilot integration after go-live?

After go-live, a project turns into a service that has to work every single day. It processes real data, and every failure affects a real case.

That changes the questions. During delivery the question is whether the mapping is right. In operation the question is whether you notice when it stops being right, and how quickly you can find out why.

The everyday work of running an interface usually looks like this:

• checking that mappers are up and can reach the systems they depend on
• spotting failures, classifying them and reprocessing the affected cases
• keeping dependencies, the Java runtime and container images current
• adjusting mappings when epilot or a surrounding system changes
• keeping documentation, runbooks and tests up to date

How such a mapper is built in the first place is covered in building mappers between epilot and surrounding systems with Quarkus. This article is about everything that comes afterwards. Which systems are typically involved is covered in connecting ERP, billing and third-party systems to epilot.

How do you monitor mappers with health checks and logging?

You monitor mappers with health checks, structured logging and alerts. Health checks tell you whether a service is ready, logs tell you what it did with a record, and alerts make sure somebody hears about it.

Health and readiness probes

Quarkus ships its own health endpoints through SmallRye Health. The Quarkus guide to SmallRye Health lists /q/health/live for whether the process is alive, /q/health/ready for whether it can accept requests, and /q/health/started for startup. A container platform polls these endpoints at regular intervals.

You can add your own checks with @Liveness or @Readiness. Here is a readiness check that reports whether a surrounding system is reachable:

@Readiness
@ApplicationScoped
public class TargetSystemReadiness implements HealthCheck {

    @Inject
    TargetSystemPing ping;

    @Override
    public HealthCheckResponse call() {
        return ping.isReachable()
                ? HealthCheckResponse.up("target-system")
                : HealthCheckResponse.down("target-system");
    }
}

TargetSystemPing is a class of the project itself. One distinction matters here: the liveness check should only look at the service’s own process. If it depends on another system, the platform will restart a perfectly healthy mapper that is simply waiting for its neighbour.

Structured logging

Structured logging means every log line carries machine-readable fields, such as the message identifier, the case and the outcome. The Quarkus logging guide explains how to switch the output to JSON.

quarkus.log.console.json.enabled=true

That lets you follow a single record through epilot, the mapper and the target system. Personal data should not end up in the logs unfiltered. Logging the identifiers you need to find the case is enough.

Alerts

Alerts turn measurements into notifications. Useful ones fire when a service stops being ready, when failed processing starts to pile up, and when a circuit breaker opens. Which tools deliver them depends on the platform you already run.

How do you find and fix errors in epilot webhooks?

You find errors in epilot webhooks in epilot’s webhook monitoring and in the mapper’s logs. You fix them by correcting the cause and replaying the event.

According to the epilot documentation on webhook monitoring, every webhook request is logged as an event with a success or failure status. In each webhook configuration you can filter events by status, inspect payload, headers and response code, and replay any event. A replay sends the same payload again and creates a new event record. Events are retained for four weeks by default.

That gives you a simple operational routine:

1. An alert or a look at the monitoring shows failed deliveries.
2. The response code and the mapper’s logs tell you what went wrong.
3. You fix the cause, for example a missing mapping or an unreachable system.
4. You replay the affected events in epilot.

For a replay to be safe, the mapper has to be idempotent. If the same payload arrives twice, the target system must not end up with a second record. The mapper checks a business key from the payload to see whether the case has already been processed.

What the mapper answers also matters. An error response to epilot should mean the case was not processed. Only then does the monitoring in epilot give you an honest picture.

How do mappers handle rate limits and outages of other systems?

Mappers handle rate limits and outages with retries and backoff, timeouts and a circuit breaker. The aim is that a short outage does not leave a permanent error behind, and a long outage does not put extra load on a system that is already struggling.

Rate limits of the epilot API

The epilot documentation on rate limits states that the API responds with HTTP 429 when a limit is exceeded. The Retry-After header indicates how many seconds until the quota resets. The documentation says limits apply per organisation, and read-only GET requests are exempt.

A mapper that writes back to epilot reads that header and waits accordingly. The documentation also recommends the dedicated batch interfaces for larger volumes of data.

Retries, timeouts and circuit breakers

For calls to surrounding systems I use SmallRye Fault Tolerance. The Quarkus guide to SmallRye Fault Tolerance covers, among others, these annotations:

Annotation	What it does in operation
@Retry	repeats a failed call
@ExponentialBackoff	increases the wait between retries step by step
@Timeout	aborts calls that take too long
@CircuitBreaker	stops calling a system that is currently failing and checks again later
@Fallback	defines what happens once all attempts have failed
@Bulkhead	limits how many calls run at the same time

Not every error deserves a retry. A record the target system rejects on business grounds stays rejected however often you send it. Those cases are logged and reported as errors so a person can look at them. Only transient failures such as timeouts or unreachable systems are retried.

Why does a running interface need regular updates?

A running interface needs regular updates because everything around it keeps moving. Libraries, the Java runtime and the container’s base image receive security fixes, and Quarkus itself ships new versions.

A mapper that is never updated will often keep running for a long time. It also keeps collecting known vulnerabilities, and the jump to a current version gets bigger every month.

Maintenance therefore covers three layers:

• build dependencies, meaning Quarkus and other libraries in Maven or Gradle
• the Java runtime the service runs on
• the container’s base image and its system packages

The Quarkus guide to container images describes how the image is produced straight from the build, for example with Jib, Docker or Podman. That makes it easy to rebuild, test and ship an image automatically after an update. Even without a change to your own code, a rebuilt image is worthwhile once the base image has been patched.

Configuration and secrets outside the code

Addresses, credentials and signing secrets do not live in the code. They come from environment variables or a secret store. The Quarkus configuration reference shows which sources Quarkus reads values from.

In operation that pays off twice. The same image runs in test and production with different configuration. And a key can be rotated without rebuilding the service.

What happens when epilot or a surrounding system changes?

When the configuration in epilot or a surrounding system changes, the mapping in the mapper often has to change too. A new field in an epilot schema, a different code in a line-of-business system or a new version of an API are typical triggers.

In epilot this happens a lot, and that is by design. Business teams adjust data models, journeys and automations themselves, because epilot is an excellent no-code and low-code platform. The interface has to keep up. Where the line between configuration and code usually runs is covered in when low-code in epilot is no longer enough.

Two agreements keep changes from coming as a surprise:

• changes to fields that an interface uses are announced in advance
• mappings are documented in the repository, so it is clear which fields are affected

Tests as a safety net

Tests are the safety net for every change in operation. Each mapping has sample data from epilot and the expected output for the target system. The Quarkus guide to testing your application shows how @QuarkusTest boots the whole service and checks its endpoints.

When a rule changes, the tests show straight away which cases are affected. After a dependency update, they show whether the mapper still behaves exactly as before.

What documentation does running an interface require?

Running an interface requires a description of the data flows, the mappings and the configuration, plus runbooks for the usual incidents. Without them, operations depend on individual people.

A runbook describes step by step what to do for a specific incident. Typical entries are:

• A surrounding system is unreachable: how you notice, what the mapper does on its own, and when events get replayed.
• Webhooks fail after a change in epilot: where the mapping lives and how it is tested.
• A secret has to be rotated: where it is stored and how the service picks up the new value.
• A new version is due: how it is built, tested and rolled back.

Documentation belongs in the same repository as the code. That way it stays current with every change, and whoever takes over the code takes over the documentation with it.

Who maintains the epilot integration after delivery?

After delivery there are two routes, and the choice is yours. In both cases the code belongs to you.

On the first route I hand the code over to your own repository. Your IT or another partner then runs it and develops it further. On the second route I look after operation and further development under a maintenance contract.

Criterion	Handover to your repository	Maintenance contract
Ownership of the code	yours	yours
Operation and monitoring	your IT or a partner of your choice	handled by me as agreed
Updates and security fixes	done by your team	done by me
Adjusting mappings	done by your team	done by me
Prerequisite	Java skills in your team or at your partner	access to repository and environments as agreed
Foundation	documentation, runbooks and tests in the repository	documentation, runbooks and tests in the repository

You can also take both routes one after the other. A maintenance contract does not rule out a later handover, because code and documentation can sit in your repository all along. The article who owns the integration code explains why ownership matters so much.

I work alongside epilot’s consulting team, not instead of it. epilot remains your point of contact for the platform. I take care of the interfaces that run outside it.

Why does stable interface operation matter so much for grid operators?

For grid operators, stable interface operation matters because cases have to move reliably and traceably through several systems. A record that silently disappears often only surfaces when a customer calls.

Three things stand out:

• Stable processes: when cases run across several systems, every hand-off has to work.
• Traceability: structured logs, epilot’s monitoring and versioned mappings show what happened to a case and which rule applied at the time. That helps with internal and external audits.
• Continuity: documentation, runbooks and tests mean operations do not hinge on a single person.

That is how I work in my current project with a grid operator in Hesse. Many processes there run outside epilot, and several third-party systems are connected through individually developed interfaces. The mapper tools are written in Java with Quarkus, run in containers and are designed to scale and stay maintainable. There is more on this in the in practice section on the home page.

The contracting party is DigitalFreedom Global LLC. For grid operators, German law is available on request. Data stays in Europe, processing follows the GDPR, and a data processing agreement is put in place where needed. The details are in the contracting party section on the home page.

How do you get started with running and maintaining your interface?

You start with a free intro call in which we go through your interfaces and how they are run today. Afterwards it is clear whether a handover to your repository, a maintenance contract, or one followed by the other fits best.

1. Intro call: free and without obligation, on Microsoft Teams.
2. Review: which mappers are running, how they are monitored, and where documentation or tests are missing.
3. Individual quote: tailored to your systems and your operational requirements.

Everything I offer is based on an individual quote. The services section on the home page lists what that covers, and you can pick a slot directly in the booking calendar.