# Test DST

**Since Camel 4.24**

The `camel-test-dst` module runs the real code of a Camel consumer under a deterministic, seeded, virtual-time scheduler. Only the external system is faked. A test runs thousands of seeds in seconds, and a failing seed replays with the same step trace.

-   Every thread pool created through the `ExecutorServiceManager` is backed by one event loop on the test thread (`DstThreadPoolFactory`). Tasks run one at a time; when several can run, the seed picks one.
    
-   Time is virtual: it jumps to the next due task. The time reads, sleeps and blocking waits of the code under test are rewritten at test time (`@DstInstrument`), and the Camel core code that waits, times tasks or throttles reads the virtual clock through its internal time source, so "wait 30 seconds" costs nothing. A world can move the wall clock forward or backward apart from the monotonic clock (see [Wall clock jumps](#_wall_clock_jumps)).
    
-   A fake of the external system (a queue, a checkpoint store, a web API) injects faults chosen by the same seed.
    
-   Invariants are checked at the end of each run.
    
-   `@DstTest(seeds = 1000)` runs a test method once per seed. A failing seed is printed with its trace and is replayed exactly with `-Ddst.seed=4711` or `@DstSeed(4711)`.
    

This is a test tool. Nothing of it runs in production. Camel core has one internal hook for it, `TimeSource` in camel-util: with its default provider it reads `System` and sleeps with `Thread.sleep`, and only an open run replaces the provider (installing it, and putting the default back, is logged at INFO with the class of the provider). `TimeSource` is not an API for components: a component gets virtual time through the core classes it already uses (the "Camel core" row of the table below); `camel-chaos` is the one exception that reads it directly.

## A test

```java
@DstInstrument("org.apache.camel.component.aws2.sqs")
class SqsVisibilityDstTest {

    @DstTest(seeds = 1000)
    void noMessageIsDeliveredAgainWhileTheConsumerHoldsIt(Dst dst) throws Exception {
        new SqsWorld(dst).run();
    }
}
```

The world (`SqsWorld`) builds a `CamelContext` with `DstCamel.newContext(dst)`, binds the fake client, starts the route, drives the run with `dst.runUntil(…​)` and `dst.runFor(…​)`, and checks the invariants with `dst.check(…​)`. The test and its world live in a package of their own (for example `org.apache.camel.component.aws2.sqs.dst`); that package is never rewritten.

A `Dst` is one run. The extension creates one per seed and closes it after the seed; a test that creates its own `new Dst(seed)` closes it in a `finally`. The virtual clock is JVM-wide, so only one run may be open at a time: a second `new Dst` while another run is open throws `IllegalStateException` ("another run (seed N) is still active"), and a DST test refuses to start when JUnit runs tests in parallel (`junit.jupiter.execution.parallel.enabled=true`).

These worlds are in the Camel code base and are the examples to copy. Each world class is annotated with `@DstWorld` (see step 10 of the guide below), and the build lists it in the `dstWorlds` section of the component in the Camel catalog. This table is generated from the catalog, which also has the faults each world injects and the invariants it checks:

    
| Component | World (tests) | Fake | What it covers | Issues |
| --- | --- | --- | --- | --- |
| camel-aws2-kinesis | `KinesisWorld` (`KinesisConsumerDstTest`) | FakeKinesis: a data stream with shards that can be split and merged (GetShardIterator, GetRecords, ListShards) bound as amazonKinesisClient | 
-   the iterator types TRIM\_HORIZON and LATEST, with one or two shards at the start and more after resharding
    
-   shard iterators that expire between two polls, while the route is suspended or an exchange is slow
    
-   a closed shard and its child shards after a split or a merge (shardClosed ignore and silent)
    
-   checkpoints with the resumable EIP and KinesisResumeStrategy, and the resume after a route restart
    
-   route restart and route suspend and resume between two polls
    





 | CAMEL-25569, CAMEL-25942, CAMEL-26968 |
| camel-aws2-sqs | `SqsWorld` (`SqsRedeliveryDstTest`, `SqsVisibilityDstTest`) | FakeSqs: a standard SQS queue (ReceiveMessage, DeleteMessage, ChangeMessageVisibilityBatch) bound as amazonSQSClient | 

-   the visibility timeout extender (extendMessageVisibility=true)
    
-   short polling with one to three concurrent consumers and up to 10 messages per poll
    
-   processing that takes longer than the visibility timeout
    
-   synchronous redelivery by the error handler after a delay longer than the visibility timeout
    





 | CAMEL-25700, CAMEL-25737, CAMEL-25738, CAMEL-25911 |
| camel-azure-eventhubs | `EventHubsWorld` (`EventHubsCheckpointDstTest`) | a mocked EventHubsClientFactory that delivers the events per partition, and a fake CheckpointStore | 

-   checkpointing by checkpointBatchSize and checkpointBatchTimeout
    
-   one or two partitions with bursts and quiet periods of events
    
-   an asynchronous downstream step
    





 | CAMEL-25683, CAMEL-25738, CAMEL-25781 |
| camel-azure-servicebus | `ServiceBusWorld` (`ServiceBusConsumerDstTest`) | FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count, dead-letter queue) behind the processor and receiver clients of ServiceBusClientFactory | 

-   the lock renewer of the consumer (PEEK\_LOCK, maxAutoLockRenewDuration) with an asynchronous route
    
-   processing that takes longer than the lock duration
    
-   complete, abandon and dead-letter, also after the lock was lost
    
-   maxConcurrentCalls of one to three (with the asynchronous route it bounds the processMessage calls)
    
-   route stop and start while messages are processed
    





 | CAMEL-25701, CAMEL-26956, CAMEL-26969, CAMEL-26970, CAMEL-27066 |
| camel-google-pubsub | `PubsubWorld` (`PubsubConsumerDstTest`) | FakePubsub: a subscription (Pull, Acknowledge, ModifyAckDeadline, ack deadlines, dead letter policy) behind the SubscriberStub of synchronousPull | 

-   synchronousPull with ackMode AUTO, and ackMode NONE with grouped messages acked or nacked together
    
-   ack deadlines that expire while a message is processed
    
-   the max delivery attempts of a subscription with a dead letter policy
    
-   route stop and start while the consumer pulls
    
-   a stop and start of the same consumer in one step, as a throttling route policy does
    





 | CAMEL-25712, CAMEL-25790, CAMEL-26647 |
| camel-kafka | `KafkaWorld` (`KafkaConsumerDstTest`) | FakeKafka: a topic with one or two partitions and a consumer group (classic protocol, eager rebalance), behind the Consumer interface of KafkaClientFactory | 

-   commits by Kafka’s auto-commit, by manual synchronous or asynchronous commits in the route, and per batch with batching=true
    
-   breakOnFirstError
    
-   pollOnError ERROR\_HANDLER, DISCARD, RETRY and RECONNECT
    
-   a poll that fails while a batch is incomplete
    
-   route restart, also while the consumer polls
    
-   route suspend and resume, with a second request while the consumer handles the first
    
-   partitions revoked and assigned again by a rebalance while records are processed
    





 | CAMEL-25313, CAMEL-25756, CAMEL-26967 |
| camel-kafka-share | `KafkaShareWorld` (`KafkaShareConsumerDstTest`) | FakeShareGroup: a share group (KIP-932) of one topic with one partition, behind ShareConsumer | 

-   explicit acknowledgement with onFailure RELEASE or REJECT
    
-   commitMode SYNC and ASYNC
    
-   pollOnError ERROR\_HANDLER, RECONNECT, RETRY and DISCARD
    
-   acquisition locks and the delivery limit
    
-   the backoff when the share consumer cannot be created
    
-   route start, restart, suspend and resume
    
-   suspend and resume of the CamelContext
    





 | CAMEL-25872, CAMEL-25910 |
| camel-mongodb | `MongoDbChangeStreamsWorld` (`MongoDbChangeStreamsDstTest`) | FakeMongo: a MongoDB collection and its change stream (resumeAfter, startAfter, drop and invalidate events, the resume of the driver), behind the MongoClient of the connectionBean option | 

-   the change streams consumer with a store of its resume token (changeStreamTokenRepository)
    
-   a dropped collection, made again by an insert
    
-   a server that cannot be reached for a while: the cursor cannot be opened, a getMore fails
    
-   route restarts (the consumer thread interrupted, then the route stopped and started), resuming after the stored token
    
-   the fullDocument option and cursorRegenerationDelay
    





 | CAMEL-25679, CAMEL-25946, CAMEL-26992 |
| camel-nats | `NatsWorld` (`NatsConsumerDstTest`) | FakeJetStream: a server with one JetStream stream and its pull consumers (AckWait, MaxDeliver, Nak), behind a jnats Connection given with the connection option | 

-   the JetStream pull consumer with explicit acks and naks, ephemeral or durable
    
-   AckWait and MaxDeliver
    
-   route stop and start, also while the consumer waits in a fetch
    
-   a server restart that deletes the ephemeral consumer, and the subscription made again
    





 | CAMEL-26884 |
| camel-paho-mqtt5 | `PahoMqtt5World` (`PahoMqtt5ConsumerDstTest`) | FakeMqtt: an MQTT 5 broker with the session of the client id (Clean Start, Session Expiry, Receive Maximum, resend on reconnect) and a Paho MqttClient subclass on it, given to the consumer by a test component | 

-   QoS 0, 1 and 2 with automatic or manual acknowledgements (manualAcksEnabled)
    
-   a session that the broker keeps (cleanStart=false, sessionExpiryInterval) or not
    
-   the automatic reconnect of Paho and the subscription made again
    
-   the reconnect of the consumer after a failed message, and Receive Maximum
    
-   route stop and start, with messages queued for a kept session
    





 | CAMEL-25793, CAMEL-25944, CAMEL-27041, CAMEL-27043 |
| camel-pulsar | `PulsarWorld` (`PulsarConsumerDstTest`) | FakePulsar: a broker and client for one topic and one durable subscription | 

-   subscription types Exclusive, Shared, Failover and Key\_Shared with one to three consumers
    
-   the message listener (messageListener=true, one consumer thread)
    
-   the ack timeout and the redelivery of a failed exchange after a negative acknowledgement
    
-   route start, restart, suspend and resume
    
-   an asynchronous downstream step
    





 | CAMEL-25869 |
| camel-slack | `SlackWorld` (`SlackSuspendDstTest`) | a fake Slack Web API (conversations.history) returned by a mocked Slack.getInstance | 

-   the polling consumer with maxResults and naturalOrder
    
-   the pages of conversations.history (oldest and cursor)
    
-   route suspend and resume while a batch is routed
    





 | CAMEL-25684, CAMEL-25738 |
| camel-zookeeper-master | `ZooKeeperMasterWorld` (`ZooKeeperMasterDstTest`) | FakeZooKeeper: a ZooKeeper ensemble (sessions, ephemeral sequential znodes, one-time watches) with the other nodes of the group as sessions, behind a CuratorFramework shared as the curator bean | 

-   the master consumer (zookeeper-master endpoint) and the MasterRoutePolicy, on the real ZooKeeperGroup
    
-   connection losses shorter and longer than the session timeout (SUSPENDED, RECONNECTED, LOST)
    
-   other nodes that join, leave and die
    
-   a start of the consumer that fails and its retry; with the master consumer, a start that takes long
    
-   route stop and start, and a route of the policy removed and added again
    





 | CAMEL-25315, CAMEL-26540, CAMEL-26802, CAMEL-26870, CAMEL-26887, CAMEL-27037, CAMEL-27044 |

## What the harness controls

<table class="tableblock frame-all grid-all stretch"><colgroup><col> <col></colgroup><tbody><tr><td class="tableblock halign-left valign-top">Thread pools</td><td class="tableblock halign-left valign-top">Every pool and scheduled executor created by Camel or the component through the <code>ExecutorServiceManager</code>. A single-thread executor keeps its FIFO order; an executor with n threads offers its first n runnable tasks to the seed.</td></tr><tr><td class="tableblock halign-left valign-top">Time reads</td><td class="tableblock halign-left valign-top"><code>System.currentTimeMillis()</code>, <code>System.nanoTime()</code>, <code>new Date()</code>, <code>Instant.now()</code>, <code>Clock.systemUTC()</code>, <code>Clock.systemDefaultZone()</code>, <code>Clock.system(ZoneId)</code> and <code>now()</code> / <code>now(ZoneId)</code> of <code>LocalDateTime</code>, <code>ZonedDateTime</code>, <code>OffsetDateTime</code>, <code>LocalDate</code>, <code>LocalTime</code> and <code>OffsetTime</code>, in the packages of <code>@DstInstrument</code>. The wall clock is <code>Dst.EPOCH_MILLIS</code> plus the virtual time, moved by the wall clock jumps of the run; <code>System.nanoTime()</code> is the monotonic clock and never jumps.</td></tr><tr><td class="tableblock halign-left valign-top">Sleeps</td><td class="tableblock halign-left valign-top"><code>Thread.sleep</code> and <code>TimeUnit.sleep</code>: on the loop thread a sleep is a virtual wait. The sleeping task keeps its thread and the other tasks run meanwhile.</td></tr><tr><td class="tableblock halign-left valign-top">Blocking waits</td><td class="tableblock halign-left valign-top"><code>CompletableFuture.get/join</code>, <code>Future.get</code>, <code>CountDownLatch.await</code>, <code>Semaphore.acquire/tryAcquire</code>, <code>DelayQueue</code> and <code>BlockingQueue</code> <code>take/poll</code>: the loop runs until the wait can end, and a timed wait gives up in virtual time. A wait that no task of the run can end fails the seed at once and names the call site.</td></tr><tr><td class="tableblock halign-left valign-top">Timed waits on tasks</td><td class="tableblock halign-left valign-top"><code>Future.get(timeout)</code> on a task of the run and <code>awaitTermination</code> time out in virtual time.</td></tr><tr><td class="tableblock halign-left valign-top">Parallel streams</td><td class="tableblock halign-left valign-top"><code>parallelStream()</code> of a <code>java.util</code> collection is rewritten to <code>stream()</code>: the elements are handled in order by the calling task, not on the common fork-join pool.</td></tr><tr><td class="tableblock halign-left valign-top">Camel core</td><td class="tableblock halign-left valign-top">The code of Camel core that waits, times tasks or throttles reads the clock and sleeps through <code>org.apache.camel.util.TimeSource</code>, an internal class of camel-util. A run installs its virtual clock there (<code>Dst</code> does it when the run is created) and puts the previous provider back when the run is closed, so no rewriting is needed: the synchronous redelivery (<code>RedeliveryPolicy</code>, <code>RedeliveryErrorHandler</code>), <code>ForegroundTask</code>, <code>BackgroundTask.schedule</code>, the budgets of both tasks, the wait loop of <code>DefaultShutdownStrategy</code>, the synchronous delayer, <code>MulticastProcessor</code>, <code>OptimisticLockRetryPolicy</code>, <code>DefaultStartupConditionStrategy</code>, <code>StopWatch</code>, <code>MonotonicClock</code>, <code>BackOffTimerTask</code>, <code>DefaultTimeoutMap</code>, the elapsed time of <code>MessageHistory</code>, the clocks of the throttlers and the counters of <code>ThrottlingExceptionRoutePolicy</code>. A sleep there is a virtual wait on the loop thread, traced with its call site. A test of camel-core (<code>TimeSourceDirectClockCallsTest</code>) fails when one of these classes reads the system clock or sleeps directly: <code>System.nanoTime()</code>, <code>System.currentTimeMillis()</code>, <code>Thread.sleep</code>, a <code>sleep</code> of a <code>TimeUnit</code> constant, <code>Instant.now()</code> and the <code>now()</code> of the other <code>java.time</code> types, <code>new Date()</code>, <code>Calendar.getInstance()</code>, <code>new GregorianCalendar()</code> and the system <code>Clock`s. It also fails when a class of a core module uses `TimeSource</code> and is not in its list. Two waits of these classes are not virtual (see below): the permit queue of <code>TotalRequestsThrottler</code> and the <code>Semaphore</code> of <code>ConcurrentRequestsThrottler</code>. The sleep of the route <code>delayer</code> option is not virtual either (see below). The half-open check of <code>ThrottlingExceptionRoutePolicy</code> runs on a scheduler of the <code>ExecutorServiceManager</code>, so the circuit is half opened after <code>halfOpenAfter</code> of virtual time.</td></tr></tbody></table>

Event timestamps, the clock of an exchange and the direct clock reads of the debugger, the tracer, management and the developer consoles stay on `System`. The durations they measure with `StopWatch` or `MonotonicClock` follow `TimeSource` (with the default provider this is the same `System.nanoTime()`), and so does the created time of `MonotonicClock`: the message history times (`MessageHistory.getTime()`) are virtual in a DST run. `@DstInstrument(alsoRewrite = …​)` applies the rewriting of the code under test to classes outside it, for example `org.apache.camel.support.ResetableClock` (the clock of an exchange) or a library the component calls. It is not needed for the core classes listed above, except for the permit queue of `TotalRequestsThrottler` (see below).

### Wall clock jumps

Without a jump the wall clock moves only forward, together with the monotonic clock. A world can move the wall clock forward or backward, as an NTP step or an operator who sets the system time does, to test code that keeps times of the wall clock: expiring entries, caches, deadlines and schedules.

-   `dst.jumpWallClock(deltaMillis)` moves it at once (back when the value is negative). Call it on the loop thread: in the test body, or in a task of the run.
    
-   `dst.jumpWallClockBetween(fromMillis, toMillis, minDeltaMillis, maxDeltaMillis)` lets the seed choose: it draws when the jump happens (between `fromMillis` and `toMillis` of virtual time from now) and how big it is, and schedules it. The same seed makes the same jump at the same time, so a failing seed replays exactly. The trace says when the jump is scheduled (its time and size) and when it happens, and the returned task has the delta as its result once the jump has happened. The jump is a pending task of the run, so `dst.dropPending()` drops it.
    
-   A jump moves the wall clock at most 100 years (`Dst.MAX_WALL_CLOCK_JUMP_MILLIS`) away from the monotonic clock.
    

After a jump, the wall clock reads see it: the rewritten reads of the code under test (`System.currentTimeMillis()`, `new Date()`, `Instant.now()`, the `java.time` reads) and, in Camel core, `TimeSource.currentTimeMillis()` and `TimeSource.instant()` (for example the expiry of `DefaultTimeoutMap` and the failure window and open state of `ThrottlingExceptionRoutePolicy`, and the created time of a `MonotonicClock`). The monotonic clock (`System.nanoTime()`, `TimeSource.nanoTime()`, `StopWatch`, `MonotonicClock.elapsed()`), `dst.now()`, the sleeps and waits, and the due times of the tasks do not: a task scheduled 10 seconds ahead still runs 10 seconds later. `dst.wallClockOffset()` is how far the jumps have moved the wall clock. Each run starts without a jump.

`dst.currentTimeMillis()` follows the jumps; `Dst.EPOCH_MILLIS + dst.now()` does not. Give the fake the one that matches the external system: a server keeps its own clock, so a jump of the clock of the Camel host leaves the timestamps of the server as they were.

An example: an entry with a timeout of 30 seconds in a `DefaultTimeoutMap` (the expiring map behind request-reply timeouts), under a backward jump of an hour 10 seconds after it was put:

```java
DefaultTimeoutMap<String, String> map = new DefaultTimeoutMap<>(dst.executor("purge", 1), 1_000);
map.start();
map.put("order-1", "pending", 30_000);
dst.runFor(10_000);
dst.jumpWallClock(-3_600_000);
dst.runFor(60_000);
// 70 s after the put the entry is still there: its expiry is a wall clock time, so it now expires an hour late
assertEquals(1, map.size());
```

The example asks `size()` and not `get()`, because `get()` refreshes the expiry of the entry, from the wall clock as it is after the jump. After a forward jump of an hour instead, the entry is evicted by the next purge after the jump (here 1 second later). In a world, let the seed choose the jump (`dst.jumpWallClockBetween(0, 60_000, -3_600_000, 3_600_000)`, which writes it to the trace), and decide from the contract of the component what a jump may change: an entry that expires late or early by the size of the jump can be what the component documents, while an exchange that is lost or completed twice is not. As for any fault, record which jump excuses which violation.

### Escapes and strict mode

Before the first seed, the harness scans the packages under test and writes every **escape point** to `target/dst/<Test>.escapes.txt`: own threads and executors, the common fork-join pool, Reactor schedulers, waits it cannot run the loop for (`Object.wait`, `LockSupport.park`, `Mono.block`), wall clock reads it does not rewrite.

Strict mode is the default: when a seed reaches one of these at runtime, or a thread outside the harness submits a task to the loop, reads the virtual clock or sleeps, the seed fails with the escape as its cause. This includes Camel core: a `StopWatch` started on such a thread reads the virtual clock through `TimeSource`. A clock read from such a thread (a Reactor or Netty thread, for example) still gets the virtual time and its sleep still takes real time, so that thread mixes the two; this is what strict mode is there to make visible. `@DstInstrument(allowEscapes = "Foo.bar")` allows an escape that cannot change the result (say why in a comment); `@DstInstrument(strict = false)` or `-Ddst.strict=false` only records escapes in `target/dst/<Test>.summary.txt`.

A wait in code that is not rewritten blocks the loop thread for real. A watchdog interrupts the loop thread after 10 s of real time without progress (`-Ddst.watchdogMillis`) and the seed fails with the stack where it was blocked. In Camel core:

-   `BackgroundTask.run` waits on a latch until its scheduled attempts are done, so it blocks the loop thread. `BackgroundTask.schedule` does not wait, and its attempts run in virtual time.
    
-   `TotalRequestsThrottler` waits for a permit on its `DelayQueue`. The delay of a permit follows the virtual clock, which does not move while the loop thread waits, so a seed that has to wait for a permit always ends on the watchdog. `asyncDelayed()` does not avoid it: a delayed exchange that finds its permit taken by another one waits on the same queue. Add `org.apache.camel.processor.TotalRequestsThrottler` to `@DstInstrument(alsoRewrite)`: the permit wait is then a virtual wait.
    
-   `ConcurrentRequestsThrottler` blocks on its `Semaphore` once `maxConcurrentRequests` exchanges are in flight. No task of the run releases a permit while the loop thread waits, so the seed ends on the watchdog. `alsoRewrite` does not help here (the call goes through a `Semaphore` subclass): keep `maxConcurrentRequests` above the number of exchanges the world has in flight.
    
-   The route `delayer` option (`CamelInternalProcessor.DelayerAdvice`) calls `Thread.sleep` directly, not through `TimeSource`: the delay takes real time on the loop thread and the virtual clock does not move. Use the `delay` EIP in a world instead.
    

### The one-stack model

All tasks run on one thread, the test thread. A task that blocks (a call of the fake with a latency, a sleep, a wait) stays on the stack while the loop runs the other tasks **nested** above it. So a blocked task can resume only after every task that blocked after it has resumed. Three consequences:

-   Timing: see the rule in the guide below.
    
-   A timed wait notices its timeout only when the tasks nested above it return. If the awaited task completes after the deadline that way, the caller still gets the `TimeoutException`, but the task’s side effects have already happened.
    
-   A route stop or suspend that runs while an exchange is blocked in a synchronous call below it on the stack cannot see that exchange finish. The Slack world therefore copies the suspend sequence of `DefaultShutdownStrategy` with a short polling wait instead of calling `suspendRoute`. With an asynchronous downstream call (`DstCamel.asyncStep`) a real `stopRoute` works.
    

Code that uses its own threads or a third-party scheduler (the Azure SDKs on Reactor) is outside the harness; fake the client at a level above those threads.

## Writing a fake world: a guide for AI agents and humans

A world is the component’s real consumer, a fake of the external system, a load (producers, operators), seeded faults and invariants. This guide uses the SQS world of camel-aws2-sqs (`FakeSqs`, `SqsWorld`, about 420 lines) as the worked example. Follow the steps in order.

### 1\. Find where the client is swapped

Look for the place where the component gets its client:

-   an endpoint option that takes a client instance, bound in the registry. camel-aws2-sqs has `amazonSQSClient`:
    
    ```java
    context.getRegistry().bind("sqs", fakeSqs);   // FakeSqs implements SqsClient
    from("aws2-sqs://dst?amazonSQSClient=#sqs&queueUrl=RAW(" + FakeSqs.QUEUE_URL + ")&extendMessageVisibility=true"
         + "&visibilityTimeout=" + visibilityTimeout + "&maxMessagesPerPoll=" + maxMessagesPerPoll
         + "&concurrentConsumers=" + concurrentConsumers + "&initialDelay=100&delay=" + delay)
    ```
    
-   else a static factory, replaced with Mockito’s `mockStatic` for the duration of the run (the Event Hubs world mocks `EventHubsClientFactory`, the Slack world mocks `Slack.getInstance`).
    

Fake the client at the level where the component calls it synchronously. If the client runs its own threads (Reactor, Netty, an HTTP client pool), fake it above them.

### 2\. Run the escape scan before writing anything

Add the agent to the module’s pom first, see [Setup](#_setup): without it every test class with `@DstInstrument` fails with "The byte code agent of camel-test-dst is not loaded".

Write the smallest test with `@DstInstrument("<component package>")` and one seed, run it, and read `target/dst/<Test>.escapes.txt`. Every line is a place where the component leaves the harness. For each one, decide:

-   **fix** it in the harness (the time reads, sleeps and waits listed above are already handled),
    
-   **avoid** it with the configuration of the world (the first SQS world kept `maxMessagesPerPoll` at 10 or less because more joined several requests with `CompletableFuture.get()`; the harness now runs the loop for that wait), or
    
-   **document** it, in the world’s Javadoc, and allow it in strict mode only if it cannot change the result.
    

The SQS scan flagged `Instant.now()` in the visibility extender. Before it was rewritten, 271 of 500 seeds on `main` failed falsely: the extender computed its deadline from real time while the fake counted in virtual time.

### 3\. Write the fake from the service’s documentation

Write the fake from the API reference of the service, not from the component’s code: a fake copied from the consumer agrees with the consumer’s bugs. Quote the rule next to the code that implements it, name the page, and mark what is assumed:

```java
// [R] "Valid values: 1 to 10. Default: 1." and "Amazon SQS never returns more messages than this value
// (however, fewer messages might be returned)"
int max = request.maxNumberOfMessages() != null ? request.maxNumberOfMessages() : 1;
...
// [D] "If you use an old ReceiptHandle, the request will succeed, but the message might not be deleted."
if (dst.chance(0.5)) {
    delete(m, "deleted (old handle)");
}
...
// [C] "ReceiptHandleIsInvalid: The specified receipt handle isn't valid." (assumed for a deleted
// message and for the handle of an earlier receive; the page does not spell these cases out)
```

Where the documentation allows several behaviours ("might", "fewer messages might be returned"), let the seed choose. List what is not modelled in the class Javadoc (`FakeSqs`: long polling, FIFO queues, the in-flight limit, retention, dead-letter queues).

### 4\. The one-stack timing rule

A blocked task resumes only after the tasks that blocked after it. A call of the fake that blocks for its latency can therefore return later than its latency, and anything it decides at the start can be stale at the end. The rule:

-   **reads take effect at the end of their latency**: `FakeSqs.receiveMessage` blocks first, then picks the visible messages and starts their visibility timeout;
    
-   **writes take effect at the start**: `deleteMessage` deletes, then blocks for its latency;
    
-   **background tasks do not block**: the visibility extender of the consumer runs on a scheduled executor, so `changeMessageVisibilityBatch` answers without a latency. If it blocked, a delayed resume could make the extension arrive after the timeout it extends, which is not what SQS does.
    

Write this down in the fake’s Javadoc, as `FakeSqs` does.

### 5\. Let the seed pick the configuration

The seed picks the endpoint configuration as well as the timing, at the start of the run, and the world writes it to the trace:

```java
this.visibilityTimeout = dst.pick(List.of(2, 3, 5, 10));
this.maxMessagesPerPoll = dst.pick(List.of(1, 2, 5, 10));
this.concurrentConsumers = dst.pick(List.of(1, 2, 3));
this.delay = dst.pick(List.of(100L, 500L, 1000L));
dst.trace("world: visibilityTimeout=" + visibilityTimeout + " maxMessagesPerPoll=" + maxMessagesPerPoll + ...);
```

This found CAMEL-25737: with `visibilityTimeout=1` the extender is always late. It is left out of the default set while that issue is open, and `-Ddst.sqs.visibilityTimeouts=1` puts it back. The seed of a run is scrambled (SplitMix64), so consecutive seeds draw unrelated configurations.

### 6\. Seeded faults, a drain phase, and fault-excuse records

Inject the faults the service documents (failed calls, slow answers, lost requests) with `dst.chance(…​)`, at a rate that still lets work get done (the SQS world: 5 % failed receives and deletes, 2 % failed visibility changes, 15 % slow processing). Then switch the faults off and let the system drain:

```java
dst.runUntil(() -> sendsLeft == 0, "all sends");
dst.runFor(dst.between(0, 3L * visibilityTimeout * 1000));
// faults cleared: the queue must drain
faults = false;
sqs.faults = false;
// drain while it makes progress: stop when no message was deleted for a while, or at a cap
long quiet = 20L * visibilityTimeout * 1000 + 30_000;
long cap = dst.now() + 3_600_000;
int left = sqs.notDeleted().size();
long lastProgress = dst.now();
while (!sqs.allDeleted() && dst.now() < cap && dst.now() - lastProgress < quiet) {
    dst.runFor(100);
    int remaining = sqs.notDeleted().size();
    if (remaining < left) {
        left = remaining;
        lastProgress = dst.now();
    }
}
```

Drain until progress stops, not for a fixed time. With a fixed drain time, 1 of 20,000 seeds failed falsely: one consumer taking one message per poll was still deleting messages when the time ran out.

Some faults legitimately cause what the invariant forbids. Record which fault excuses which violation, and only those: a failed `ChangeMessageVisibilityBatch` lets SQS deliver the messages of that batch again, so `FakeSqs` reports their ids (`onExtensionFault`) and the world does not count a redelivery of those messages.

### 7\. The invariant comes from the component’s contract

Write the property the component promises, not what the code happens to do. The visibility extender promises that a message is not delivered again while the consumer holds it. The world records what happened in a `DstLedger` and checks:

```java
dst.check(
        Invariants.of("no message is delivered again while the consumer holds it",
                () -> redeliveredWhileHeld.isEmpty() ? null : redeliveredWhileHeld.toString()),
        Invariants.everyOfferedRoutedAtLeastOnce(ledger, "sent", "completed"),
        Invariants.of("the queue is drained once the faults cleared",
                () -> sqs.allDeleted() ? null : "not deleted: " + sqs.notDeleted()));
```

Start from the reusable `Invariants` (`everyOfferedRoutedAtLeastOnce`, `noDuplicates`, `lastCompletedAcknowledgedWhenQuiet`, `acknowledgedWithin`, `routeStarted`) and add the component’s own with `Invariants.of(name, check)`. A violation message carries the configuration, so the failure report is readable on its own.

### 8\. The self-check

A world is only evidence when it can fail. Before trusting a pass on `main`:

1.  **Fail on the code before a past fix.** Find a fixed bug of the same behaviour with `git log` on the component (the SQS world: CAMEL-25700, commit d09593b4a584). Revert the fix locally and run the world: it must fail (1397 of 5000 seeds), and only in the configurations the fix describes (for CAMEL-25700: `maxMessagesPerPoll >= 2` and `concurrentConsumers >= 2`).
    
2.  **Pass on main**, with thousands of seeds (`-Ddst.seeds=5000`).
    
3.  **Replay**: run a failing seed twice with `-Ddst.seed=<seed>`. The trace files in `target/dst/` must have the same SHA-256 (it is on their second line).
    

If the world cannot fail on the old code, its fake, its load or its invariant is too weak.

### 9\. Triage before reporting

A failing seed is not yet a bug. Before reporting one:

1.  Group the failing seeds by configuration (the `world:` line of each trace, or the violation message). A failure that appears only in one configuration points at that configuration (CAMEL-25737 was all `visibilityTimeout=1`).
    
2.  Read one trace from the start: what the world chose, then each step with its virtual time and executor.
    
3.  Tell three causes apart:
    
    -   a **harness artifact**: an escape (check the summary), the one-stack rule broken by the fake, a time read that is not rewritten (before the `Instant.now()` rewrite, half of the SQS seeds failed this way);
        
    -   a **fake mistake**: the fake does something the service documentation does not allow;
        
    -   a **real bug**: the consumer breaks its contract against a fake that follows the documentation.
        
    
4.  Reproduce a real bug outside DST too, with a plain unit test, before filing it.
    

### 10\. Describe the world with @DstWorld

Annotate the world class (the top-level class named after the file) with `@DstWorld`, so tools can find it: the build of the component reads the annotation from the test sources and writes the world into the `dstWorlds` section of the component’s JSON in the Camel catalog, with the DST tests of the same package that use it, and the table of worlds on this page is generated from the catalog.

```java
@DstWorld(component = "aws2-sqs",
          fake = "FakeSqs: a standard SQS queue (ReceiveMessage, DeleteMessage, ChangeMessageVisibilityBatch) bound as"
                 + " amazonSQSClient",
          covers = { "the visibility timeout extender (extendMessageVisibility=true)", ... },
          faults = { "ReceiveMessage fails", "ReceiveMessage answers slowly", "DeleteMessage is lost", ... },
          invariants = { "no message is delivered again while the consumer holds it", ... },
          issues = { "CAMEL-25700", "CAMEL-25737", "CAMEL-25738", "CAMEL-25911" })
final class SqsWorld {
```

`component` is the scheme of the component (a scheme the module does not have fails the build); the invariants are named as in `dst.check(…​)`; the faults are the ones the world injects, not the ones it could. The values must be string literals (joined with `+` if needed), as the build reads the source. Commit what the build regenerates: the component’s JSON (built with the component), and its copy in the catalog and this page (built with `camel-catalog`).

### Pitfalls seen when agents wrote worlds

Agents wrote nine worlds with this guide (not all of them are in the code base). These mistakes came up:

-   **Timestamps at 0.** Give the times of the fake virtual wall clock time (`Dst.EPOCH_MILLIS + dst.now()`, or `dst.currentTimeMillis()`, which follows the wall clock jumps of the run), not 0. A Service Bus fake with lock times in 1970 made the lock renewer renew on every run, so the world could not fail. The self-check (fail on the code before the fix) caught it.
    
-   **No end to the run.** A consumer that loops forever needs an unwind at the end of the run, or the run never ends: cancel its task through the task registry, or stop the route at the end of the drain phase. A consumer task that never returns (a pull loop) sits above the test thread on the one stack, so a `dst.runFor` or `dst.runUntil` in the test body does not return while it loops, and the seed fails on the step budget. Make the phases of the run (clear the faults, drain, end) scheduled tasks, and end the run by having the fake end the loop (close the subscription), then `dst.runUntil(() → closed, "subscription closed")`.
    
-   **A fake that blocks under a lock.** A call of the fake made while the consumer holds a lock must not block: with the one-stack rule, the tasks nested above it run while the lock is held.
    
-   **A violation without its excuse.** Every invariant needs the record of which injected faults excuse a violation of it. Without it, a message dead-lettered after injected acknowledgement faults showed up as a false failure.
    
-   **Old traces.** Clear `target/dst` between runs, or the traces of an earlier run end up in the grouping of the failing seeds.
    
-   **A loud test log.** A consumer looping under a buggy fake wrote a 31 GB test log and filled the disk. Set the component to `WARN` in the test `log4j2` configuration, or cap the log.
    
-   **A fixed configuration.** Let the seed pick the endpoint configuration, edge values included. That is how CAMEL-25737 (`visibilityTimeout=1`) and CAMEL-25741 (aws2-ddbstream, found with a fake DynamoDB Streams world) were found.
    

Later worlds (IMAP, JMS, NATS JetStream, Azure Service Bus) showed more:

-   **An empty answer at once.** A fake that answers at once with nothing (no message, no responder) lets a consumer with its own receive loop (JMS, a NATS fetch) spin without virtual time moving; the seed fails on the step budget or the watchdog instead of reaching its end. Give every call of the fake a round-trip latency.
    
-   **A proxy that throws what the interface does not declare.** A `java.lang.reflect.Proxy` keeps the fake of a wide interface (JMS) small. A checked exception that the interface method does not declare comes out of the proxy as `UndeclaredThrowableException`; do what the library does instead (for an `InterruptedException` in a fetch: set the interrupt flag again and return).
    
-   **Only the protocol.** Model what the client library does with the objects of an earlier connection, not only what the server does: after a reconnect, a message or handle from before may still work, fail, or now refer to something else. Read the library (its source, or its byte code) for that.
    
-   **One excuse for every stop.** Each kind of operator stop needs its own excuse record: a consumer stopped through JMX and a route stop that hits the shutdown timeout leave different work unfinished (a lock renewer cancelled under exchanges still in flight, for example).
    
-   **A synchronous consumer over an asynchronous step.** With `DstCamel.asyncStep` in the route, a consumer that calls the route synchronously waits on a latch of `org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager`: add it to `alsoRewrite`, or the watchdog fails the seed. Let the asynchronous exchanges in flight finish before `dst.dropPending()`, which drops their answers too, so that the shutdown does not wait for exchanges that cannot complete.
    

### Checklist

-   The module’s pom loads the Byte Buddy agent with `-javaagent` (see [Setup](#_setup)).
    
-   The client is swapped through an endpoint option or a static mock, at a synchronous level.
    
-   The escape scan was read; every escape is fixed, avoided or documented; the test runs in strict mode.
    
-   Every behaviour of the fake quotes the documentation; assumptions are marked; what is not modelled is listed.
    
-   Reads take effect at the end of their latency, writes at the start; background tasks do not block.
    
-   The seed picks the configuration, edge values included, and the world traces it.
    
-   The fake’s timestamps are virtual wall clock time (`Dst.EPOCH_MILLIS` based), not 0.
    
-   A consumer that loops forever is unwound at the end of the run.
    
-   Faults are seeded, switched off for a drain phase, and recorded where they excuse a violation.
    
-   The component logs at `WARN` in the test log configuration, and `target/dst` is cleared between runs.
    
-   The invariant states the component’s contract.
    
-   The world fails on the code before a past fix, passes on `main`, and a seed replays with the same SHA-256.
    
-   The world class has `@DstWorld`, and the regenerated component JSON and catalog are committed.
    

## Running the tests

The worlds are tests of their components, in a `dst` package. Build the harness in the same reactor as the component (`-pl` both), so that the component’s tests see it. `-Dsurefire.rerunFailingTestsCount=0` stops surefire from running each failing seed again. A component whose tests depend on a `camel-test-infra-*` module (camel-nats, camel-azure-storage-queue) needs that module in `-pl` too when it is not in the local repository.

```bash
DST=components/camel-test/camel-test-dst
SQS=components/camel-aws/camel-aws2-sqs
KINESIS=components/camel-aws/camel-aws2-kinesis
EH=components/camel-azure/camel-azure-eventhubs
SLACK=components/camel-slack
SHARE=components/camel-kafka-share
PULSAR=components/camel-pulsar
MONGODB=components/camel-mongodb
MVN="./mvnw -B -Papache-snapshots test -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.rerunFailingTestsCount=0"

# the harness self-tests
$MVN -pl $DST -Dtest='Dst*Test'

# the worlds, 1000 seeds each, then 5000 seeds without stopping at the first failing one
$MVN -pl $DST,$SQS,$KINESIS,$EH,$SLACK,$SHARE,$PULSAR,$MONGODB -Dtest='*DstTest'
$MVN -pl $DST,$SQS,$KINESIS,$EH,$SLACK,$SHARE,$PULSAR,$MONGODB -Dtest='*DstTest' -Ddst.seeds=5000 -Ddst.maxFailures=100000

# the self-check: revert the main code of a past fix locally (do not commit), run the world, restore
git show d09593b4a584 -- $SQS/src/main | git apply -R          # CAMEL-25700
$MVN -pl $DST,$SQS -Dtest=SqsVisibilityDstTest -Ddst.seeds=5000 -Ddst.maxFailures=100000
git checkout -- $SQS/src/main

# replay one seed twice and compare the traces
$MVN -pl $DST,$SQS -Dtest=SqsVisibilityDstTest -Ddst.seed=8
cp $SQS/target/dst/SqsVisibilityDstTest.*/seed-8.trace /tmp/seed-8.first
$MVN -pl $DST,$SQS -Dtest=SqsVisibilityDstTest -Ddst.seed=8
diff /tmp/seed-8.first $SQS/target/dst/SqsVisibilityDstTest.*/seed-8.trace && echo identical
```

For camel-mongodb (`MongoDbChangeStreamsDstTest`) the past fixes are the main code of 5f10c225652e (CAMEL-25946) and of 3af2c90d63ad (CAMEL-25679), reverted by hand. Its world also found CAMEL-26992 (fixed by 1eb59175e600): a cursor that the consumer opened again before any exchange completed started at the current time and skipped the events in between.

The past fixes of the other worlds are 83aaa80754f9 (CAMEL-25683, Event Hubs), 68329f0919bc (CAMEL-25684, Slack), the main code of b62e90ddb2d9 (CAMEL-25869, Pulsar), the main code of 20c94435789e (CAMEL-25872, Kafka share, whose world was also checked with planted faults), and the main code of 043ac9ec84ab (CAMEL-25569) and of 847ee9affb4c (CAMEL-25942) for Kinesis (`KinesisConsumerDstTest`). Some system properties belong to a world rather than to the harness: `dst.sqs.visibilityTimeouts` (the visibility timeouts the SQS world picks from, default `2,3,5,10`; `1` shows CAMEL-25737), `dst.replayCheckSeed` (the seed of the in-JVM replay test of the Event Hubs world, default 1), `dst.share.pollOnError` (the pollOnError strategies the Kafka share world picks from) and `dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar world, `0` runs without subscribe failures).

For camel-paho-mqtt5 (`components/camel-paho-mqtt5`, `PahoMqtt5ConsumerDstTest`) the past fixes are the main code of 6044f38166f9 (CAMEL-25944) and the change of b7594f06e0b7 (CAMEL-25793), taken out by hand, and the main code of CAMEL-27041 (a route stopped while the client connects). Its world has one property, which adds back a case that the default set leaves out: `dst.paho.strictCallbackThread` (`true` checks a client whose connect hangs while the callback thread is still in messageArrived, CAMEL-27043).

For camel-zookeeper-master (`-pl components/camel-zookeeper-master`, `ZooKeeperMasterDstTest`) the past fixes are `MasterConsumer` of e670025a49a4^ (CAMEL-25315), `MasterRoutePolicy` of 706beec6d5ab^ (CAMEL-26540), 136ffd929263 (CAMEL-26802) and 28f3170dcc6a (CAMEL-26887) reverted, and the `unregisterAsListener()` of 5ecab86a58ba (CAMEL-26870) taken out by hand. Its world has three properties, each adding back a case that the default set leaves out: `dst.zookeeper.stopWhileSuspended` (a route stopped while the client is disconnected, CAMEL-27037), `dst.zookeeper.stopDuringCreate` (a route stopped while the group waits for the answer of a create, or a create made after the close interrupted the thread of the group, CAMEL-27037) and `dst.zookeeper.startStartedRoute` (a started route of the policy started again, CAMEL-27044).

## Limits

-   Interleaving is per task: the harness does not preempt a task in the middle of a method.
    
-   The one-stack model above: a blocked task resumes only after the tasks that blocked after it.
    
-   Code on its own threads or on a third-party scheduler is outside the harness; strict mode makes reaching it fail the seed.
    
-   One run per JVM at a time: the active run, and so the virtual clock, is a static field, and every rewritten clock read in the JVM sees it, as does Camel core through `TimeSource` while the run is open. A world must therefore not run concurrently with other tests in a module with `camel.surefire.parallel=true`: the other tests in that surefire fork would read the virtual clock while a run is active. To host a DST test in a parallel module, tag the test with `@Tag("not-parallel")` and configure Surefire with two executions — a parallel pass excluding `not-parallel` and a serial pass (`<groups>not-parallel</groups>`) with `junit.jupiter.execution.parallel.enabled = false` in its `<configurationParameters>`. For the same reason, a `CamelContext` left running in the same fork by an earlier non-DST test would read the virtual time too and could fail a strict seed; stop it in that test. The harness enforces this: a second run while one is open fails, and a DST test refuses to start when JUnit runs tests in parallel.
    
-   Locks do not exclude: every task runs on the test thread, so a `ReentrantLock` or `synchronized` block held by a blocked task is entered again by the tasks nested above it. A critical section is atomic only because the fake does not block inside it, and races between two code paths that each hold a lock cannot be explored.
    
-   The rewriting stays for the rest of the surefire fork: the classes of the `@DstInstrument` packages (the component’s main classes, and its test classes in the same package) stay rewritten for every test that runs after the DST test, the same as the `alsoRewrite` entries. Outside a run they read the real clock and sleep for real, so the other tests of the component still pass.
    
-   The results are only as good as the fake of the external system.
    

## The classes

<table class="tableblock frame-all grid-all stretch"><colgroup><col> <col></colgroup><tbody><tr><td class="tableblock halign-left valign-top"><code>Dst</code></td><td class="tableblock halign-left valign-top">the event loop: virtual clock, wall clock jumps, seeded random source, step trace with its SHA-256, timed and untimed waits, strict mode</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstThreadPoolFactory</code>, <code>DstExecutor</code>, <code>DstTask</code></td><td class="tableblock halign-left valign-top">Camel’s pools and schedulers backed by the loop</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstInstrumentation</code>, <code>DstClock</code>, <code>DstWaits</code></td><td class="tableblock halign-left valign-top">the byte code rewrite of the code under test: time reads, sleeps and blocking waits go to the loop; <code>DstClock</code> is also the <code>TimeSource</code> provider of Camel core during a run</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstEscapes</code>, <code>DstWatchdog</code></td><td class="tableblock halign-left valign-top">the escape scan, runtime escapes (strict mode), and the watchdog for waits in code that is not rewritten</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstLedger</code>, <code>Invariant</code>, <code>Invariants</code></td><td class="tableblock halign-left valign-top">what happened to each message, and reusable invariants</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstTest</code>, <code>DstSeed</code>, <code>DstInstrument</code>, <code>DstExtension</code></td><td class="tableblock halign-left valign-top">the JUnit integration</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstWorld</code></td><td class="tableblock halign-left valign-top">the description of a world, which the build lists in the catalog</td></tr><tr><td class="tableblock halign-left valign-top"><code>DstCamel</code></td><td class="tableblock halign-left valign-top">a <code>CamelContext</code> on the loop, and an asynchronous route step with a seeded latency</td></tr></tbody></table>

## Options

  
| System property | Meaning | Default |
| --- | --- | --- |
| `dst.seeds` | number of seeds | `@DstTest(seeds)` |
| `dst.firstSeed` | first seed | `@DstTest(firstSeed)`, 1 |
| `dst.maxFailures` | stop after this many failing seeds | `@DstTest(maxFailures)`, 1 |
| `dst.seed` | comma separated seeds to replay; their full traces are written | \- |
| `dst.traceDir` | where the reports and traces go | `target/dst` |
| `dst.strict` | strict mode | `@DstInstrument(strict)`, true |
| `dst.alsoRewrite` | comma separated classes or packages outside the code under test to rewrite too | \- |
| `dst.watchdogMillis` | real time without progress after which a waiting loop thread fails the seed | 10000 |

Reports in `target/dst/`: `<Test>.escapes.txt` (the scan), `<Test>.summary.txt` (seeds run, failing seeds, threads left alive, runtime escapes) and `<Test>.<method>/seed-<n>.trace` (the trace of each failing or replayed seed, with its SHA-256 on the second line).

## Setup

Add the module with test scope:

```xml
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-test-dst</artifactId>
    <scope>test</scope>
</dependency>
```

The rewriting needs the Byte Buddy agent, loaded with `-javaagent` when the JVM of the tests starts. The harness does not attach it to the running JVM: the JDK warns about an agent loaded that way, and is to refuse it by default (JEP 451). Let the `properties` goal of the `maven-dependency-plugin` set the path of the agent jar (the property `net.bytebuddy:byte-buddy-agent:jar`; `byte-buddy-agent` comes with `camel-test-dst`), and add the agent to the surefire `argLine`. In a project of your own:

```xml
<properties>
    <!-- empty by default: a plugin such as JaCoCo sets it, and surefire adds it through @{argLine} -->
    <argLine/>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>dst-agent-path</id>
                    <goals>
                        <goal>properties</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <argLine>"-javaagent:${net.bytebuddy:byte-buddy-agent:jar}" @{argLine}</argLine>
            </configuration>
        </plugin>
    </plugins>
</build>
```

The `properties` goal runs in the `initialize` phase, so `mvn test` sets the property before surefire starts. Keep `@{argLine}` in the `argLine`, so that an agent another plugin sets (the JaCoCo agent of `prepare-agent`) is still loaded; the empty `<argLine/>` property keeps surefire from passing `@{argLine}` as it is when no plugin sets it. The quotes keep a path with a space in one argument (surefire splits the `argLine` at the spaces outside quotes).

In the Camel build, a module with DST tests adds the agent to `camel.surefire.fork.additional-vmargs`, which the `argLine` of the parent includes (with `@{argLine}`):

```xml
<properties>
    <camel.surefire.fork.additional-vmargs>"-javaagent:${net.bytebuddy:byte-buddy-agent:jar}"</camel.surefire.fork.additional-vmargs>
</properties>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>dst-agent-path</id>
                    <goals>
                        <goal>properties</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
```

Without the agent, every test class with `@DstInstrument` fails with "The byte code agent of camel-test-dst is not loaded". To run a DST test from an IDE, add `-javaagent:<path of byte-buddy-agent.jar>` to the JVM options of the run configuration.