Engine

The BPMN engine runs inside every Enterprise replica, on the same PostgreSQL database as the rest of the platform. It keeps its state in two schemas, engine and engine_catalog, which the backend migrates on startup like its own. There is no other service to run. When one database is no longer enough, the runtime data spreads over extra shard databases, see Scaling the engine.

How instances run

Every process instance lives on one partition, a logical slice of the engine's tables. The partition is part of every ID the engine hands out, for example bpmn-002a-... for an instance on partition 42.

At any moment one replica owns each partition, and only the owner advances its instances. It advances an instance in steps. A step reads the events waiting in the instance's inbox, runs the process until every token waits again, and commits the new state with its history, jobs, user tasks and incidents in one transaction.

Everything else only adds to an inbox:

• An API call (a start, a job completion, a user task completion, a resolve, a modification) stores an event in the instance's inbox, on whichever replica receives the call.
• One instance reaches another through an outbox. A call activity, a message, a signal and an operator command are written to the sender's outbox in the sender's step, and a relay moves them to the receiver's inbox.
• A message or signal goes first to a correlation partition chosen by its name and, for a message, its correlation value. That partition's owner matches it to the waiting instance or buffers it.

The owner steps an instance as soon as an event is stored on the same replica, and finds the events stored on other replicas and the timers that came due within ENGINE_POLL_INTERVAL.

Replicas

Replicas share the partitions through leases in the engine_catalog schema. Every replica takes an even share and renews it every few seconds. When a replica joins, the others hand over partitions until the shares are even again.

• A stopping replica finishes its running steps and hands its partitions back, so the other replicas take them over within seconds. REST_SHUTDOWN_TIMEOUT in Configuration bounds the shutdown.
• A replica that crashes keeps its partitions until its leases run out, after about 10 seconds. Its instances wait for that time and continue on another replica. No step is lost, since a step that did not commit left nothing behind.

Any replica answers any API call and any worker poll. A worker polls the jobs of every partition, whichever replica it reaches.

Partitions

The partition count is set when the engine first starts on an empty database and is fixed from then on. The default is 256 and the maximum is 65,536. The partitions live on the shards, which is the app database alone until you add a shard.

Leave ENGINE_PARTITIONS unset unless you plan for more than 256 replicas. Unset, a replica accepts the count the database already has. Set to a different count than the database has, the replica refuses to start with the count is fixed when the cluster bootstraps.

Scaling

PostgreSQL does the work: every step is one transaction, and every job poll and completion is a query. Size the database for the instance rate you expect, and watch its CPU and I/O before anything else.

• Replicas add step workers and spread the HTTP load. Each replica opens its own pool of up to DB_MAX_CONNS connections, so keep replicas times DB_MAX_CONNS under the database's max_connections.
• Step workers (ENGINE_STEP_WORKERS) are how many instances one replica steps at once. Each holds one pool connection while it steps, so keep the count well under DB_MAX_CONNS. Raise it when bpmn_engine_step_workers shows them busy while steps stay fast. When steps slow down as workers rise, the database is the limit.
• Shards add database capacity once the database is the limit. Scaling the engine covers when to add which and how to size them.

Operations lists the engine metrics and how to read them.

Settings

Variable	YAML key	Default	Purpose
ENGINE_PARTITIONS	engine.partitions	256	Partition count of a new database. Fixed once the engine has started on it.
ENGINE_SHARDS	engine.shards	none	Extra shard databases next to the app database, a JSON array of {"id", "url"} in the variable. See Configuring shards.
ENGINE_SHARD_MAX_CONNS	engine.shardMaxConns	ENGINE_STEP_WORKERS + 16	Connections one replica may open to each extra shard. A shard URL that sets pool_max_conns keeps its own.
ENGINE_REPLICA_ID	engine.replicaID	random	This replica's name in the partition lease table. A restart under the same name takes over its partitions at once instead of waiting for the leases to run out. Two running replicas must never share a name. The Helm chart sets the pod name.
ENGINE_STEP_WORKERS	engine.stepWorkers	8	Instances one replica steps at once. Keep it well under DB_MAX_CONNS.
ENGINE_POLL_INTERVAL	engine.pollInterval	100ms	How often a replica scans its partitions for due timers and for events stored on other replicas.
ENGINE_DEFINITION_CACHE	engine.definitionCache	100	Deployed process versions kept parsed in memory, and as many deployed DMN versions kept compiled.
ENGINE_MAX_CALL_DEPTH	engine.maxCallDepth	64	How deep call activities may nest. A call past it raises an incident on the call activity.
ENGINE_JOB_RETRY_DELAY	engine.jobRetryDelay	10s	How long a job a worker failed waits before it is offered again, while it has retries left. ENGINE_JOB_RETRY_DELAY=0s offers it again at once.
ENGINE_JOB_MAX_HOLD	engine.jobMaxHold	30m	The longest one worker attempt can keep a job by heartbeating. Reaching it counts as a failed attempt. ENGINE_JOB_MAX_HOLD=0s removes the limit.
ENGINE_WORKER_LOCK_MIN	engine.workerLockMin	10s	Lower bound of the lockDuration a worker asks for.
ENGINE_WORKER_LOCK_MAX	engine.workerLockMax	5m	Upper bound of the lockDuration a worker asks for.
ENGINE_WORKER_LOCK_DEFAULT	engine.workerLockDefault	30s	The lock a worker gets when it asks for none.
ENGINE_RETENTION_MAX_AGE	engine.retention.maxAge	2160h	How long ended instances are kept. See Data retention.
ENGINE_RETENTION_DRAFT_MAX_AGE	engine.retention.draftMaxAge	168h	How long test runs from the modeler are kept.

A zero takes effect only as an environment variable, see Configuration file.

External workers describes what happens when a worker's lock runs out.

In the Helm chart, the retention settings are typed values under config.engine.retention, the shards come from engineShards and their pool size from config.engine.shardMaxConns. Set the others through backend.env:

backend:
  env:
    ENGINE_STEP_WORKERS: "16"