How a BPMN Engine Works, and What You Can Build With One

BPMN engines can look intimidating at first. There are process diagrams, several kinds of events and tasks, gateways, variables, integrations, and a whole ecosystem of concepts that make it hard to know where to start.

This article goes back to basics. First, how a BPMN engine actually runs a process. Then, more importantly, what you can build with one: from simple workflows to processes involving data, external systems, and automation.

Let's start with the basics.

How a process flows

To understand how a BPMN engine works, it helps to stop thinking of the diagram as something the engine "executes" top to bottom. Instead, think of the process as a map through which tokens travel.

Tokens

A token represents the current execution of a process. It moves from one element of the diagram to the next, and as it moves, the engine keeps track of where the token is, what has happened so far, and what data belongs to that execution.

Here is the simplest possible example:

In this process we fetch customer data from an external system, then send a welcome email to the address that lookup returned.

When an instance starts, the engine creates a token at the start event. The token then moves through the process:

At each step the engine decides what the token should do next. With a single flow there is not much deciding to do, so once an activity completes, the token continues along the outgoing sequence flow.

Process state

A running process also carries state and a data scope.

Suppose our process starts with this input:

{
   "customerId": 123
}

The token starts with access to this data. When it reaches Get Customer Data, that step produces additional information, whether by calling a database, hitting an external service, or asking a person:

{
   "email": "customer_mail@mail.com",
   "firstName": "Customer",
   "lastName": "Test",
   "phoneNumber": "123456789"
}

The important part is that the task did not just move the token forward. It changed the state of the process by producing data that later steps can use. The next task, Send Welcome Email, can now read what Get Customer Data filled in.

In QuantumBPM

Variables live in a scope chain rather than one flat bag. Each sub-process, multi-instance iteration, and boundary handler gets its own scope, and a FEEL expression resolves a name from the innermost scope outward. Writes inside a scope stay there unless you opt in to propagate them. See Variables and scope.

Tasks and workers

A task is a step where there is work to be done. A task can:

• queue up a job for a worker to execute,
• run a script,
• ask a user to provide information,
• wait for an event or another process to supply something.

The work itself is usually delegated to a human user or a software worker. Workers are typically small applications, written in any language, that poll the engine for available jobs.

This is by design. Keeping the actual work outside the engine gives developers freedom to choose their own tools and languages, and it makes it easier to scale individual parts of a process independently when more performance is needed.

A worker might call an API, prompt an AI agent, query a database, calculate something, send an email, update a record, or generate a document.

In QuantumBPM

Workers pull jobs over long-polling HTTP, so anything that can speak HTTP can be a worker. Each worker registers for one or more task types and only ever sees jobs for those types. There are SDKs for Go, Java, JavaScript, and Python, and the full contract is in External workers.

One process, multiple tokens

A process can have several tokens running at once. This usually happens when the flow splits into parallel paths:

Each path gets its own token, but both tokens still belong to the same process instance. Work done by one token can affect the overall process state and the data available to the other.

For example, both tasks might modify the same process variables:

{
  "customerId": 123,
  "invoiceReady": true,
  "shipmentReady": true
}

That is useful, but it means you have to be careful with shared state. If two parallel tokens modify the same variable at the same time, the result depends on which update lands last.

The same applies to scope. Some data belongs to the whole instance, other data is scoped to a particular execution or activity. When working with parallel paths it is important to know where a variable lives and who is allowed to modify it. A good rule of thumb: avoid having parallel paths modify the same variables unnecessarily, and where they genuinely need to share data, make the ownership and the expected updates explicit.

Once the parallel paths meet again, the engine joins their executions and continues with the resulting state.

The engine loop

For a simple process, you can picture the engine repeatedly doing something like:

1. Where is the token?
2. What element is it at?
3. What needs to happen there?
4. What state and data does that activity produce?
5. Where should the token go next?
6. Repeat.

Real engines have considerably more machinery around this: persistence, jobs, timers, events, parallel execution, retries, and external workers. But with this model in place, the more advanced concepts get much easier to follow.

What can you build with a BPMN engine?

Now that the mechanics are clear, the more interesting question is what you can actually build. Here are seven patterns worth knowing.

AI-powered processes

AI is a good example of something you can use inside a business process without making the AI responsible for the entire workflow.

Consider a customer support process:

The AI worker could classify the request, extract relevant information, summarize the conversation, or draft a suggested response. The BPMN process stays responsible for the overall flow. It can define that simple requests are handled automatically while complex ones require a human to review the result.

That distinction matters. The AI does not need to own the workflow. It can simply be one of the workers participating in it.

The same pattern works for extracting information from invoices, classifying incoming applications, summarizing conversations, generating document drafts, checking whether a request contains the required information, enriching data, or routing cases to the right team.

The result is a process where AI handles the parts that benefit from probabilistic reasoning, while the BPMN model controls the business flow.

In QuantumBPM

An AI step is an ordinary service task, so the model stays deterministic and replayable even though the model call is not. Every prompt, response, and routing decision is recorded in the instance history for audit. We covered the full pattern, including ad-hoc sub-processes for reasoning loops, in Orchestrating AI agents with BPMN.

Error handling and retries

Another practical use case is dealing with failure. In real applications external calls fail constantly. APIs time out, services go down, workers crash, connections disappear.

Imagine a process that charges a customer's payment method:

The process defines what happens when the payment service is temporarily unavailable. The first failure might trigger a retry after a few seconds. A second failure could retry after a longer delay. After several unsuccessful attempts the process could take an alternative path and create a human task.

The important part is that the retry is not a loop buried in application code. It is part of the process. The engine persists the state while the process waits for the next attempt, so no application thread has to stay alive just because something should happen again in five minutes.

The same approach helps with retrying API calls, waiting for a service to recover, handling failed document generation, retrying notifications, recovering from temporary infrastructure problems, and escalating to a human after repeated attempts. It gets particularly valuable when different failures need different recovery strategies.

In QuantumBPM

Retries come at two levels. A quantum:taskDefinition retries count handles transient worker failures automatically, and when it runs out the instance raises an incident that an operator can inspect and resolve from the Operations UI. For backoff and escalation you model it explicitly, as above, with timer and error boundary events.

Long-running business processes

Some processes do not fit the request-response model at all. A customer might start an application today, provide extra information tomorrow, wait days for external verification, and finally need a human approval.

This might take minutes, days, or weeks. A BPMN engine is built to keep track of the process while it waits. No code runs continuously while waiting for a document or a person. The engine persists the state and resumes when the awaited event arrives.

That makes BPMN a good fit for loan applications, insurance claims, employee and customer onboarding, procurement, compliance reviews, contract approval, mortgage processing, and subscription lifecycles.

In QuantumBPM

Durability comes from Temporal underneath the BPMN layer, so a process waiting fourteen days for a document costs nothing while it waits and survives restarts and deploys. Human steps are user tasks with their own lifecycle and API, which you drive from your own inbox UI.

Orchestrating multiple systems

Another common use is coordinating several services that must work together. Imagine an e-commerce order:

These operations might live in completely different systems. Payment could be owned by one team, inventory by another, shipping by an external provider. The BPMN engine acts as the coordinator. It does not necessarily perform the work itself, it tells the right workers what needs to happen and keeps track of the whole.

This becomes especially useful with asynchronous operations. The payment provider might respond immediately while the shipping provider takes several minutes to confirm. The process continues where it can and waits where it must.

In QuantumBPM

Cross-system waits are message events with correlation keys, so an external system can resume the right instance out of thousands by publishing a message. Processes can also call other processes through call activities, which keeps each system's flow in its own diagram.

Document processing

Document-heavy processes are another natural fit. Consider invoice processing:

Different workers handle different parts. One extracts data from a PDF, another validates it against company records, an AI worker classifies it. If something does not look right, the process creates a human task instead of failing outright.

The same shape applies to contracts, applications, claims, forms, identity documents, and purchase orders. The advantage is that the pipeline becomes an explicit process rather than a collection of hidden callbacks and background jobs.

In QuantumBPM

Keep the documents themselves out of the process. Variables are capped by Temporal's per-event size limit, so store files externally and put a reference in the variable, as described under supported types. Validation rules that business users need to change are a good fit for DMN decision tables rather than worker code.

Scheduled and time-based processes

BPMN also helps when something needs to happen exactly on time. Take a subscription:

Instead of building a separate scheduling mechanism for every business requirement, the process states explicitly what should happen and when.

This suits reminders, renewals, deadlines, escalations, scheduled notifications, payment collection, periodic checks, and expiration handling.

In QuantumBPM

Timers accept ISO 8601 in all three forms: timeDuration to wait a fixed span, timeDate for an exact moment, and timeCycle such as R/PT1H to fire repeatedly. Cyclic timers work as start events for recurring processes and as non-interrupting boundary events for nudges that must not cancel the work underneath. See Events.

Saga-like distributed processes

BPMN also helps when an operation spans several systems and there is no single transaction that can atomically roll everything back.

What happens if payment succeeds but shipment creation fails? You cannot roll everything back with a database transaction when the operations happen in different systems. Instead, the process defines compensating actions:

The BPMN process models these failure and compensation paths explicitly. This is what you want for distributed business operations where consistency comes from coordination and compensation rather than one large transaction.

In QuantumBPM

Compensation is a first-class engine feature, not something you hand-roll. Handlers run in reverse completion order against a snapshot of the variables as they were when the host activity finished, and compensation propagates across call activities and sub-processes, which is where Camunda 8 stops. See Compensation and our deeper write-up, The saga pattern in BPMN.

Putting it all together

Each of these patterns is useful on its own. They get considerably more interesting in combination, and that is where a BPMN engine like QuantumBPM earns its place. A single process can pull an AI classification, a retry with backoff, a two week wait for a human, a compensating rollback, and a recurring timer into one model that you can read, version, and audit.

QuantumBPM is built for exactly that: BPMN and DMN on top of Temporal's durable execution, workers in any language over plain HTTP, and an operations UI where you can watch a live instance, inspect its variables, and resolve incidents. If you want to try the patterns from this article, the getting started guide will get you a running process quickly.

With the right mental model and a capable engine, BPMN stops being just a way to draw processes. It becomes a practical foundation for building scalable, reliable, and intelligent business automation.

Frequently Asked Questions

How does a BPMN engine work?

A BPMN engine treats the diagram as a map rather than a script. It creates a token at the start event and moves it from element to element, and at each step it asks the same few questions: where is the token, what element is it at, what work happens there, what data does that produce, and where does the token go next. Around that loop sits the machinery that makes it durable: persistence, jobs, timers, events, retries, and external workers.

What is a token in BPMN?

A token represents one execution of a process. It marks where the process currently is, and the engine tracks what has happened so far and what data belongs to that execution. A process instance can hold several tokens at once, which is what happens when a parallel gateway splits the flow. All of those tokens still share the same instance state, so parallel paths that write the same variable can overwrite each other.

How does a BPMN engine handle long-running workflows and state management?

It persists the process state and stops executing between steps. A process waiting three days for a document or a human approval is not holding a thread or a database connection, it is durable state that the engine resumes when the awaited event arrives. Variables live in a scope chain rather than one flat bag, so sub-processes and parallel branches get their own scope and have to opt in to write outward. In QuantumBPM that durability comes from Temporal under the BPMN layer, so instances survive restarts and deploys.

What is the difference between a BPMN engine and a task queue?

A task queue moves individual units of work. A BPMN engine owns the flow between those units: the order, the branching conditions, the timers, the error and compensation paths, and the state that carries across them. The two are complementary rather than competing, and a BPMN engine usually has a queue inside it. The difference shows up when you need to answer what happened on a specific instance six weeks ago, or resume something that was waiting for a person.

Do I need to write external workers to run a BPMN process?

For anything that touches the outside world, yes, and that is deliberate. The engine dispatches a job and a worker picks it up, which keeps your business logic in your own language and lets you scale individual steps independently. In QuantumBPM workers pull jobs over long-polling HTTP, so anything that speaks HTTP qualifies, and there are SDKs for Go, Java, JavaScript and Python. Steps that are pure decisions do not need a worker at all, they can be DMN decision tables evaluated by the engine.