Importer & Archiver. Scaling Operate

Operate consists of three modules:

  • Webapp - contains the UI and operation executor functionality
  • Importer - is responsible for importing data from Zeebe
  • Archiver - is responsible for archiving "old" data (finished workflow instances and user operations) (see Data retention).

Modules can be run together or separately in any combination and can be scaled. When you run Operate instance, by default, all modules are enabled. To disable them you can use following configuration parameters:

Configuration parameterDescriptionDefault value
camunda.operate.importerEnabledWhen true, Importer module is enabledtrue
camunda.operate.archiverEnabledWhen true, Archiver module is enabledtrue
camunda.operate.webappEnabledWhen true, Webapp module is enabledtrue

Additionally you can have several importer and archiver nodes to increase throughput. Internally they will spread their work based on Zeebe partitions.

E.g. if your Zeebe runs 10 partitions and you configure 2 importer nodes, they will import data from 5 partitions each. Each single importer/archiver node must be configured with the use of following configuration parameters:

Configuration parameterDescriptionDefault value
camunda.operate.clusterNode.partitionIdsArray of Zeebe partition ids, this Importer (or Archiver) node must be responsible forempty array, meaning all partitions data is loaded
camunda.operate.clusterNode.nodeCountTotal amount of Importer (or Archiver) nodes in the cluster1
camunda.operate.clusterNode.currentNodeIdId of current Importer (or Archiver) node, starting from 00

It's enough to configure either partitionIds or pair of nodeCount and currentNodeId. In case you provide nodeCount and currentNodeId, each node will automatically guess Zeebe partitions it is responsible for.

Note nodeCount always represents the number of nodes of one specific type.

E.g. configuration of the cluster with 1 Webapp node, 2 Importer nodes and 1 Archiver node could look like this:

Webapp node

camunda.operate:
  archiverEnabled: false
  importerEnabled: false
  #other configuration...

Importer node #1

camunda.operate:
  archiverEnabled: false
  webappEnabled: false
  clusterNode:
    nodeCount: 2
    currentNodeId: 0
  #other configuration...
  
Importer node #2

camunda.operate:
  archiverEnabled: false
  webappEnabled: false
  clusterNode:
    nodeCount: 2
    currentNodeId: 1
  #other configuration...
  
Archiver node

camunda.operate:
  webappEnabled: false
  importerEnabled: false
  

You can further parallelize archiver and(or) importer within one node by using following configuration parameters:

Configuration parameterDescriptionDefault value
camunda.operate.archiver.threadsCountNumber of threads, in which data will be archived1
camunda.operate.importer.threadsCountNumber of threads, in which data will be importe3

Note Parallelization of import and archiving within one node will also happen based on Zeebe partitions, meaning that only configurations with (number of nodes) * (threadsCount) <= (total number of Zeebe partitions) will make sense. Too many threads and nodes will still work, but some of them will be idle.