Control Plane Metrics

This page consolidates Testkube Control Plane Prometheus metrics, including scheduler metrics.

In v2.7+, Control Plane metrics are enabled by default with:

• TK_CLOUD_METRICS_ENABLED=true

For context on the source-of-truth migration in v2.7, see Testkube Resource Management.

Availability and Enablement

Control Plane metrics are available in Testkube Cloud and Enterprise connected deployments and are exposed on the control plane /metrics endpoint.

• Path: /metrics on the Control Plane API service.
• TK_CLOUD_METRICS_ENABLED controls control-plane metrics emission (default true in v2.7+).
• TK_SCHEDULER_METRICS_ENABLED controls scheduler metrics emission (default true).
• TK_SCHEDULER_METRICS_INCLUDE_EXECUTION_ID enables execution-identity queue metric emission (default false).
• TK_WORKFLOW_CUSTOM_LABEL_METRICS_ENABLED enables exporting custom TestWorkflow labels as Prometheus metadata metrics (default false).
• TK_WORKFLOW_CUSTOM_LABEL_METRICS_MAX_PER_RUN caps how many custom labels are exported per workflow execution (default 10).

To enable custom workflow label metrics, set:

testkube-cloud-api:
  api:
    workflowCustomLabelMetricsEnabled: true
    workflowCustomLabelMetricsMaxPerRun: 10

Label Model (Control Plane)

• environment_name.
• workflow_namespace on workflow execution metrics and on the lifecycle (CRUD) counters. See What workflow_namespace reports.
• runner_id (stable), runner_name (readable).
• No dynamic maps/arrays: dropped labels, tags, testworkflow_uri, trigger causes.
• triggered_by is normalized to manual, schedule, or trigger.
• CRUD counters drop resource names to keep cardinality bounded.
• testkube_testworkflow_custom_label_info is optional and exports selected custom workflow labels as label_key / label_value.

What workflow_namespace reports

workflow_namespace is the namespace of the Kubernetes object behind the metric, not the namespace of whoever triggered the action. What ends up in it depends on how the resource was created.

Applied as a manifest (kubectl apply, Argo CD, Flux, or any other GitOps tool): the label carries the namespace the manifest was applied into. Teams that each own a namespace are distinguishable here, so this is the path to use when you want per-team attribution inside a shared cluster.

Created through the dashboard or the API: the agent assigns its own namespace to the resource before storing it, regardless of what the request contained. Every resource created this way therefore reports the namespace the agent runs in, which is a single value per environment. It tells you the resource came in through the agent; it does not separate one team from another. To split activity by team for resources created this way, give each team its own environment and group by environment_name.

Environments where the Control Plane is the source of truth: no Kubernetes object exists, so there is no namespace to report and the label is empty. See Testkube Resource Management for which environments these are.

Available Metrics

Metric	Type	Description	Labels
Scheduler
testkube_scheduler_environment_queue_limit	gauge	Configured queue limit for the environment	environment_name
testkube_scheduler_queued_executions	gauge	Number of queued executions awaiting assignment	environment_name
testkube_scheduler_queued_executions_by_workflow	gauge	Number of queued executions awaiting assignment, grouped by workflow name	environment_name, workflow_name
testkube_scheduler_queued_execution_info	gauge	Identity metric for queued executions (value is always 1); emitted only when TK_SCHEDULER_METRICS_INCLUDE_EXECUTION_ID=true	environment_name, workflow_name, execution_id, execution_name
testkube_scheduler_active_executions	gauge	Number of currently active executions	environment_name
testkube_scheduler_queue_oldest_age_seconds	gauge	Age in seconds of the oldest queued execution (0 when none)	environment_name
testkube_scheduler_runner_active_executions	gauge	Active executions attributed to each runner	environment_name, runner_id, runner_name
Workflow Executions
testkube_testworkflow_executions_total	counter	Total number of test workflow executions	environment_name, workflow_name, workflow_namespace, workflow_result, triggered_by, runner_id, runner_name
testkube_testworkflow_executions_duration_seconds	histogram	Duration of test workflow executions	environment_name, workflow_name, workflow_namespace, workflow_result, triggered_by, runner_id, runner_name
testkube_testworkflow_aborts_total	counter	Total number of test workflow aborts	environment_name, workflow_name, workflow_namespace, workflow_result, triggered_by, runner_id, runner_name
testkube_testworkflow_custom_label_info	gauge	Optional metadata metric exporting selected custom TestWorkflow labels	execution labels + label_key, label_value
Workflow Execution Steps
testkube_testworkflow_execution_steps_total	counter	Total number of test workflow execution steps	execution labels + step_name, step_status
testkube_testworkflow_execution_steps_duration_seconds	histogram	Duration of test workflow execution steps	execution labels + step_name, step_status
testkube_testworkflow_execution_steps_start_time_seconds	histogram	Start time of test workflow execution steps	execution labels + step_name, step_status
testkube_testworkflow_execution_steps_finish_time_seconds	histogram	Finish time of test workflow execution steps	execution labels + step_name, step_status
Workflow Lifecycle (CRUD)
testkube_testworkflow_creations_total	counter	Total number of test workflows created	environment_name, result, workflow_namespace
testkube_testworkflow_updates_total	counter	Total number of test workflows updated	environment_name, result, workflow_namespace
testkube_testworkflow_deletes_total	counter	Total number of test workflows deleted	environment_name, result, workflow_namespace
Workflow Templates
testkube_testworkflowtemplate_creations_total	counter	Total number of test workflow templates created	environment_name, workflow_template, result, workflow_namespace
testkube_testworkflowtemplate_updates_total	counter	Total number of test workflow templates updated	environment_name, workflow_template, result, workflow_namespace
testkube_testworkflowtemplate_deletes_total	counter	Total number of test workflow templates deleted	environment_name, workflow_template, result, workflow_namespace
Triggers
testkube_testtrigger_creations_total	counter	Total number of test trigger created events	environment_name, result, workflow_namespace
testkube_testtrigger_events_total	counter	Total number of TestTrigger-fired workflow executions	environment_name, trigger_name, workflow_namespace
testkube_testtrigger_updates_total	counter	Total number of test trigger updated events	environment_name, result, workflow_namespace
testkube_testtrigger_deletes_total	counter	Total number of test trigger deleted events	environment_name, result, workflow_namespace
testkube_testtrigger_bulk_updates_total	counter	Total number of test trigger bulk update events	environment_name, result
testkube_testtrigger_bulk_deletes_total	counter	Total number of test trigger bulk delete events	environment_name, result
Webhooks
testkube_webhook_executions_total	counter	Total number of webhook executions	environment_name, webhook_name, event_type, result
testkube_webhook_creations_total	counter	Total number of webhook create attempts	environment_name, result, workflow_namespace
testkube_webhook_updates_total	counter	Total number of webhook update attempts	environment_name, result, workflow_namespace
testkube_webhook_deletes_total	counter	Total number of webhook delete attempts	environment_name, result, workflow_namespace

Examples

# scheduler queue depth
testkube_scheduler_queued_executions{environment_name="prod-env"} 4

# queued executions by workflow
testkube_scheduler_queued_executions_by_workflow{
  environment_name="prod-env",
  workflow_name="metrics-demo"
} 2

# queued execution identity (requires TK_SCHEDULER_METRICS_INCLUDE_EXECUTION_ID=true)
testkube_scheduler_queued_execution_info{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  execution_id="01904b4d-4f8a-7f98-9dd7-991820db74d8",
  execution_name="metrics-demo-42"
} 1

# workflow execution (control plane)
testkube_testworkflow_executions_total{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  workflow_namespace="testkube",
  workflow_result="passed",
  triggered_by="manual",
  runner_id="tkc_run_123",
  runner_name="runner-a"
} 1

# workflow custom label metadata (requires TK_WORKFLOW_CUSTOM_LABEL_METRICS_ENABLED=true)
testkube_testworkflow_custom_label_info{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  workflow_namespace="testkube",
  workflow_result="passed",
  triggered_by="manual",
  runner_id="tkc_run_123",
  runner_name="runner-a",
  label_key="team",
  label_value="platform"
} 1

# workflow step duration (seconds, histogram)
testkube_testworkflow_execution_steps_duration_seconds_bucket{
  environment_name="prod-env",
  workflow_name="metrics-demo",
  workflow_namespace="testkube",
  workflow_result="passed",
  triggered_by="manual",
  runner_id="tkc_run_123",
  runner_name="runner-a",
  step_name="build",
  step_status="passed",
  le="1"
} 0

# trigger-fired workflow execution
testkube_testtrigger_events_total{
  environment_name="prod-env",
  trigger_name="smoke-on-deploy",
  workflow_namespace="team-a"
} 1

# webhook execution
testkube_webhook_executions_total{
  environment_name="prod-env",
  webhook_name="notify",
  event_type="END_TESTWORKFLOW_SUCCESS",
  result="success"
} 3

# webhook create, applied as a manifest into the team's own namespace
testkube_webhook_creations_total{
  environment_name="prod-env",
  result="created",
  workflow_namespace="team-a"
} 1

# webhook create through the dashboard, stamped with the namespace the agent runs in
testkube_webhook_creations_total{
  environment_name="prod-env",
  result="created",
  workflow_namespace="testkube"
} 1

Runner to Control-Plane Mapping

Every Runner metric has a corresponding Control Plane metric. The mapping follows a small set of consistent conventions described below.

Naming Conventions

• Counter suffix: Runner metrics ending in _count become _total on the Control Plane (e.g. testkube_testworkflow_executions_count → testkube_testworkflow_executions_total).
• Trigger name normalization: Runner trigger metrics using the plural testtriggers become singular testtrigger on the Control Plane (e.g. testkube_testtriggers_updates_count → testkube_testtrigger_updates_total).
• Duration units: Runner timing metrics in milliseconds (_ms) become histograms in seconds (_seconds) on the Control Plane for cross-replica aggregation (e.g. testkube_testworkflow_executions_duration_ms → testkube_testworkflow_executions_duration_seconds).

Label Changes

All Control Plane metrics gain environment_name for environment-level scoping. Beyond that:

• Workflow execution and step metrics gain runner_id, runner_name, and workflow_namespace for finer scoping, and rename runner labels for clarity: name → workflow_name, result → workflow_result, status → step_status.
• triggered_by is normalized to one of manual, schedule, or trigger.
• High-cardinality labels are dropped: labels, tags, testworkflow_uri, and trigger causes are not carried to the Control Plane.
• CRUD counters use only environment_name and result (workflow names are omitted to keep cardinality bounded). Template metrics additionally include workflow_template.
• Webhook metrics rename name → webhook_name and eventType → event_type.
• Optional custom workflow label metadata is exposed separately via testkube_testworkflow_custom_label_info using label_key and label_value.

Metrics Not Mapped

Runner-only legacy test and test-suite metric families (testkube_test_*, testkube_testsuite_*) do not have Control Plane equivalents.

Full Mapping Reference

Runner Metric	Control-Plane Metric
testkube_testworkflow_executions_count	testkube_testworkflow_executions_total
testkube_testworkflow_executions_duration_ms	testkube_testworkflow_executions_duration_seconds
testkube_testworkflow_aborts_count	testkube_testworkflow_aborts_total
testkube_testworkflow_execution_steps_count	testkube_testworkflow_execution_steps_total
testkube_testworkflow_execution_steps_duration_ms	testkube_testworkflow_execution_steps_duration_seconds
testkube_testworkflow_execution_steps_start_time_ms	testkube_testworkflow_execution_steps_start_time_seconds
testkube_testworkflow_execution_steps_finish_time_ms	testkube_testworkflow_execution_steps_finish_time_seconds
testkube_testworkflow_creations_count	testkube_testworkflow_creations_total
testkube_testworkflow_updates_count	testkube_testworkflow_updates_total
testkube_testworkflow_deletes_count	testkube_testworkflow_deletes_total
testkube_testworkflowtemplate_creations_count	testkube_testworkflowtemplate_creations_total
testkube_testworkflowtemplate_updates_count	testkube_testworkflowtemplate_updates_total
testkube_testworkflowtemplate_deletes_count	testkube_testworkflowtemplate_deletes_total
testkube_testtrigger_creations_count	testkube_testtrigger_creations_total
testkube_testtrigger_event_count	testkube_testtrigger_events_total
testkube_testtriggers_updates_count	testkube_testtrigger_updates_total
testkube_testtriggers_deletes_count	testkube_testtrigger_deletes_total
testkube_testtriggers_bulk_updates_count	testkube_testtrigger_bulk_updates_total
testkube_testtriggers_bulk_deletes_count	testkube_testtrigger_bulk_deletes_total
testkube_webhook_executions_count	testkube_webhook_executions_total