Get Pipeline

GET/v0/pipelines/{pipeline_name}

Authentication

JSON web token (JWT) or API key (http bearer)

Parameters

pipeline_name (path, string, required)

selector (query, string)

Retrieve a pipeline. Configure which fields are included using the selector query parameter.

Authentication

Scheme	Type	Details
`JSON web token (JWT) or API key`	Bearer token	Authorization header · Use a JWT token obtained via an OAuth2/OIDC login workflow or an API key obtained via the `/v0/api-keys` endpoint.

Parameters

Path parameters

Name	Type	Required	Description
`pipeline_name`	string	Yes	Unique pipeline name

Query parameters

Name	Type	Required	Description
`selector`	PipelineFieldSelector	No	The `selector` parameter limits which fields are returned for a pipeline. Limiting which fields is particularly handy for instance when frequently monitoring over low bandwidth connections while being only interested in pipeline status.

Response

application/json

Pipeline retrieved successfully

object
Pipeline information which has a selected subset of optional fields. It both includes fields which are user-provided and system-generated. If an optional field is not selected (i.e., is `None`), it will not be serialized.
- connectorsobject
- created_atstring (date-time)required
- deployment_desired_statusstringrequired
- deployment_desired_status_sincestring (date-time)required
- deployment_errorobject
- deployment_idstring (uuid)
- deployment_initialobject
- deployment_resources_desired_statusstringrequired
- deployment_resources_desired_status_sincestring (date-time)required
- deployment_resources_statusstringrequired
  Pipeline resources status. ```text /start (early start failed) ┌───────────────────┐ │ ▼ Stopped ◄────────── Stopping /start │ ▲ │ │ /stop?force=true │ │ OR: timeout (from Provisioning) ▼ │ OR: fatal runtime or resource error ⌛Provisioning ────────────│ OR: runtime status is Suspended │ │ │ │ ▼ │ Provisioned ─────────────┘ ``` ### Desired and actual status We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the **desired** status, which represents what the user would like the pipeline to do, and the **current** status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards. There are two desired statuses: - `Provisioned` (set by invoking `/start`) - `Stopped` (set by invoking `/stop?force=true`) The user can monitor the current status of the pipeline via the `GET /v0/pipelines/{name}` endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the `/start` endpoint, and then polls the `GET /v0/pipelines/{name}` endpoint to monitor the actual status of the pipeline until its `deployment_resources_status` attribute changes to `Provisioned` indicating that the pipeline has been successfully provisioned, or `Stopped` with `deployment_error` being set.
- deployment_resources_status_detailsobject
- deployment_resources_status_sincestring (date-time)required
- deployment_runtime_desired_statusobject
- deployment_runtime_desired_status_sincestring (date-time)
- deployment_runtime_statusobject
- deployment_runtime_status_detailsobject
- deployment_runtime_status_sincestring (date-time)
- deployment_statusstringrequired
- deployment_status_sincestring (date-time)required
- descriptionstringrequired
- idstring (uuid)required
  Pipeline identifier.
- namestringrequired
- platform_versionstringrequired
- program_codestring
- program_configobject
- program_errorobject
- program_infoobject
- program_statusstringrequired
  Program compilation status.
- program_status_sincestring (date-time)required
- program_versioninteger (int64)required
  Version number.
- refresh_versioninteger (int64)required
  Version number.
- runtime_configobject
- storage_statusstringrequired
  Storage status. The storage status can only transition when the resources status is `Stopped`. ```text Cleared ───┐ ▲ │ /clear │ │ │ │ Clearing │ ▲ │ │ │ InUse ◄───┘ ```
- storage_status_detailsobject
- udf_ruststring
- udf_tomlstring
- versioninteger (int64)required
  Version number.

curl -X GET '/v0/pipelines/{pipeline_name}?selector=%3Cselector%3E' \
  -H 'Authorization: Bearer YOUR_TOKEN'

Pipeline retrieved successfully


{
  "id": "67e55044-10b1-426f-9247-bb680e5fe0c8",
  "name": "example1",
  "description": "Description of the pipeline example1",
  "created_at": "1970-01-01T00:00:00Z",
  "version": 4,
  "platform_version": "v0",
  "runtime_config": {
    "workers": 16,
    "max_rss_mb": null,
    "hosts": 1,
    "storage": {
      "backend": {
        "name": "default"
      },
      "min_storage_bytes": null,
      "min_step_storage_bytes": null,
      "compression": "default",
      "cache_mib": null
    },
    "fault_tolerance": {
      "model": "none",
      "checkpoint_interval_secs": 60
    },
    "cpu_profiler": true,
    "tracing": false,
    "tracing_endpoint_jaeger": "",
    "min_batch_size_records": 0,
    "max_buffering_delay_usecs": 0,
    "resources": {
      "cpu_cores_min": null,
      "cpu_cores_max": null,
      "memory_mb_min": null,
      "memory_mb_max": null,
      "storage_mb_max": null,
      "storage_class": null,
      "service_account_name": null,
      "namespace": null
    },
    "clock_resolution_usecs": 1000000,
    "pin_cpus": [],
    "provisioning_timeout_secs": null,
    "max_parallel_connector_init": null,
    "init_containers": null,
    "checkpoint_during_suspend": true,
    "http_workers": null,
    "io_workers": null,
    "env": {},
    "dev_tweaks": {},
    "logging": null,
    "pipeline_template_configmap": null
  },
  "program_code": "CREATE TABLE table1 ( col1 INT );",
  "udf_rust": "",
  "udf_toml": "",
  "program_config": {
    "profile": "optimized",
    "cache": true,
    "runtime_version": null
  },
  "program_version": 2,
  "program_status": "Pending",
  "program_status_since": "1970-01-01T00:00:00Z",
  "program_error": {
    "sql_compilation": null,
    "rust_compilation": null,
    "system_error": null
  },
  "program_info": null,
  "deployment_error": null,
  "refresh_version": 4,
  "storage_status": "Cleared",
  "storage_status_details": null,
  "deployment_id": null,
  "deployment_initial": null,
  "deployment_status": "Stopped",
  "deployment_status_since": "1970-01-01T00:00:00Z",
  "deployment_desired_status": "Stopped",
  "deployment_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_status": "Stopped",
  "deployment_resources_status_details": null,
  "deployment_resources_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_desired_status": "Stopped",
  "deployment_resources_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_runtime_status": null,
  "deployment_runtime_status_details": null,
  "deployment_runtime_status_since": null,
  "deployment_runtime_desired_status": null,
  "deployment_runtime_desired_status_since": null
}