CTRL K
Skip to Content
API ReferencePipeline CrudPatch Pipeline

Patch Pipeline

PATCH/v0/pipelines/{pipeline_name}
Authentication
Parameters
Request body (application/json)

Partially update a pipeline.

Authentication

SchemeTypeDetails
JSON web token (JWT) or API keyBearer tokenAuthorization header · Use a JWT token obtained via an OAuth2/OIDC login workflow or an API key obtained via the `/v0/api-keys` endpoint.

Parameters

Path parameters

NameTypeRequiredDescription
pipeline_namestringYesUnique pipeline name

Request body

Content type: application/json

  • object
    Partially update the pipeline (PATCH). Note that the patching only applies to the main fields, not subfields. For instance, it is not possible to update only the number of workers; it is required to again pass the whole runtime configuration with the change.
    • descriptionstring
    • namestring
    • program_codestring
    • program_configobject
    • runtime_configobject
    • udf_ruststring
    • udf_tomlstring

Response

application/json
Pipeline successfully updated
  • object
    Pipeline information. It both includes fields which are user-provided and system-generated.
    • created_atstring (date-time)required
    • deployment_desired_statusstringrequired
    • deployment_desired_status_sincestring (date-time)required
    • deployment_errorobject
    • deployment_idstring (uuid)
    • deployment_initialobject
    • deployment_resources_desired_statusstringrequired
    • deployment_resources_desired_status_sincestring (date-time)required
    • deployment_resources_statusstringrequired
      Pipeline resources status. ```text /start (early start failed) ┌───────────────────┐ │ ▼ Stopped ◄────────── Stopping /start │ ▲ │ │ /stop?force=true │ │ OR: timeout (from Provisioning) ▼ │ OR: fatal runtime or resource error ⌛Provisioning ────────────│ OR: runtime status is Suspended │ │ │ │ ▼ │ Provisioned ─────────────┘ ``` ### Desired and actual status We use the desired state model to manage the lifecycle of a pipeline. In this model, the pipeline has two status attributes associated with it: the **desired** status, which represents what the user would like the pipeline to do, and the **current** status, which represents the actual (last observed) status of the pipeline. The pipeline runner service continuously monitors the desired status field to decide where to steer the pipeline towards. There are two desired statuses: - `Provisioned` (set by invoking `/start`) - `Stopped` (set by invoking `/stop?force=true`) The user can monitor the current status of the pipeline via the `GET /v0/pipelines/{name}` endpoint. In a typical scenario, the user first sets the desired status, e.g., by invoking the `/start` endpoint, and then polls the `GET /v0/pipelines/{name}` endpoint to monitor the actual status of the pipeline until its `deployment_resources_status` attribute changes to `Provisioned` indicating that the pipeline has been successfully provisioned, or `Stopped` with `deployment_error` being set.
    • deployment_resources_status_detailsobject
    • deployment_resources_status_sincestring (date-time)required
    • deployment_runtime_desired_statusobject
    • deployment_runtime_desired_status_sincestring (date-time)
    • deployment_runtime_statusobject
    • deployment_runtime_status_detailsobject
    • deployment_runtime_status_sincestring (date-time)
    • deployment_statusstringrequired
    • deployment_status_sincestring (date-time)required
    • descriptionstringrequired
    • idstring (uuid)required
      Pipeline identifier.
    • namestringrequired
    • platform_versionstringrequired
    • program_codestringrequired
    • program_configobjectrequired
      Program configuration.
    • program_errorobjectrequired
      Log, warning and error information about the program compilation.
    • program_infoobject
    • program_statusstringrequired
      Program compilation status.
    • program_status_sincestring (date-time)required
    • program_versioninteger (int64)required
      Version number.
    • refresh_versioninteger (int64)required
      Version number.
    • runtime_configobjectrequired
      Global pipeline configuration settings. This is the publicly exposed type for users to configure pipelines.
    • storage_statusstringrequired
      Storage status. The storage status can only transition when the resources status is `Stopped`. ```text Cleared ───┐ ▲ │ /clear │ │ │ │ Clearing │ ▲ │ │ │ InUse ◄───┘ ```
    • storage_status_detailsobject
    • udf_ruststringrequired
    • udf_tomlstringrequired
    • versioninteger (int64)required
      Version number.
Request
curl -X PATCH '/v0/pipelines/{pipeline_name}' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"name":null,"description":"This is a new description","runtime_config":null,"program_code":"CREATE TABLE table3 ( col3 INT );","udf_rust":null,"udf_toml":null,"program_config":null}'
Response
Pipeline successfully updated
{
  "id": "67e55044-10b1-426f-9247-bb680e5fe0c8",
  "name": "example1",
  "description": "Description of the pipeline example1",
  "created_at": "1970-01-01T00:00:00Z",
  "version": 4,
  "platform_version": "v0",
  "runtime_config": {
    "workers": 16,
    "max_rss_mb": null,
    "hosts": 1,
    "storage": {
      "backend": {
        "name": "default"
      },
      "min_storage_bytes": null,
      "min_step_storage_bytes": null,
      "compression": "default",
      "cache_mib": null
    },
    "fault_tolerance": {
      "model": "none",
      "checkpoint_interval_secs": 60
    },
    "cpu_profiler": true,
    "tracing": false,
    "tracing_endpoint_jaeger": "",
    "min_batch_size_records": 0,
    "max_buffering_delay_usecs": 0,
    "resources": {
      "cpu_cores_min": null,
      "cpu_cores_max": null,
      "memory_mb_min": null,
      "memory_mb_max": null,
      "storage_mb_max": null,
      "storage_class": null,
      "service_account_name": null,
      "namespace": null
    },
    "clock_resolution_usecs": 1000000,
    "pin_cpus": [],
    "provisioning_timeout_secs": null,
    "max_parallel_connector_init": null,
    "init_containers": null,
    "checkpoint_during_suspend": true,
    "http_workers": null,
    "io_workers": null,
    "env": {},
    "dev_tweaks": {},
    "logging": null,
    "pipeline_template_configmap": null
  },
  "program_code": "CREATE TABLE table1 ( col1 INT );",
  "udf_rust": "",
  "udf_toml": "",
  "program_config": {
    "profile": "optimized",
    "cache": true,
    "runtime_version": null
  },
  "program_version": 2,
  "program_status": "Pending",
  "program_status_since": "1970-01-01T00:00:00Z",
  "program_error": {
    "sql_compilation": null,
    "rust_compilation": null,
    "system_error": null
  },
  "program_info": null,
  "deployment_error": null,
  "refresh_version": 4,
  "storage_status": "Cleared",
  "storage_status_details": null,
  "deployment_id": null,
  "deployment_initial": null,
  "deployment_status": "Stopped",
  "deployment_status_since": "1970-01-01T00:00:00Z",
  "deployment_desired_status": "Stopped",
  "deployment_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_status": "Stopped",
  "deployment_resources_status_details": null,
  "deployment_resources_status_since": "1970-01-01T00:00:00Z",
  "deployment_resources_desired_status": "Stopped",
  "deployment_resources_desired_status_since": "1970-01-01T00:00:00Z",
  "deployment_runtime_status": null,
  "deployment_runtime_status_details": null,
  "deployment_runtime_status_since": null,
  "deployment_runtime_desired_status": null,
  "deployment_runtime_desired_status_since": null
}
Upsert PipelineDelete Pipeline