pipelines

Creates, updates, deletes or gets a pipeline resource or lists pipelines in a region

Overview

Name	pipelines
Type	Resource
Description	An OpenSearch Ingestion Service Data Prepper pipeline running Data Prepper.
Id	awscc.osis.pipelines

Fields

get (all properties)

Name	Datatype	Description
▶buffer_options	object	Key-value pairs to configure buffering.
▶encryption_at_rest_options	object	Key-value pairs to configure encryption at rest.
▶log_publishing_options	object	Key-value pairs to configure log publishing.
max_units	integer	The maximum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs).
min_units	integer	The minimum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs).
pipeline_configuration_body	string	The Data Prepper pipeline configuration.
pipeline_name	string	Name of the OpenSearch Ingestion Service pipeline to create. Pipeline names are unique across the pipelines owned by an account within an AWS Region.
▶tags	array	An array of key-value pairs to apply to this resource.
▶vpc_options	object	Container for the values required to configure VPC access for the pipeline. If you don't specify these values, OpenSearch Ingestion Service creates the pipeline with a public endpoint.
▶vpc_endpoints	array	The VPC interface endpoints that have access to the pipeline.
vpc_endpoint_service	string	The VPC endpoint service name for the pipeline.
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline.
ingest_endpoint_urls	array	A list of endpoints that can be used for ingesting data into a pipeline
region	string	AWS region.

list (identifiers only)

Name	Datatype	Description
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline.
region	string	AWS region.

For more information, see AWS::OSIS::Pipeline.

Methods

Name	Resource	Accessible by	Required Params
create_resource	pipelines	INSERT	MaxUnits, MinUnits, PipelineConfigurationBody, PipelineName, region
delete_resource	pipelines	DELETE	Identifier, region
update_resource	pipelines	UPDATE	Identifier, PatchDocument, region
list_resources	pipelines_list_only	SELECT	region
get_resource	pipelines	SELECT	Identifier, region

SELECT examples

get (all properties)

Gets all properties from an individual pipeline.

SELECT
  region,
  buffer_options,
  encryption_at_rest_options,
  log_publishing_options,
  max_units,
  min_units,
  pipeline_configuration_body,
  pipeline_name,
  tags,
  vpc_options,
  vpc_endpoints,
  vpc_endpoint_service,
  pipeline_arn,
  ingest_endpoint_urls
FROM awscc.osis.pipelines
WHERE
  region = '{{ region }}' AND
  Identifier = '{{ pipeline_arn }}';

list (identifiers only)

Lists all pipelines in a region.

SELECT
  region,
  pipeline_arn
FROM awscc.osis.pipelines_list_only
WHERE
  region = '{{ region }}';

INSERT example

Use the following StackQL query and manifest file to create a new pipeline resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO awscc.osis.pipelines (
  MaxUnits,
  MinUnits,
  PipelineConfigurationBody,
  PipelineName,
  region
)
SELECT
  '{{ max_units }}',
  '{{ min_units }}',
  '{{ pipeline_configuration_body }}',
  '{{ pipeline_name }}',
  '{{ region }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

All Properties

/*+ create */
INSERT INTO awscc.osis.pipelines (
  BufferOptions,
  EncryptionAtRestOptions,
  LogPublishingOptions,
  MaxUnits,
  MinUnits,
  PipelineConfigurationBody,
  PipelineName,
  Tags,
  VpcOptions,
  region
)
SELECT
  '{{ buffer_options }}',
  '{{ encryption_at_rest_options }}',
  '{{ log_publishing_options }}',
  '{{ max_units }}',
  '{{ min_units }}',
  '{{ pipeline_configuration_body }}',
  '{{ pipeline_name }}',
  '{{ tags }}',
  '{{ vpc_options }}',
  '{{ region }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

Manifest

version: 1
name: stack name
description: stack description
providers:
- aws
globals:
- name: region
  value: '{{ vars.AWS_REGION }}'
resources:
- name: pipeline
  props:
    - name: buffer_options
      value:
        persistent_buffer_enabled: '{{ persistent_buffer_enabled }}'
    - name: encryption_at_rest_options
      value:
        kms_key_arn: '{{ kms_key_arn }}'
    - name: log_publishing_options
      value:
        is_logging_enabled: '{{ is_logging_enabled }}'
        cloud_watch_log_destination:
          log_group: '{{ log_group }}'
    - name: max_units
      value: '{{ max_units }}'
    - name: min_units
      value: '{{ min_units }}'
    - name: pipeline_configuration_body
      value: '{{ pipeline_configuration_body }}'
    - name: pipeline_name
      value: '{{ pipeline_name }}'
    - name: tags
      value:
        - key: '{{ key }}'
          value: '{{ value }}'
    - name: vpc_options
      value:
        security_group_ids:
          - '{{ security_group_ids[0] }}'
        subnet_ids:
          - '{{ subnet_ids[0] }}'
        vpc_endpoint_management: '{{ vpc_endpoint_management }}'
        vpc_attachment_options:
          attach_to_vpc: '{{ attach_to_vpc }}'
          cidr_block: '{{ cidr_block }}'

UPDATE example

Use the following StackQL query and manifest file to update a pipeline resource, using stack-deploy.

/*+ update */
UPDATE awscc.osis.pipelines
SET PatchDocument = string('{{ {
    "BufferOptions": buffer_options,
    "EncryptionAtRestOptions": encryption_at_rest_options,
    "LogPublishingOptions": log_publishing_options,
    "MaxUnits": max_units,
    "MinUnits": min_units,
    "PipelineConfigurationBody": pipeline_configuration_body,
    "Tags": tags,
    "VpcOptions": vpc_options
} | generate_patch_document }}')
WHERE
  region = '{{ region }}' AND
  Identifier = '{{ pipeline_arn }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

DELETE example

/*+ delete */
DELETE FROM awscc.osis.pipelines
WHERE
  Identifier = '{{ pipeline_arn }}' AND
  region = '{{ region }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

Additional Parameters

Mutable resources in the Cloud Control provider support additional optional parameters which can be supplied with INSERT, UPDATE, or DELETE operations. These include:

Parameter	Description
ClientToken	A unique identifier to ensure the idempotency of the resource request. This allows the provider to accurately distinguish between retries and new requests. A client token is valid for 36 hours once used. After that, a resource request with the same client token is treated as a new request. If you do not specify a client token, one is generated for inclusion in the request.
RoleArn	The ARN of the IAM role used to perform this resource operation. The role specified must have the permissions required for this operation. If you do not specify a role, a temporary session is created using your AWS user credentials.
TypeVersionId	For private resource types, the type version to use in this resource operation. If you do not specify a resource version, the default version is used.

Permissions

To operate on the pipelines resource, the following permissions are required:

Create

osis:CreatePipeline,
osis:GetPipeline,
osis:TagResource,
osis:ListTagsForResource,
iam:PassRole,
iam:CreateServiceLinkedRole,
logs:CreateLogDelivery,
kms:DescribeKey

Read

osis:GetPipeline,
osis:ListTagsForResource

Update

osis:UpdatePipeline,
osis:GetPipeline,
osis:ListTagsForResource,
osis:TagResource,
osis:UntagResource,
iam:PassRole,
logs:GetLogDelivery,
logs:UpdateLogDelivery,
logs:ListLogDeliveries,
kms:DescribeKey

Delete

osis:DeletePipeline,
osis:GetPipeline,
logs:GetLogDelivery,
logs:DeleteLogDelivery,
logs:ListLogDeliveries

List

osis:ListPipelines