pipelines

Creates, updates, deletes or gets a pipeline resource or lists pipelines in a region

Overview

Name	pipelines
Type	Resource
Description	An OpenSearch Ingestion Service Data Prepper pipeline running Data Prepper.
Id	awscc.osis.pipelines

Fields

get (all properties)

Name	Datatype	Description
▶buffer_options	object	Key-value pairs to configure buffering.
▶encryption_at_rest_options	object	Key-value pairs to configure encryption at rest.
▶log_publishing_options	object	Key-value pairs to configure log publishing.
max_units	integer	The maximum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs).
min_units	integer	The minimum pipeline capacity, in Ingestion OpenSearch Compute Units (OCUs).
pipeline_configuration_body	string	The Data Prepper pipeline configuration.
pipeline_name	string	Name of the OpenSearch Ingestion Service pipeline to create. Pipeline names are unique across the pipelines owned by an account within an AWS Region.
▶tags	array	An array of key-value pairs to apply to this resource.
▶vpc_options	object	Container for the values required to configure VPC access for the pipeline. If you don't specify these values, OpenSearch Ingestion Service creates the pipeline with a public endpoint.
▶vpc_endpoints	array	The VPC interface endpoints that have access to the pipeline.
vpc_endpoint_service	string	The VPC endpoint service name for the pipeline.
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline.
ingest_endpoint_urls	array	A list of endpoints that can be used for ingesting data into a pipeline
region	string	AWS region.

list (identifiers only)

Name	Datatype	Description
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline.
region	string	AWS region.

For more information, see AWS::OSIS::Pipeline.

Methods

Name	Resource	Accessible by	Required Params
create_resource	pipelines	INSERT	MaxUnits, MinUnits, PipelineConfigurationBody, PipelineName, region
delete_resource	pipelines	DELETE	Identifier, region
update_resource	pipelines	UPDATE	Identifier, PatchDocument, region
list_resources	pipelines_list_only	SELECT	region
get_resource	pipelines	SELECT	Identifier, region

SELECT examples

get (all properties)

Gets all properties from an individual pipeline.

SELECT
  region,
  buffer_options,
  encryption_at_rest_options,
  log_publishing_options,
  max_units,
  min_units,
  pipeline_configuration_body,
  pipeline_name,
  tags,
  vpc_options,
  vpc_endpoints,
  vpc_endpoint_service,
  pipeline_arn,
  ingest_endpoint_urls
FROM awscc.osis.pipelines
WHERE
  region = 'us-east-1' AND
  Identifier = '{{ pipeline_arn }}';

list (identifiers only)

Lists all pipelines in a region.

SELECT
  region,
  pipeline_arn
FROM awscc.osis.pipelines_list_only
WHERE
  region = 'us-east-1';

INSERT example

Use the following StackQL query and manifest file to create a new pipeline resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO awscc.osis.pipelines (
  MaxUnits,
  MinUnits,
  PipelineConfigurationBody,
  PipelineName,
  region
)
SELECT
  '{{ max_units }}',
  '{{ min_units }}',
  '{{ pipeline_configuration_body }}',
  '{{ pipeline_name }}',
  '{{ region }}';

All Properties

/*+ create */
INSERT INTO awscc.osis.pipelines (
  BufferOptions,
  EncryptionAtRestOptions,
  LogPublishingOptions,
  MaxUnits,
  MinUnits,
  PipelineConfigurationBody,
  PipelineName,
  Tags,
  VpcOptions,
  region
)
SELECT
  '{{ buffer_options }}',
  '{{ encryption_at_rest_options }}',
  '{{ log_publishing_options }}',
  '{{ max_units }}',
  '{{ min_units }}',
  '{{ pipeline_configuration_body }}',
  '{{ pipeline_name }}',
  '{{ tags }}',
  '{{ vpc_options }}',
  '{{ region }}';

Manifest

version: 1
name: stack name
description: stack description
providers:
  - aws
globals:
  - name: region
    value: '{{ vars.AWS_REGION }}'
resources:
  - name: pipeline
    props:
      - name: buffer_options
        value:
          persistent_buffer_enabled: '{{ persistent_buffer_enabled }}'
      - name: encryption_at_rest_options
        value:
          kms_key_arn: '{{ kms_key_arn }}'
      - name: log_publishing_options
        value:
          is_logging_enabled: '{{ is_logging_enabled }}'
          cloud_watch_log_destination:
            log_group: '{{ log_group }}'
      - name: max_units
        value: '{{ max_units }}'
      - name: min_units
        value: '{{ min_units }}'
      - name: pipeline_configuration_body
        value: '{{ pipeline_configuration_body }}'
      - name: pipeline_name
        value: '{{ pipeline_name }}'
      - name: tags
        value:
          - key: '{{ key }}'
            value: '{{ value }}'
      - name: vpc_options
        value:
          security_group_ids:
            - '{{ security_group_ids[0] }}'
          subnet_ids:
            - '{{ subnet_ids[0] }}'
          vpc_endpoint_management: '{{ vpc_endpoint_management }}'
          vpc_attachment_options:
            attach_to_vpc: '{{ attach_to_vpc }}'
            cidr_block: '{{ cidr_block }}'

UPDATE example

Use the following StackQL query and manifest file to update a pipeline resource, using stack-deploy.

/*+ update */
UPDATE awscc.osis.pipelines
SET PatchDocument = string('{{ {
    "BufferOptions": buffer_options,
    "EncryptionAtRestOptions": encryption_at_rest_options,
    "LogPublishingOptions": log_publishing_options,
    "MaxUnits": max_units,
    "MinUnits": min_units,
    "PipelineConfigurationBody": pipeline_configuration_body,
    "Tags": tags,
    "VpcOptions": vpc_options
} | generate_patch_document }}')
WHERE
  region = '{{ region }}' AND
  Identifier = '{{ pipeline_arn }}';

DELETE example

/*+ delete */
DELETE FROM awscc.osis.pipelines
WHERE
  Identifier = '{{ pipeline_arn }}' AND
  region = 'us-east-1';

Permissions

To operate on the pipelines resource, the following permissions are required:

Create

osis:CreatePipeline,
osis:GetPipeline,
osis:TagResource,
osis:ListTagsForResource,
iam:PassRole,
iam:CreateServiceLinkedRole,
logs:CreateLogDelivery,
kms:DescribeKey

Read

osis:GetPipeline,
osis:ListTagsForResource

Update

osis:UpdatePipeline,
osis:GetPipeline,
osis:ListTagsForResource,
osis:TagResource,
osis:UntagResource,
iam:PassRole,
logs:GetLogDelivery,
logs:UpdateLogDelivery,
logs:ListLogDeliveries,
kms:DescribeKey

Delete

osis:DeletePipeline,
osis:GetPipeline,
logs:GetLogDelivery,
logs:DeleteLogDelivery,
logs:ListLogDeliveries

List

osis:ListPipelines