datasets

Creates, updates, deletes or gets a dataset resource or lists datasets in a region

Overview

Name	datasets
Type	Resource
Description	Resource schema for AWS::DataBrew::Dataset.
Id	awscc.databrew.datasets

Fields

get (all properties)

Name	Datatype	Description
name	string	Dataset name
format	string	Dataset format
▶format_options	object	Format options for dataset
▶input	object	Input
source	string	Source type of the dataset
▶path_options	object	PathOptions
▶tags	array
region	string	AWS region.

list (identifiers only)

Name	Datatype	Description
name	string	Dataset name
region	string	AWS region.

For more information, see AWS::DataBrew::Dataset.

Methods

Name	Resource	Accessible by	Required Params
create_resource	datasets	INSERT	Name, Input, region
delete_resource	datasets	DELETE	Identifier, region
update_resource	datasets	UPDATE	Identifier, PatchDocument, region
list_resources	datasets_list_only	SELECT	region
get_resource	datasets	SELECT	Identifier, region

SELECT examples

get (all properties)

Gets all properties from an individual dataset.

SELECT
  region,
  name,
  format,
  format_options,
  input,
  source,
  path_options,
  tags
FROM awscc.databrew.datasets
WHERE
  region = 'us-east-1' AND
  Identifier = '{{ name }}';

list (identifiers only)

Lists all datasets in a region.

SELECT
  region,
  name
FROM awscc.databrew.datasets_list_only
WHERE
  region = 'us-east-1';

INSERT example

Use the following StackQL query and manifest file to create a new dataset resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO awscc.databrew.datasets (
  Name,
  Input,
  region
)
SELECT
  '{{ name }}',
  '{{ input }}',
  '{{ region }}';

All Properties

/*+ create */
INSERT INTO awscc.databrew.datasets (
  Name,
  Format,
  FormatOptions,
  Input,
  Source,
  PathOptions,
  Tags,
  region
)
SELECT
  '{{ name }}',
  '{{ format }}',
  '{{ format_options }}',
  '{{ input }}',
  '{{ source }}',
  '{{ path_options }}',
  '{{ tags }}',
  '{{ region }}';

Manifest

version: 1
name: stack name
description: stack description
providers:
  - aws
globals:
  - name: region
    value: '{{ vars.AWS_REGION }}'
resources:
  - name: dataset
    props:
      - name: name
        value: '{{ name }}'
      - name: format
        value: '{{ format }}'
      - name: format_options
        value:
          json:
            multi_line: '{{ multi_line }}'
          excel:
            sheet_names:
              - '{{ sheet_names[0] }}'
            sheet_indexes:
              - '{{ sheet_indexes[0] }}'
            header_row: '{{ header_row }}'
          csv:
            delimiter: '{{ delimiter }}'
            header_row: '{{ header_row }}'
      - name: input
        value:
          s3_input_definition:
            bucket: '{{ bucket }}'
            key: '{{ key }}'
          data_catalog_input_definition:
            catalog_id: '{{ catalog_id }}'
            database_name: '{{ database_name }}'
            table_name: '{{ table_name }}'
            temp_directory: null
          database_input_definition:
            glue_connection_name: '{{ glue_connection_name }}'
            database_table_name: '{{ database_table_name }}'
            temp_directory: null
            query_string: '{{ query_string }}'
          metadata:
            source_arn: '{{ source_arn }}'
      - name: source
        value: '{{ source }}'
      - name: path_options
        value:
          files_limit:
            max_files: '{{ max_files }}'
            ordered_by: '{{ ordered_by }}'
            order: '{{ order }}'
          last_modified_date_condition:
            expression: '{{ expression }}'
            values_map:
              - value_reference: '{{ value_reference }}'
                value: '{{ value }}'
          parameters:
            - path_parameter_name: '{{ path_parameter_name }}'
              dataset_parameter:
                name: null
                type: '{{ type }}'
                datetime_options:
                  format: '{{ format }}'
                  timezone_offset: '{{ timezone_offset }}'
                  locale_code: '{{ locale_code }}'
                create_column: '{{ create_column }}'
                filter: null
      - name: tags
        value:
          - key: '{{ key }}'
            value: '{{ value }}'

UPDATE example

Use the following StackQL query and manifest file to update a dataset resource, using stack-deploy.

/*+ update */
UPDATE awscc.databrew.datasets
SET PatchDocument = string('{{ {
    "Format": format,
    "FormatOptions": format_options,
    "Input": input,
    "Source": source,
    "PathOptions": path_options,
    "Tags": tags
} | generate_patch_document }}')
WHERE
  region = '{{ region }}' AND
  Identifier = '{{ name }}';

DELETE example

/*+ delete */
DELETE FROM awscc.databrew.datasets
WHERE
  Identifier = '{{ name }}' AND
  region = 'us-east-1';

Permissions

To operate on the datasets resource, the following permissions are required:

Create

databrew:CreateDataset,
databrew:DescribeDataset,
databrew:TagResource,
databrew:UntagResource,
glue:GetConnection,
glue:GetTable,
iam:PassRole

Read

databrew:DescribeDataset,
iam:ListRoles

Update

databrew:UpdateDataset,
databrew:TagResource,
databrew:UntagResource,
glue:GetConnection,
glue:GetTable

Delete

databrew:DeleteDataset

List

databrew:ListDatasets,
databrew:ListTagsForResource,
iam:ListRoles