processing_jobs

Creates, updates, deletes or gets a processing_job resource or lists processing_jobs in a region

Overview

Name	processing_jobs
Type	Resource
Description	Resource Type definition for AWS::SageMaker::ProcessingJob
Id	awscc.sagemaker.processing_jobs

Fields

get (all properties)

Name	Datatype	Description
▶app_specification	object	Configures the processing job to run a specified Docker container image.
environment	object	Sets the environment variables in the Docker container
▶experiment_config	object	Associates a SageMaker job as a trial component with an experiment and trial.
▶network_config	object	Networking options for a job, such as network traffic encryption between containers, whether to allow inbound and outbound network calls to and from containers, and the VPC subnets and security groups to use for VPC-enabled jobs.
▶processing_inputs	array	An array of inputs configuring the data to download into the processing container.
processing_job_name	string	The name of the processing job. The name must be unique within an AWS Region in the AWS account.
▶processing_output_config	object	Configuration for uploading output from the processing container.
▶processing_resources	object	Identifies the resources, ML compute instances, and ML storage volumes to deploy for a processing job. In distributed training, you specify more than one instance.
role_arn	string	The Amazon Resource Name (ARN) of an IAM role that Amazon SageMaker can assume to perform tasks on your behalf.
▶stopping_condition	object	Configures conditions under which the processing job should be stopped, such as how long the processing job has been running. After the condition is met, the processing job is stopped.
▶tags	array	(Optional) An array of key-value pairs. For more information, see Using Cost Allocation Tags(https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html#allocation-whatURL) in the AWS Billing and Cost Management User Guide.
processing_job_arn	string	The Amazon Resource Name (ARN) of the processing job.
auto_ml_job_arn	string	The ARN of an AutoML job associated with this processing job.
exit_message	string	An optional string, up to one KB in size, that contains metadata from the processing container when the processing job exits.
failure_reason	string	A string, up to one KB in size, that contains the reason a processing job failed, if it failed.
monitoring_schedule_arn	string	The ARN of a monitoring schedule for an endpoint associated with this processing job.
training_job_arn	string	The ARN of a training job associated with this processing job
processing_job_status	string	Provides the status of a processing job.
creation_time	string	The time at which the processing job was created.
last_modified_time	string	The time at which the processing job was last modified.
processing_start_time	string	The time at which the processing job started.
processing_end_time	string	The time at which the processing job completed.
region	string	AWS region.

list (identifiers only)

Name	Datatype	Description
processing_job_arn	string	The Amazon Resource Name (ARN) of the processing job.
region	string	AWS region.

For more information, see AWS::SageMaker::ProcessingJob.

Methods

Name	Resource	Accessible by	Required Params
create_resource	processing_jobs	INSERT	AppSpecification, ProcessingResources, RoleArn, region
delete_resource	processing_jobs	DELETE	Identifier, region
list_resources	processing_jobs_list_only	SELECT	region
get_resource	processing_jobs	SELECT	Identifier, region

SELECT examples

get (all properties)

Gets all properties from an individual processing_job.

SELECT
  region,
  app_specification,
  environment,
  experiment_config,
  network_config,
  processing_inputs,
  processing_job_name,
  processing_output_config,
  processing_resources,
  role_arn,
  stopping_condition,
  tags,
  processing_job_arn,
  auto_ml_job_arn,
  exit_message,
  failure_reason,
  monitoring_schedule_arn,
  training_job_arn,
  processing_job_status,
  creation_time,
  last_modified_time,
  processing_start_time,
  processing_end_time
FROM awscc.sagemaker.processing_jobs
WHERE
  region = 'us-east-1' AND
  Identifier = '{{ processing_job_arn }}';

list (identifiers only)

Lists all processing_jobs in a region.

SELECT
  region,
  processing_job_arn
FROM awscc.sagemaker.processing_jobs_list_only
WHERE
  region = 'us-east-1';

INSERT example

Use the following StackQL query and manifest file to create a new processing_job resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO awscc.sagemaker.processing_jobs (
  AppSpecification,
  ProcessingResources,
  RoleArn,
  region
)
SELECT
  '{{ app_specification }}',
  '{{ processing_resources }}',
  '{{ role_arn }}',
  '{{ region }}';

All Properties

/*+ create */
INSERT INTO awscc.sagemaker.processing_jobs (
  AppSpecification,
  Environment,
  ExperimentConfig,
  NetworkConfig,
  ProcessingInputs,
  ProcessingJobName,
  ProcessingOutputConfig,
  ProcessingResources,
  RoleArn,
  StoppingCondition,
  Tags,
  region
)
SELECT
  '{{ app_specification }}',
  '{{ environment }}',
  '{{ experiment_config }}',
  '{{ network_config }}',
  '{{ processing_inputs }}',
  '{{ processing_job_name }}',
  '{{ processing_output_config }}',
  '{{ processing_resources }}',
  '{{ role_arn }}',
  '{{ stopping_condition }}',
  '{{ tags }}',
  '{{ region }}';

Manifest

version: 1
name: stack name
description: stack description
providers:
  - aws
globals:
  - name: region
    value: '{{ vars.AWS_REGION }}'
resources:
  - name: processing_job
    props:
      - name: app_specification
        value:
          container_arguments:
            - '{{ container_arguments[0] }}'
          container_entrypoint:
            - '{{ container_entrypoint[0] }}'
          image_uri: '{{ image_uri }}'
      - name: environment
        value: {}
      - name: experiment_config
        value:
          experiment_name: '{{ experiment_name }}'
          trial_name: '{{ trial_name }}'
          trial_component_display_name: '{{ trial_component_display_name }}'
          run_name: '{{ run_name }}'
      - name: network_config
        value:
          enable_inter_container_traffic_encryption: '{{ enable_inter_container_traffic_encryption }}'
          enable_network_isolation: '{{ enable_network_isolation }}'
          vpc_config:
            security_group_ids:
              - '{{ security_group_ids[0] }}'
            subnets:
              - '{{ subnets[0] }}'
      - name: processing_inputs
        value:
          - s3_input:
              local_path: '{{ local_path }}'
              s3_compression_type: '{{ s3_compression_type }}'
              s3_data_distribution_type: '{{ s3_data_distribution_type }}'
              s3_data_type: '{{ s3_data_type }}'
              s3_input_mode: '{{ s3_input_mode }}'
              s3_uri: '{{ s3_uri }}'
            dataset_definition:
              athena_dataset_definition:
                catalog: '{{ catalog }}'
                database: '{{ database }}'
                output_s3_uri: '{{ output_s3_uri }}'
                query_string: '{{ query_string }}'
                work_group: '{{ work_group }}'
                output_format: '{{ output_format }}'
                kms_key_id: '{{ kms_key_id }}'
                output_compression: '{{ output_compression }}'
              redshift_dataset_definition:
                database: '{{ database }}'
                db_user: '{{ db_user }}'
                query_string: '{{ query_string }}'
                cluster_id: '{{ cluster_id }}'
                cluster_role_arn: '{{ cluster_role_arn }}'
                output_s3_uri: '{{ output_s3_uri }}'
                output_format: '{{ output_format }}'
                kms_key_id: '{{ kms_key_id }}'
                output_compression: '{{ output_compression }}'
              data_distribution_type: '{{ data_distribution_type }}'
              input_mode: '{{ input_mode }}'
              local_path: '{{ local_path }}'
            input_name: '{{ input_name }}'
            app_managed: '{{ app_managed }}'
      - name: processing_job_name
        value: '{{ processing_job_name }}'
      - name: processing_output_config
        value:
          kms_key_id: '{{ kms_key_id }}'
          outputs:
            - output_name: '{{ output_name }}'
              app_managed: '{{ app_managed }}'
              s3_output:
                local_path: '{{ local_path }}'
                s3_upload_mode: '{{ s3_upload_mode }}'
                s3_uri: '{{ s3_uri }}'
              feature_store_output:
                feature_group_name: '{{ feature_group_name }}'
      - name: processing_resources
        value:
          cluster_config:
            instance_count: '{{ instance_count }}'
            instance_type: '{{ instance_type }}'
            volume_size_in_gb: '{{ volume_size_in_gb }}'
            volume_kms_key_id: '{{ volume_kms_key_id }}'
      - name: role_arn
        value: '{{ role_arn }}'
      - name: stopping_condition
        value:
          max_runtime_in_seconds: '{{ max_runtime_in_seconds }}'
      - name: tags
        value:
          - value: '{{ value }}'
            key: '{{ key }}'

DELETE example

/*+ delete */
DELETE FROM awscc.sagemaker.processing_jobs
WHERE
  Identifier = '{{ processing_job_arn }}' AND
  region = 'us-east-1';

Permissions

To operate on the processing_jobs resource, the following permissions are required:

Create

ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission,
ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission,
ec2:DescribeNetworkInterfaces,
ec2:DescribeVpcs,
ec2:DescribeDhcpOptions,
ec2:DescribeSubnets,
ec2:DescribeSecurityGroups,
kms:CreateGrant,
kms:Decrypt,
kms:DescribeKey,
sagemaker:CreateProcessingJob,
sagemaker:DescribeProcessingJob,
sagemaker:AddTags,
sagemaker:ListTags,
iam:PassRole

Read

sagemaker:DescribeProcessingJob,
sagemaker:ListTags

Delete

ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission,
ec2:DescribeNetworkInterfaces,
ec2:DescribeVpcs,
ec2:DescribeSubnets,
ec2:DescribeSecurityGroups,
sagemaker:StopProcessingJob,
sagemaker:DescribeProcessingJob,
sagemaker:ListTags,
sagemaker:DeleteTags

List

sagemaker:ListProcessingJobs,
sagemaker:DescribeProcessingJob,
sagemaker:ListTags