processing_jobs

Creates, updates, deletes or gets a processing_job resource or lists processing_jobs in a region

Overview

Name	processing_jobs
Type	Resource
Description	Resource Type definition for AWS::SageMaker::ProcessingJob
Id	awscc.sagemaker.processing_jobs

Fields

get (all properties)

Name	Datatype	Description
▶app_specification	object	Configures the processing job to run a specified Docker container image.
environment	object	Sets the environment variables in the Docker container
▶experiment_config	object	Associates a SageMaker job as a trial component with an experiment and trial.
▶network_config	object	Networking options for a job, such as network traffic encryption between containers, whether to allow inbound and outbound network calls to and from containers, and the VPC subnets and security groups to use for VPC-enabled jobs.
▶processing_inputs	array	An array of inputs configuring the data to download into the processing container.
processing_job_name	string	The name of the processing job. The name must be unique within an AWS Region in the AWS account.
▶processing_output_config	object	Configuration for uploading output from the processing container.
▶processing_resources	object	Identifies the resources, ML compute instances, and ML storage volumes to deploy for a processing job. In distributed training, you specify more than one instance.
role_arn	string	The Amazon Resource Name (ARN) of an IAM role that Amazon SageMaker can assume to perform tasks on your behalf.
▶stopping_condition	object	Configures conditions under which the processing job should be stopped, such as how long the processing job has been running. After the condition is met, the processing job is stopped.
▶tags	array	(Optional) An array of key-value pairs. For more information, see Using Cost Allocation Tags(https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html#allocation-whatURL) in the AWS Billing and Cost Management User Guide.
processing_job_arn	string	The Amazon Resource Name (ARN) of the processing job.
auto_ml_job_arn	string	The ARN of an AutoML job associated with this processing job.
exit_message	string	An optional string, up to one KB in size, that contains metadata from the processing container when the processing job exits.
failure_reason	string	A string, up to one KB in size, that contains the reason a processing job failed, if it failed.
monitoring_schedule_arn	string	The ARN of a monitoring schedule for an endpoint associated with this processing job.
training_job_arn	string	The ARN of a training job associated with this processing job
processing_job_status	string	Provides the status of a processing job.
creation_time	string	The time at which the processing job was created.
last_modified_time	string	The time at which the processing job was last modified.
processing_start_time	string	The time at which the processing job started.
processing_end_time	string	The time at which the processing job completed.
region	string	AWS region.

list (identifiers only)

Name	Datatype	Description
processing_job_arn	string	The Amazon Resource Name (ARN) of the processing job.
region	string	AWS region.

For more information, see AWS::SageMaker::ProcessingJob.

Methods

Name	Resource	Accessible by	Required Params
create_resource	processing_jobs	INSERT	AppSpecification, ProcessingResources, RoleArn, region
delete_resource	processing_jobs	DELETE	Identifier, region
list_resources	processing_jobs_list_only	SELECT	region
get_resource	processing_jobs	SELECT	Identifier, region

SELECT examples

get (all properties)

Gets all properties from an individual processing_job.

SELECT
  region,
  app_specification,
  environment,
  experiment_config,
  network_config,
  processing_inputs,
  processing_job_name,
  processing_output_config,
  processing_resources,
  role_arn,
  stopping_condition,
  tags,
  processing_job_arn,
  auto_ml_job_arn,
  exit_message,
  failure_reason,
  monitoring_schedule_arn,
  training_job_arn,
  processing_job_status,
  creation_time,
  last_modified_time,
  processing_start_time,
  processing_end_time
FROM awscc.sagemaker.processing_jobs
WHERE
  region = '{{ region }}' AND
  Identifier = '{{ processing_job_arn }}';

list (identifiers only)

Lists all processing_jobs in a region.

SELECT
  region,
  processing_job_arn
FROM awscc.sagemaker.processing_jobs_list_only
WHERE
  region = '{{ region }}';

INSERT example

Use the following StackQL query and manifest file to create a new processing_job resource, using stack-deploy.

Required Properties

/*+ create */
INSERT INTO awscc.sagemaker.processing_jobs (
  AppSpecification,
  ProcessingResources,
  RoleArn,
  region
)
SELECT
  '{{ app_specification }}',
  '{{ processing_resources }}',
  '{{ role_arn }}',
  '{{ region }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

All Properties

/*+ create */
INSERT INTO awscc.sagemaker.processing_jobs (
  AppSpecification,
  Environment,
  ExperimentConfig,
  NetworkConfig,
  ProcessingInputs,
  ProcessingJobName,
  ProcessingOutputConfig,
  ProcessingResources,
  RoleArn,
  StoppingCondition,
  Tags,
  region
)
SELECT
  '{{ app_specification }}',
  '{{ environment }}',
  '{{ experiment_config }}',
  '{{ network_config }}',
  '{{ processing_inputs }}',
  '{{ processing_job_name }}',
  '{{ processing_output_config }}',
  '{{ processing_resources }}',
  '{{ role_arn }}',
  '{{ stopping_condition }}',
  '{{ tags }}',
  '{{ region }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

Manifest

version: 1
name: stack name
description: stack description
providers:
- aws
globals:
- name: region
  value: '{{ vars.AWS_REGION }}'
resources:
- name: processing_job
  props:
    - name: app_specification
      value:
        container_arguments:
          - '{{ container_arguments[0] }}'
        container_entrypoint:
          - '{{ container_entrypoint[0] }}'
        image_uri: '{{ image_uri }}'
    - name: environment
      value: {}
    - name: experiment_config
      value:
        experiment_name: '{{ experiment_name }}'
        trial_name: '{{ trial_name }}'
        trial_component_display_name: '{{ trial_component_display_name }}'
        run_name: '{{ run_name }}'
    - name: network_config
      value:
        enable_inter_container_traffic_encryption: '{{ enable_inter_container_traffic_encryption }}'
        enable_network_isolation: '{{ enable_network_isolation }}'
        vpc_config:
          security_group_ids:
            - '{{ security_group_ids[0] }}'
          subnets:
            - '{{ subnets[0] }}'
    - name: processing_inputs
      value:
        - s3_input:
            local_path: '{{ local_path }}'
            s3_compression_type: '{{ s3_compression_type }}'
            s3_data_distribution_type: '{{ s3_data_distribution_type }}'
            s3_data_type: '{{ s3_data_type }}'
            s3_input_mode: '{{ s3_input_mode }}'
            s3_uri: '{{ s3_uri }}'
          dataset_definition:
            athena_dataset_definition:
              catalog: '{{ catalog }}'
              database: '{{ database }}'
              output_s3_uri: '{{ output_s3_uri }}'
              query_string: '{{ query_string }}'
              work_group: '{{ work_group }}'
              output_format: '{{ output_format }}'
              kms_key_id: '{{ kms_key_id }}'
              output_compression: '{{ output_compression }}'
            redshift_dataset_definition:
              database: '{{ database }}'
              db_user: '{{ db_user }}'
              query_string: '{{ query_string }}'
              cluster_id: '{{ cluster_id }}'
              cluster_role_arn: '{{ cluster_role_arn }}'
              output_s3_uri: '{{ output_s3_uri }}'
              output_format: '{{ output_format }}'
              kms_key_id: '{{ kms_key_id }}'
              output_compression: '{{ output_compression }}'
            data_distribution_type: '{{ data_distribution_type }}'
            input_mode: '{{ input_mode }}'
            local_path: '{{ local_path }}'
          input_name: '{{ input_name }}'
          app_managed: '{{ app_managed }}'
    - name: processing_job_name
      value: '{{ processing_job_name }}'
    - name: processing_output_config
      value:
        kms_key_id: '{{ kms_key_id }}'
        outputs:
          - output_name: '{{ output_name }}'
            app_managed: '{{ app_managed }}'
            s3_output:
              local_path: '{{ local_path }}'
              s3_upload_mode: '{{ s3_upload_mode }}'
              s3_uri: '{{ s3_uri }}'
            feature_store_output:
              feature_group_name: '{{ feature_group_name }}'
    - name: processing_resources
      value:
        cluster_config:
          instance_count: '{{ instance_count }}'
          instance_type: '{{ instance_type }}'
          volume_size_in_gb: '{{ volume_size_in_gb }}'
          volume_kms_key_id: '{{ volume_kms_key_id }}'
    - name: role_arn
      value: '{{ role_arn }}'
    - name: stopping_condition
      value:
        max_runtime_in_seconds: '{{ max_runtime_in_seconds }}'
    - name: tags
      value:
        - value: '{{ value }}'
          key: '{{ key }}'

DELETE example

/*+ delete */
DELETE FROM awscc.sagemaker.processing_jobs
WHERE
  Identifier = '{{ processing_job_arn }}' AND
  region = '{{ region }}'
RETURNING
  ErrorCode,
  EventTime,
  Identifier,
  Operation,
  OperationStatus,
  RequestToken,
  ResourceModel,
  RetryAfter,
  StatusMessage,
  TypeName
;

Additional Parameters

Mutable resources in the Cloud Control provider support additional optional parameters which can be supplied with INSERT, UPDATE, or DELETE operations. These include:

Parameter	Description
ClientToken	A unique identifier to ensure the idempotency of the resource request. This allows the provider to accurately distinguish between retries and new requests. A client token is valid for 36 hours once used. After that, a resource request with the same client token is treated as a new request. If you do not specify a client token, one is generated for inclusion in the request.
RoleArn	The ARN of the IAM role used to perform this resource operation. The role specified must have the permissions required for this operation. If you do not specify a role, a temporary session is created using your AWS user credentials.
TypeVersionId	For private resource types, the type version to use in this resource operation. If you do not specify a resource version, the default version is used.

Permissions

To operate on the processing_jobs resource, the following permissions are required:

Create

ec2:CreateNetworkInterface,
ec2:CreateNetworkInterfacePermission,
ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission,
ec2:DescribeNetworkInterfaces,
ec2:DescribeVpcs,
ec2:DescribeDhcpOptions,
ec2:DescribeSubnets,
ec2:DescribeSecurityGroups,
kms:CreateGrant,
kms:Decrypt,
kms:DescribeKey,
sagemaker:CreateProcessingJob,
sagemaker:DescribeProcessingJob,
sagemaker:AddTags,
sagemaker:ListTags,
iam:PassRole

Read

sagemaker:DescribeProcessingJob,
sagemaker:ListTags

Delete

ec2:DeleteNetworkInterface,
ec2:DeleteNetworkInterfacePermission,
ec2:DescribeNetworkInterfaces,
ec2:DescribeVpcs,
ec2:DescribeSubnets,
ec2:DescribeSecurityGroups,
sagemaker:StopProcessingJob,
sagemaker:DescribeProcessingJob,
sagemaker:ListTags,
sagemaker:DeleteTags

List

sagemaker:ListProcessingJobs,
sagemaker:DescribeProcessingJob,
sagemaker:ListTags