Skip to main content

S3 Publisher Specification

Bacalhau's S3 Publisher provides users with a secure and efficient method to publish task results to any S3-compatible storage service. This publisher supports not just AWS S3, but other S3-compatible services offered by cloud providers like Google Cloud Storage and Azure Blob Storage, as well as open-source options like MinIO. The integration is designed to be highly flexible, ensuring users can choose the storage option that aligns with their needs, privacy preferences, and operational requirements.

Publisher Parameters​

  • Bucket (string: <required>): The name of the S3 bucket where the task results will be stored.
  • Key (string: <required>): The object key within the specified bucket where the task results will be stored.
  • Endpoint (string: <optional>): The endpoint URL of the S3 service (useful for S3-compatible services).
  • Region (string: <optional>): The region where the S3 bucket is located.

Published Result Spec​

Results published to S3 are stored as objects that can also be used as inputs to other Bacalhau jobs by using S3 Input Source. The published result specification includes the following parameters:

  • Bucket: Confirms the name of the bucket containing the stored results.
  • Key: Identifies the unique object key within the specified bucket.
  • Region: Notes the AWS region of the bucket.
  • Endpoint: Records the endpoint URL for S3-compatible storage services.
  • VersionID: The version ID of the stored object, enabling versioning support for retrieving specific versions of stored data.
  • ChecksumSHA256: The SHA-256 checksum of the stored object, providing a method to verify data integrity.

Dynamic Naming​

With the S3 Publisher in Bacalhau, you have the flexibility to use dynamic naming for the objects you publish to S3. This allows you to incorporate specific job and execution details into the object key, making it easier to trace, manage, and organize your published artifacts.

Bacalhau supports the following dynamic placeholders that will be replaced with their actual values during the publishing process:

  • {executionID}: Replaced with the specific execution ID.
  • {jobID}: Replaced with the ID of the job.
  • {nodeID}: Replaced with the ID of the node where the execution took place
  • {date}: Replaced with the current date in the format YYYYMMDD.
  • {time}: Replaced with the current time in the format HHMMSS.

Additionally, if you are publishing an archive and the object key does not end with .tar.gz, it will be automatically appended. Conversely, if you're not archiving and the key doesn't end with a /, a trailing slash will be added.

Example

Imagine you've specified the following object key pattern for publishing:

results/{jobID}/{date}/{time}/

Given a job with ID abc123, executed on 2023-09-26 at 14:05:30, the published object key would be:

results/abc123/20230926/140530/

This dynamic naming feature offers a powerful way to create organized, intuitive naming conventions for your Bacalhau published objects in S3.

Examples​

Declarative Examples​

Here’s an example YAML configuration that outlines the process of using the S3 Publisher with Bacalhau:

Publisher:
  Type: "s3"
  Params:
    Bucket: "my-task-results"
    Key: "task123/result.tar.gz"
    Endpoint: "https://s3.us-west-2.amazonaws.com"

In this configuration, task results will be published to the specified S3 bucket and object key. If you’re using an S3-compatible service, simply update the Endpoint parameter with the appropriate URL.

The results will be compressed into a single object, and the published result specification will look like:

PublishedResult:
  Type: "s3"
  Params:
    Bucket: "my-task-results"
    Key: "task123/result.tar.gz"
    Endpoint: "https://s3.us-west-2.amazonaws.com"
    Region: "us-west-2"
    ChecksumSHA256: "0x9a3a..."
    VersionID: "3/L4kqtJlcpXroDTDmJ+rmDbwQaHWyOb..."

Imperative Examples​

The Bacalhau command-line interface (CLI) provides an imperative approach to specify the S3 Publisher. Below are a few examples showcasing how to define an S3 publisher using CLI commands:

  1. Basic Docker job writing to S3 with default configurations:

    bacalhau docker run -p s3://bucket/key ubuntu ...

    This command writes to the S3 bucket using default endpoint and region settings.

  2. Docker job writing to S3 with a specific endpoint and region:

    bacalhau docker run -p s3://bucket/key,opt=endpoint=http://s3.example.com,opt=region=us-east-1 ubuntu ...

    This command specifies a unique endpoint and region for the S3 bucket.

  3. Using naming placeholders:

    bacalhau docker run -p s3://bucket/result-{date}-{jobID} ubuntu ...

    Dynamic naming placeholders like {date} and {jobID} allow for organized naming structures, automatically replacing these placeholders with appropriate values upon execution.

Remember to replace the placeholders like bucket, key, and other parameters with your specific values. These CLI commands offer a quick and customizable way to submit jobs and specify how the results should be published to S3.

Credential Requirements​

To support this storage provider, no extra dependencies are necessary. However, valid AWS credentials are essential to sign the requests. The storage provider employs the default credentials chain to retrieve credentials, primarily sourcing them from:

  1. Environment variables: AWS credentials can be specified using AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

  2. Credentials file: The credentials file typically located at ~/.aws/credentials can also be used to fetch the necessary AWS credentials.

  3. IAM Roles for Amazon EC2 Instances: If you're running your tasks within an Amazon EC2 instance, IAM roles can be utilized to provide the necessary permissions and credentials.

For a more detailed overview on AWS credential management and other ways to provide these credentials, please refer to the AWS official documentation on standardized credentials.

Required IAM Policies​

Compute Nodes​

Compute nodes must run with the following policies to publish to S3:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        }
    ]
}

  • PutObject Permissions: The s3:PutObject permission is necessary to publish objects to the specified S3 bucket.

  • Resource: The Resource field in the policy specifies the Amazon Resource Name (ARN) of the S3 bucket. The /* suffix is necessary to allow publishing with any prefix within the bucket or can be replaced with a prefix to limit the scope of the policy. You can also specify multiple resources in the policy to allow publishing to multiple buckets, or * to allow publishing to all buckets in the account.

Requester Node​

To enable downloading published results using bacalhau get <job_id> command, the requester node must run with the following policies:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        }
    ]
}
  • GetObject Permissions: The s3:GetObject permission is necessary for the requester node to provide a pre-signed URL to download the published results by the client.

For more information on IAM policies specific to Amazon S3 buckets and users, please refer to the AWS documentation on Using IAM Policies with Amazon S3.