Description

The bacalhau job describe command provides a detailed description of a specific job in YAML format. This description can be particularly useful when wanting to understand the attributes and current status of a specific job. To list all available jobs, the bacalhau job list command can be used.

Usage

bacalhau job describe [id] [flags]

Flags

• -h, --help:

  • Description: Display help for the describe command.

• --output format:

  • Description: Specifies the desired output format for the command. Supported values are json and yaml.

  • Default: yaml

• --pretty:

  • Description: Pretty prints the output. This option is applicable only to json and yaml output formats.

Global Flags

• --api-host string:

  • Description: Specifies the host for the client and server to communicate through via REST. If the BACALHAU_API_HOST environment variable is set, this flag will be ignored.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for the client and server to communicate on using REST. If the BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Specifies the desired log format. Supported values include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. Describe a Job with Full ID:

   Copy

   bacalhau job describe j-e3f8c209-d683-4a41-b840-f09b88d087b9

2. Describe a Job with Shortened ID:

   Copy

   bacalhau job describe j-47805f5c

3. Describe a Job with JSON Output:

   Copy

   bacalhau job describe --output json --pretty j-b6ad164a

Description

The bacalhau job executions command retrieves a list of executions for a specific job based on its ID. This can be essential when tracking the various runs and their respective states for a particular job.

Usage

Flags

• -h, --help:

  • Description: Display help for the executions command.

• --hide-header:

  • Description: Do not print the column headers when displaying the results.

• --limit uint32:

  • Description: Restricts the number of results returned.

  • Default: 20

• --next-token string:

  • Description: Uses the specified token for pagination. Useful for fetching the next set of results.

• --no-style:

  • Description: Removes all styling from the table output, displaying raw data.

• --order-by string:

  • Description: Orders results based on a specific field. Valid fields are: modify_time, create_time, id, and state.

• --order-reversed:

  • Description: Reverses the order of the results. Useful in conjunction with --order-by.

• --output format:

  • Description: Specifies the desired output format for the command. Supported values are table, csv, json, and yaml.

  • Default: table

• --pretty:

  • Description: Pretty prints the output. This option is applicable only to json and yaml output formats.

• --wide:

  • Description: Prints full values in the table results without truncating any information.

Global Flags

• --api-host string:

  • Description: Specifies the host for the client and server to communicate through via REST. If the BACALHAU_API_HOST environment variable is set, this flag will be ignored.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for the client and server to communicate on using REST. If the BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Specifies the desired log format. Supported values include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. List executions for a specific Job:

   Expected output:
2. Order executions by state for a specific job:

   Execute the command:

   Expected output:
3. List executions with YAML output:

   Expected output:

Description

The bacalhau job list command provides a listing of all submitted jobs. This command offers an overview of all tasks and processes registered in the system, allowing users to monitor and manage their jobs effectively.

Usage

bacalhau job list [flags]

Flags

• -h, --help:

  • Description: Display help for the list command.

• --hide-header:

  • Description: Opts out of printing the column headers in the results.

• --labels string:

  • Description: Filters jobs by labels. It's designed to function similar to Kubernetes label selectors.

  • Default: bacalhau_canary != true

• --limit uint32:

  • Description: Limits the number of results returned.

  • Default: 10

• --next-token string:

  • Description: Uses the provided token for pagination.

• --no-style:

  • Description: Strips all styling from the table output.

• --order-by string:

  • Description: Organizes results based on a chosen field. Valid fields are id and created_at.

• --order-reversed:

  • Description: Reverses the order of the displayed results.

• --output format:

  • Description: Dictates the desired output format for the command. Options are table, csv, json, and yaml.

  • Default: table

• --pretty:

  • Description: Offers a more visually pleasing output for json and yaml formats.

• --wide:

  • Description: Presents full values in the table results, preventing truncation.

Global Flags

• --api-host string:

  • Description: Defines the host for client-server communication via REST. Overridden by the BACALHAU_API_HOST environment variable, if set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Sets the port for RESTful communication between the client and server. The BACALHAU_API_PORT environment variable takes precedence if set.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Designates the desired log format. Options include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Points to the bacalhau repository location.

  • Default: $HOME/.bacalhau

Examples

1. List all jobs:

   Execute the command to list all the jobs:

   Copy

   bacalhau job list

   Expected output:

   Copy

   CREATED   ID          JOB     TYPE   STATE
   08:19:07  d78a4cb4    docker  batch  Completed
   04:17:21  e45f31a7    docker  batch  Completed
   04:53:50  f4993f62    docker  batch  Completed
   ... (trimmed for brevity) ...

2. Limit the list to the last two jobs:

   Limit the list to display only the last two jobs:

   Copy

   bacalhau job list --limit 2

   Expected output:

   Copy

   CREATED   ID          JOB     TYPE   STATE
   03:14:16  19a26187    docker  batch  Completed
   21:47:21  2a53a13b    docker  batch  Completed

3. Order the list by creation date in descending order:

   Order the jobs by their creation date in a descending manner:

   Copy

   bacalhau job list --order-by created_at --order-reversed

   Expected output:

   Copy

   CREATED   ID          JOB     TYPE   STATE
   17:44:16  90e14efd    docker  batch  Completed
   17:44:08  8204570c    docker  batch  Completed
   17:43:50  f196521d    docker  batch  Completed
   ... (trimmed for brevity) ...

4. Filter the jobs by specific labels:

   Display jobs that have specific labels:

   Copy

   bacalhau job list --labels "region in (us-east-1, us-east-2),env = prod"

   Expected output:

   Copy

   ... (filtered jobs) ...

5. Display the list in JSON format with pretty printing:

   Get a limited list of jobs in a formatted JSON output:

   Copy

   bacalhau job list --limit 3 --output json --pretty

   Expected output:

   Copy

   ... [The JSON formatted output] ...

Description

The bacalhau job stop command allows users to terminate a previously submitted job. This is useful in scenarios where there's a need to halt a running job, perhaps due to misconfiguration or changed priorities.

Usage

bacalhau job stop [id] [flags]

Flags

• --quiet:

  • Description: If provided, the command will not display any output, neither to the standard output (stdout) nor to the standard error (stderr).

• -h, --help:

  • Description: Displays help information for the stop command.

Global Flags

• --api-host string:

  • Description: Specifies the host used for RESTful communication between the client and server. The flag is disregarded if BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for REST communication. If BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Selects the desired log format. Options include: default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. Stop a Specific Job:

   If you wish to halt the execution of a job, you can utilize the stop command. Here's how you can achieve that:

   Command:

   Copy

   bacalhau job stop j-10eb97de-14cd-4db4-96ec-561bb943309a

   Expected Output:

   Copy

   Checking job status
   
   	Connecting to network  ................  done ✅  0.0s
   	  Verifying job state  ................  done ✅  0.2s
   	          Stopping job ................  done ✅  0.1s
   
   Job stop successfully submitted with evaluation ID: 397fd425-8b1a-491e-952a-0632492e7ece

2. Silently Stop a Job:

   If you prefer to terminate a job without seeing any verbose feedback or messages, the --quiet option can be used.

   Command:

   Copy

   bacalhau job stop j-63b5ec0c-b5bf-4398-a152-b46c07abe52a --quiet

   Expected Output:

   Copy

   [No output displayed as the operation is run quietly.]

Description

The bacalhau job history command lists the history events of a specific job based on its ID. This feature allows users to track changes, executions, and other significant milestones associated with a particular job.

Usage

bacalhau job history [id] [flags]

Flags

• --event-type string:

  • Description: Specifies the type of history events to retrieve. Available options include all, job, and execution.

  • Default: all

• --execution-id string:

  • Description: Filters results by a specific execution ID.

• -h, --help:

  • Description: Display help for the history command.

• --hide-header:

  • Description: Opts out of printing the column headers in the results.

• --limit uint32:

  • Description: Limits the number of results returned.

• --next-token string:

  • Description: Uses the provided token for pagination.

• --no-style:

  • Description: Strips all styling from the table output.

• --node-id string:

  • Description: Filters the results by a specific node ID.

• --order-by string:

  • Description: Organizes results based on a chosen field.

• --order-reversed:

  • Description: Reverses the order of the displayed results.

• --output format:

  • Description: Dictates the desired output format for the command. Options are table, csv, json, and yaml.

  • Default: table

• --pretty:

  • Description: Offers a more visually pleasing output for json and yaml formats.

• --wide:

  • Description: Presents full values in the table results, preventing truncation.

Global Flags

• --api-host string:

  • Description: Defines the host for client-server communication via REST. Overridden by the BACALHAU_API_HOST environment variable, if set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Sets the port for RESTful communication between the client and server. The BACALHAU_API_PORT environment variable takes precedence if set.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Designates the desired log format. Options include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Points to the bacalhau repository location.

  • Default: $HOME/.bacalhau

Examples

1. Retrieve the history of a specific job:

   Execute the command to get the job history:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881

   Expected output:

   Copy

   TIME      LEVEL           EXEC. ID    ...     NEW STATE          COMMENT
   ... [The output rows like the ones you've shown] ...
   16:46:04  JobLevel                              2     Pending            Completed

2. Filter the history by event type:

   Filter the job history by the event type:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --event-type job

   Expected output:

   Copy

   TIME      LEVEL     EXEC. ID  NODE ID  REV.  PREVIOUS STATE  NEW STATE  COMMENT
   16:46:03  JobLevel                     1     Pending         Pending    Job created
   16:46:04  JobLevel                     2     Pending         Completed

3. Filter the history by execution ID:

   Filter the job history by a specific execution ID:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --execution-id e-99362435

   Expected output:

   Copy

   TIME      LEVEL           EXEC. ID    ...     NEW STATE          COMMENT
   ... [The output rows for the specific execution ID] ...
   16:46:04  ExecutionLevel  e-99362435  QmTSJgdN  6     BidAccepted        Completed

4. Retrieve the history in YAML format:

   Get the job history in YAML format:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --output yaml

   Expected output:

   Copy

   ... [The YAML formatted output] ...

Description

The bacalhau job run command facilitates the initiation of a job from a file or directly from the standard input (stdin). The command supports both JSON and YAML data formats. This command is particularly useful for quickly executing a job without the need for manual configurations.

Usage

bacalhau job run [flags]

Flags

• --dry-run:

  • Description: With this flag, the job will not be submitted. Instead, it will display what would have been submitted, providing a way to preview before actual submission.

• -f, --follow:

  • Description: If provided, the command will continuously display the output from the job as it runs.

• --id-only:

  • Description: On successful job submission, only the Job ID will be printed.

• --no-template:

  • Disable the templating feature. When this flag is set, the job spec will be used as-is, without any placeholder replacements

• --node-details:

  • Description: Displays details of all nodes. Note that this flag is overridden if --id-only is provided.

• --show-warnings:

  • Description: Shows any warnings that occur during the job submission.

• -E, --template-envs:

  • Specify a regular expression pattern for selecting environment variables to be included as template variables in the job spec. e.g. --template-envs ".*" will include all environment variables.

• -V, --template-vars:

  • Replace a placeholder in the job spec with a value. e.g. --template-vars foo=bar

• --wait:

  • Description: Waits for the job to finish execution. To set this to false, use --wait=false

  • Default: true

• --wait-timeout-secs int:

  • Description: If --wait is provided, this flag sets the maximum time (in seconds) the command will wait for the job to finish before it terminates.

  • Default: 600 seconds

• -h, --help:

  • Description: Displays help information for the run command.

Global Flags

• --api-host string:

  • Description: Specifies the host used for RESTful communication between the client and server. The flag is disregarded if BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for REST communication. If BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Selects the desired log format. Options include: default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

Sample Job (job.yaml)

A sample job used in the following examples is provided below:

cat job.yaml

name: A Simple Docker Job
type: batch
count: 1
tasks:
  - name: My main task
    engine:
      type: docker
      params:
        Image: ubuntu:latest
        Entrypoint:
          - /bin/bash
        Parameters:
          - -c
          - echo Hello Bacalhau!

This configuration describes a batch job that runs a Docker task. It utilizes the ubuntu:latest image and executes the command echo Hello Bacalhau!.

1. Running a Job using a YAML Configuration:

   To run a job with a configuration provided in a job.yaml file:

   Command:

   Copy

   bacalhau job run job.yaml

   Expected Output:

   Copy

   Job successfully submitted. Job ID: j-2d0f513a-9eb1-49c2-8bc8-246c6fb41520
   Checking job status... (Enter Ctrl+C to exit at any time, your job will continue running):
   
    Communicating with the network  ................  done ✅  0.1s
       Creating job for submission  ................  done ✅  0.6s
   
   To get more details about the run, execute:
    bacalhau job describe j-2d0f513a-9eb1-49c2-8bc8-246c6fb41520
   
   To get more details about the run executions, execute:
    bacalhau job executions j-2d0f513a-9eb1-49c2-8bc8-246c6fb41520

2. Running a Job and Following its Logs:

   Command:

   Copy

   bacalhau job run job.yaml --follow

   Expected Output:

   Copy

   Job successfully submitted. Job ID: j-b89df816-7564-4f04-b270-e6cda89eda72
   Waiting for logs... (Enter Ctrl+C to exit at any time, your job will continue running):
   
   Hello Bacalhau!

3. Running a Job Without Waiting:

   Command:

   Copy

   bacalhau job run job.yaml --wait=false

   Expected Output:

   Copy

   j-3fd396b3-e92e-42ca-bd87-0dc9eb15e6f9

4. Fetching Only the Job ID Upon Submission:

   Command:

   Copy

   bacalhau job run job.yaml --id-only

   Expected Output:

   Copy

   j-5976ffb6-3465-4fec-8b3b-2c822cbaf417

5. Fetching Only the Job ID and Wait for Completion:

   Command:

   Copy

   bacalhau job run job.yaml --id-only --wait

   Expected Output:

   Copy

   j-293f1302-3298-4aca-b06d-33fd1e3f9d2c

6. Running a Job with Node Details:

   Command:

   Copy

   bacalhau job run job.yaml --node-details

   Expected Output:

   Copy

   Job successfully submitted. Job ID: j-05e65dd3-4e9e-4e20-a104-3c91ba934435
   Checking job status... (Enter Ctrl+C to exit at any time, your job will continue running):
   
    Communicating with the network  ................  done ✅  0.1s
       Creating job for submission  ................  done ✅  0.6s
   
   Job Results By Node:
   • Node QmVXwmdZ:
    Hello Bacalhau!
   
   To get more details about the run, execute:
    bacalhau job describe j-05e65dd3-4e9e-4e20-a104-3c91ba934435
   
   To get more details about the run executions, execute:
    bacalhau job executions j-05e65dd3-4e9e-4e20-a104-3c91ba934435

7. Rerunning a previously submitting job:

   Command:

   Copy

   bacalhau job describe j-05e65dd3-4e9e-4e20-a104-3c91ba934435 | bacalhau job run

   Expected Output:

   Copy

   Reading from /dev/stdin; send Ctrl-d to stop.Job successfully submitted. Job ID: j-d8625929-83f4-411a-b9aa-7bcfecb27a8b
   Checking job status... (Enter Ctrl+C to exit at any time, your job will continue running):
   
    Communicating with the network  ................  done ✅  0.1s
       Creating job for submission  ................  done ✅  0.6s
   
   To get more details about the run, execute:
    bacalhau job describe j-d8625929-83f4-411a-b9aa-7bcfecb27a8b
   
   To get more details about the run executions, execute:
    bacalhau job executions j-d8625929-83f4-411a-b9aa-7bcfecb27a8b

Templating

bacalhau job run providing users with the ability to dynamically inject variables into their job specifications. This feature is particularly useful when running multiple jobs with varying parameters such as S3 buckets, prefixes, and time ranges without the need to edit each job specification file manually. You can find more information about templating here.

Description

The bacalhau job command provides a suite of sub-commands to submit, query, and manage jobs within Bacalhau. Users can deploy jobs, obtain job details, track execution logs, and more.

Usage

bacalhau job [command]

Available Commands

1. describe:

   • Description: Retrieves detailed information of a job using its ID.

   • Usage:

     Copy

     bacalhau job describe

2. executions:

   • Description: Lists all executions associated with a job, identified by its ID.

   • Usage:

     Copy

     bacalhau job executions

3. history:

   • Description: Enumerates the historical events related to a job, identified by its ID.

   • Usage:

     Copy

     bacalhau job history

4. list:

   • Description: Provides an overview of all submitted jobs.

   • Usage:

     Copy

     bacalhau job list

5. logs:

   • Description: Fetches and streams the logs from a currently executing job.

   • Usage:

     Copy

     bacalhau job logs

6. run:

   • Description: Submits a job for execution using either a JSON or YAML configuration file.

   • Usage:

     Copy

     bacalhau job run

7. stop:

   • Description: Halts a previously submitted job.

   • Usage:

     Copy

     bacalhau job stop

For comprehensive details on any of the sub-commands, run:

bacalhau job [command] --help

Flags

• -h, --help:

  • Description: Shows the help information for the job command.

Global Flags

• --api-host string:

  • Description: Determines the host for RESTful communication between the client and server. This flag is overlooked if the BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Designates the port for RESTful communication. This flag is bypassed if the BACALHAU_API_PORT environment variable is active.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Chooses the preferred log format. Available choices are: default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Specifies the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Description

The bacalhau exec command allows for the specification of jobs to be executed from the command line, without the need for a job specification file (see ).

Usage

Flags

• -h, --help:

  • Description: Displays help information for the exec sub-command.

• --code:

  • Includes the specified code in the job. This can be a single file, or a directory containing many files. There is a limit of 10Mb on the size of the uploaded code.

• -f, --follow:

  • Description: If provided, the command will continuously display the output from the job as it runs.

• --wait

  • Description: Wait for the job to finish. Use --wait=false to return as soon as the job is submitted.

• --wait-timeout-secs

  • Description: When using --wait, how many seconds to wait for the job to complete before giving up.

• --node-details

  • Description: Print out details of all nodes (overridden by --id-only).

• --id-only:

  • Description: On successful job submission, only the Job ID will be printed.

• -p, --publisher

  • Description: Where to publish the result of the job.

    Examples:

    Publish to IPFS

    -p ipfs

    Publish to S3

    -p s3://bucket/key

• -i, --input

  • Description: Mount URIs as inputs to the job. Can be specified multiple times. Format: src=URI,dst=PATH[,opt=key=value]

    Examples:

    Mount IPFS CID to /inputs directory

    -i ipfs://QmeZRGhe4PmjctYVSVHuEiA9oSXnqmYa4kQubSHgWbjv72

    Mount S3 object to a specific path

    -i s3://bucket/key,dst=/my/input/path

    Mount S3 object with specific endpoint and region

    -i src=s3://bucket/key,dst=/my/input/path,opt=endpoint=https://s3.example.com,opt=region=us-east-1

• -o, --output

  • Description: name:path of the output data volumes. 'outputs:/outputs' is always added unless '/outputs' is mapped to a different name.

• -e, --env

  • Description: The environment variables to supply to the job (e.g. --env FOO=bar --env BAR=baz)

• --timeout

  • Description: Job execution timeout in seconds (e.g. 300 for 5 minutes)

• -l, --labels

  • Description: List of labels for the job. Enter multiple in the format '-l a -l 2'. All characters not matching /a-zA-Z0-9_:|-/ and all emojis will be stripped.

• -s, --selector

  • Description: Selector (label query) to filter nodes on which this job can be executed, supports '=', '==', and '!='.(e.g. -s key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.

Global Flags

• --api-host string:

  • Description: Specifies the host used for RESTful communication between the client and server. The flag is disregarded if the BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Specifies the port for REST communication. If the BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Sets the desired log format. Options are: default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: ``$HOME/.bacalhau`

Examples

Running python tasks

1. Basic Usage:

   Command:

   Output:
2. Single file Python:

   Command:

   where app.py is

   Output:

   As red text

Running duckdb queries

1. Basic Usage:

   Command:

   Output:

Description

The bacalhau job logs command allows users to retrieve logs from a job that has been previously submitted. This command is useful for tracking and debugging the progress and state of a running or completed job.

Usage

Flags

• -f, --follow:

  • Description: This flag allows the user to follow the logs in real-time after fetching the current logs. It provides a continuous stream of log updates, similar to tail -f in Unix-like systems.

• -h, --help:

  • Description: Display help information for the logs command.

Global Flags

• --api-host string:

  • Description: Specifies the host for the client and server to communicate through REST. This flag is disregarded if the BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Sets the port for RESTful communication between the client and server. If the BACALHAU_API_PORT environment variable is available, this flag is ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Determines the desired log format. Available options include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Specifies the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. Display Logs for a Previously Submitted Job with Full ID:

   Command:

   Expected Output:
2. Follow Logs in Real-Time:

   Command:

   Expected Output:
3. Display Logs Using a Shortened ID:

   Command:

   Expected Output: