Description

The bacalhau job executions command retrieves a list of executions for a specific job based on its ID. This can be essential when tracking the various runs and their respective states for a particular job.

Usage

bacalhau job executions [id] [flags]

Flags

• -h, --help:

  • Description: Display help for the executions command.

• --hide-header:

  • Description: Do not print the column headers when displaying the results.

• --limit uint32:

  • Description: Restricts the number of results returned.

  • Default: 20

• --next-token string:

  • Description: Uses the specified token for pagination. Useful for fetching the next set of results.

• --no-style:

  • Description: Removes all styling from the table output, displaying raw data.

• --order-by string:

  • Description: Orders results based on a specific field. Valid fields are: modify_time, create_time, id, and state.

• --order-reversed:

  • Description: Reverses the order of the results. Useful in conjunction with --order-by.

• --output format:

  • Description: Specifies the desired output format for the command. Supported values are table, csv, json, and yaml.

  • Default: table

• --pretty:

  • Description: Pretty prints the output. This option is applicable only to json and yaml output formats.

• --wide:

  • Description: Prints full values in the table results without truncating any information.

Global Flags

• --api-host string:

  • Description: Specifies the host for the client and server to communicate through via REST. If the BACALHAU_API_HOST environment variable is set, this flag will be ignored.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for the client and server to communicate on using REST. If the BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Specifies the desired log format. Supported values include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. List executions for a specific Job:

   Copy

   bacalhau job executions j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881

   Expected output:

   Copy

   CREATED   MODIFIED  ID          NODE ID   REV.  COMPUTE    DESIRED  COMMENT
                                                   STATE      STATE
   16:46:03  16:46:04  e-99362435  QmTSJgdN  6     Completed  Stopped
   16:46:03  16:46:04  e-75dd20bb  QmXRdLru  6     Completed  Stopped
   16:46:03  16:46:04  e-03870df5  QmVXwmdZ  6     Completed  Stopped

2. Order executions by state for a specific job:

   Execute the command:

   Copy

   bacalhau job executions j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --order-by state

   Expected output:

   Copy

   CREATED   MODIFIED  ID          NODE ID   REV.  COMPUTE    DESIRED  COMMENT
                                                   STATE      STATE
   16:46:03  16:46:04  e-03870df5  QmVXwmdZ  6     Completed  Stopped
   16:46:03  16:46:04  e-75dd20bb  QmXRdLru  6     Completed  Stopped
   16:46:03  16:46:04  e-99362435  QmTSJgdN  6     Completed  Stopped

3. List executions with YAML output:

   Copy

   bacalhau job executions j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --output yaml

   Expected output:

   Copy

   ... [The YAML formatted output] ...

Description

The bacalhau job describe command provides a detailed description of a specific job in YAML format. This description can be particularly useful when wanting to understand the attributes and current status of a specific job. To list all available jobs, the bacalhau job list command can be used.

Usage

Flags

• -h, --help:

  • Description: Display help for the describe command.

• --output format:

  • Description: Specifies the desired output format for the command. Supported values are json and yaml.

  • Default: yaml

• --pretty:

  • Description: Pretty prints the output. This option is applicable only to json and yaml output formats.

Global Flags

• --api-host string:

  • Description: Specifies the host for the client and server to communicate through via REST. If the BACALHAU_API_HOST environment variable is set, this flag will be ignored.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for the client and server to communicate on using REST. If the BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Specifies the desired log format. Supported values include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. Describe a Job with Full ID:
2. Describe a Job with Shortened ID:
3. Describe a Job with JSON Output:

Description

The bacalhau job command provides a suite of sub-commands to submit, query, and manage jobs within Bacalhau. Users can deploy jobs, obtain job details, track execution logs, and more.

Usage

bacalhau job [command]

Available Commands

1. describe:

   • Description: Retrieves detailed information of a job using its ID.

   • Usage:

     Copy

     bacalhau job describe

2. executions:

   • Description: Lists all executions associated with a job, identified by its ID.

   • Usage:

     Copy

     bacalhau job executions

3. history:

   • Description: Enumerates the historical events related to a job, identified by its ID.

   • Usage:

     Copy

     bacalhau job history

4. list:

   • Description: Provides an overview of all submitted jobs.

   • Usage:

     Copy

     bacalhau job list

5. logs:

   • Description: Fetches and streams the logs from a currently executing job.

   • Usage:

     Copy

     bacalhau job logs

6. run:

   • Description: Submits a job for execution using either a JSON or YAML configuration file.

   • Usage:

     Copy

     bacalhau job run

7. stop:

   • Description: Halts a previously submitted job.

   • Usage:

     Copy

     bacalhau job stop

For comprehensive details on any of the sub-commands, run:

bacalhau job [command] --help

Flags

• -h, --help:

  • Description: Shows the help information for the job command.

Global Flags

1. --api-host string:

   • Description: Determines the host for RESTful communication between the client and server. This flag is overlooked if the BACALHAU_API_HOST environment variable is set.

   • Default: bootstrap.production.bacalhau.org

2. --api-port int:

   • Description: Designates the port for RESTful communication. This flag is bypassed if the BACALHAU_API_PORT environment variable is active.

   • Default: 1234

3. --log-mode logging-mode:

   • Description: Chooses the preferred log format. Available choices are: default, station, json, combined, and event.

   • Default: default

4. --repo string:

   • Description: Specifies the path to the bacalhau repository.

   • Default: $HOME/.bacalhau

Description

The bacalhau job list command provides a listing of all submitted jobs. This command offers an overview of all tasks and processes registered in the system, allowing users to monitor and manage their jobs effectively.

Usage

bacalhau job list [flags]

Flags

• -h, --help:

  • Description: Display help for the list command.

• --hide-header:

  • Description: Opts out of printing the column headers in the results.

• --labels string:

  • Description: Filters jobs by labels. It's designed to function similar to Kubernetes label selectors.

  • Default: bacalhau_canary != true

• --limit uint32:

  • Description: Limits the number of results returned.

  • Default: 10

• --next-token string:

  • Description: Uses the provided token for pagination.

• --no-style:

  • Description: Strips all styling from the table output.

• --order-by string:

  • Description: Organizes results based on a chosen field. Valid fields are id and created_at.

• --order-reversed:

  • Description: Reverses the order of the displayed results.

• --output format:

  • Description: Dictates the desired output format for the command. Options are table, csv, json, and yaml.

  • Default: table

• --pretty:

  • Description: Offers a more visually pleasing output for json and yaml formats.

• --wide:

  • Description: Presents full values in the table results, preventing truncation.

Global Flags

• --api-host string:

  • Description: Defines the host for client-server communication via REST. Overridden by the BACALHAU_API_HOST environment variable, if set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Sets the port for RESTful communication between the client and server. The BACALHAU_API_PORT environment variable takes precedence if set.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Designates the desired log format. Options include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Points to the bacalhau repository location.

  • Default: $HOME/.bacalhau

Examples

1. List all jobs:

   Execute the command to list all the jobs:

   Copy

   bacalhau job list

   Expected output:

   Copy

   CREATED   ID          JOB     TYPE   STATE
   08:19:07  d78a4cb4    docker  batch  Completed
   04:17:21  e45f31a7    docker  batch  Completed
   04:53:50  f4993f62    docker  batch  Completed
   ... (trimmed for brevity) ...

2. Limit the list to the last two jobs:

   Limit the list to display only the last two jobs:

   Copy

   bacalhau job list --limit 2

   Expected output:

   Copy

   CREATED   ID          JOB     TYPE   STATE
   03:14:16  19a26187    docker  batch  Completed
   21:47:21  2a53a13b    docker  batch  Completed

3. Order the list by creation date in descending order:

   Order the jobs by their creation date in a descending manner:

   Copy

   bacalhau job list --order-by created_at --order-reversed

   Expected output:

   Copy

   CREATED   ID          JOB     TYPE   STATE
   17:44:16  90e14efd    docker  batch  Completed
   17:44:08  8204570c    docker  batch  Completed
   17:43:50  f196521d    docker  batch  Completed
   ... (trimmed for brevity) ...

4. Filter the jobs by specific labels:

   Display jobs that have specific labels:

   Copy

   bacalhau job list --labels "region in (us-east-1, us-east-2),env = prod"

   Expected output:

   Copy

   ... (filtered jobs) ...

5. Display the list in JSON format with pretty printing:

   Get a limited list of jobs in a formatted JSON output:

   Copy

   bacalhau job list --limit 3 --output json --pretty

   Expected output:

   Copy

   ... [The JSON formatted output] ...

Description

The bacalhau job history command lists the history events of a specific job based on its ID. This feature allows users to track changes, executions, and other significant milestones associated with a particular job.

Usage

bacalhau job history [id] [flags]

Flags

• --event-type string:

  • Description: Specifies the type of history events to retrieve. Available options include all, job, and execution.

  • Default: all

• --execution-id string:

  • Description: Filters results by a specific execution ID.

• -h, --help:

  • Description: Display help for the history command.

• --hide-header:

  • Description: Opts out of printing the column headers in the results.

• --limit uint32:

  • Description: Limits the number of results returned.

• --next-token string:

  • Description: Uses the provided token for pagination.

• --no-style:

  • Description: Strips all styling from the table output.

• --node-id string:

  • Description: Filters the results by a specific node ID.

• --order-by string:

  • Description: Organizes results based on a chosen field.

• --order-reversed:

  • Description: Reverses the order of the displayed results.

• --output format:

  • Description: Dictates the desired output format for the command. Options are table, csv, json, and yaml.

  • Default: table

• --pretty:

  • Description: Offers a more visually pleasing output for json and yaml formats.

• --wide:

  • Description: Presents full values in the table results, preventing truncation.

Global Flags

• --api-host string:

  • Description: Defines the host for client-server communication via REST. Overridden by the BACALHAU_API_HOST environment variable, if set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Sets the port for RESTful communication between the client and server. The BACALHAU_API_PORT environment variable takes precedence if set.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Designates the desired log format. Options include default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Points to the bacalhau repository location.

  • Default: $HOME/.bacalhau

Examples

1. Retrieve the history of a specific job:

   Execute the command to get the job history:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881

   Expected output:

   Copy

   TIME      LEVEL           EXEC. ID    ...     NEW STATE          COMMENT
   ... [The output rows like the ones you've shown] ...
   16:46:04  JobLevel                              2     Pending            Completed

2. Filter the history by event type:

   Filter the job history by the event type:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --event-type job

   Expected output:

   Copy

   TIME      LEVEL     EXEC. ID  NODE ID  REV.  PREVIOUS STATE  NEW STATE  COMMENT
   16:46:03  JobLevel                     1     Pending         Pending    Job created
   16:46:04  JobLevel                     2     Pending         Completed

3. Filter the history by execution ID:

   Filter the job history by a specific execution ID:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --execution-id e-99362435

   Expected output:

   Copy

   TIME      LEVEL           EXEC. ID    ...     NEW STATE          COMMENT
   ... [The output rows for the specific execution ID] ...
   16:46:04  ExecutionLevel  e-99362435  QmTSJgdN  6     BidAccepted        Completed

4. Retrieve the history in YAML format:

   Get the job history in YAML format:

   Copy

   bacalhau job history j-6f2bf0ea-ebcd-4490-899a-9de9d8d95881 --output yaml

   Expected output:

   Copy

   ... [The YAML formatted output] ...

Description

The bacalhau exec command allows for the specification of jobs to be executed from the command line, without the need for a job specification file (see job run).

Usage

bacalhau exec [flags] [job-type] arguments

Flags

1. -h, --help:

   • Description: Displays help information for the exec sub-command.

2. --code:

   • Includes the specified code in the job. This can be a single file, or a directory containing many files. There is a limit of 10Mb on the size of the uploaded code.

3. -f, --follow:

   • Description: If provided, the command will continuously display the output from the job as it runs.

4. --wait

   • Description: Wait for the job to finish. Use --wait=false to return as soon as the job is submitted.

5. --wait-timeout-secs

   • Description: When using --wait, how many seconds to wait for the job to complete before giving up.

6. --node-details

   • Description: Print out details of all nodes (overridden by --id-only).

7. --id-only:

   • Description: On successful job submission, only the Job ID will be printed.

8. -p, --publisher

   • Description: Where to publish the result of the job.

     Examples:

     Publish to IPFS

     -p ipfs

     Publish to S3

     -p s3://bucket/key

9. -i, --input

   • Description: Mount URIs as inputs to the job. Can be specified multiple times. Format: src=URI,dst=PATH[,opt=key=value]

     Examples:

     Mount IPFS CID to /inputs directory

     -i ipfs://QmeZRGhe4PmjctYVSVHuEiA9oSXnqmYa4kQubSHgWbjv72

     Mount S3 object to a specific path

     -i s3://bucket/key,dst=/my/input/path

     Mount S3 object with specific endpoint and region

     -i src=s3://bucket/key,dst=/my/input/path,opt=endpoint=https://s3.example.com,opt=region=us-east-1

10. -o, --output

    • Description: name:path of the output data volumes. 'outputs:/outputs' is always added unless '/outputs' is mapped to a different name.

11. -e, --env

    • Description: The environment variables to supply to the job (e.g. --env FOO=bar --env BAR=baz)

12. --timeout

    • Description: Job execution timeout in seconds (e.g. 300 for 5 minutes)

13. -l, --labels

    • Description: List of labels for the job. Enter multiple in the format '-l a -l 2'. All characters not matching /a-zA-Z0-9_:|-/ and all emojis will be stripped.

14. -s, --selector

    • Description: Selector (label query) to filter nodes on which this job can be executed, supports '=', '==', and '!='.(e.g. -s key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.

Global Flags

1. --api-host string:

   • Description: Specifies the host used for RESTful communication between the client and server. The flag is disregarded if the BACALHAU_API_HOST environment variable is set.

   • Default: bootstrap.production.bacalhau.org

2. --api-port int:

   • Description: Specifies the port for REST communication. If the BACALHAU_API_PORT environment variable is set, this flag will be ignored.

   • Default: 1234

3. --log-mode logging-mode:

   • Description: Sets the desired log format. Options are: default, station, json, combined, and event.

   • Default: default

4. --repo string:

   • Description: Defines the path to the bacalhau repository.

   • Default: ``$HOME/.bacalhau`

Examples

Running python tasks

1. Basic Usage:

   Command:

   Copy

   bacalhau exec python -- -c "import this"

   Output:

   Copy

   The Zen of Python, by Tim Peters
   
   Beautiful is better than ugly.
   Explicit is better than implicit.
   Simple is better than complex.
   Complex is better than complicated.
   Flat is better than nested.
   ....

2. Single file Python:

   Command:

   Copy

   bacalhau exec --code=app.py python app.py

   where app.py is

   Copy

   """
   pip install colorama
   """
   
   from colorama import Fore
   print(Fore.RED + "Hello World")

   Output:

   As red text

   Copy

   Hello World

Running duckdb queries

1. Basic Usage:

   Command:

   Copy

   cat describe.sql
     DESCRIBE TABLE '/inputs/world-cities_csv.csv';
   
   bacalhau exec --code=describe.sql -i src=https://datahub.io/core/world-cities/r/world-cities.csv,dst=/inputs duckdb -- -init /code/describe.sql

   Output:

   Copy

        ┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
        │ column_name │ column_type │  null   │   key   │ default │  extra  │
        │   varchar   │   varchar   │ varchar │ varchar │ varchar │ varchar │
        ├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
        │ name        │ VARCHAR     │ YES     │         │         │         │
        │ country     │ VARCHAR     │ YES     │         │         │         │
        │ subcountry  │ VARCHAR     │ YES     │         │         │         │
        │ geonameid   │ BIGINT      │ YES     │         │         │         │
        └─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘

Description

The bacalhau job logs command allows users to retrieve logs from a job that has been previously submitted. This command is useful for tracking and debugging the progress and state of a running or completed job.

Usage

bacalhau job logs [id] [flags]

Flags

1. -f, --follow:

   • Description: This flag allows the user to follow the logs in real-time after fetching the current logs. It provides a continuous stream of log updates, similar to tail -f in Unix-like systems.

2. -h, --help:

   • Description: Display help information for the logs command.

Global Flags

1. --api-host string:

   • Description: Specifies the host for the client and server to communicate through REST. This flag is disregarded if the BACALHAU_API_HOST environment variable is set.

   • Default: bootstrap.production.bacalhau.org

2. --api-port int:

   • Description: Sets the port for RESTful communication between the client and server. If the BACALHAU_API_PORT environment variable is available, this flag is ignored.

   • Default: 1234

3. --log-mode logging-mode:

   • Description: Determines the desired log format. Available options include default, station, json, combined, and event.

   • Default: default

4. --repo string:

   • Description: Specifies the path to the bacalhau repository.

   • Default: $HOME/.bacalhau

Examples

1. Display Logs for a Previously Submitted Job with Full ID:

   Command:

   Copy

   bacalhau job logs j-51225160-807e-48b8-88c9-28311c7899e1

   Expected Output:

   Copy

   [2023-09-24 09:01:32] INFO - Application started successfully.
   [2023-09-24 09:01:33] DEBUG - Initializing database connections.
   [2023-09-24 09:01:35] WARN - API rate limit approaching.
   [2023-09-24 09:02:01] ERROR - Failed to retrieve data from endpoint: /api/v1/data.
   [2023-09-24 09:05:00] INFO - Data sync completed with 4500 new records.

2. Follow Logs in Real-Time:

   Command:

   Copy

   bacalhau job logs --follow j-51225160-807e-48b8-88c9-28311c7899e1

   Expected Output:

   Copy

   [2023-09-24 11:30:02] INFO - User 'john_doe' logged in successfully.
   [2023-09-24 11:30:15] DEBUG - Fetching data from cache for key: userSettings_john_doe.
   [2023-09-24 11:31:05] WARN - High memory usage detected: 85% of allocated resources.
   ... [Logs continue to appear in real-time] ...

3. Display Logs Using a Shortened ID:

   Command:

   Copy

   bacalhau job logs j-ebd9bf2f

   Expected Output:

   Copy

   [2023-09-24 10:15:12] INFO - Application initialization sequence started.
   [2023-09-24 10:15:13] DEBUG - Loading configurations from /config/app.json.
   [2023-09-24 10:15:14] INFO - Connected to message broker successfully.
   [2023-09-24 10:16:00] ERROR - Failed to send email notification to user@example.com.

Description

The bacalhau job stop command allows users to terminate a previously submitted job. This is useful in scenarios where there's a need to halt a running job, perhaps due to misconfiguration or changed priorities.

Usage

Flags

• --quiet:

  • Description: If provided, the command will not display any output, neither to the standard output (stdout) nor to the standard error (stderr).

• -h, --help:

  • Description: Displays help information for the stop command.

Global Flags

• --api-host string:

  • Description: Specifies the host used for RESTful communication between the client and server. The flag is disregarded if BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for REST communication. If BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Selects the desired log format. Options include: default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

1. Stop a Specific Job:

   If you wish to halt the execution of a job, you can utilize the stop command. Here's how you can achieve that:

   Command:

   Expected Output:
2. Silently Stop a Job:

   If you prefer to terminate a job without seeing any verbose feedback or messages, the --quiet option can be used.

   Command:

   Expected Output:

Description

The bacalhau job run command facilitates the initiation of a job from a file or directly from the standard input (stdin). The command supports both JSON and YAML data formats. This command is particularly useful for quickly executing a job without the need for manual configurations.

Usage

Flags

• --dry-run:

  • Description: With this flag, the job will not be submitted. Instead, it will display what would have been submitted, providing a way to preview before actual submission.

• -f, --follow:

  • Description: If provided, the command will continuously display the output from the job as it runs.

• --id-only:

  • Description: On successful job submission, only the Job ID will be printed.

• --no-template:

  • Disable the templating feature. When this flag is set, the job spec will be used as-is, without any placeholder replacements

• --node-details:

  • Description: Displays details of all nodes. Note that this flag is overridden if --id-only is provided.

• --show-warnings:

  • Description: Shows any warnings that occur during the job submission.

• -E, --template-envs:

  • Specify a regular expression pattern for selecting environment variables to be included as template variables in the job spec. e.g. --template-envs ".*" will include all environment variables.

• -V, --template-vars:

  • Replace a placeholder in the job spec with a value. e.g. --template-vars foo=bar

• --wait:

  • Description: Waits for the job to finish execution. To set this to false, use --wait=false

  • Default: true

• --wait-timeout-secs int:

  • Description: If --wait is provided, this flag sets the maximum time (in seconds) the command will wait for the job to finish before it terminates.

  • Default: 600 seconds

• -h, --help:

  • Description: Displays help information for the run command.

Global Flags

• --api-host string:

  • Description: Specifies the host used for RESTful communication between the client and server. The flag is disregarded if BACALHAU_API_HOST environment variable is set.

  • Default: bootstrap.production.bacalhau.org

• --api-port int:

  • Description: Determines the port for REST communication. If BACALHAU_API_PORT environment variable is set, this flag will be ignored.

  • Default: 1234

• --log-mode logging-mode:

  • Description: Selects the desired log format. Options include: default, station, json, combined, and event.

  • Default: default

• --repo string:

  • Description: Defines the path to the bacalhau repository.

  • Default: $HOME/.bacalhau

Examples

Sample Job (job.yaml)

A sample job used in the following examples is provided below:

This configuration describes a batch job that runs a Docker task. It utilizes the ubuntu:latest image and executes the command echo Hello Bacalhau!.

1. Running a Job using a YAML Configuration:

   To run a job with a configuration provided in a job.yaml file:

   Command:

   Expected Output:
2. Running a Job and Following its Logs:

   Command:

   Expected Output:
3. Running a Job Without Waiting:

   Command:

   Expected Output:
4. Fetching Only the Job ID Upon Submission:

   Command:

   Expected Output:
5. Fetching Only the Job ID and Wait for Completion:

   Command:

   Expected Output:
6. Running a Job with Node Details:

   Command:

   Expected Output:
7. Rerunning a previously submitting job:

   Command:

   Expected Output:

Templating