Authentication and authorization

Bacalhau includes a flexible auth system that supports multiple methods of auth that are appropriate for different deployment environments.

By default

With no specific authentication configuration supplied, Bacalhau runs in "anonymous mode" – which allows unidentified users limited control over the system. "Anonymous mode" is only appropriate for testing or evaluation setups.

In anonymous mode, Bacalhau will allow:

• Users identified by a self-generated private key to submit any job and cancel their own jobs.

• Users not identified by any key to access other read-only endpoints, such as to read job lists, describe jobs, and query node or agent information.

Restricting anonymous access

Bacalhau auth is controlled by policies. Configuring the auth system is done by supplying a different policy file.

Restricting API access to only users that have authenticated requires specifying a new authorization policy. You can download a policy that restricts anonymous access and install it by using:

curl -sL https://raw.githubusercontent.com/bacalhau-project/bacalhau/main/pkg/authz/policies/policy_ns_anon.rego -o ~/.bacalhau/no-anon.rego
bacalhau config set Auth.AccessPolicyPath ~/.bacalhau/no-anon.rego

Once the node is restarted, accessing the node APIs will require the user to be authenticated, but by default will still allow users with a self-generated key to authenticate themselves.

Restricting the list of keys that can authenticate to only a known set requires specifying a new authentication policy. You can download a policy that restricts key-based access and install it by using:

curl -sL https://raw.githubusercontent.com/bacalhau-project/bacalhau/main/pkg/authn/challenge/challenge_ns_no_anon.rego -o ~/.bacalhau/challenge_ns_no_anon.rego
bacalhau config set Auth.Methods '\{Method: ClientKey, Policy: \{Type: challenge, PolicyPath: ~/.bacalhau/challenge_ns_no_anon.rego\}\}'

Then, modify the allowed_clients variable in challange_ns_no_anon.rego to include acceptable client IDs, found by running bacalhau id.

bacalhau id | jq -rc .ClientID

Once the node is restarted, only keys in the allowed list will be able to access any API.

Username and password access

Users can authenticate using a username and password instead of specifying a private key for access. Again, this requires installation of an appropriate policy on the server.

curl -sL https://raw.githubusercontent.com/bacalhau-project/bacalhau/main/pkg/authn/ask/ask_ns_password.rego -o ~/.bacalhau/ask_ns_password.rego
bacalhau config set Auth.Methods '\{Method: Password, Policy: \{Type: ask, PolicyPath: ~/.bacalhau/ask_ns_password.rego\}\}'

Passwords are not stored in plaintext and are salted. The downloaded policy expects password hashes and salts generated by scrypt. To generate a salted password, the helper script in pkg/authn/ask/gen_password can be used:

cd pkg/authn/ask/gen_password && go run .

This will ask for a password and generate a salt and hash to authenticate with it. Add the encoded username, salt and hash into the ask_ns_password.rego.

Writing custom policies

In principle, Bacalhau can implement any auth scheme that can be described in a structured way by a policy file.

Policies are written in a language called Rego, also used by Kubernetes. Users who want to write their own policies should get familiar with the Rego language.

Custom authentication policies

Bacalhau will pass information pertinent to the current request into every authentication policy query as a field on the input variable. The exact information depends on the type of authentication used.

challenge authentication

challenge authentication uses identifies the user by the presence of a private key. The user is asked to sign an input phrase to prove they have the key they are identifying with.

Policies used for challenge authentication do not need to actually implement the challenge verification logic as this is handled by the core code. Instead, they will only be invoked if this verification passes.

Policies for this type will need to implement these rules:

• bacalhau.authn.token: if the user should be authenticated, an access token they should use in subsequent requests. If the user should not be authenticated, should be undefined.

They should expect as fields on the input variable:

• clientId: an ID derived from the user's private key that identifies them uniquely

• nodeId: the ID of the requester node that this user is authenticating with

• signingKey: the private key (as a JWK) that should be used to sign any access tokens to be returned

The simplest possible policy might therefore be this policy that returns the same opaque token for all users:

package bacalhau.authn

token := "anything"

A more realistic example that returns a signed JWT is in challenge_ns_anon.rego.

ask authentication

ask authentication uses credentials supplied manually by the user as identification. For example, an ask policy could require a username and password as input and check these against a known list. ask policies do all the verification of the supplied credentials.

Policies for this type will need to implement these rules:

• bacalhau.authn.token: if the user should be authenticated, an access token they should use in subsequent requests. If the user should not be authenticated, should be undefined.

• bacalhau.authn.schema: a static JSON schema that should be used to collect information about the user. The type of declared fields may be used to pick the input method, and if a field is marked as writeOnly then it will be collected in a secure way (e.g. not shown on screen). The schema rule does not receive any input data.

They should expect as fields on the input variable:

• ask: a map of field names from the JSON schema to strings supplied by the user. The policy should validate these credentials.

• nodeId: the ID of the requester node that this user is authenticating with

• signingKey: the private key (as a JWK) that should be used to sign any access tokens to be returned

The simplest possible policy might therefore be one that asks for no data and returns the same opaque token for every user:

package bacalhau.authn

schema := {}
token := "anything"

A more realistic example that returns a signed JWT is in ask_ns_example.rego.

Custom authorization policies

Authorization policies do not vary depending on the type of authentication used – Bacalhau uses one authz policy for all API requests.

Authz policies are invoked for every API request. Authz policies should check the validity of any supplied access tokens and issue an authz decision for the requested API endpoint. It is not required that authz policies enforce that an access token is present – they may choose to grant access to unauthorized users.

Policies will need to implement these rules:

• bacalhau.authz.token_valid: true if the access token in the request is "valid" (but does not necessarily grant access for this request), or false if it is invalid for every request (e.g. because it has expired) and should be discarded.

• bacalhau.authz.allow: true if the user should be permitted to carry out the input request, false otherwise.

They should expect as fields on the input variable for both rules:

• http: details of the user's HTTP request:

  • host: the hostname used in the HTTP request

  • method: the HTTP method (e.g. GET, POST)

  • path: the path requested, as an array of path components without slashes

  • query: a map of URL query parameters to their values

  • headers: a map of HTTP header names to arrays representing their values

  • body: a blob of any content submitted as the body

• constraints: details about the receiving node that should be used to validate any supplied tokens:

  • cert: keys that the input token should have been signed with

  • iss: the name of a node that this node will recognize as the issuer of any signed tokens

  • aud: the name of this node that is receiving the request

Notably, the constraints data is appropriate to be passed directly to the Rego io.jwt.decode_verify method which will validate the access token as a JWT against the given constraints.

The simplest possible authz policy might be this one that allows all users to access all endpoints:

package bacalhau.authz

allow := true
token_valid := true

A more realistic example (which is the Bacalhau "anonymous mode" default) is in policy_ns_anon.rego.

Bacalhau supports GPUs out of the box and defaults to allowing execution on all GPUs installed on the node.

Prerequisites

Bacalhau makes the assumption that you have installed all the necessary drivers and tools on your node host and have appropriately configured them for use by Docker.

In general for GPUs from any vendor, the Bacalhau client requires:

• Docker

• Permission to access Docker

Nvidia

• Drivers for your GPU

• NVIDIA Container Toolkit (nvidia-docker2)

• nvidia-smi installed and functional

You can test whether you have a working GPU setup with the following command:

docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

AMD

Bacalhau requires AMD drivers to be appropriately installed and access to the rocm-smi tool.

You can test whether you have a working GPU setup with the following command, which should print details of your GPUs:

docker run --rm --device=/dev/kfd --device=/dev/dri --entrypoint=rocm-smi rocm/rocm-terminal

Intel

Bacalhau requires appropriate Intel drivers to be installed and access to the xpu-smi tool.

You can test whether you have a working GPU setup with the following command, which should print details of your GPUs:

docker run --rm --device=/dev/dri --entrypoint=/bin/bash intel/xpumanager -- -c 'xpumd & sleep 5; xpumcli discovery'

GPU Node Configuration

Access to GPUs can be controlled using resource limits. To limit the number of GPUs that can be used per job, set a job resource limit. To limit access to GPUs from all jobs, set a total resource limit.

Bacalhau employs the viper and cobra libraries for configuration management. Users can configure their Bacalhau node through a combination of command-line flags, environment variables, and the dedicated configuration file.

The Bacalhau Repo

Bacalhau manages its configuration, metadata, and internal state within a specialized repository named .bacalhau. Serving as the heart of the Bacalhau node, this repository holds the data and settings that determine node behavior. It's located on the filesystem, and by default, Bacalhau initializes this repository at $HOME/.bacalhau, where $HOME is the home directory of the user running the bacalhau process.

To customize this location, users can:

1. Set the BACALHAU_DIR environment variable to specify their desired path.

2. Utilize the --repo command line flag to specify their desired path.

Upon executing a Bacalhau command for the first time, the system will initialize the .bacalhau repository. If such a repository already exists, Bacalhau will seamlessly access its contents.

Structure of a Newly Initialized .bacalhau Repository

Below is the structure of a freshly initialized .bacalhau repository:

$ tree ~/.bacalhau
├── QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-compute/
│   ├── executions.db
│   └── jobStats.json
├── QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-requester/
│   └── jobs.db
├── config.yaml
├── executor_storages/
├── libp2p_private_key
├── plugins/
├── repo.version
└── user_id.pem

This repository comprises four directories and seven files:

Files

1. user_id.pem:

   • This file houses the Bacalhau node user's cryptographic private key, used for signing requests sent to a Requester Node.

   • Format: PEM.

2. repo.version:

   • Indicates the version of the Bacalhau node's repository.

   • Format: JSON, e.g., {"Version":1}.

3. libp2p_private_key:

   • Stores the Bacalhau node's libp2p private key, essential for its network identity. The NodeID of a Bacalhau node is derived from this key.

   • Format: Base64 encoded RSA private key.

4. config.yaml:

   • Contains configuration settings for the Bacalhau node.

   • Format: YAML.

5. update.json:

   • A file containing the date/time when the last version check was made.

   • Format: JSON, e.g., {"LastCheck":"2024-01-24T11:06:14.631816Z"}

6. tokens.json:

   • A file containing the tokens obtained through authenticating with bacalhau clusters.

Directories

1. QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-compute:

   • Contains the BoltDB executions.db database, which aids the Compute node in state persistence. Additionally, the jobStats.json file records the Compute Node's completed jobs tally.

   • Note: The segment QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv is a unique NodeID for each Bacalhau node, derived from the libp2p_private_key.

2. QmdGUjsMHEgtAfdtw7U62yPEcAZFtA33tKMsczLToegZtv-requester:

   • Contains the BoltDB jobs.db database for the Requester node's state persistence.

   • Note: NodeID derivation is similar to the Compute directory.

3. executor_storages:

   • Storage for data handled by Bacalhau storage drivers.

4. plugins:

   • Houses binaries that allow the Compute node to execute specific tasks.

   • Note: This feature is currently experimental and isn't active during standard node operations.

Configuring a Bacalhau Node

Within a .bacalhau repository, a config.yaml file may be present. This file serves as the configuration source for the bacalhau node and adheres to the YAML format.

Although the config.yaml file is optional, its presence allows Bacalhau to load custom configurations; otherwise, Bacalhau is configured with built-in default values, environment variables and command line flags.

Modifications to the config.yaml file will not be dynamically loaded by the Bacalhau node. A restart of the node is required for any changes to take effect. Bacalhau determines its configuration based on the following precedence order, with each item superseding the subsequent:

1. Command-line Flag

2. Environment Variable

3. Config File

4. Defaults

Relationship Between config.yaml and Bacalhau Environment Variables

Bacalhau establishes a direct relationship between the value-bearing keys within the config.yaml file and corresponding environment variables. For these keys that have no further sub-keys, the environment variable name is constructed by capitalizing each segment of the key, and then joining them with underscores, prefixed with BACALHAU_.

For example, a YAML key with the path Node.IPFS.Connect translates to the environment variable BACALHAU_NODE_IPFS_CONNECT and is represented in a file like:

Node:
    IPFS:
        Connect: value

There is no corresponding environment variable for either Node or Node.IPFS. Config values may also have other environment variables that set them for simplicity or to maintain backwards compatibility.

Environments

• Bacalhau leverages the BACALHAU_ENVIRONMENT environment variable to determine the specific environment configuration when initializing a repository. Notably, if a .bacalhau repository has already been initialized, the BACALHAU_ENVIRONMENT setting will be ignored.

  By default, if the BACALHAU_ENVIRONMENT variable is not explicitly set by the user, Bacalhau will adopt the production environment settings.

  Below is a breakdown of the configurations associated with each environment:

  1. Production (public network)

  • Environment Variable: BACALHAU_ENVIRONMENT=production

  • Configurations:

    • Node.ClientAPI.Host: "bootstrap.production.bacalhau.org"

    • Node.Client.API.Host: 1234

    • ...other configurations specific to this environment...

  2. Staging (staging network)

  • Environment Variable: BACALHAU_ENVIRONMENT=staging

  • Configurations:

    • Node.ClientAPI.Host: "bootstrap.staging.bacalhau.org"

    • Node.Client.API.Host: 1234

    • ...other configurations specific to this environment...

  3. Development (development network)

  • Environment Variable: BACALHAU_ENVIRONMENT=development

  • Configurations:

    • Node.ClientAPI.Host: "bootstrap.development.bacalhau.org"

    • Node.Client.API.Host: 1234

    • ...other configurations specific to this environment...

  4. Local (private or local networks)

  • Environment Variable: BACALHAU_ENVIRONMENT=local

  • Configurations:

    • Node.ClientAPI.Host: "0.0.0.0"

    • Node.Client.API.Host: 1234

    • ...other configurations specific to this environment...

  ---

  Note: The above configurations provided for each environment are not exhaustive. Consult the specific environment documentation for a comprehensive list of configurations.

Usage Examples

How to initialize a Bacalhau Server for a local private network

$ env BACALHAU_ENVIRONMENT=local ./bin/darwin_arm64/bacalhau serve
INF pkg/repo/fs.go:187 > Initializing repo at '/Users/frrist/.bacalhau' for environment 'local'

How to initialize a Bacalhau Server with a custom repo path

$ bacalhau --repo=/path/to/repo serve
INF pkg/repo/fs.go:187 > Initializing repo at '/path/to/repo' for environment 'production'

Or

$ export BACALHAU_DIR=/path/to/repo
$ bacalhau serve
INF pkg/repo/fs.go:187 > Initializing repo at '/path/to/repo' for environment 'production'

How to start a Bacalhau Server with DEBUG logs

$ env LOG_LEVEL=debug ./bin/darwin_arm64/bacalhau serve
DBG pkg/system/environment.go:53 > Defaulting to production environment: os.Args: [./bin/darwin_arm64/bacalhau serve]

These are the flags that control the capacity of the Bacalhau node, and the limits for jobs that might be run.

  --limit-job-cpu string                 Job CPU core limit for single job (e.g. 500m, 2, 8).
  --limit-job-gpu string                 Job GPU limit for single job (e.g. 1, 2, or 8).
  --limit-job-memory string              Job Memory limit for single job  (e.g. 500Mb, 2Gb, 8Gb).
  --limit-total-cpu string               Total CPU core limit to run all jobs (e.g. 500m, 2, 8).
  --limit-total-gpu string               Total GPU limit to run all jobs (e.g. 1, 2, or 8).
  --limit-total-memory string            Total Memory limit to run all jobs  (e.g. 500Mb, 2Gb, 8Gb).

The --limit-total-* flags control the total system resources you want to give to the network. If left blank, the system will attempt to detect these values automatically.

The --limit-job-* flags control the maximum amount of resources a single job can consume for it to be selected for execution.

Resource limits are not supported for Docker jobs running on Windows. Resource limits will be applied at the job bid stage based on reported job requirements but will be silently unenforced. Jobs will be able to access as many resources as requested at runtime.

Bacalhau is a peer-to-peer network of computing providers that will run jobs submitted by users. A Compute Provider (CP) is anyone who is running a Bacalhau compute node participating in the Bacalhau compute network, regardless of whether they are hosting any Filecoin data.

This section will show you how to configure and run a Bacalhau node and start accepting and running jobs.

To bootstrap your node and join the network as a CP you can leap right into the Ubuntu 22.04 quick start below, or find more setup details in these guides:

• Networking

• Storage Providers

• Job Selection Policy

• Resource Limits

• GPU Support

• Windows Support (with limitations)

Quick start (Ubuntu 22.04)

Estimated time for completion: 10 min. Tested on: Ubuntu 22.04 LTS (x86/64) running on a GCP e2-standard-4 (4 vCPU, 16 GB memory) instance with 40 GB disk size.

Prerequisites

• Docker Engine - to take on Docker workloads

• Connection to storage provider - for storing job's results

• Firewall - to ensure your node can communicate with the rest of the network

• Physical hardware, Virtual Machine, or cloud-based host. A Bacalhau compute node is not intended to be run from within a Docker container.

Install Docker

To run docker-based workloads, you should have docker installed and running.

If you already have it installed and want to configure the connection to Docker with the following environment variables:

• DOCKER_HOST to set the URL to the docker server.

• DOCKER_API_VERSION to set the version of the API to reach, leave empty for "latest".

• DOCKER_CERT_PATH to load the TLS certificates from.

• DOCKER_TLS_VERIFY to enable or disable TLS verification, off by default.

If you do not have Docker on your system, you can follow the official docker installation instructions or just use the snippet below:

# install dependencies
sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# add package repo and key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# install docker
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Now make Docker manageable by a non-root user:

sudo groupadd docker
sudo usermod -aG docker $USER

Ensure your Storage Server is Running

We will need to connect our Bacalhau node to a storage server for this we will be using IPFS server so we can run jobs that consume CIDs as inputs.

You can either install and run it locally or you can connect to a remote IPFS server.

In both cases - we should have a multiaddress for our IPFS server that should look something like this:

export IPFS_CONNECT=/ip4/10.1.10.10/tcp/80/p2p/QmVcSqVEsvm5RR9mBLjwpb2XjFVn5bPdPL69mL8PH45pPC

To install a single IPFS node locally on Ubuntu you can follow the official instructions, or follow the steps below. We advise running the same IPFS version as the Bacalhau main network.

Using the command below:

export IPFS_VERSION=$(wget -q -O - https://raw.githubusercontent.com/filecoin-project/bacalhau/main/ops/terraform/prod.tfvars | grep --color=never ipfs_version | awk -F'"' '{print $2}')

Install:

wget "https://dist.ipfs.tech/go-ipfs/${IPFS_VERSION}/go-ipfs_${IPFS_VERSION}_linux-amd64.tar.gz"
tar -xvzf "go-ipfs_${IPFS_VERSION}_linux-amd64.tar.gz"
cd go-ipfs
sudo bash install.sh
ipfs --version

Configure:

sudo mkdir -p /data/ipfs
export IPFS_PATH=/data/ipfs
sudo chown $(id -un):$(id -gn) ${IPFS_PATH} # change ownership of ipfs directory
ipfs init

Now launch the IPFS daemon in a separate terminal (make sure to export the IPFS_PATH environment variable there as well):

ipfs daemon

Don't forget we need to fetch an IPFS multiaddress pointing to our local node.

Use the following command to print out a number of addresses.

ipfs id

{
        "ID": "12D3KooWNhRU6H1eeqvT1jQAXFAdFDvT5H4AEGmHV7N1cYbzYh1F",
        "PublicKey": "CAESIL9gmDyR6IgM7ym1JmJ8JKL7NvEgIEGaWwssanl1ieuW",
        "Addresses": [
                "/ip4/10.128.0.11/tcp/4001/p2p/12D3KooWNhRU6H1eeqvT1jQAXFAdFDvT5H4AEGmHV7N1cYbzYh1F",
                "/ip4/10.128.0.11/udp/4001/quic/p2p/12D3KooWNhRU6H1eeqvT1jQAXFAdFDvT5H4AEGmHV7N1cYbzYh1F",
                "/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWNhRU6H1eeqvT1jQAXFAdFDvT5H4AEGmHV7N1cYbzYh1F",
                "/ip4/127.0.0.1/udp/4001/quic/p2p/12D3KooWNhRU6H1eeqvT1jQAXFAdFDvT5H4AEGmHV7N1cYbzYh1F",
                ...
        ],
        "AgentVersion": "go-ipfs/0.12.2/",
        "ProtocolVersion": "ipfs/0.1.0",
        "Protocols": [
                "/ipfs/bitswap",
                ...
                "/p2p/id/delta/1.0.0",
                "/x/"
        ]
}

I pick the record that combines 127.0.0.1 and tcp but I replace port 4001 with 5001:

export IPFS_CONNECT=/ip4/127.0.0.1/tcp/5001/p2p/12D3KooWNhRU6H1eeqvT1jQAXFAdFDvT5H4AEGmHV7N1cYbzYh1F

Configure firewall

To ensure that our node can communicate with other nodes on the network - we need to make sure the 1235 port is open.

(Optional) To ensure the CLI can communicate with our node directly (i.e. bacalhau --api-host <MY_NODE_PUBLIC_IP> version) - we need to make sure the 1234 port is open.

Firewall configuration is very specific to your network and we can't provide generic instructions for this step but if you need any help feel free to reach out on Slack!

Install the Bacalhau Binary

Install the bacalhau binary to run bacalhau serve.

Run bacalhau

Now we can run our bacalhau node:

LOG_LEVEL=debug BACALHAU_ENVIRONMENT=production \
  bacalhau serve \
    --node-type compute \
    --ipfs-connect $IPFS_CONNECT \
    --private-internal-ipfs=false \
    --peer env

Alternatively, you can run the following Docker command:

docker run -it --rm \
  -e LOG_LEVEL=debug \
  -e BACALHAU_ENVIRONMENT=production \
  ghcr.io/bacalhau-project/bacalhau:latest serve \
    --node-type compute \
    --ipfs-connect $IPFS_CONNECT \
    --private-internal-ipfs=false \
    --peer env

Check your node works

Even though the CLI (by default) submits jobs, each node listens for events on the global network and possibly bids for taking a job: your logs should therefore show the activity of your node bidding for incoming jobs.

To quickly check your node runs properly, let's submit the following dummy job:

bacalhau docker run ubuntu echo Test

If you see logs of your compute node bidding for the job above it means you've successfully joined Bacalhau as a Compute Provider!

What's next?

At this point, you probably have a number of questions for us. What incentive should you expect for running a public Bacalhau node? Please contact us on Slack to further discuss this topic and for sharing your valuable feedback.

When running a node, you can choose which jobs you want to run by using configuration options, environment variables or flags to specify a job selection policy.

Job selection probes

If you want more control over making the decision to take on jobs, you can use the --job-selection-probe-exec and --job-selection-probe-http flags.

These are external programs that are passed the following data structure so that they can make a decision about whether or not to take on a job:

{
  "node_id": "XXX",
  "job_id": "XXX",
  "spec": {
    "engine": "docker",
    "verifier": "ipfs",
    "job_spec_vm": {
      "image": "ubuntu:latest",
      "entrypoint": ["cat", "/file.txt"]
    },
    "inputs": [{
      "engine": "ipfs",
      "cid": "XXX",
      "path": "/file.txt"
    }]
  }
}

The exec probe is a script to run that will be given the job data on stdin, and must exit with status code 0 if the job should be run.

The http probe is a URL to POST the job data to. The job will be rejected if the HTTP request returns a non-positive status code (e.g. >= 400).

If the HTTP response is a JSON blob, it should match the following schema and will be used to respond to the bid directly:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "shouldBid": {
      "description": "If the job should be accepted",
      "type": "boolean"
    },
    "shouldWait": {
      "description": "If the node should wait for an async response that will come later. `shouldBid` will be ignored",
      "type": "boolean",
      "default": false,
    },
    "reason": {
      "description": "Human-readable string explaining why the job should be accepted or rejected, or why the wait is required",
      "type": "string"
    }
  },
  "required": [
    "shouldBid",
    "reason"
  ]
}

For example, the following response will reject the job:

{
  "shouldBid": false,
  "reason": "The job did not pass this specific validation: ...",
}

If the HTTP response is not a JSON blob, the content is ignored and any non-error status code will accept the job.

Good news everyone! You can now run your Bacalhau-IPFS stack in Docker.

This page describes several ways in which to operate Bacalhau. You can choose the method that best suits your needs. The methods are:

Pre-Prerequisites

• This guide works best on a Linux machine. If you're trying to run this on a Mac, you may encounter issues. Remember that network host mode doesn't work.

• You need to have Docker installed. If you don't have it, you can .

Connect to the Public Bacalhau Network Using Docker

This method is appropriate for those who:

• Provide compute resources to the public Bacalhau network

This is not appropriate for:

• Testing and development

• Running a private network

Prerequisites

(Optional) Start a Public IPFS Node

This will start a local IPFS node and connect it to the public DHT. If you already have an IPFS node running, then you can skip this step.

Some notes about this command:

• It wipes the $(pwd)/ipfs directory to make sure you have a clean slate

• It runs the IPFS container in the specified Docker network

• It exposes the IPFS API port to the world on port 4002, to avoid clashes with Bacalhau

• It exposes the admin RPC API to the local host only, on port 5001

• We are not specifying or removing the bootstrap nodes, so it will default to connecting to public machines

Start a Public Bacalhau Node

Bacalhau consists of two parts: a "requester" that is responsible for operating the API and managing jobs, and a "compute" element that is responsible for executing jobs. In a public context, you'd typically just run a compute node, and allow the public requesters to handle the traffic.

Notes about the command:

• It runs the Bacalhau container in "host" mode. This means that the container will use the same network as the host.

• It uses the root user, which is the default system user that has access to the Docker socket on a Mac. You may need to change this to suit your environment.

• It mounts the Docker Socket

• It mounts the /tmp directory

• It exposes the Bacalhau API ports to the world

• The container version should match that of the current release

• The IPFS connect string points to the RPC port of the IPFS node in Docker. Because Bacalhau is running in the same network, it can use DNS to find the IPFS container IP. If you're running your own node, replace it

• The --node-type flag is set to compute because we only want to run a compute node

• The --labels flag is used to set a human-readable label for the node, and so we can run jobs on our machine later

• We specify the --peer env flag so that it uses the environment specified by BACALHAU_ENVIRONMENT=production and therefore connects to the public network peers

There are several ways to ensure that the Bacalhau compute node is connected to the network.

First, check that the Bacalhau libp2p port is open and connected. On Linux you can run lsof and it should look something like this:

Note the three established connections at the bottom. These are the production bootstrap nodes that Bacalhau is now connected to.

You can also check that the node is connected by listing the current network peers and grepping for your IP address or node ID. The node ID can be obtained from the Bacalhau logs. It will look something like this:

Finally, submit a job with the label you specified when you ran the compute node. If this label is unique, there should be only one node with this label. The job should succeed. Run the following:

If instead, your job fails with the following error, it means that the compute node is not connected to the network:

Run a Private Bacalhau Network Using Docker (Insecure)

This method is appropriate for:

• Testing and development

• Evaluating the Bacalhau platform before scaling jobs via the public network

This method is useful for testing and development. It's easier to use because it doesn't require a secret IPFS swarm key -- this is essentially an authentication token that allows you to connect to the node.

This method is not appropriate for:

• Secure, private use

• Production use

Prerequisites

Start a Local IPFS Node (Insecure)

To run an insecure, private node, you need to initialize your IPFS configuration by removing all of the default public bootstrap nodes. Then we run the node in the normal way, without the special LIBP2P_FORCE_PNET flag that checks for a secure private connection.

Some notes about this command:

• It wipes the $(pwd)/ipfs directory to make sure you have a clean slate

• It removes the default bootstrap nodes

• It runs the IPFS container in the specified Docker network

• It exposes the IPFS API port to the local host only, to prevent accidentally exposing the IPFS node, on 4002, to avoid clashes with Bacalhau

• It exposes the admin RPC API to the local host only, on port 5001

Start a Private Bacalhau Node

Bacalhau consists of two parts: a "requester" that is responsible for operating the API and managing jobs, and a "compute" element that is responsible for executing jobs. In a public context, you'd typically just run a compute node, and allow the public requesters to handle the traffic. But in a private context, you'll want to run both.

Notes about the command:

• It runs the Bacalhau container in the specified Docker network

• It uses the root user, which is the default system user that has access to the Docker socket on a Mac. You may need to change this to suit your environment

• It mounts the Docker Socket

• It mounts the /tmp directory and specifies this as the location where Bacalhau will write temporary execution data (BACALHAU_NODE_COMPUTESTORAGEPATH)

• It exposes the Bacalhau API ports to the local host only, to prevent accidentally exposing the API to the public internet

• The container version should match that of the Bacalhau installed on your system

• The IPFS connect string points to the RPC port of the IPFS node. Because Bacalhau is running in the same network, it can use DNS to find the IPFS container IP.

• The --node-type flag is set to requester,compute because we want to run both a requester and a compute node

Run a Job on the Private Network

Now it's time to run a job. Recall that you exposed the Bacalhau API on the default ports to the local host only. So you'll need to use the --api-host flag to tell Bacalhau where to find the API. Everything else is a standard part of the Bacalhau CLI.

The job should succeed. Run it again but this time capture the job ID to make it easier to retrieve the results.

Retrieve the Results on the Private Network (Insecure)

To retrieve the results using the Bacalhau CLI, you need to know the p2p swarm multiaddress of the IPFS node because you don't want to connect to the public global IPFS network. To do that you can run the IPFS id command (and parse to remove the trub at the bottom of the barrel):

Note that the command above changes the reported port from 4001 to 4002. This is because the IPFS node is running on port 4002, but the IPFS id command reports the port as 4001.

Now get the results:

Alternatively, you can use the Docker container, mount the results volume, and change the --api-host to the name of the Bacalhau container and the --ipfs-swarm-addrs back to port 4001:

Run a Private Bacalhau Network Using Docker (Secure)

Running a private secure network is useful in a range of scenarios, including:

• Running a private network for a private project

You need two things. A private IPFS node to store data and a Bacalhau node to execute over that data. To keep the nodes private you need to tell the nodes to shush and use a secret key. This is a bit harder to use, and a bit more involved than the insecure version.

Prerequisites

Start a Private IPFS Node (Secure)

First, you need to bootstrap a new IPFS cluster for your own private use. This consists of a process of generating a swarm key, removing any bootstrap nodes, and then starting the IPFS node.

Some notes about this command:

• It wipes the $(pwd)/ipfs directory to make sure you have a clean slate

• It generates a new swarm key -- this is the token that is required to connect to this node

• It removes the default bootstrap nodes

• It runs the IPFS container in the specified Docker network

• It exposes the IPFS API port to the local host only, to prevent accidentally exposing the IPFS node, on 4002, to avoid clashes with Bacalhau

• It exposes the admin RPC API to the local host only, on port 5001

Start a Private Bacalhau Node (Secure)

Run a Job on the Private Network (Secure)

Retrieve the Results on the Private Network (Secure)

The same process as above can be used to retrieve results from the IPFS node as long as the Bacalhau get command has access to the IPFS swarm key.

Running the Bacalhau binary from outside of Docker:

Alternatively, you can use the Docker container, mount the results volume, and change the --api-host to the name of the Bacalhau container and the --ipfs-swarm-addrs back to port 4001:

Common Prerequisites

Create a New Docker Network

Without this, inter-container DNS will not work, and internet access may not work either.

Test that the IPFS Node is Working

You can now browse the IPFS web UI at http://127.0.0.1:5001/webui.

Test that the Bacalhau Node is Working

Ensure that the Bacalhau logs (docker logs bacalhau) have no errors.

Check that your Bacalhau installation is the same version:

The versions should match. Alternatively, you can use the Docker container:

Perform a list command to ensure you can connect to the Bacalhau API.

It should return empty.

Authenticate with docker hub

Should you wish to authenticate with Docker Hub when pulling images, you can do so by specifying credentials as environment variables wherever your compute node is running.

Bacalhau can limit the total time a job spends executing. A job that spends too long executing will be cancelled and no results will be published.

By default, a Bacalhau node does not enforce any limit on job execution time. Both node operators and job submitters can supply a maximum execution time limit. If a job submitter asks for a longer execution time than permitted by a node operator, their job will be rejected.

Applying job timeouts allows node operators to more fairly distribute the work submitted to their nodes. It also protects users from transient errors that results in their jobs waiting indefinitely.

Configuring execution time limits for a job

Job submitters can pass the --timeout flag to any Bacalhau job submission CLI to set a maximum job execution time. The supplied value should be a whole number of seconds with no unit.

The timeout can also be added to an existing job spec by adding the Timeout property to the Spec.

Configuring execution time limits for a node

Node operators can pass the --max-job-execution-timeout flag to bacalhau serve to configure the maximum job time limit. The supplied value should be a numeric value followed by a time unit (one of s for seconds, m for minutes or h for hours).

Node operators can also use configuration properties to configure execution limits.

Compute nodes will use the properties:

Requester nodes will use the properties:

Bacalhau has two ways to make use of external storage providers:

• Inputs storage resources consumed as inputs to jobs

• Publishers storage resources created with the results of jobs

Inputs

IPFS

To start, you'll need to connect the Bacalhau node to an IPFS server so that you can run jobs that consume CIDs as inputs.

You can either and run it locally, or you can connect to a remote IPFS server.

In both cases, you should have an for the IPFS server that should look something like this:

You can then configure your Bacalhau node to use this IPFS server by passing the --ipfs-connect argument to the serve command:

Or, set the Node.IPFS.Connect property in the Bacalhau configuration file.

Publishers

IPFS

The IPFS publisher works using the same setup as above - you'll need to have an IPFS server running and a multiaddress for it. You'll then you pass that multiaddress using the --ipfs-connect argument to the serve command.

If you are publishing to a public IPFS node, you can use bacalhau get with no further arguments to download the results. However, you may experience a delay in results becoming available as indexing of new data by public nodes takes time.

To speed up the download or to retrieve results from a private IPFS node, pass the swarm multiaddress to bacalhau get to download results.

Pass the swarm key to bacalhau get if the IPFS swarm is a private swarm.

By default, the requester node APIs used by the Bacalhau CLI are accessible over HTTP, but it is possible to configure it to use Transport Level Security (TLS) so that they are accessible over HTTPS instead. There are several ways to obtain the necessary certificates and keys, and Bacalhau supports obtaining them via ACME and Certificate Authorities or even self-signing them.

Getting a certificate from Let's Encrypt with ACME

Automatic Certificate Management Environment (ACME) is a protocol that allows for automating the deployment of Public Key Infrastructure, and is the protocol used to obtain a free certificate from the Certificate Authority.

Using the --autocert [hostname] parameter to the CLI (in the serve and devstack commands), a certificate is obtained automatically from Lets Encrypt. The provided hostname should be a comma-separated list of hostnames, but they should all be publicly resolvable as Lets Encrypt will attempt to connect to the server to verify ownership (using the challenge). On the very first request this can take a short time whilst the first certificate is issued, but afterwards they are then cached in the bacalhau repository.

Alternatively, you may set these options via the environment variable, BACALHAU_AUTO_TLS. If you are using a configuration file, you can set the values inNode.ServerAPI.TLS.AutoCert instead.

A cache of ACME data is held in the config repository, by default ~/.bacalhau/autocert-cache, and this will be used to manage renewals to avoid rate limits.

Getting a certificate from a Certificate Authority

Obtaining a TLS certificate from a Certificate Authority (CA) without using the Automated Certificate Management Environment (ACME) protocol involves a manual process that typically requires the following steps:

1. Choose a Certificate Authority: First, you need to select a trusted Certificate Authority that issues TLS certificates. Popular CAs include DigiCert, GlobalSign, Comodo (now Sectigo), and others. You may also consider whether you want a free or paid certificate, as CAs offer different pricing models.

2. Generate a Certificate Signing Request (CSR): A CSR is a text file containing information about your organization and the domain for which you need the certificate. You can generate a CSR using various tools or directly on your web server. Typically, this involves providing details such as your organization's name, common name (your domain name), location, and other relevant information.

3. Submit the CSR: Access your chosen CA's website and locate their certificate issuance or order page. You'll typically find an option to "Submit CSR" or a similar option. Paste the contents of your CSR into the provided text box.

4. Verify Domain Ownership: The CA will usually require you to verify that you own the domain for which you're requesting the certificate. They may send an email to one of the standard domain-related email addresses (e.g., admin@yourdomain.com, webmaster@yourdomain.com). Follow the instructions in the email to confirm domain ownership.

5. Complete Additional Verification: Depending on the CA's policies and the type of certificate you're requesting (e.g., Extended Validation or EV certificates), you may need to provide additional documentation to verify your organization's identity. This can include legal documents or phone calls from the CA to confirm your request.

6. Payment and Processing: If you're obtaining a paid certificate, you'll need to make the payment at this stage. Once the CA has received your payment and completed the verification process, they will issue the TLS certificate.

Once you have obtained your certificates, you will need to put two files in a location that bacalhau can read them. You need the server certificate, often called something like server.cert or server.cert.pem, and the server key which is often called something like server.key or server.key.pem.

Once you have these two files available, you must start bacalhau serve which two new flags. These are tlscert and tlskey flags, whose arguments should point to the relevant file. An example of how it is used is:

Alternatively, you may set these options via the environment variables, BACALHAU_TLS_CERT and BACALHAU_TLS_KEY. If you are using a configuration file, you can set the values inNode.ServerAPI.TLS.ServerCertificate and Node.ServerAPI.TLS.ServerKey instead.

Self-signed certificates

Once you have generated the necessary files, the steps are much like above, you must start bacalhau serve which two new flags. These are tlscert and tlskey flags, whose arguments should point to the relevant file. An example of how it is used is:

Alternatively, you may set these options via the environment variables, BACALHAU_TLS_CERT and BACALHAU_TLS_KEY. If you are using a configuration file, you can set the values inNode.ServerAPI.TLS.ServerCertificate and Node.ServerAPI.TLS.ServerKey instead.

If you use self-signed certificates, it is unlikely that any clients will be able to verify the certificate when connecting to the Bacalhau APIs. There are three options available to work around this problem:

1. Provide a CA certificate file of trusted certificate authorities, which many software libraries support in addition to system authorities.

2. Install the CA certificate file in the system keychain of each machine that needs access to the Bacalhau APIs.

3. Instruct the software library you are using not to verify HTTPS requests.

Before you join the main Bacalhau network, you can test locally.

To test, you can use the bacalhau devstack command, which offers a way to get a 3 node cluster running locally.

In another window, export the following environment variables so that the Bacalhau client binary connects to our local development cluster:

You can now interact with Bacalhau - all jobs are running by the local devstack cluster.

Both compute nodes, and requester nodes, maintain state. How that state is maintained is configurable, although the defaults are likely adequate for most use-cases. This page describes how to configure the persistence of compute and requester nodes should the defaults not be suitable.

Compute node persistence

The computes nodes maintain information about the work that has been allocated to them, including:

• The current state of the execution, and

• The original job that resulted in this allocation

This information is used by the compute and requester nodes to ensure allocated jobs are completed successfully. By default, compute nodes store their state in a bolt-db database and this is located in the bacalhau repository along with configuration data. For a compute node whose ID is "abc", the database can be found in ~/.bacalhau/abc-compute/executions.db.

In some cases, it may be preferable to maintain the state in memory, with the caveat that should the node restart, all state will be lost. This can be configured using the environment variables in the table below.

Requester node persistence

When running a requester node, it maintains state about the jobs it has been requested to orchestrate and schedule, the evaluation of those jobs, and the executions that have been allocated. By default, this state is stored in a bolt db database that, with a node ID of "xyz" can be found in ~/.bacalhau/xyz-requester/jobs.db.

Bacalhau supports the three main 'pillars' of observability - logging, metrics, and tracing. Bacalhau uses the for metrics and tracing, which can be configured using the . Exporting metrics and traces can be as simple as setting the OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_ENDPOINT environment variables. Custom code is used for logging as the .

Logging

Logging in Bacalhau outputs in human-friendly format to stderr at INFO level by default, but this can be changed by two environment variables:

• LOG_LEVEL - Can be one of trace, debug, error, warn or fatal to output more or fewer logging messages as required

• LOG_TYPE - Can be one of the following values:

  • default - output logs to stderr in a human-friendly format

  • json - log messages outputted to stdout in JSON format

  • combined - log JSON formatted messages to stdout and human-friendly format to stderr

Log statements should include the relevant trace, span and job ID so it can be tracked back to the work being performed.

Metrics

Bacalhau produces a number of different metrics including those around the libp2p resource manager (rcmgr), performance of the requester HTTP API and the number of jobs accepted/completed/received.

Tracing

Traces are produced for all major pieces of work when processing a job, although the naming of some spans is still being worked on. You can find relevant traces covering working on a job by searching for the jobid attribute.

Viewing

The metrics and traces can easily be forwarded to a variety of different services as we use OpenTelemetry, such as Honeycomb or Datadog.

To view the data locally, or simply to not use a SaaS offering, you can start up Jaeger and Prometheus placing these three files into a directory then running docker compose start while running Bacalhau with the OTEL_EXPORTER_OTLP_PROTOCOL=grpc and OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 environment variables.

Running a Windows-based node is not officially supported, so your mileage may vary. Some features (like resource limits) are not present in Windows-based nodes.

Bacalhau currently makes the assumption that all containers are Linux-based. Users of the Docker executor will need to manually ensure that their Docker engine is running and configured appropriately to support Linux containers, e.g. using the WSL-based backend.

Overview

The Bacalhau WebUI offers an intuitive interface for interacting with the Bacalhau network. This guide provides comprehensive instructions for setting up, deploying, and utilizing the WebUI.

For contributing to the WebUI's development, please refer to the Bacalhau WebUI GitHub Repository.

Spinning Up the WebUI Locally

Prerequisites

• Ensure you have a Bacalhau v1.1.7 or later installed.

Running the WebUI

To launch the WebUI locally, execute the following command:

bacalhau serve --node-type=requester,compute --web-ui

This command initializes a requester and compute node, configured to listen on HOST=0.0.0.0 and PORT=1234.

Accessing the Local WebUI

Once started, the WebUI is accessible at http://127.0.0.1/. This local instance allows you to interact with your local Bacalhau network setup.

Accessing the WebUI from the Browser

For observational purposes, a development version of the WebUI is available at bootstrap.development.bacalhau.org. This instance displays jobs from the development server.

N.b. The development version of the WebUI is for observation only and may not reflect the latest changes or features available in the local setup.