1 of 6

Deployment Guide

Setting Up a Cluster on Amazon Web Services (AWS) with Terraform 🚀

Welcome to the guide for setting up your own Bacalhau cluster across multiple AWS regions! This guide will walk you through creating a robust, distributed compute cluster that's perfect for running your Bacalhau workloads.

What You'll Build

Think of this as building your own distributed supercomputer! Your cluster will provision compute nodes spread across different AWS regions for global coverage.

Before You Start

You'll need a few things ready:

Terraform (version 1.0.0 or newer)
AWS CLI installed and configured
An active AWS account with appropriate permissions
Your AWS credentials configured
An SSH key pair for securely accessing your nodes
A Bacalhau network

Quick Setup Guide

First, set up an orchestrator node. We recommend using for this! But you can always set up your own
Create your environment configuration file:
```
cp env.tfvars.example.json env.tfvars.json
```

Fill in your AWS details in env.tfvars.json:

{
    "app_name": "bacalhau",
    "app_tag": "bacalhau-cluster",
    "bacalhau_installation_id": "EXAMPLE-c5c1-44fd-a0f3-1b90488f1b68",
    "bacalhau_data_dir": "/bacalhau_data",
    "bacalhau_node_dir": "/bacalhau_node",
    "username": "bacalhau-runner",
    "public_key_path": "/path/to/your/.ssh/id_rsa.pub",
    "private_key_path": "/path/to/your/.ssh/id_rsa"
}

Configure your desired regions in locations.yaml. Here's an example (we have a full list of these in all_locations.yaml):

- us-west-2:
    region: us-west-2
    zone: us-west-2a
    instance_type: t3.medium
    instance_ami: ami-07d9cf938edb0739b
    node_count: 1
- eu-central-1:
    region: eu-central-1
    zone: eu-central-1a
    instance_type: t3.medium
    instance_ami: ami-0e54671bdf3c8ed8d
    node_count: 1

Make sure the AMI exists in the region you need it to! You can confirm this by executing the following command:

REGION=us-east-1
IMAGE_SUBSTRING=amzn2-ami-hvm
ARCH=x86_64
aws ec2 describe-images \
--region $REGION \
--filters "Name=name,Values=*$IMAGE_SUBSTRING*" "Name=architecture,Values=$ARCH" \
--query 'sort_by(Images, &CreationDate)[-10:].{AMI:ImageId, Name:Name, CreationDate:CreationDate}' \
--output table \
--no-cli-pager

Update your Bacalhau config/config.yaml (the defaults are mostly fine, just update the Orchestrator, and Token lines):

NameProvider: puuid
API:
  Port: 1234
Compute:
  Enabled: true
  Orchestrators:
    - nats://EXAMPLE-6ed7-4d95-8871-b46153007057.us1.cloud.expanso.dev:4222
  Auth:
    Token: "EXAMPLE.EFukWVffnf5jb9QkpNnwfiBWEk3475csM7ysudpbFTzYBap5c7sWr6"
  TLS:
    RequireTLS: true
  AllowListedLocalPaths:
    - /bacalhau_data:rw
JobAdmissionControl:
  AcceptNetworkedJobs: true

Deploy your cluster using the Python deployment script:
```
./deploy.py create
```

Understanding the Configuration

Why use a deployment script? Why not use Terraform directly?

Terraform on AWS requires switching to different workspaces when deploying to different availability zones. As a result, we had to setup a separate deploy.py script which switches to each workspace for you under the hood, to make it easier.

Core Configuration Files

env.tfvars.json: Your main configuration file containing AWS-specific settings`
locations.yaml: Defines which regions to deploy to and instance configurations
config/config.yaml: Bacalhau node configuration

Essential Settings in env.tfvars.json

app_name: Name for your cluster resources
app_tag: Tag for resource management
bacalhau_installation_id: Unique identifier for your cluster
username: SSH username for instances
public_key_path: Path to your SSH public key
private_key_path: Path to your SSH private key
bacalhau_config_file_path: Path to the config file for this compute node (should point at the orchestrator and have the right token)

Location Configuration (locations.yaml)

Each region entry requires:

region: AWS region (e.g., us-west-2)
zone: Availability zone (e.g., us-west-2a)
instance_type: EC2 instance type (e.g., t3.medium)
instance_ami: AMI ID for the region
node_count: Number of instances to deploy

Taking Your Cluster for a Test Drive

Once everything's up and running, let's make sure it works!

Configure your Bacalhau client:

bacalhau config set -c API.Host=<orchestrator-ip> # Should be the same orchestrator
                                                  # IP from your config.yaml

List your compute nodes:
```
bacalhau node list
```

Run a test job:

bacalhau docker run ubuntu echo "Hello from my cluster!"

Check job status:
```
bacalhau list
```

Troubleshooting Tips

Deployment Issues

Verify AWS credentials are properly configured:
```
aws sts get-caller-identity
```
Check IAM permissions
Ensure you have quota available in target regions

Node Health Issues

SSH into a node:

ssh -i ~/.ssh/id_rsa bacalhau-runner@<public-ip> # Or whatever username you set

Check Bacalhau service logs:
```
journalctl -u bacalhau-startup.service
```

Check Docker container status:

docker ps
docker logs bacalhau-node-bacalhau-node-1

Network Issues

Verify security group rules (ports 22, 80, and 4222 should be open)
Check VPC and subnet configurations
Ensure internet gateway is properly attached

Common Solutions

If nodes aren't joining the network:
- Check NATS connection string in config.yaml
- Verify security group allows port 4222
- Ensure nodes can reach the orchestrator
If jobs aren't running:
- Check compute is enabled in node config
- Verify Docker is running properly
- Check available disk space
If deployment fails:
- Look for errors in Terraform output
- Check AWS service quotas
- Verify AMI availability in chosen regions

Cleanup

Remove all resources:

./deploy.py destroy

Monitoring

Check node health:
```
curl http://<node-ip>/healthz
```

Understanding the Directory Structure

setting-up-bacalhau-with-terraform-on-AWS/
├── modules/
│   ├── instance/       # EC2 instance configuration
│   ├── network/        # VPC and subnet setup
│   ├── region/         # Region-specific resources
│   └── securityGroup/  # Security group rules
├── config/             # Bacalhau configuration
├── scripts/           # Deployment and utility scripts
├── locations.yaml     # Region definitions
└── env.tfvars.json    # Environment configuration

Need Help?

If you get stuck or have questions:

We're here to help you get your cluster running smoothly! 🌟

Setting Up a Cluster on Google Cloud Platform (GCP) With Terraform 🚀

Welcome to the guide for setting up your own Bacalhau cluster across multiple Google Cloud Platform (GCP) regions! This guide will walk you through creating a robust, distributed compute cluster that's perfect for running your Bacalhau workloads.

What You'll Build

Think of this as building your own distributed supercomputer! Your cluster will provision compute nodes spread across different GCP regions for global coverage.

Before You Start

You'll need a few things ready:

Terraform (version 1.0.0 or newer)
A running Bacalhau orchestrator node
Google Cloud SDK installed and set up
An active GCP billing account
Your organization ID handy
An SSH key pair for securely accessing your nodes

Quick Setup Guide

Make sure you are logged in with GCP. This could involve both of the following commands:

gcloud auth login
gcloud auth application-default login

Clone the examples repo to your machine and go into the GCP directory.

gh repo clone bacalhau-project/examples
cd setting-up-bacalhau-cluster/setting-up-bacalhau-with-terraform-on-GC

Now, make a copy of the example environment file:
```
cp env.json.example env.json
```
Open up env.json and fill in your GCP details (more on this below!)

Update your config/config.yaml with your orchestrator information. Specifically, these lines:

  Orchestrators:
    - nats://EXAMPLE-7a02-4083-bf08-bcc2f5fbc025.us1.cloud.expanso.dev:4222
  Auth:
    Token: "EXAMPLE-aEEFukWVffnf5jb9QkpNnwfiBWEk3475csM7ysudpbFTzYBap5c7sWr6"

Let Terraform get everything ready:
```
terraform init --env-file env.json
```
Launch your cluster:
```
terraform apply --env-file env.json
```

The entire process takes about 8 minutes, but should end with something like the below:

Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

deployment_status = {
  "asia-northeast1" = {
    "external_ip" = "35.221.64.233"
    "health_check" = "healthy"
    "name" = "bacalhau-cluster-2501020854-asia-northeast1-vm"
  }
  
[...]

  "europe-west12" = {
    "external_ip" = "34.17.50.110"
    "internal_ip" = "10.210.0.2"
    "name" = "bacalhau-cluster-2501020854-europe-west12-vm"
    "zone" = "europe-west12-a"
  }
}

You're good to go!

Customizing Your Network

The env.json file is where all the magic happens. Here's what you'll need to fill in:

Essential Settings

bootstrap_project_id: Your existing GCP project (just used for setup)
base_project_name: What you want to call your new project
gcp_billing_account_id: Where the charges should go
gcp_user_email: Your GCP email address
org_id: Your organization's ID
app_tag: A friendly name for your resources (like "bacalhau-demo")

Node Configuration

bacalhau_data_dir: Where job data should be stored
bacalhau_node_dir: Where node configs should live
username: Your SSH username
public_key: Path to your SSH public key

Location Settings

You can set up nodes in different regions with custom configurations:

"locations": {
  "us-central1": {
    "zone": "us-central1-a",
    "node_count": 3,
    "machine_type": "e2-standard-4"
  }
}

Taking Your Cluster for a Test Drive

Once everything's up and running, let's make sure it works!

First, make sure you have the Bacalhau CLI installed. You can read more about installing the CLI here.

First configure the CLI to use your cluster:

bacalhau config set -c API.Host=<orchestrator-ip>

Check on the health of your nodes:
```
bacalhau node list
```
If you're using the Expanso Cloud hosted orchestrator (Recommended!), you can look at your nodes on the Expanso Cloud dashboard in real-time.

Run a simple test job:

bacalhau docker run ubuntu echo "Hello from my cluster!"

Check on your jobs:
```
bacalhau list
```
Get your results:
```
bacalhau get <job-id>
```

Troubleshooting Tips

Having issues? Here are some common solutions:

Deployment Problems

Double-check your GCP permissions
Make sure your billing account is active
Verify that all needed APIs are turned on in GCP

Node Health Issues

Look at the logs on a node: journalctl -u bacalhau-startup.service
Check Docker logs on a node: docker logs <container-id>
Make sure that port 4222 isn't blocked

Job Running Troubles

Verify your NATS connection settings
Check if nodes are properly registered
Make sure compute is enabled in your config

Cleaning Up

When you're done, clean everything up with:

terraform destroy --env-file env.json

Need to Check on Things?

If you need to peek under the hood, here's how:

Find your node IPs:
```
terraform output instance_ips
```
SSH into a node:
```
ssh -i ~/.ssh/id_rsa ubuntu@<public-ip>
```
Check on Docker:
```
docker ps
```

Go into the container on the node:

CONTAINER_ID=$(docker ps --filter name=^/bacalhau_node --format '{{.ID}}' | head -n1)
docker exec -it $CONTAINER_ID /bin/bash

Understanding the Configuration Files

Here's what each important file does in your setup:

Core Files

main.tf: Your main Terraform configuration
variables.tf: Where input variables are defined
outputs.tf: What information Terraform will show you
config/config.yaml: How your Bacalhau nodes are configured
scripts/startup.sh: Gets your nodes ready to run
scripts/bacalhau-startup.service: Manages the Bacalhau service

Cloud-Init and Docker Setup

cloud-init/init-vm.yml: Sets up your VM environment, installs packages, and gets services running
config/docker-compose.yml: Runs Bacalhau in a privileged container with all the right volumes and health checks

The neat thing is that most of your configuration happens in just one file: env.json. Though if you want to get fancy, there's lots more you can customize!

Need Help?

If you get stuck or have questions:

Open an issue in our GitHub repository
Join our Slack

We're here to help you get your cluster running smoothly! 🌟

Setting Up a Cluster on Azure with Terraform 🚀

Welcome to the guide for setting up your own Bacalhau cluster across multiple Azure regions! This guide will walk you through creating a robust, distributed compute cluster that's perfect for running your Bacalhau workloads.

What You'll Build

Think of this as building your own distributed supercomputer! Your cluster will provision compute nodes spread across different Azure regions for global coverage.

Before You Start

You'll need a few things ready:

Terraform (version 1.0.0 or newer)
A running Bacalhau orchestrator node
Azure CLI installed and set up
An active Azure subscription
Your subscription ID handy
An SSH key pair for securely accessing your nodes

Quick Setup Guide

First, create a terraform.tfvars.json file with your Azure details:
```
cp terraform.tfvars.example.json terraform.tfvars.json
```

Open up terraform.tfvars.json and fill in your Azure details:

 "subscription_id": "f67231aa-8387-498b-9ca3-EXAMPLE",
 "app_tag": "bacalhau-cluster",
 "resource_group_region": "eastus",
 "username": "bacalhau-runner",
 "public_key": "~/.ssh/id_rsa.pub",
 "bacalhau_data_dir": "/bacalhau_data",
 "bacalhau_node_dir": "/bacalhau_node",
 "bacalhau_config_file_path": "./config/config.yaml",

 "locations": {
     "eastus": {
         "machine_type": "Standard_D4_v4",
         "node_count": 1
     }
}

Update your config/config.yaml with your orchestrator information. Specifically, these lines:

  Orchestrators:
    - nats://EXAMPLE-7a02-4083-bf08-bcc2f5fbc025.us1.cloud.expanso.dev:4222
  Auth:
    Token: "EXAMPLE-aEEFukWVffnf5jb9QkpNnwfiBWEk3475csM7ysudpbFTzYBap5c7sWr6"

Let Terraform get everything ready:
```
terraform init
```
Launch your cluster:
```
terraform apply
```

Understanding the Configuration

The infrastructure is organized into modules:

Network: Creates VNets and subnets in each region
Security Group: Sets up NSGs with rules for SSH, HTTP, and NATS
Instance: Provisions VMs with cloud-init configuration

Taking Your Cluster for a Test Drive

Once everything's up and running, let's make sure it works!

Setup your configuration to point at your orchestrator node:
```
bacalhau config set -c API.Host=<ip-address-of-orchestrator>
```
Check on the health of your nodes:
```
bacalhau node list
```

Run a simple test job:

bacalhau docker run docker.io/bacalhauproject/hello-world

Check on your jobs:
```
bacalhau list
```
Get your results:
```
bacalhau get <job-id>
```

Troubleshooting Tips

Having issues? Here are some common solutions:

Deployment Problems

Double-check your Azure permissions
Make sure your subscription is active
Verify that all needed resource providers are registered

Node Health Issues

Look at the logs on a node: journalctl -u bacalhau-startup.service
Check Docker logs on a node: docker logs <container-id>
Make sure that port 4222 isn't blocked

Job Running Troubles

Verify your NATS connection settings
Check if nodes are properly registered
Make sure compute is enabled in your config

Cleaning Up

When you're done, clean everything up with:

terraform destroy

Need to Check on Things?

If you need to peek under the hood, here's how:

Find your node IPs:
```
terraform output deployment_status
```

SSH into a node:

ssh -i ~/.ssh/id_rsa <username>@<public-ip>

Check on Docker:
```
docker ps
```

Go into the container on the node:

CONTAINER_ID=$(docker ps --filter name=^/bacalhau_node --format '{{.ID}}' | head -n1)
docker exec -it $CONTAINER_ID /bin/bash

Understanding the Configuration Files

Here's what each important file does in your setup:

Core Files

main.tf: Your main Terraform configuration
variables.tf: Where input variables are defined
outputs.tf: What information Terraform will show you

Modules

modules/network: Handles VNet and subnet creation
modules/securityGroup: Manages network security groups
modules/instance: Provisions VMs with cloud-init

Cloud-Init and Docker Setup

cloud-init/init-vm.yml: Sets up your VM environment, installs packages, and gets services running
config/docker-compose.yml: Runs Bacalhau in a privileged container with all the right volumes and health checks

Azure Specific Commands

For ensuring that you have configured your Azure CLI correctly, here are some commands you can use:

Get available Azure locations

az account list-locations --query "[].{Name:name, DisplayName:displayName}" -o table

Get available VM sizes in a region

az vm list-sizes --location eastus --query "[].{Name:name, vCPUs:numberOfCores, MemoryGB:memoryInMb, GPUs:gpus}" --output table

Get your subscription ID

az account show --query "id"

Need Help?

If you get stuck or have questions:

We're here to help you get your cluster running smoothly! 🌟

Marketplace Deployments

Google Cloud Marketplace

Introduction

Well done on deploying your Bacalhau cluster! Now that the deployment is finished, this document will help with the next steps. It provides important information on how to interact with and manage the cluster. You'll find details on the outputs from the deployment, including how to set up and connect a Bacalhau Client, and how to authorize and connect a Bacalhau Compute node to the cluster. This guide gives everything needed to start using the Bacalhau setup

Deployment Outputs

After completing the deployment, several outputs will be presented. Below is a description of each output and instructions on how to configure your Bacalhau node using them.

Requester Public IP

Description: The IP address of the Requester node for the deployment and the endpoint where the Bacalhau API is served.

Usage: Configure the Bacalhau Client to connect to this IP address in the following ways:

Setting the --api-host CLI Flag:

$ bacalhau --api-host $REQUESTER_IP [CMD]

Setting the BACALHAU_API_HOST environment variable:

$ export BACALHAU_API_HOST=$REQUESTER_IP
$ bacalhau [CMD]

Modifying the Bacalhau Configuration File:

$ bacalhau config set node.clientapi.host $REQUESTER_IP
$ bacalhau [CMD]

Requester API Token

Description: The token used to authorize a client when accessing the Bacalhau API.

Usage: The Bacalhau client prompts for this token when a command is first issued to the Bacalhau API. For example:

$ bacalhau agent version
token: $REQUESTER_API_TOKEN

Compute API Token

Description: The token used to authorize a Bacalhau Compute node to connect to the Requester Node.

Usage: A Bacalhau Compute node can be connected to the Requester Node using the following command:

bacalhau serve --node-type=compute --orchestrators=nats://$COMPUTE_API_TOKEN@$REQUESTER_IP

Setting Up a Cluster on Google Cloud Platform (GCP) With Terraform 🚀

What You'll Build

Think of this as building your own distributed supercomputer! Your cluster will provision compute nodes spread across different GCP regions for global coverage.

Before You Start

You'll need a few things ready:

Terraform (version 1.0.0 or newer)
A running Bacalhau orchestrator node
Google Cloud SDK installed and set up
An active GCP billing account
Your organization ID handy
An SSH key pair for securely accessing your nodes

Quick Setup Guide

Make sure you are logged in with GCP. This could involve both of the following commands:

gcloud auth login
gcloud auth application-default login

Clone the examples repo to your machine and go into the GCP directory.

gh repo clone bacalhau-project/examples
cd setting-up-bacalhau-cluster/setting-up-bacalhau-with-terraform-on-GC

Now, make a copy of the example environment file:
```
cp env.json.example env.json
```
Open up env.json and fill in your GCP details (more on this below!)

Update your config/config.yaml with your orchestrator information. Specifically, these lines:

  Orchestrators:
    - nats://EXAMPLE-7a02-4083-bf08-bcc2f5fbc025.us1.cloud.expanso.dev:4222
  Auth:
    Token: "EXAMPLE-aEEFukWVffnf5jb9QkpNnwfiBWEk3475csM7ysudpbFTzYBap5c7sWr6"

Let Terraform get everything ready:
```
terraform init --env-file env.json
```
Launch your cluster:
```
terraform apply --env-file env.json
```

The entire process takes about 8 minutes, but should end with something like the below:

Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

deployment_status = {
  "asia-northeast1" = {
    "external_ip" = "35.221.64.233"
    "health_check" = "healthy"
    "name" = "bacalhau-cluster-2501020854-asia-northeast1-vm"
  }
  
[...]

  "europe-west12" = {
    "external_ip" = "34.17.50.110"
    "internal_ip" = "10.210.0.2"
    "name" = "bacalhau-cluster-2501020854-europe-west12-vm"
    "zone" = "europe-west12-a"
  }
}

You're good to go!

Customizing Your Network

The env.json file is where all the magic happens. Here's what you'll need to fill in:

Essential Settings

bootstrap_project_id: Your existing GCP project (just used for setup)
base_project_name: What you want to call your new project
gcp_billing_account_id: Where the charges should go
gcp_user_email: Your GCP email address
org_id: Your organization's ID
app_tag: A friendly name for your resources (like "bacalhau-demo")

Node Configuration

bacalhau_data_dir: Where job data should be stored
bacalhau_node_dir: Where node configs should live
username: Your SSH username
public_key: Path to your SSH public key

Location Settings

You can set up nodes in different regions with custom configurations:

"locations": {
  "us-central1": {
    "zone": "us-central1-a",
    "node_count": 3,
    "machine_type": "e2-standard-4"
  }
}

Taking Your Cluster for a Test Drive

Once everything's up and running, let's make sure it works!

First, make sure you have the Bacalhau CLI installed. You can read more about installing the CLI here.

First configure the CLI to use your cluster:

bacalhau config set -c API.Host=<orchestrator-ip>

Check on the health of your nodes:
```
bacalhau node list
```
If you're using the Expanso Cloud hosted orchestrator (Recommended!), you can look at your nodes on the Expanso Cloud dashboard in real-time.

Run a simple test job:

bacalhau docker run ubuntu echo "Hello from my cluster!"

Check on your jobs:
```
bacalhau list
```
Get your results:
```
bacalhau get <job-id>
```

Troubleshooting Tips

Having issues? Here are some common solutions:

Deployment Problems

Double-check your GCP permissions
Make sure your billing account is active
Verify that all needed APIs are turned on in GCP

Node Health Issues

Look at the logs on a node: journalctl -u bacalhau-startup.service
Check Docker logs on a node: docker logs <container-id>
Make sure that port 4222 isn't blocked

Job Running Troubles

Verify your NATS connection settings
Check if nodes are properly registered
Make sure compute is enabled in your config

Cleaning Up

When you're done, clean everything up with:

terraform destroy --env-file env.json

Need to Check on Things?

If you need to peek under the hood, here's how:

Find your node IPs:
```
terraform output instance_ips
```
SSH into a node:
```
ssh -i ~/.ssh/id_rsa ubuntu@<public-ip>
```
Check on Docker:
```
docker ps
```

Go into the container on the node:

CONTAINER_ID=$(docker ps --filter name=^/bacalhau_node --format '{{.ID}}' | head -n1)
docker exec -it $CONTAINER_ID /bin/bash

Understanding the Configuration Files

Here's what each important file does in your setup:

Core Files

main.tf: Your main Terraform configuration
variables.tf: Where input variables are defined
outputs.tf: What information Terraform will show you
config/config.yaml: How your Bacalhau nodes are configured
scripts/startup.sh: Gets your nodes ready to run
scripts/bacalhau-startup.service: Manages the Bacalhau service

Cloud-Init and Docker Setup

cloud-init/init-vm.yml: Sets up your VM environment, installs packages, and gets services running
config/docker-compose.yml: Runs Bacalhau in a privileged container with all the right volumes and health checks

The neat thing is that most of your configuration happens in just one file: env.json. Though if you want to get fancy, there's lots more you can customize!

Need Help?

If you get stuck or have questions:

Open an issue in our GitHub repository
Join our Slack

We're here to help you get your cluster running smoothly! 🌟

Setting Up a Cluster on Azure with Terraform 🚀

What You'll Build

Think of this as building your own distributed supercomputer! Your cluster will provision compute nodes spread across different Azure regions for global coverage.

Before You Start

You'll need a few things ready:

Terraform (version 1.0.0 or newer)
A running Bacalhau orchestrator node
Azure CLI installed and set up
An active Azure subscription
Your subscription ID handy
An SSH key pair for securely accessing your nodes

Quick Setup Guide

First, create a terraform.tfvars.json file with your Azure details:
```
cp terraform.tfvars.example.json terraform.tfvars.json
```

Open up terraform.tfvars.json and fill in your Azure details:

 "subscription_id": "f67231aa-8387-498b-9ca3-EXAMPLE",
 "app_tag": "bacalhau-cluster",
 "resource_group_region": "eastus",
 "username": "bacalhau-runner",
 "public_key": "~/.ssh/id_rsa.pub",
 "bacalhau_data_dir": "/bacalhau_data",
 "bacalhau_node_dir": "/bacalhau_node",
 "bacalhau_config_file_path": "./config/config.yaml",

 "locations": {
     "eastus": {
         "machine_type": "Standard_D4_v4",
         "node_count": 1
     }
}

Update your config/config.yaml with your orchestrator information. Specifically, these lines:

  Orchestrators:
    - nats://EXAMPLE-7a02-4083-bf08-bcc2f5fbc025.us1.cloud.expanso.dev:4222
  Auth:
    Token: "EXAMPLE-aEEFukWVffnf5jb9QkpNnwfiBWEk3475csM7ysudpbFTzYBap5c7sWr6"

Let Terraform get everything ready:
```
terraform init
```
Launch your cluster:
```
terraform apply
```

Understanding the Configuration

The infrastructure is organized into modules:

Network: Creates VNets and subnets in each region
Security Group: Sets up NSGs with rules for SSH, HTTP, and NATS
Instance: Provisions VMs with cloud-init configuration

Taking Your Cluster for a Test Drive

Once everything's up and running, let's make sure it works!

First, make sure you have the Bacalhau CLI installed. You can read more about installing the CLI .
Setup your configuration to point at your orchestrator node:
```
bacalhau config set -c API.Host=<ip-address-of-orchestrator>
```
Check on the health of your nodes:
```
bacalhau node list
```

Run a simple test job:

bacalhau docker run docker.io/bacalhauproject/hello-world

Check on your jobs:
```
bacalhau list
```
Get your results:
```
bacalhau get <job-id>
```

Troubleshooting Tips

Having issues? Here are some common solutions:

Deployment Problems

Double-check your Azure permissions
Make sure your subscription is active
Verify that all needed resource providers are registered

Node Health Issues

Look at the logs on a node: journalctl -u bacalhau-startup.service
Check Docker logs on a node: docker logs <container-id>
Make sure that port 4222 isn't blocked

Job Running Troubles

Verify your NATS connection settings
Check if nodes are properly registered
Make sure compute is enabled in your config

Cleaning Up

When you're done, clean everything up with:

terraform destroy

Need to Check on Things?

If you need to peek under the hood, here's how:

Find your node IPs:
```
terraform output deployment_status
```

SSH into a node:

ssh -i ~/.ssh/id_rsa <username>@<public-ip>

Check on Docker:
```
docker ps
```

Go into the container on the node:

CONTAINER_ID=$(docker ps --filter name=^/bacalhau_node --format '{{.ID}}' | head -n1)
docker exec -it $CONTAINER_ID /bin/bash

Understanding the Configuration Files

Here's what each important file does in your setup:

Core Files

main.tf: Your main Terraform configuration
variables.tf: Where input variables are defined
outputs.tf: What information Terraform will show you

Modules

modules/network: Handles VNet and subnet creation
modules/securityGroup: Manages network security groups
modules/instance: Provisions VMs with cloud-init

Cloud-Init and Docker Setup

cloud-init/init-vm.yml: Sets up your VM environment, installs packages, and gets services running
config/docker-compose.yml: Runs Bacalhau in a privileged container with all the right volumes and health checks

Azure Specific Commands

For ensuring that you have configured your Azure CLI correctly, here are some commands you can use:

Get available Azure locations

az account list-locations --query "[].{Name:name, DisplayName:displayName}" -o table

Get available VM sizes in a region

az vm list-sizes --location eastus --query "[].{Name:name, vCPUs:numberOfCores, MemoryGB:memoryInMb, GPUs:gpus}" --output table

Get your subscription ID

az account show --query "id"

Need Help?

If you get stuck or have questions:

Open an issue in our
Join our

We're here to help you get your cluster running smoothly! 🌟

Setting Up a Cluster on Amazon Web Services (AWS) with Terraform 🚀

What You'll Build

Think of this as building your own distributed supercomputer! Your cluster will provision compute nodes spread across different AWS regions for global coverage.

Before You Start

You'll need a few things ready:

Terraform (version 1.0.0 or newer)
AWS CLI installed and configured
An active AWS account with appropriate permissions
Your AWS credentials configured
An SSH key pair for securely accessing your nodes
A Bacalhau network

Quick Setup Guide

First, set up an orchestrator node. We recommend using for this! But you can always set up your own
Create your environment configuration file:
```
cp env.tfvars.example.json env.tfvars.json
```

Fill in your AWS details in env.tfvars.json:

{
    "app_name": "bacalhau",
    "app_tag": "bacalhau-cluster",
    "bacalhau_installation_id": "EXAMPLE-c5c1-44fd-a0f3-1b90488f1b68",
    "bacalhau_data_dir": "/bacalhau_data",
    "bacalhau_node_dir": "/bacalhau_node",
    "username": "bacalhau-runner",
    "public_key_path": "/path/to/your/.ssh/id_rsa.pub",
    "private_key_path": "/path/to/your/.ssh/id_rsa"
}

Configure your desired regions in locations.yaml. Here's an example (we have a full list of these in all_locations.yaml):

- us-west-2:
    region: us-west-2
    zone: us-west-2a
    instance_type: t3.medium
    instance_ami: ami-07d9cf938edb0739b
    node_count: 1
- eu-central-1:
    region: eu-central-1
    zone: eu-central-1a
    instance_type: t3.medium
    instance_ami: ami-0e54671bdf3c8ed8d
    node_count: 1

Make sure the AMI exists in the region you need it to! You can confirm this by executing the following command:

REGION=us-east-1
IMAGE_SUBSTRING=amzn2-ami-hvm
ARCH=x86_64
aws ec2 describe-images \
--region $REGION \
--filters "Name=name,Values=*$IMAGE_SUBSTRING*" "Name=architecture,Values=$ARCH" \
--query 'sort_by(Images, &CreationDate)[-10:].{AMI:ImageId, Name:Name, CreationDate:CreationDate}' \
--output table \
--no-cli-pager

Update your Bacalhau config/config.yaml (the defaults are mostly fine, just update the Orchestrator, and Token lines):

NameProvider: puuid
API:
  Port: 1234
Compute:
  Enabled: true
  Orchestrators:
    - nats://EXAMPLE-6ed7-4d95-8871-b46153007057.us1.cloud.expanso.dev:4222
  Auth:
    Token: "EXAMPLE.EFukWVffnf5jb9QkpNnwfiBWEk3475csM7ysudpbFTzYBap5c7sWr6"
  TLS:
    RequireTLS: true
  AllowListedLocalPaths:
    - /bacalhau_data:rw
JobAdmissionControl:
  AcceptNetworkedJobs: true

Deploy your cluster using the Python deployment script:
```
./deploy.py create
```

Understanding the Configuration

Why use a deployment script? Why not use Terraform directly?

Core Configuration Files

env.tfvars.json: Your main configuration file containing AWS-specific settings`
locations.yaml: Defines which regions to deploy to and instance configurations
config/config.yaml: Bacalhau node configuration

Essential Settings in env.tfvars.json

app_name: Name for your cluster resources
app_tag: Tag for resource management
bacalhau_installation_id: Unique identifier for your cluster
username: SSH username for instances
public_key_path: Path to your SSH public key
private_key_path: Path to your SSH private key
bacalhau_config_file_path: Path to the config file for this compute node (should point at the orchestrator and have the right token)

Location Configuration (locations.yaml)

Each region entry requires:

region: AWS region (e.g., us-west-2)
zone: Availability zone (e.g., us-west-2a)
instance_type: EC2 instance type (e.g., t3.medium)
instance_ami: AMI ID for the region
node_count: Number of instances to deploy

Taking Your Cluster for a Test Drive

Once everything's up and running, let's make sure it works!

First, make sure you have the Bacalhau CLI installed. You can read more about installing the CLI .

Configure your Bacalhau client:

bacalhau config set -c API.Host=<orchestrator-ip> # Should be the same orchestrator
                                                  # IP from your config.yaml

List your compute nodes:
```
bacalhau node list
```

Run a test job:

bacalhau docker run ubuntu echo "Hello from my cluster!"

Check job status:
```
bacalhau list
```

Troubleshooting Tips

Deployment Issues

Verify AWS credentials are properly configured:
```
aws sts get-caller-identity
```
Check IAM permissions
Ensure you have quota available in target regions

Node Health Issues

SSH into a node:

ssh -i ~/.ssh/id_rsa bacalhau-runner@<public-ip> # Or whatever username you set

Check Bacalhau service logs:
```
journalctl -u bacalhau-startup.service
```

Check Docker container status:

docker ps
docker logs bacalhau-node-bacalhau-node-1

Network Issues

Verify security group rules (ports 22, 80, and 4222 should be open)
Check VPC and subnet configurations
Ensure internet gateway is properly attached

Common Solutions

If nodes aren't joining the network:
- Check NATS connection string in config.yaml
- Verify security group allows port 4222
- Ensure nodes can reach the orchestrator
If jobs aren't running:
- Check compute is enabled in node config
- Verify Docker is running properly
- Check available disk space
If deployment fails:
- Look for errors in Terraform output
- Check AWS service quotas
- Verify AMI availability in chosen regions

Cleanup

Remove all resources:

./deploy.py destroy

Monitoring

Check node health:
```
curl http://<node-ip>/healthz
```

Understanding the Directory Structure

setting-up-bacalhau-with-terraform-on-AWS/
├── modules/
│   ├── instance/       # EC2 instance configuration
│   ├── network/        # VPC and subnet setup
│   ├── region/         # Region-specific resources
│   └── securityGroup/  # Security group rules
├── config/             # Bacalhau configuration
├── scripts/           # Deployment and utility scripts
├── locations.yaml     # Region definitions
└── env.tfvars.json    # Environment configuration

Need Help?

If you get stuck or have questions:

Open an issue in our
Join our

We're here to help you get your cluster running smoothly! 🌟