GPU Support

Bacalhau supports GPUs out of the box and defaults to allowing execution on all GPUs installed on the node.

Prerequisites

Bacalhau makes the assumption that you have installed all the necessary drivers and tools on your node host and have appropriately configured them for use by Docker.

In general for GPUs from any vendor, the Bacalhau client requires:

• Docker

• Permission to access Docker

Nvidia

• Drivers for your GPU

• NVIDIA Container Toolkit (nvidia-docker2)

• nvidia-smi installed and functional

You can test whether you have a working GPU setup with the following command:

docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

AMD

Bacalhau requires AMD drivers to be appropriately installed and access to the rocm-smi tool.

You can test whether you have a working GPU setup with the following command, which should print details of your GPUs:

docker run --rm --device=/dev/kfd --device=/dev/dri --entrypoint=rocm-smi rocm/rocm-terminal

Intel

Bacalhau requires appropriate Intel drivers to be installed and access to the xpu-smi tool.

You can test whether you have a working GPU setup with the following command, which should print details of your GPUs:

docker run --rm --device=/dev/dri --entrypoint=/bin/bash intel/xpumanager -- -c 'xpumd & sleep 5; xpumcli discovery'

GPU Node Configuration

Access to GPUs can be controlled using resource limits. To limit the number of GPUs that can be used per job, set a job resource limit. To limit access to GPUs from all jobs, set a total resource limit.