Bacalhau Docs
GithubSlackBlogEnterprise
v1.6.x
  • Documentation
  • Use Cases
  • CLI & API
  • References
  • Community
v1.6.x
  • Welcome
  • Getting Started
    • How Bacalhau Works
    • Getting Started
      • Step 1: Install the Bacalhau CLI
      • Step 2: Running Your Own Job
      • Step 3: Checking on the Status of Your Job
    • Creating Your Own Bacalhau Network
      • Setting Up a Cluster on Amazon Web Services (AWS) with Terraform πŸš€
      • Setting Up a Cluster on Google Cloud Platform (GCP) With Terraform πŸš€
      • Setting Up a Cluster on Azure with Terraform πŸš€
    • Hardware Setup
    • Container Onboarding
      • Docker Workloads
      • WebAssembly (Wasm) Workloads
  • Setting Up
    • Running Nodes
      • Node Onboarding
      • GPU Installation
      • Job selection policy
      • Access Management
      • Node persistence
      • Configuring Your Input Sources
      • Configuring Transport Level Security
      • Limits and Timeouts
      • Test Network Locally
      • Bacalhau WebUI
      • Private IPFS Network Setup
    • Workload Onboarding
      • Container
        • Docker Workload Onboarding
        • WebAssembly (Wasm) Workloads
        • Bacalhau Docker Image
        • How To Work With Custom Containers in Bacalhau
      • Python
        • Building and Running Custom Python Container
        • Running Pandas on Bacalhau
        • Running a Python Script
        • Running Jupyter Notebooks on Bacalhau
        • Scripting Bacalhau with Python
      • R (language)
        • Building and Running your Custom R Containers on Bacalhau
        • Running a Simple R Script on Bacalhau
      • Run CUDA programs on Bacalhau
      • Running a Prolog Script
      • Reading Data from Multiple S3 Buckets using Bacalhau
      • Running Rust programs as WebAssembly (WASM)
      • Generate Synthetic Data using Sparkov Data Generation technique
    • Networking Instructions
      • Accessing the Internet from Jobs
      • Utilizing NATS.io within Bacalhau
    • GPU Workloads Setup
    • Automatic Update Checking
    • Marketplace Deployments
      • Google Cloud Marketplace
    • Inter-Nodes TLS
  • Guides
    • Configuration Management
    • Write a config.yaml
    • Write a SpecConfig
    • Using Labels and Constraints
  • Examples
    • Table of Contents for Bacalhau Examples
    • Data Engineering
      • Using Bacalhau with DuckDB
      • Ethereum Blockchain Analysis with Ethereum-ETL and Bacalhau
      • Convert CSV To Parquet Or Avro
      • Simple Image Processing
      • Oceanography - Data Conversion
      • Video Processing
      • Bacalhau and BigQuery
    • Data Ingestion
      • Copy Data from URL to Public Storage
      • Pinning Data
      • Running a Job over S3 data
    • Model Inference
      • EasyOCR (Optical Character Recognition) on Bacalhau
      • Running Inference on Dolly 2.0 Model with Hugging Face
      • Speech Recognition using Whisper
      • Stable Diffusion on a GPU
      • Stable Diffusion on a CPU
      • Object Detection with YOLOv5 on Bacalhau
      • Generate Realistic Images using StyleGAN3 and Bacalhau
      • Stable Diffusion Checkpoint Inference
      • Running Inference on a Model stored on S3
    • Model Training
      • Training Pytorch Model with Bacalhau
      • Training Tensorflow Model
      • Stable Diffusion Dreambooth (Finetuning)
    • Molecular Dynamics
      • Running BIDS Apps on Bacalhau
      • Coresets On Bacalhau
      • Genomics Data Generation
      • Gromacs for Analysis
      • Molecular Simulation with OpenMM and Bacalhau
    • Systems Engineering
      • Ad-hoc log query using DuckDB
  • References
    • Jobs Guide
      • Job Specification
        • Job Types
        • Task Specification
          • Engines
            • Docker Engine Specification
            • WebAssembly (WASM) Engine Specification
          • Publishers
            • IPFS Publisher Specification
            • Local Publisher Specification
            • S3 Publisher Specification
          • Sources
            • IPFS Source Specification
            • Local Source Specification
            • S3 Source Specification
            • URL Source Specification
          • Network Specification
          • Input Source Specification
          • Resources Specification
          • ResultPath Specification
        • Constraint Specification
        • Labels Specification
        • Meta Specification
      • Job Templates
      • Queuing & Timeouts
        • Job Queuing
        • Timeouts Specification
      • Job Results
        • State
    • CLI Guide
      • Single CLI commands
        • Agent
          • Agent Overview
          • Agent Alive
          • Agent Node
          • Agent Version
        • Config
          • Config Overview
          • Config Auto-Resources
          • Config Default
          • Config List
          • Config Set
        • Job
          • Job Overview
          • Job Describe
          • Job Executions
          • Job History
          • Job List
          • Job Logs
          • Job Run
          • Job Stop
        • Node
          • Node Overview
          • Node Approve
          • Node Delete
          • Node List
          • Node Describe
          • Node Reject
      • Command Migration
    • API Guide
      • Bacalhau API overview
      • Best Practices
      • Agent Endpoint
      • Orchestrator Endpoint
      • Migration API
    • Node Management
    • Authentication & Authorization
    • Database Integration
    • Debugging
      • Debugging Failed Jobs
      • Debugging Locally
    • Running Locally In Devstack
    • Setting up Dev Environment
  • Help & FAQ
    • Bacalhau FAQs
    • Glossary
    • Release Notes
      • v1.5.0 Release Notes
      • v1.4.0 Release Notes
  • Integrations
    • Apache Airflow Provider for Bacalhau
    • Lilypad
    • Bacalhau Python SDK
    • Observability for WebAssembly Workloads
  • Community
    • Social Media
    • Style Guide
    • Ways to Contribute
Powered by GitBook
LogoLogo

Use Cases

  • Distributed ETL
  • Edge ML
  • Distributed Data Warehousing
  • Fleet Management

About Us

  • Who we are
  • What we value

News & Blog

  • Blog

Get Support

  • Request Enterprise Solutions

Expanso (2025). All Rights Reserved.

On this page
  • Introduction
  • Prerequisite​
  • Downloading datasets​
  • Uploading the datasets to IPFS​
  • Running a Bacalhau Job​
  • Structure of the command​
  • Declarative job description​
  • Checking the State of your Jobs​
  • Viewing your Job Output​
  • Support​

Was this helpful?

Export as PDF
  1. Examples
  2. Molecular Dynamics

Running BIDS Apps on Bacalhau

PreviousMolecular DynamicsNextCoresets On Bacalhau

Was this helpful?

Introduction

In this example tutorial, we will look at how to run BIDS App on Bacalhau. BIDS (Brain Imaging Data Structure) is an emerging standard for organizing and describing neuroimaging datasets. is a container image capturing a neuroimaging pipeline that takes a BIDS formatted dataset as input. Each BIDS App has the same core set of command line arguments, making them easy to run and integrate into automated platforms. BIDS Apps are constructed in a way that does not depend on any software outside of the image other than the container engine.

Prerequisite​

To get started, you need to install the Bacalhau client, see more information

Downloading datasets​

For this tutorial, download file ds005.tar from this Bids dataset and untar it in a directory:

mkdir data
tar -xf ds005.tar -C data 

Let's take a look at the structure of the data directory:

data
└── ds005
    β”œβ”€β”€ CHANGES
    β”œβ”€β”€ dataset_description.json
    β”œβ”€β”€ participants.tsv
    β”œβ”€β”€ README
    β”œβ”€β”€ sub-01
    β”‚   β”œβ”€β”€ anat
    β”‚   β”‚   β”œβ”€β”€ sub-01_inplaneT2.nii.gz
    β”‚   β”‚   └── sub-01_T1w.nii.gz
    β”‚   └── func
    β”‚       β”œβ”€β”€ sub-01_task-mixedgamblestask_run-01_bold.nii.gz
    β”‚       β”œβ”€β”€ sub-01_task-mixedgamblestask_run-01_events.tsv
    β”‚       β”œβ”€β”€ sub-01_task-mixedgamblestask_run-02_bold.nii.gz
    β”‚       β”œβ”€β”€ sub-01_task-mixedgamblestask_run-02_events.tsv
    β”‚       β”œβ”€β”€ sub-01_task-mixedgamblestask_run-03_bold.nii.gz
    β”‚       └── sub-01_task-mixedgamblestask_run-03_events.tsv
    β”œβ”€β”€ sub-02
    β”‚   β”œβ”€β”€ anat
    β”‚   β”‚   β”œβ”€β”€ sub-02_inplaneT2.nii.gz
    β”‚   β”‚   └── sub-02_T1w.nii.gz
    ...

Uploading the datasets to IPFS​

When you pin your data, you'll get a CID which is in a format like this QmaNyzSpJCt1gMCQLd3QugihY6HzdYmA8QMEa45LDBbVPz. Copy the CID as it will be used to access your data

Running a Bacalhau Job​

export JOB_ID=$(bacalhau docker run \
    --id-only \
    --wait \
    --timeout 3600 \
    --wait-timeout-secs 3600 \
    -i ipfs://QmaNyzSpJCt1gMCQLd3QugihY6HzdYmA8QMEa45LDBbVPz:/data \
    nipreps/mriqc:latest \
    -- mriqc ../data/ds005 ../outputs participant --participant_label 01 02 03)

Structure of the command​

Let's look closely at the command above:

  1. bacalhau docker run: call to bacalhau

  2. -i ipfs://QmaNyzSpJCt1gMCQLd3QugihY6HzdYmA8QMEa45LDBbVPz:/data: mount the CID of the dataset that is uploaded to IPFS and mount it to a folder called data on the container

  3. nipreps/mriqc:latest: the name and the tag of the docker image we are using

  4. ../data/ds005: path to input dataset

  5. ../outputs: path to the output

  6. participant --participant_label 01 02 03: run the mriqc on subjects with participant labels 01, 02, and 03

When a job is submitted, Bacalhau prints out the related job_id. We store that in an environment variable so that we can reuse it later on.

Declarative job description​

Copy

name: Running BIDS
type: batch
count: 1
tasks:
  - name: My main task
    Engine:
      type: docker
      params:
        Image: nipreps/mriqc:latest
        Entrypoint:
          - /bin/bash
        Parameters:
          - -c
          - mriqc ../data/ds005 ../outputs participant --participant_label 01 02 03
    Publisher:
      Type: ipfs
    ResultPaths:
      - Name: outputs
        Path: /outputs
    InputSources:
      - Target: "/data"
        Source:
          Type: "ipfs"
          Params:
            CID: "QmaNyzSpJCt1gMCQLd3QugihY6HzdYmA8QMEa45LDBbVPz"

The job description should be saved in .yaml format, e.g. bids.yaml, and then run with the command:

bacalhau job run bids.yaml

Checking the State of your Jobs​

Job status: You can check the status of the job using bacalhau job list.

bacalhau job list --id-filter ${JOB_ID} --wide

When it says Published or Completed, that means the job is done, and we can get the results.

Job information: You can find out more information about your job by using bacalhau job describe.

bacalhau job describe ${JOB_ID}

Job download: You can download your job results directly by using bacalhau job get. Alternatively, you can choose to create a directory to store your results. In the command below, we created a directory (results) and downloaded our job output to be stored in that directory.

rm -rf results && mkdir -p results
bacalhau job get $JOB_ID --output-dir results

Viewing your Job Output​

To view the file, run the following command:

ls results/ # list the contents of the current directory 
cat results/stdout # displays the contents of the current directory 

Support​

The simplest way to upload the data to IPFS is to use a third-party service to "pin" data to the IPFS network, to ensure that the data exists and is available. To do this, you need an account with a pinning service like or . Once registered, you can use their UI or API or SDKs to upload files.

Alternatively, you can upload your dataset to IPFS using , but the recommended approach is to use a pinning service as we have mentioned above.

The same job can be presented in the format. In this case, the description will look like this:

If you have questions or need support or guidance, please reach out to the (#general channel).

BIDS App
here
folder
Pinata
nft.storage
IPFS CLI
declarative
Bacalhau team via Slack