V1.7.0 Release Notes

Enterprise-Ready Distributed Computing

We are thrilled to announce the release of Bacalhau v1.7.0, marking a significant milestone in our journey to make distributed computing accessible and enterprise-ready. This release introduces enterprise support options, enhanced job management capabilities, significant performance improvements, and a host of new features that make it easier than ever for organizations to deploy and manage distributed compute networks at scale.

Enterprise Support through Expanso

Bacalhau v1.7.0 introduces enterprise-grade support options through a flexible licensing model. Organizations can now either leverage the fully managed orchestration provided by Expanso Cloud or purchase licenses for their on-premises Bacalhau networks. Through a new self-service model in Expanso Cloud, organizations can easily select and purchase support licenses based on their node count requirements. This offering ensures that enterprises have access to professional support, enabling them to confidently deploy and maintain their distributed compute infrastructure, whether in the cloud or on-premises.

Our tiered licensing model provides flexibility for organizations of all sizes, from growing startups to large enterprises, ensuring they receive the level of support that matches their needs.

Expanso Cloud: Enhanced Job Management and Templates

The Expanso Cloud platform now offers comprehensive job submission capabilities through its web interface. Going beyond basic job specifications, users can now choose from a library of pre-defined job templates and create their own custom workflows. The platform includes templates for common use cases such as log processing, data analysis with DuckDB, Apache Iceberg operations, and more. This enhancement significantly reduces the complexity of common distributed computing tasks while maintaining the flexibility to create custom solutions.

Enhanced Job Execution with Partitioned Workloads

Bacalhau v1.7.0 enhances our partitioning capabilities, allowing you to efficiently process large datasets and compute-intensive tasks across distributed nodes. This powerful feature intelligently distributes work across compute nodes with built-in partition management, failure handling, and execution context.

By distributing work across multiple compute nodes, partitioning enables horizontal scaling and parallel processing, significantly improving processing speed and resource utilization. It also provides granular failure recovery by isolating errors within individual partitions. If one partition fails, the system continues processing other partitions without interruption, with automatic retry mechanisms for failed partitions.

Each partition runs independently with its own lifecycle and error handling, while receiving essential context through environment variables. This allows your code to identify its assigned partition, access job-level information, and implement partition-specific logic without complex coordination code.

S3 Storage Input Partitioning

Processing large datasets from S3 just got a whole lot easier in Bacalhau v1.7.0. Our enhanced S3 partitioning feature automatically handles data distribution across multiple executions, complete with graceful failure handling and independent retry of failed partitions.

Before this feature, processing large S3 datasets required creating multiple jobs or writing custom code to split your data processing. Now, Bacalhau orchestrates all of this for you. Simply specify your partitioning strategy, and each execution automatically receives its designated subset of S3 objects while your processing code remains clean and focused on its core task.

Bacalhau supports multiple partitioning strategies to meet different needs:

• Object-Based Distribution: Even distribution of files without specific grouping

• Date-Based Partitioning: Process each day's data in parallel for time-series analysis

• Regex Pattern Matching: Distribute by matching patterns in file paths (e.g., regions)

• Substring-Based Partitioning: Organize by customer segments or categories

Jobs can also combine partitioned data with shared reference data, giving you complete flexibility in how you distribute your data processing workloads without writing complex partition-aware code.

S3 Publishing Enhancements

Bacalhau v1.7.0 introduces important enhancements to our S3 publisher, focusing on improved flexibility and simplified integration with existing workflows.

Our S3 publisher now supports plain encoding, providing more flexibility in how job results are stored in S3-compatible storage. This enhancement allows for direct access to job outputs without additional processing, publishing files individually rather than as a compressed archive. This is particularly valuable when subsequent jobs need to access individual files or when building data pipelines with partial result access.

While plain encoding offers benefits like efficient access to individual files and no decompression overhead, it does involve some tradeoffs such as more S3 PUT requests and higher storage costs. The default gzip encoding remains recommended when results are typically accessed as a complete set or when storage and transfer costs are a concern.

Enhanced Networking Capabilities

Bacalhau v1.7.0 brings significant improvements to networking capabilities, making it easier to run jobs that require network connectivity while providing greater control over network configuration.

Networking is now enabled by default, removing barriers to running jobs that require internet access. This change shifts networking from opt-in to opt-out, with a new configuration option for environments that need to restrict network access. We've also added experimental support for port mapping and advanced network configuration for jobs, with multiple network modes (bridge, host, none) and flexible port mapping options.

Comprehensive Authentication and Authorization

Bacalhau v1.7.0 transforms the user experience with a completely redesigned authentication and authorization system. This major enhancement dramatically simplifies the onboarding process while providing enterprise-grade security controls that organizations require.

We now support multiple authentication methods to meet diverse organizational needs, including basic authentication, API tokens, Single Sign-On (SSO), and fine-grained authorization controls to define precise user capabilities. New CLI commands have been introduced to manage authentication, making it easier to view status information and handle SSO operations.

These authentication improvements provide the security controls that organizations require while ensuring a smooth experience for both new and existing users. The system establishes a foundation for future enterprise authentication features and simplifies integration with existing identity management systems.

Host Environment Variable Forwarding

A powerful new feature in v1.7.0 is Host Environment Variable Forwarding, which allows jobs to access specified environment variables from the host system. This enables secure credential passing from the host to jobs through a controlled allowlist mechanism. Users can reference host variables using the env: prefix in job specifications, while compute nodes can define allowlists to control which variables can be forwarded.

Improved WebAssembly Support

Bacalhau v1.7.0 significantly enhances our WebAssembly (WASM) executor, making it a more powerful option for secure, portable distributed computing workloads. We've completely refactored the WASM job specification to improve separation of concerns and align with our other executors. Remote modules are now defined in the job's input sources like any other input source, with the WASM engine spec simply referencing where those modules are mounted. Legacy job specifications are automatically transformed for compatibility, ensuring a smooth transition for existing users. A major advancement in this release is the addition of HTTP networking support for WebAssembly workloads through a new host function implementation. This enables web requests directly from any WASM module, regardless of the language or compiler used. The feature includes configurable network access modes, security controls like domain allowlists, and resource protection measures. For developers working with Go, we've also provided a TinyGo-compatible client library that makes it easy to integrate HTTP functionality into your WASM modules.

Docker DinD Integration for Simplified Deployment

Bacalhau v1.7.0 introduces Docker-in-Docker (DinD) images that make it easier to deploy Bacalhau compute nodes within containerized environments. We now offer two image variants: the standard base image optimized for client usage and orchestrator nodes, and the new DinD image specifically designed for compute nodes that need to run Docker workloads.

The DinD image includes a pre-configured Docker daemon and provides full Docker execution capabilities, making it simple to deploy compute nodes as containers in your existing Docker ecosystem. This integration streamlines the process of scaling out compute resources in containerized environments and provides greater flexibility for organizations using container orchestration platforms like Kubernetes or Docker Swarm.

Performance and Quality of Life Improvements

Bacalhau v1.7.0 includes performance optimizations and usability enhancements that improve the overall experience. Enhanced resource management and more efficient state tracking allow organizations to build larger networks without sacrificing reliability, particularly valuable for large-scale data processing needs and edge computing deployments.

We've also made several usability improvements to enhance the daily experience with Bacalhau. The count field now defaults to 1 if not provided, eliminating a common source of confusion where jobs would silently fail to run with the previous default of 0. Additionally, we've added a NetworkDefault type that allows compute nodes to determine appropriate networking based on their capabilities and the executor type. These changes collectively make Bacalhau more accessible to new users while streamlining workflows for experienced users.

Comprehensive Documentation Overhaul

We've completely rebuilt our documentation with a clear, task-based organization. The new documentation includes improved architectural guides, comprehensive API references, and better navigation to help users find the information they need. We've also enhanced our troubleshooting guidance and simplified the overall structure to make it easier for both new and experienced users to get the most out of Bacalhau.

FAQ

Enterprise Support

Q: What license tiers are available? A: Bacalhau offers a free tier supporting up to 5 nodes, making it easy for teams to get started. Beyond that, organizations can purchase licenses based on their node count requirements through the Expanso Cloud self-service portal.

Q: What happens if my network exceeds the licensed node count? A: Your network will continue to operate normally even if you exceed your licensed node count. The system will display a friendly notice about the overage, giving you time to adjust your license as needed.

Q: How do I purchase a license? A: Licenses can be purchased directly through the Expanso Cloud self-service portal. Simply log in, select your desired node count, and complete the purchase process.

Q: How are licenses deployed? A: Users can download their license directly from Expanso Cloud and deploy it to their network. Only the orchestrator needs the license deployed, eliminating the need to manage licenses on compute nodes or clients. A new configuration option allows you to specify the local license path on the orchestrator.

Expanso Cloud Templates

Q: What pre-defined templates are included? A: Expanso Cloud provides templates for common use cases including log processing, data analysis with DuckDB, and Apache Iceberg operations. These templates serve as starting points for your distributed computing workloads.

Q: How do templates handle customization? A: Templates feature customizable placeholders that allow you to reuse the same template with different values. For example, you can use the same log processing template with different input sources, output destinations, or processing parameters.

Q: Can I create my own templates? A: Yes, the platform provides an intuitive interface for creating custom templates. You can define job specifications, add parameters, include documentation, and then share these templates across your organization.

Partitioned Executions and S3 Input

Q: How do partitioned executions work? A: The system provides environment variables to each execution that indicate its partition number and the total number of partitions, allowing implementations to deterministically divide work across nodes.

Q: What happens if a partition fails? A: Bacalhau automatically retries just that partition while preserving the results from successful executions, ensuring you don't lose progress when processing large datasets.

Q: What partitioning strategies are available for S3 data? A: Bacalhau supports object-based distribution, date-based partitioning, regex pattern matching, and substring-based partitioning, with the ability to combine partitioned data with shared reference data.

S3 Publishing

Q: What storage services are supported? A: The pre-signed URL system works with any S3 API-compatible storage service, including Amazon S3, MinIO, and Google Cloud Storage.

Q: How does plain encoding for S3 publishing work? A: When plain encoding is selected, job outputs are stored directly in S3 without compression, with each file stored individually rather than as a single archive.

Q: When should I use plain encoding versus the default gzip encoding? A: Use plain encoding when subsequent jobs need to access individual files or when building data pipelines with partial result access. Use gzip encoding when results are typically accessed as a complete set or when storage costs are a concern.

Networking and Connectivity

Q: What networking capabilities are enabled by default? A: Basic outbound network connectivity is now enabled by default, allowing jobs to access external resources without additional configuration. This is a shift from previous versions where networking was disabled by default.

Q: How do I disable networking if needed? A: Compute nodes can be configured to reject networked jobs by setting JobAdmissionControl.RejectNetworkedJobs to true. Jobs can explicitly disable networking by setting network.type to none.

Q: How do I use the experimental port mapping feature?

A: Port mapping can be configured in job specifications using the network.ports field, which allows defining named ports with optional static host port assignments and required container target ports. Each port receives automatic environment variables for discovery.

Q: What network modes are supported? A: Bacalhau supports three network modes:

• bridge: Container networking with port mapping (default for Docker)

• host: Direct access to host network stack (Linux only)

• none: Complete network isolation

Authentication and Authorization

Q: How do I switch between authentication methods? A: The system provides clear CLI commands for managing authentication. You can view your current authentication status with auth info and switch methods as needed.

Q: How does SSO integration work? A: SSO integration follows standard OAuth flows, with commands for initiating login (auth sso login), managing tokens (auth sso token), and logging out (auth sso logout).

Q: What authorization controls are available? A: The system now provides fine-grained controls for defining user capabilities, allowing administrators to limit which actions users can perform within the system.

Host Environment Variables

Q: How does the environment variable forwarding work? A: Host environment variables can be forwarded to jobs by using the env: prefix in job environment variable definitions. For example, API_KEY: "env:API_KEY" forwards the host's API_KEY to the job.

Q: What security controls are in place? A: Compute nodes must explicitly allowlist environment variables or patterns (e.g., API_*) that can be forwarded to jobs. Any attempt to access non-allowlisted variables will cause the job to fail during the bid phase.

Q: Can I use this for secrets management? A: Yes, this feature provides a secure way to pass credentials and configuration from the host to jobs through a controlled mechanism.

WebAssembly Support

Q: What improvements have been made to the WASM executor? A: The WASM executor has been refactored for better separation of concerns, with remote modules now defined as input sources like other inputs, aligning with other executors in the system.

Q: How does the HTTP capability for WASM work? A: WebAssembly modules can now make HTTP requests through a new host function implementation that works with any WASM module, regardless of the language or compiler used. This capability includes configurable security controls like domain allowlists, size limits, and timeouts.

Q: Is there a client library available for the HTTP functionality? A: Yes, we provide a TinyGo-compatible client library that makes it easy to integrate HTTP functionality into your WASM modules, though the underlying host function will work with any WebAssembly module.

Q: How do I migrate from the old WASM job definition to the new one? A: Existing job specifications will continue to work as we automatically transform them for compatibility. We recommend using the new format where modules are specified as input sources.

Docker DinD Integration

Q: What are the requirements for using Docker DinD images? A: Docker DinD images require Docker to be installed on the host system and appropriate privileges (privileged mode) for the container to run nested Docker instances.

Q: When should I use the DinD image versus the standard image? A: Use the DinD image for compute nodes that need to run Docker workloads. Use the standard image for client usage, orchestrator nodes, and compute nodes running non-Docker workloads.

Performance and Quality of Life Improvement

Q: What performance improvements are included in this release? A: Enhanced resource management and more efficient state tracking allow organizations to build larger networks without sacrificing reliability, particularly valuable for large-scale data processing needs.

Q: What usability improvements have been made? A: The count field now defaults to 1 if not provided, and we've added a NetworkDefault type that allows compute nodes to determine appropriate networking based on their capabilities.

Documentation and Use Cases

Q: What improvements have been made to the documentation? A: We've completely rebuilt our documentation with a task-based organization, improved architectural guides, comprehensive API references, and better navigation to help users find information more easily.

Q: How is the documentation structured? A: The documentation is organized around common tasks and use cases, with dedicated sections for getting started, core concepts, advanced usage, and troubleshooting.