Worker Build Process

Worker Build Process

Back to Home

This document describes the ByteBiota worker build process, including how to create standalone binaries with custom server configurations.

Overview {#overview}

The ByteBiota worker build process creates standalone, self-contained binaries that can run on any compatible system without requiring Python installation. The build process automatically configures the worker to connect to a specified server URL with full SSL/TLS support.

Build Scripts {#build-scripts}

ByteBiota provides platform-specific build scripts for different operating systems:

Unix Build Script (macOS/Linux)

The Unix build script is located at scripts/build_worker_unix.sh and handles:

  • Platform detection (Linux, macOS)
  • Architecture detection (amd64, arm64)
  • Version management
  • Server URL configuration
  • Binary compilation with PyInstaller
  • Archive creation and checksum generation

Windows Build Scripts

For Windows, there are two build scripts:

  • scripts/build_worker.ps1 - PowerShell script (recommended)
  • scripts/build_worker.bat - Batch wrapper that calls PowerShell script

Both handle:
- Windows platform detection
- Architecture detection (amd64, arm64)
- Version management
- Server URL configuration
- Binary compilation with PyInstaller
- ZIP archive creation and checksum generation

Configuration {#configuration}

Default Server URL

By default, workers are configured to connect to https://www.bytebiota.com:

Unix (macOS/Linux):

# Default build
bash scripts/build_worker_unix.sh

Windows:

REM Default build (using batch wrapper)
scripts\build_worker.bat

REM Or directly with PowerShell
powershell -ExecutionPolicy Bypass -File scripts\build_worker.ps1

Custom Server URL

To build a worker for a different server:

Unix (macOS/Linux):

# Custom server URL
SERVER_URL=https://your-server.com bash scripts/build_worker_unix.sh

# Local development server
SERVER_URL=http://localhost:8080 bash scripts/build_worker_unix.sh

Windows:

REM Custom server URL
set SERVER_URL=https://your-server.com
scripts\build_worker.bat

REM Local development server
set SERVER_URL=http://localhost:8080
scripts\build_worker.bat

Version Control

Unix (macOS/Linux):

# Custom version
VERSION=2025.01.15 bash scripts/build_worker_unix.sh

Windows:

REM Custom version
set VERSION=2025.01.15
scripts\build_worker.bat

Automatic Version Detection

All build scripts now automatically detect existing builds and increment version numbers:

  • First build of the day: Uses base version (e.g., 2025.01.15)
  • Subsequent builds: Auto-increments (e.g., 2025.01.15.1, 2025.01.15.2)
  • Cross-platform: Each platform maintains its own version sequence

Windows Requirements {#windows-requirements}

PowerShell Execution Policy

Windows users may need to configure PowerShell execution policy to run the build scripts:

# Allow local scripts to run (recommended)
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Or bypass policy for this session only
powershell -ExecutionPolicy Bypass -File scripts\build_worker.ps1

Prerequisites

  • PowerShell 5.1+ (included with Windows 10/11)
  • Python 3.9+ with PyInstaller
  • Git for Windows (optional, for version control)

Troubleshooting

"Execution of scripts is disabled" error:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

"PowerShell is not available" error:
- Ensure PowerShell is installed and in PATH
- Try running powershell from Command Prompt to verify

Build Output {#build-output}

The build process creates several artifacts:

Binary Distribution

dist/bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}/
├── bytebiota-worker[.exe]     # Main executable
├── worker.conf                # Default configuration
└── bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}.sha256  # Checksum

Archive Files

releases/v{VERSION}/
├── bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}.tar.gz  # Linux/macOS
└── bytebiota-worker-{VERSION}-windows-amd64.zip         # Windows

SSL/TLS Support {#ssl-support}

The worker automatically supports secure connections:

HTTPS Support

  • HTTP API calls use HTTPS when server URL starts with https://
  • SSL certificate validation is enabled by default
  • Supports modern TLS protocols (1.2, 1.3)

WebSocket SSL (WSS)

  • WebSocket connections automatically use WSS for HTTPS servers
  • URL conversion: https://server.com → wss://server.com
  • Trailing slashes are normalized to prevent double slashes in WebSocket paths
  • Maintains secure connection throughout worker lifecycle

Configuration

# Secure connection (automatic SSL/TLS)
SERVER_URL=https://secure-server.com bash scripts/build_worker.sh

# Insecure connection (development only)
SERVER_URL=http://localhost:8080 bash scripts/build_worker.sh

Build Process Details {#build-details}

Environment Variables

The build script uses environment variables to configure the worker binary without modifying source files:

# Set during build
export WORKER_VERSION="$VERSION"
export SERVER_URL="$SERVER_URL"

# worker_entry.py reads from environment variables
BUILD_VERSION = os.environ.get('WORKER_VERSION', '0.1.0')
BUILD_SERVER_URL = os.environ.get('SERVER_URL', 'http://localhost:8080')

Clean Build Process

The build process is designed to be clean and reproducible:

  • No file modifications: The build script never modifies source files
  • Environment-based configuration: All build-time values are passed via environment variables
  • Template-based approach: worker_entry.py serves as a template that reads from environment variables
  • Reproducible builds: The same source code produces consistent results across different machines

PyInstaller Configuration

The build uses worker.spec for PyInstaller configuration:

  • Includes all required dependencies
  • Creates single-file executable
  • Handles platform-specific requirements
  • Optimizes for size and performance

Platform Support

Platform Architecture Binary Format Archive Format
Linux amd64, arm64 ELF .tar.gz
macOS amd64, arm64 Mach-O .tar.gz
Windows amd64 PE .zip

Usage Examples {#usage-examples}

Basic Build

Unix (macOS/Linux):

# Build for current platform
bash scripts/build_worker_unix.sh

# Output: dist/bytebiota-worker-2025.01.15-macos-amd64/

Windows:

REM Build for current platform
scripts\build_worker.bat

REM Output: dist\bytebiota-worker-2025.01.15-windows-amd64\

Production Build

Unix (macOS/Linux):

# Production server with custom version
VERSION=2025.01.15 SERVER_URL=https://prod.bytebiota.com bash scripts/build_worker_unix.sh

# Verify build
cd dist/bytebiota-worker-2025.01.15-macos-amd64/
./bytebiota-worker --help
./bytebiota-worker --server-url https://prod.bytebiota.com

Windows:

REM Production server with custom version
set VERSION=2025.01.15
set SERVER_URL=https://prod.bytebiota.com
scripts\build_worker.bat

REM Verify build
cd dist\bytebiota-worker-2025.01.15-windows-amd64\
bytebiota-worker.exe --help
bytebiota-worker.exe --server-url https://prod.bytebiota.com

Development Build

Unix (macOS/Linux):

# Local development server
SERVER_URL=http://localhost:8080 bash scripts/build_worker_unix.sh

# Test locally
cd dist/bytebiota-worker-2025.01.15-macos-amd64/
./bytebiota-worker --server-url http://localhost:8080 --verbose

Windows:

REM Local development server
set SERVER_URL=http://localhost:8080
scripts\build_worker.bat

REM Test locally
cd dist\bytebiota-worker-2025.01.15-windows-amd64\
bytebiota-worker.exe --server-url http://localhost:8080 --verbose

Troubleshooting {#troubleshooting}

Build Failures

# Check dependencies
pip install -r requirements.txt pyinstaller

# Clean previous builds
rm -rf build/ dist/

# Verbose build
bash -x scripts/build_worker.sh

SSL Issues

# Test SSL connection
curl -I https://www.bytebiota.com

# Check certificate
openssl s_client -connect www.bytebiota.com:443 -servername www.bytebiota.com

Platform Issues

# Check platform detection
uname -s  # Should return Linux, Darwin, or CYGWIN/MSYS
uname -m  # Should return x86_64, amd64, or arm64

Best Practices {#best-practices}

Security

  • Always use HTTPS for production deployments
  • Verify SSL certificates are valid
  • Use strong server authentication

Performance

  • Build on target platform when possible
  • Test binaries before distribution
  • Monitor resource usage in production

Maintenance

  • Keep build dependencies updated
  • Test with different server configurations
  • Document custom build parameters

Integration {#integration}

CI/CD Pipelines

# Example GitHub Actions workflow
- name: Build Worker Binary
  run: |
    VERSION=${{ github.sha }} \
    SERVER_URL=https://api.bytebiota.com \
    bash scripts/build_worker.sh

Automated Deployment

# Build and deploy script
#!/bin/bash
VERSION=$(date +%Y.%m.%d)
SERVER_URL=https://prod.bytebiota.com

# Build
bash scripts/build_worker.sh

# Deploy
scp releases/v${VERSION}/*.tar.gz deploy@server:/opt/bytebiota/