Deployment Guide

Deployment Guide

Back to Home

Use this guide to deploy ByteBiota for local research runs, shared servers, or cloud experiments. This covers both Python-based deployment and standalone binary deployment.

Environments {#deployment-environments}

Local Development

  • Target: Developer workstations, research machines
  • Method: Python virtual environment with pip install -e .
  • Configuration: Environment variables and CLI flags
  • Monitoring: Local log files and web UI

Production Servers

  • Target: Dedicated servers, cloud instances
  • Method: Standalone worker binaries or Python deployment
  • Configuration: Systemd services, environment files
  • Monitoring: Centralized logging, health checks

Distributed Clusters

  • Target: Multi-machine deployments
  • Method: Mixed deployment (server on main node, workers on compute nodes)
  • Configuration: Network configuration, load balancing
  • Monitoring: Distributed metrics collection

Binary Deployment {#binary-deployment}

Building Worker Binaries

For production deployments, build standalone worker binaries:

# Install build dependencies
pip install -r requirements.txt pyinstaller

# Build for current platform (defaults to https://www.bytebiota.com)
bash scripts/build_worker.sh

# Build with custom version
VERSION=2025.01.15 bash scripts/build_worker.sh

# Build with custom server URL
SERVER_URL=https://your-server.com bash scripts/build_worker.sh

The build process automatically configures the worker to connect to the specified server URL. By default, workers are configured to connect to https://www.bytebiota.com with full SSL/TLS support for both HTTP and WebSocket connections.

Binary Distribution

  1. Archive the build output:
    bash # The build script creates archives automatically # Linux/macOS: bytebiota-worker-{version}-{platform}-{arch}.tar.gz # Windows: bytebiota-worker-{version}-windows-amd64.zip

  2. Transfer to target machines:
    bash # Copy archive and checksum file scp bytebiota-worker-2025.01.15-linux-amd64.tar.gz user@server:/opt/bytebiota/ scp bytebiota-worker-2025.01.15-linux-amd64.sha256 user@server:/opt/bytebiota/

  3. Verify and install:
    bash # On target machine cd /opt/bytebiota tar -xzf bytebiota-worker-2025.01.15-linux-amd64.tar.gz sha256sum -c bytebiota-worker-2025.01.15-linux-amd64.sha256

Binary Configuration

Worker binaries use the same configuration as Python deployments. The build process creates a default configuration file (worker.conf) with the server URL:

# Environment variables (override defaults)
export SERVER_URL=https://your-server.com
export WORKER_ID=worker-001
export RESOURCE_PRESET=standard

# Or command-line flags
./bytebiota-worker --server-url https://your-server.com --preset standard --worker-id worker-001

# Or use the included configuration file
# Edit worker.conf to customize settings

SSL/TLS Support: Workers automatically support HTTPS and WebSocket SSL (WSS) connections. When you specify an https:// URL, the worker will:
- Use HTTPS for HTTP API calls
- Use WSS (WebSocket Secure) for real-time communication
- Validate SSL certificates by default

Python Deployment {#python-deployment}

Virtual Environment Setup

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate  # Linux/macOS
# .venv\Scripts\activate   # Windows

# Install ByteBiota
pip install -e .

# Install additional dependencies
pip install -r requirements.txt

Server Deployment

# Start server
python -m bytebiota server --host 0.0.0.0 --port 8080 --no-reload

# With custom configuration
SERVER_HOST=0.0.0.0 SERVER_PORT=8080 MAX_WORKERS=50 \
  python -m bytebiota server --no-reload

Worker Deployment

# Start worker
python -m bytebiota worker --server-url http://server:8080 --preset standard

# With custom resource limits
python -m bytebiota worker \
  --server-url http://server:8080 \
  --cpu-percent 75 \
  --memory-mb 2048 \
  --batch-size 10

Production Service Management {#service-management}

Systemd Services

Server Service (/etc/systemd/system/bytebiota-server.service):

[Unit]
Description=ByteBiota Distributed Server
After=network.target

[Service]
Type=simple
User=bytebiota
Group=bytebiota
WorkingDirectory=/opt/bytebiota
Environment=PYTHONPATH=/opt/bytebiota/src
Environment=SERVER_HOST=0.0.0.0
Environment=SERVER_PORT=8080
Environment=MAX_WORKERS=100
Environment=ENABLE_WEB_UI=true
ExecStart=/opt/bytebiota/.venv/bin/python -m bytebiota server --no-reload
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Worker Service (/etc/systemd/system/bytebiota-worker@.service):

[Unit]
Description=ByteBiota Worker %i
After=network.target

[Service]
Type=simple
User=bytebiota
Group=bytebiota
WorkingDirectory=/opt/bytebiota
Environment=PYTHONPATH=/opt/bytebiota/src
Environment=WORKER_ID=worker-%i
Environment=SERVER_URL=http://localhost:8080
Environment=RESOURCE_PRESET=standard
ExecStart=/opt/bytebiota/.venv/bin/python -m bytebiota worker
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Binary Worker Service (/etc/systemd/system/bytebiota-worker-binary@.service):

[Unit]
Description=ByteBiota Worker Binary %i
After=network.target

[Service]
Type=simple
User=bytebiota
Group=bytebiota
WorkingDirectory=/opt/bytebiota
Environment=SERVER_URL=http://localhost:8080
Environment=RESOURCE_PRESET=standard
ExecStart=/opt/bytebiota/bytebiota-worker --server-url http://localhost:8080 --preset standard --worker-id worker-%i
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Service Management Commands

# Enable and start services
sudo systemctl daemon-reload
sudo systemctl enable bytebiota-server
sudo systemctl start bytebiota-server

# Start multiple workers
sudo systemctl enable bytebiota-worker@1
sudo systemctl start bytebiota-worker@1
sudo systemctl enable bytebiota-worker@2
sudo systemctl start bytebiota-worker@2

# Check status
sudo systemctl status bytebiota-server
sudo systemctl status bytebiota-worker@1

# View logs
sudo journalctl -u bytebiota-server -f
sudo journalctl -u bytebiota-worker@1 -f

Configuration Management {#configuration}

Environment Files

Create /opt/bytebiota/.env:

# Server configuration
SERVER_HOST=0.0.0.0
SERVER_PORT=8080
MAX_WORKERS=100
SEED_BANK_SIZE=2000
CHECKPOINT_INTERVAL=300
ENABLE_WEB_UI=true
LOG_LEVEL=INFO

# Worker configuration
SERVER_URL=http://localhost:8080
RESOURCE_PRESET=standard
CPU_PERCENT=50
MEMORY_MB=1024
POLL_INTERVAL=1.0
HEARTBEAT_INTERVAL=10.0

Resource Presets

Preset CPU % Memory Use Case
minimal 10% 256MB Light background processing
background 25% 512MB Background tasks
standard 50% 1024MB Normal operation
full 100% 4096MB Maximum performance
auto Detected Detected Automatically detects optimal settings

Monitoring and Health Checks {#monitoring}

Health Check Endpoints

# Server health
curl -f http://localhost:8080/api/simulation/stats

# Worker connectivity
curl -f http://localhost:8080/api/workers/stats

# Distributed overview
curl -f http://localhost:8080/api/distributed/overview

Log Monitoring

# View all logs
tail -f logs/*.log

# View specific service logs
tail -f logs/server_*.log
tail -f logs/worker_*.log

# Follow logs with timestamps
tail -f -t logs/*.log

Metrics Dashboard

Access the web interface at:
- http://localhost:8080 - Main dashboard
- http://localhost:8080/distributed - Distributed system overview
- http://localhost:8080/api/distributed/overview - API overview

Scaling {#scaling}

Horizontal Scaling

Add more workers by starting additional processes:

# Python workers
python -m bytebiota worker --preset standard --server-url http://server:8080

# Binary workers
./bytebiota-worker --server-url http://server:8080 --preset standard

# Systemd workers
sudo systemctl start bytebiota-worker@3
sudo systemctl start bytebiota-worker@4

Vertical Scaling

Adjust resource limits:

# High-performance worker
export CPU_PERCENT=100
export MEMORY_MB=4096
export BATCH_SIZE=20
python -m bytebiota worker --server-url http://server:8080

Backup and Recovery {#backup}

Checkpoint Backup

# Backup checkpoints
tar -czf checkpoints-backup-$(date +%Y%m%d).tar.gz server_checkpoints/ worker_checkpoints/

# Restore checkpoints
tar -xzf checkpoints-backup-20240101.tar.gz

Configuration Backup

# Backup configuration
cp .env .env.backup
cp -r config/ config.backup/

# Backup systemd services
sudo cp /etc/systemd/system/bytebiota-*.service /opt/bytebiota/backup/

Update Procedures {#updates}

Python Deployment Updates

# Pull latest code
git pull origin main

# Update dependencies
pip install -e .

# Restart services
sudo systemctl restart bytebiota-server
sudo systemctl restart bytebiota-worker@*

Binary Deployment Updates

# Build new binary
bash scripts/build_worker.sh

# Stop workers
sudo systemctl stop bytebiota-worker@*

# Replace binary
sudo cp dist/bytebiota-worker-*/bytebiota-worker /opt/bytebiota/

# Start workers
sudo systemctl start bytebiota-worker@*

Troubleshooting {#troubleshooting}

Common Issues

  1. Workers can't connect to server:
    ```bash
    # Check network connectivity
    ping server
    curl -f http://server:8080/api/simulation/stats

# Check server logs
tail -f logs/server_*.log
```

  1. High resource usage:
    ```bash
    # Monitor resource usage
    htop
    ps aux | grep bytebiota

# Adjust resource limits
export CPU_PERCENT=25
export MEMORY_MB=512
```

  1. Checkpoint failures:
    ```bash
    # Check disk space
    df -h

# Check checkpoint directory permissions
ls -la server_checkpoints/
```

Debug Mode

# Server with debug logging
LOG_LEVEL=DEBUG python -m bytebiota server --no-reload

# Worker with debug logging
LOG_LEVEL=DEBUG python -m bytebiota worker

Security Considerations {#security}

Network Security

# Use firewall to restrict access
sudo ufw allow 8080/tcp
sudo ufw deny from 0.0.0.0/0 to any port 8080

Resource Limits

# Set process limits
ulimit -n 4096
ulimit -u 1024

User Permissions

# Create dedicated user
sudo useradd -r -s /bin/false bytebiota
sudo chown -R bytebiota:bytebiota /opt/bytebiota

Maintenance {#maintenance}

Cleanup

# Remove old checkpoints
find checkpoints/ -name "*.json" -mtime +7 -delete

# Clean up logs
find logs/ -name "*.log" -mtime +30 -delete

Monitoring Scripts

Create monitoring scripts:

#!/bin/bash
# monitor.sh
while true; do
  echo "$(date): $(ps aux | grep bytebiota | grep -v grep | wc -l) processes running"
  sleep 60
done