Hybrid tuning subsystem

Back to Home

Hybrid tuning subsystem {#overview}

The hybrid tuning subsystem lets the distributed server discover, author, and broadcast every tunable parameter while workers stay read-only holders. It combines metric-driven health evaluation, deterministic decision making, and replay manifests so that any update stream can be reproduced offline. Online edits are intentionally conservative; operators can opt into larger offline sweeps that feed new baselines into the seed bank.

Tunables {#tunables}

All tunables originate from the dataclass hierarchy in Config plus the runtime overrides defined in RuntimeTuning. Discovery walks both server and worker modules to locate constants annotated with # TUNABLE so that documentation and tuning_report.json stay in sync with code. Each tunable records its owner, default, and whether it is safe for online adjustment.
The canonical parameter list and defaults live in docs/tuning-parameters.md.

Snapshots {#snapshots}

The server publishes snapshots whenever a worker reconnects or explicitly requests /api/simulation/config. Snapshots include the full simulation dataclass tree, current runtime overrides, the whitelist of online-safe keys, and a signature derived from a stable hash. Workers treat snapshots as authoritative and validate their local state accordingly.

Deltas {#deltas}

Online nudges are sent as deltas that carry only the paths which changed plus a signature and monotonically increasing version. Workers reject stale or unknown keys, logging a CONFLICT_OVERRIDDEN message whenever a server value replaces a CLI or environment default. Offline runs can still apply broad changes by setting online=False when issuing deltas from the server CLI.

Runtime tunables {#runtime-tunables}

Runtime overrides let the server adjust worker cadences without rebuilding executors mid-cycle. The authoritative payload exposes executor_checkpoint_interval, state_save_interval, max_worker_error_threshold, and the new accept_worker_tuning toggle; workers forward these through ConfigSyncManager.get_runtime_config() so that the LocalExecutor can pick up new values at generation boundaries. Result snapshot cadence is no longer a runtime override—operators should tune the adaptive batching profile (target_interval_seconds, min_batch_size, max_batch_size) when they need to reshape network load.

Authoritative config {#authoritative-config}

AuthoritativeConfig owns the baseline Config, applies deltas with coercion and clamping, records history for replay manifests, and emits snapshot/delta payloads. Unsafe online edits throw UnsafeParameterError immediately, ensuring the control plane never leaks unexpected fields into workers.

Discovery {#discovery}

discover_tunables() walks the repository, merges dataclass defaults with worker constants, and writes structured metadata. It powers both documentation and the bytebiota tune discover command, keeping server operators aware of every knob the tuning controller might touch.

Metrics {#metrics}

Metric helpers derive diversity (Shannon index, dominant family ratio), adaptation momentum (net birth rate, mutation pressure), and stability (rolling slope/variance) from the state aggregator. Adaptation metrics now look at the most recent history window rather than lifetime totals, and mutation pressure uses the aggregate mutation rate that the server computes as total_mutations / global_instructions. The health evaluator combines these figures with configurable thresholds to flag warning or critical states.

Health {#health}

HybridHealthEvaluator converts raw metrics into a normalized score, collecting any degradation messages and suggesting remedial actions. A warning prompts cautious online tuning; a critical score pushes the controller to schedule offline exploration.

Policy {#policy}

Online policy heuristics apply small, reversible adjustments that respect the whitelist and per-parameter clamps. Offline policy prepares larger moves (e.g., widening scheduler bounds) that should be rolled out through deterministic restarts. Every decision records a confidence value and reason for auditability.

Controller {#controller}

HybridTuningController binds metrics, health evaluation, policy planning, invariant checks, and delta application. It maintains replay manifests so that operators can compare online behaviour against offline replays or regression tests.

Invariants {#invariants}

Invariant checks guard against unsafe values such as negative mutation rates, scheduler windows that collapse, or runaway submission cadences. Violations raise a runtime exception in the controller and are also logged through the dedicated tuning logger.

Determinism {#determinism}

Deterministic helpers seed Python (and NumPy when present), capture the history of applied deltas, and produce replay manifests. Offline validation harnesses can feed these manifests back into the controller to guarantee identical outcomes across environments.

Checkpointing {#checkpointing}

Every server checkpoint embeds the latest tuning snapshot and delta history via TuningCheckpointManager. Restoring a checkpoint therefore restores not only organism state but the exact tuning context that produced it.

Server integration {#server-integration}

The server enables tuning through --tuning CLI or environment flags. During result ingestion the controller is invoked at a configurable tick interval, computes health metrics, applies safe online deltas, and broadcasts payloads over the existing WebSocket channel. The REST API exposes the pending tuning payload alongside traditional simulation metadata.

Tuning Trigger Optimization: The tuning system only runs when results contain actual simulation steps (step_count > 0). Event fast-lane results (births, deaths, seed submissions) with step_count=0 bypass tuning assessment to prevent excessive tuning activity while maintaining real-time responsiveness for critical events.

Worker integration {#worker-integration}

Workers now inherit tuning acceptance from the runtime override runtime.accept_worker_tuning that the server broadcasts in each snapshot (default: true). ConfigSyncManager queues and validates deltas, applies changes at generation boundaries, logs conflicts, and exposes runtime overrides to LocalExecutor. When updates would have been unsafe, workers simply retain their previous values and raise a validation warning. Operators can pause live tuning for a cohort by issuing a delta that sets runtime.accept_worker_tuning=false; the server restores the previous value once conditions improve.

CLI {#cli}

Use bytebiota tune status to inspect the authoritative version, bytebiota tune discover to regenerate documentation, and bytebiota tune offline-opt to plan deterministic offline searches. None of these commands mutates the running simulation; they provide the observability and inputs required for deliberate operator interventions.

Operations {#operations}

The basic execution loop requires one server and at least one worker. Run the server first so that it can accept worker registrations and broadcast snapshots. The commands below assume you are inside the project root with your virtual environment activated.

python -m bytebiota server --tuning --tuning-tick-interval 250 --tuning-online --port 18080

Then launch a worker pointed at the server:

python -m bytebiota worker --server-url http://127.0.0.1:18080 --preset auto

Leave both processes running; stop them with Ctrl+C when finished. To confirm the tuning loop is active while they run:

  • Tail server audits: tail -f {data_dir}/logs/tuning_actions.ndjson (default: data/logs/tuning_actions.ndjson)
  • Tail worker receipts: tail -f {data_dir}/logs/tuning_deltas.ndjson (default: data/logs/tuning_deltas.ndjson)
  • Poll the current payload: curl -s http://127.0.0.1:18080/api/simulation/config | jq '.tuning'
  • Spot-check health: Use the web dashboard at http://127.0.0.1:18080/monitoring for real-time health monitoring
  • Open the web dashboard at http://<server>:<port>/tuning to view live health, metrics, and recent deltas.

Persisting tuned parameters {#persist-tuned-params}

Tuning changes are ephemeral—each server restart rebuilds the controller from the baseline Config. Before shutting down a run you want to keep, capture the tuned values using one of these methods:

  1. REST snapshot (quick and scriptable)

bash curl -s http://<server>:<port>/api/monitoring/tuning | jq '.pending_payload.config'

The simulation block mirrors the dataclass tree; copy the updated fields into your baseline config file or environment variables.

  1. Tuning history log

The server echoes each evaluation to {data_dir}/logs/tuning_actions.ndjson (created automatically once tuning runs). The log directory is configurable via the DATA_BASE_DIR environment variable. Grab the latest entry:

bash tail -n1 {data_dir}/logs/tuning_actions.ndjson | jq '.changes'

Apply those key→value pairs to your baseline configuration to reuse the tuned defaults.

  1. Replay manifest

Access HybridTuningController.generate_manifest() (either via a small script or an admin endpoint) to obtain a replayable history. Reapply that manifest automatically on startup if you prefer code-driven persistence.

After you’ve captured the values, update the baseline Config (or your .env) so the next server boot starts with the tuned parameters and new workers inherit them immediately. Optionally, rerun bytebiota tune discover to regenerate documentation with the new defaults.

Mapping values back to configuration

  • Editing Config defaults: translate the dotted path into attribute access. Example: a delta on reaper.reap_chance means changing Config.reaper.reap_chance (see src/bytebiota/config.py).
  • Using environment variables: convert each dotted path to uppercase with underscores (e.g. reaper.reap_chance → REAPER_REAP_CHANCE, mutation.copy_bit_flip_rate → MUTATION_COPY_BIT_FLIP_RATE). Place those in your .env or deployment environment so Config.from_env() picks them up.
  • Runtime-only keys: entries under runtime.* (e.g. runtime.state_save_interval) affect workers at runtime only; they’re not parsed from .env. Persist them via your replay manifest or a startup delta if needed.

Always re-run the relevant tests (pytest tests/tuning -q) after adjusting defaults to make sure the new baseline sits within the expected invariants.

Two quick diagnostics help verify determinism:

  1. python -m bytebiota tune offline-opt --space mutation.copy_bit_flip_rate=0.01,0.0125,0.015 --space scheduler.base_slice=5,8,10 --seed 42
  2. pytest tests/tuning -q

If logs show repeated stale or not_whitelisted entries, review the whitelist in docs/tuning-parameters.md. When health remains critical for several cycles, schedule an offline baseline search and roll forward with a new snapshot.

Web UI layout {#web-ui-layout}

The /tuning page now mirrors the /distributed dashboard layout for consistency. The container, header card, page title gradient, card spacing, and table styles reuse the same design tokens as distributed.css. See src/bytebiota/ui/static/css/tuning.css for the aligned rules. This improves visual parity and reduces cognitive switching when navigating between control surfaces.