Panel Inference Fabric

The Fabric page is part of the shared native Panel shell. Read-only visitors can inspect descriptions and permitted state but cannot submit privileged planner or worker-control requests. Administrators see only actions allowed by their role and the exact endpoint scope of their credential; every submitted action invokes the existing Fabric route owner rather than a duplicate simulator. A guided step, including its undo, must pass the same route-specific authorization check.

The Inference Fabric page at /rust/fabric (also /rust/advanced/fabric) lets operators inspect registered workers, see heartbeats and recent request telemetry, plan capacity and placement, inspect timing traces, or—in an administrator session—change worker registry state and send requests. It uses the existing fabric endpoints rather than a separate simulation. Numerical outputs are plans and estimates, not measurements or proof of available hardware.

Quick start

1. Open Inference Fabric and read Registered workers, heartbeats, cache and telemetry. The state view shows registration expiry, load snapshots, cache hints and recent telemetry.
2. Open Fabric capabilities to see which operations are exposed.
3. Choose a planner such as Route a request, Admit or queue, Scale replicas, or Place replicas on devices. Edit Request (JSON) using your own snapshots, then select Run planner. The example values are illustrative; do not treat them as measured performance.
4. Inspect Result and Request sent to verify the exact input and any returned warnings. Invalid JSON is reported without a route call.
5. Only administrators can use the Confirm and run buttons for changes. Check the confirmation box first. Register a live worker adds metadata; it does not install or start a runtime. Heartbeat a worker refreshes that metadata, and Remove a worker stops the worker being considered for new routing.
6. Dispatch a request and Live streaming test send real requests and can consume compute resources. Use them only when a suitable endpoint is available. A failed request is shown as an error rather than a successful inference.

Guided operations

Open /rust/fabric/guides in an administrator session.

Register two workers and plan a deployment

1. Enter First worker ID, First worker endpoint, Second worker ID, Second worker endpoint and Loaded model label. The two IDs must be new and different. Endpoints must meet the registry's local/fleet address restrictions and include a port.
2. Check I have checked the actual worker state and authorize these changes. Then choose Run guided task.
3. The guide uses Register a live worker twice and checks each saved ID, then reads Registered workers, heartbeats, cache and telemetry.
4. It invokes Place replicas on devices with two representative device slots. Successful placement is checked using unscheduled_replicas = 0. The plan does not allocate hardware or start processes; replace the example memory numbers with measured inventory when making an actual deployment decision.
5. If a step fails, the guide attempts to remove only the worker registrations created by this run. Any rollback failure appears explicitly in the result. On success, ensure the actual workers continue to send heartbeats; registrations expire without them.

Find a prefill bottleneck and apply a routing hint

1. Make sure the worker is already registered and ready, and verify its real cached prefixes independently. This task requires a worker with no existing fabric cache hints so that rollback cannot erase an earlier hint.
2. Enter Registered worker ID, Loaded model label, Measured request telemetry JSON, Verified cache prefix hashes (JSON array) and Verified resident token count.
3. Confirm and select Run guided task. Bottleneck of one request attributes time to queue, prefill, decode or transfer. This guide only proceeds if prefill is the largest contributor.
4. Record cached prefixes applies an advisory routing hint; it cannot create a cache in the runtime. Route a request then recalculates the destination using that hint, and the guide verifies the metadata in the state endpoint.
5. If a later check fails, the new cache hint is removed. Observe new requests under similar load before treating the hint as an improvement. Decode, queue and transfer bottlenecks need different corrective actions; use the individual planners to investigate them.

Administrators should not run guided actions against a worker that they did not authorize. Viewers can read the fabric state and use planning forms, but state-changing forms and these guided tasks are hidden or refused.

Forms and controls

Every operation has its own form under Inference Fabric with editable JSON input, a one-sentence explanation, results and a request echo. State-changing forms additionally display a confirmation checkbox.

The Request (JSON) editor accepts up to 256 KiB. Run planner executes read-only computations. Confirm and run is administrator-only for worker registration, heartbeat, removal, cache changes, dispatch and sparsification. Live streaming test requires separate consent and a running endpoint. Sparsify a weights file writes a new file and refuses to overwrite the source or an existing destination.

Editor fields, buttons and guided task tooltips

Every editable field and run or confirmation control displays a ? help marker beside its label. The following entries are also registered, with the exact tooltip displayed by the native page.

Interpreting results

• Admission and scheduling: these decide whether and where a request could run; they do not send it. Dispatch a request does send it.
• Capacity and deployment: variants, replica counts and device placement depend on your submitted demand and inventory. Scale replicas makes a recommendation but does not start or stop replicas.
• Cache and offload: cache hints are advisory. Where to keep a cache estimates memory, storage and recomputation trade-offs; it does not move a cache.
• Optimization: Model transformations to try, Rank measured transformations, Activation-aware precision and Distillation plan describe candidate changes or stages. Only Sparsify a weights file writes an artifact.
• Telemetry: Aggregate telemetry, Bottleneck of one request and Trace one request consume supplied request measurements. Distinguish observed timing from simulated request samples.

For broader instance and deployment controls see Panel Instances, Panel Fleet and Panel Fit Lab.