01The problem
Moving a model behind an endpoint introduces capacity, security, recovery and cost questions. Those behaviours need direct measurement.
02The approach
- Provision dedicated AWS networking and servers with Terraform; deploy speech and chat workloads through Helm on k3s.
- Use authenticated gateways, resource limits, probes, constrained uploads and model concurrency. Observe metrics with Prometheus and Grafana.
- Exercise one- and two-node deployments with k6 load, HPA scale changes, pod deletion and verified teardown.
03The evidence
- Both AWS topologies completed measured load, scaling and recovery experiments; their known resources were removed.
- One historical short-audio workload achieved 0.208 successful requests/s at 3,592 ms p95, estimated at $0.1387 per 1,000 successes.
- The selected fine-tuned int8 artifact subsequently passed a real-audio local gateway check.
04Scope & status
The historical AWS run used a pretrained tiny model and a fixed 5.76-second audio clip. Final images/model integration and production sign-off remain pending. The two-node comparison also had more total vCPU capacity.
Public source is available for independent inspection.