Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Architectures for Government

• Evaluating hosted API versus self-managed infrastructure trade-offs
• Implementing on-premises, edge, and air-gapped operational patterns
• Comparing single model implementation versus multi-model portfolio strategies
• Assessing cost-effectiveness per decision outcome

Hardware Sizing and Capacity Planning

• Determining memory requirements based on model parameter counts
• Analyzing CPU-only performance versus GPU acceleration benefits
• Planning for concurrent request handling and throughput targets
• Allocating capacity headroom for increasing task complexity

Serving Stack Selection

• Reviewing vLLM and SGLang implementation options
• Evaluating llama.cpp and GGUF execution environments
• Utilizing Ollama for local development and testing
• Selecting infrastructure based on specific decision workload characteristics

Quantization and Calibration Integrity

• Understanding precision formats and associated memory savings
• Measuring impact on output quality and calibration accuracy
• Implementing per-tensor and importance-aware quantization variants
• Conducting acceptance testing to verify post-quantization performance

Latency Optimization for Government Services

• Differentiating between median and tail latency metrics
• Managing latency impacts as context length increases
• Optimizing warm start processes and model caching strategies
• Batching multiple decision queries to improve efficiency

Edge and Constrained Environment Deployment

• Deploying small decision models on CPU resources
• Implementing solutions on Apple Silicon and embedded hardware
• Establishing offline operation protocols and update strategies
• Managing weight distribution in air-gapped environments

Containerization and Scalability

• Designing container images optimized for model serving
• Managing storage volumes and cache efficiency
• Implementing horizontal scaling and traffic routing mechanisms
• Defining resource limits and GPU sharing policies

Observability and Monitoring

• Tracking request volume, latency, and error rates
• Monitoring confidence distribution metrics
• Logging decision outcomes and escalation events
• Configuring alerts for calibration drift detection

Version Control and Deployment Management

• Managing model and adapter versioning processes
• Implementing shadow and canary deployment strategies
• Pinning model files to ensure audit readiness
• Defining safe rollback procedures for incidents

Security and Multi-Tenancy Controls

• Enforcing authentication standards and rate limiting
• Ensuring tenant isolation for prompts and operational logs
• Implementing network isolation and egress traffic controls
• Maintaining comprehensive audit logs of all decisions