Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
AI Sovereignty and Local Deployment of LLMs
- Examination of risks associated with cloud-based LLMs, including data retention policies, input utilization for training, and exposure to foreign jurisdictions.
- Overview of Ollama architecture, specifically the model server, registry components, and OpenAI-compatible API endpoints.
- Technical comparison between Ollama and alternative frameworks such as vLLM, llama.cpp, and Text Generation Inference.
- Analysis of licensing terms for major models, including Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Deployment of Ollama on Linux environments with support for CUDA and ROCm acceleration.
- Configuration of CPU-only fallback mechanisms and optimization for AVX/AVX2 instruction sets.
- Implementation of Docker-based deployments and strategies for persistent volume mapping.
- Design of multi-GPU configurations and protocols for VRAM allocation.
Model Management Procedures
- Retrieval of models from the Ollama registry, including specific commands such as ollama pull llama3.
- Importing GGUF format models from external repositories such as HuggingFace and TheBloke.
- Evaluation of quantization levels, specifically Q4_K_M, Q5_K_M, and Q8_0, and their associated tradeoffs.
- Management of model switching and adherence to concurrent model loading limits.
Custom Modelfile Development
- Syntax standards for Modelfiles, covering FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Calibration of inference parameters, including temperature, top_p, and repeat_penalty.
- Engineering of system prompts to define role-specific behaviors for government applications.
- Creation and publication of custom models to a local registry.
API Integration Standards
- Utilization of the OpenAI-compatible /v1/chat/completions endpoint for service integration.
- Implementation of streaming responses and strict JSON output modes.
- Integration of local LLMs with frameworks such as LangChain, LlamaIndex, and proprietary applications.
- Configuration of authentication and rate limiting mechanisms via reverse proxies.
Performance Optimization Strategies
- Determination of context window sizes and management of KV cache resources.
- Implementation of batch inference and protocols for parallel request handling.
- Optimization of CPU thread allocation and adherence to NUMA topology awareness.
- Monitoring of GPU utilization rates and memory pressure indicators.
Security and Compliance Frameworks
- Establishment of network isolation protocols for model serving endpoints for government environments.
- Implementation of input filtering and output moderation pipelines to ensure data integrity.
- Creation of comprehensive audit logs for all prompts and generated completions.
- Verification of model provenance and integrity through hash validation.
Requirements
- Intermediate proficiency in Linux system administration and container management.
- A conceptual understanding of machine learning and transformer architectures.
- Familiarity with RESTful APIs and JSON data formats.
Target Audience
- AI engineers and developers seeking to replace cloud-based LLM APIs with secure alternatives.
- Organizations with strict data sensitivity requirements that preclude the use of public cloud models.
- Government and defense sectors requiring air-gapped, sovereign language model capabilities.
14 Hours
