Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Assessing the risks associated with cloud LLMs, including data retention, input training usage, and jurisdictional concerns.
- Exploring Ollama’s architecture, covering its model server, registry, and OpenAI-compatible API.
- Comparing Ollama with alternatives such as vLLM, llama.cpp, and Text Generation Inference.
- Reviewing licensing terms for models like Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Setting up Ollama on Linux with support for CUDA and ROCm.
- Implementing CPU-only fallbacks and optimizing for AVX/AVX2.
- Deploying via Docker with persistent volume mapping.
- Configuring multi-GPU setups and strategies for VRAM allocation.
Model Management
- Retrieving models from the Ollama registry using commands like 'ollama pull llama3'.
- Importing GGUF models from sources such as HuggingFace and TheBloke.
- Evaluating the tradeoffs between quantization levels: Q4_K_M, Q5_K_M, and Q8_0.
- Managing model switching and understanding concurrent model loading limitations.
Creating Custom Modelfiles
- Mastery of Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to define role-specific behaviors.
- Publishing custom models to a local registry.
API Integration
- Leveraging the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enabling JSON mode.
- Integrating with frameworks like LangChain, LlamaIndex, and custom applications.
- Implementing authentication and rate limiting using reverse proxies.
Performance Optimization
- Managing context window sizing and KV cache operations.
- Optimizing batch inference and parallel request handling.
- Allocating CPU threads with NUMA awareness.
- Monitoring GPU utilization and addressing memory pressure.
Security and Compliance
- Establishing network isolation for model serving endpoints.
- Implementing input filtering and output moderation pipelines.
- Maintaining audit logs for prompts and completions.
- Verifying model provenance and hash integrity.
Requirements
- Intermediate proficiency in Linux and container management.
- A foundational understanding of machine learning concepts and transformer architectures.
- Working knowledge of REST APIs and JSON data formats.
Target Audience
- AI engineers and developers seeking to substitute cloud-based LLM APIs with local solutions.
- Organizations with strict data sensitivity constraints that preclude the use of cloud models.
- Government and defense units necessitating air-gapped language model capabilities.
