MLOps vs. LLMOps & Production Infrastructure

• Traditional MLOps (Left): Focuses on training models from scratch, managing tabular feature stores, serialized model pickles, and automated continuous retraining (CT).
• LLMOps (Right): Centers on foundation model routing, prompt template versioning, vector database embeddings, and parameter-efficient fine-tuning (LoRA).
• Operational Lifecycle: Shifts continuous operations from compute-intensive retraining to golden benchmark evaluation, token cost optimization, and safety guardrail tripwires.

• Interviewer Question: "Our team wants to migrate our fraud detection pipeline from XGBoost to an LLM. How does our CI/CD and deployment lifecycle change between classical MLOps and LLMOps?"
• Candidate Depth: Junior: Notes that LLMs use prompts rather than model training code. Senior: Explains shifting from continuous retraining (CT) on labeled data to golden evaluation datasets, prompt regression suites, and token cost budgeting. Principal: Details architectural decoupling where foundation model APIs require semantic version pinning, LoRA adapter registries, and runtime output filtering rather than full weight rebuilds.
• Common Trap: Assuming LLMOps eliminates feature engineering when it actually replaces tabular feature stores with embedding pipelines and vector retrieval.

Implement an automated evaluation harness (e.g., DeepEval, Ragas) integrated into CI/CD that blocks pull requests when semantic drift exceeds 2% against a versioned golden benchmark dataset.

> Axiom: In MLOps, code and data produce a model; in LLMOps, a frozen foundation model, prompts, and external data produce an application.

• Git-Based Semantic Versioning: System prompts are stored as standalone configuration artifacts (e.g., `prompt-v2.4.1.yaml`) decoupled from application logic.
• Traceable Metadata Headers: Every template explicitly records author identity, target foundation model, approved sampling temperature, and changelog history.
• Automated Regression Gates: Git push triggers CI/CD benchmark runs evaluating 500 golden dataset cases and requiring approval from both domain experts and security engineers.

• Interviewer Question: "A developer modifies a production customer support prompt to sound friendlier, inadvertently opening a jailbreak vulnerability. How does your infrastructure prevent this?"
• Candidate Depth: Junior: Suggests manual review before updating prompts in code. Senior: Enforces GitOps prompt repos where pull requests trigger automated adversarial injection tests and semantic regression checks. Principal: Enforces cryptographic digest hashing at the API gateway, rejecting any prompt invocation that lacks an approved registry signature and evaluation attestation.
• Common Trap: Treating prompt templates as informal string variables rather than immutable, audited software dependencies.

Deploy an immutable prompt registry with GitOps CI/CD pipelines that execute automated red-teaming and prompt regression test suites before tagging releases.

> Axiom: Hardcoding prompts in application source code is technical debt; treat prompts as immutable, versioned configuration artifacts.

• Cryptographic Provenance: Details model identity (e.g., `Llama-3.3-70B-Instruct-v1.2`) verified against official SHA-256 weight checksums and paired with approved LoRA adapter checkpoints (`hr-domain-adapter-r16-v3`).
• Evaluation Scorecard: Embeds quantitative operational metrics directly in the metadata catalog, displaying Safety (99.8%), Groundedness (96.2%), and Latency (1.8s).
• Lifecycle Gate Progression: Governs promotion transitions (`Staging` → `Canary` → `Production` → `Archived`) bound to authorized internal vLLM inference worker clusters.

• Interviewer Question: "How do you prevent engineering teams from deploying arbitrary unvetted open-source foundation model weights across production microservices?"
• Candidate Depth: Junior: Blocks outbound internet access to Hugging Face on inference servers. Senior: Implements an internal artifact repository where clusters only serve approved model checkpoints. Principal: Enforces mTLS service mesh policies where inference clusters reject execution requests unless the requested model-adapter digest is in the signed 'APPROVED' registry state.
• Common Trap: Registering base foundation model weights while failing to track fine-tuned LoRA adapters, hyperparameter configs, and evaluation scorecards.

Configure production vLLM inference clusters and API gateways to authorize only model digests bearing an active, cryptographically signed 'APPROVED' status in the enterprise registry.

> Axiom: Applications in production may only invoke models and adapters that carry an explicit 'APPROVED' tag in the model registry.

• Five-Node Provenance Pipeline: Traces data flow from raw Source Document (`SHA: a8f9b2`) through Text Chunking (#14, 380 tokens), Vector Embedding (1536-d index), Retrieval Event (0.89 similarity), to final Generated Answer.
• Granular Audit Identifiers: Each pipeline stage stamps an immutable audit hash (`a8f9b2` → `7c4e9b` → `d3a4f8` → `9b1e2c`), logging timestamp, user ID (9841), and document version.
• Verifiable Citation Grounding: Pairs generated user responses with clickable citations directly grounded in the authenticated source chunk.

• Interviewer Question: "A customer claims our enterprise assistant gave negligent HR policy guidance six months ago. How do you prove whether the error was due to an out-of-date source document, a bad embedding chunk, or model hallucination?"
• Candidate Depth: Junior: Searches chat database logs for the user session. Senior: Uses OpenTelemetry span attributes to correlate user query, retrieved chunk IDs, and model parameters. Principal: Reconstructs the complete lineage DAG—validating document SHA-256, chunk token offsets, vector similarity score, and system prompt hash active at that precise timestamp.
• Common Trap: Logging only user prompts and raw model completions without recording chunk hashes, embedding index snapshots, and retrieval similarity scores.

Inject source document SHA-256 checksums and chunk IDs into vector payload metadata, propagating immutable trace IDs across all retrieval spans.

> Axiom: When an audit asks "Why did the AI say this 6 months ago?", data lineage provides the exact document, chunk, and model version used.

• Intended Use & Provenance (Sections 1-2): Defines approved business workflows and explicitly prohibited use cases, alongside dataset cutoff dates, licensing, and preprocessing steps.
• Quantitative Benchmarks (Section 3): Documents empirical performance across accuracy, F1 score, perplexity, and demographic fairness parity across subgroup cohorts.
• Safety Boundaries & Compute Impact (Sections 4-5): Discloses known hallucination risks, adversarial attack vulnerabilities, hardware configurations, and carbon footprints.

• Interviewer Question: "Your product team wants to use an internal customer service LLM to screen job applicant resumes. How does the model card stop this?"
• Candidate Depth: Junior: States that hiring discrimination is illegal. Senior: Cites the model card's 'Prohibited Use Cases' section and absence of hiring fairness evaluations. Principal: Integrates model card metadata into deployment admission controllers, automatically blocking API deployment requests whose declared workload category violates the card's verified use boundaries.
• Common Trap: Treating model cards as static, post-hoc documentation rather than machine-readable governance contracts.

Publish machine-readable JSON-LD model cards in the central registry, executing CI/CD admission controllers that evaluate deployment scope against prohibited use lists.

> Axiom: A model card complements system documentation; it communicates what the underlying model can and cannot safely do.

• Traffic Splitting Gateway: Dynamically partitions 100% incoming user traffic between Stable V1 (95%) and Canary V2 (5%) at the API gateway layer.
• Automated Validation Triad: Continuously measures latency/error spikes (Normal), hallucination guardrail tripwires (Zero), and user thumbs-down signals (Zero).
• Progressive Promotion Schedule: Governs automated gate decisions, expanding canary traffic from 5% to 25% to 100% over a 48-hour observation window.

• Interviewer Question: "We are upgrading our enterprise code-generation model from v1 to v2. How do you architect a safe canary rollout, and what specific metrics trigger an immediate abort?"
• Candidate Depth: Junior: Routes 10% traffic to v2 and monitors server HTTP 500 errors. Senior: Configures a 5% canary split monitoring p99 latency, guardrail trigger frequency, and negative feedback spikes. Principal: Implements dynamic gateway weighted routing with automated circuit-breakers that instantly revert traffic to v1 if semantic tripwires (e.g., syntax error rate, guardrail violations) exceed baseline thresholds by 1.5%.
• Common Trap: Relying strictly on HTTP error codes (e.g., 200 OK) while failing to inspect semantic failure modes, hallucination rates, and user sentiment.

Implement Envoy or Kong dynamic weighted routing linked to Prometheus alerts that automatically revert canary traffic to 0% upon any guardrail tripwire breach.

> Axiom: Never flip 100% of enterprise users to a new model version at once; roll out gradually with real-time safety monitoring.

• Rollback (State & Code Reversion): Reverts deployment from V2 back to V1 when canary evaluations or post-deployment monitoring detect regression test failures.
• Fallback (Runtime Fault Tolerance): Reroutes live inflight requests from a failing primary API to a secondary approved model, rule-based cache, or human agent.
• Architectural Security Invariant: The secondary fallback pathway must strictly enforce identical security, data privacy, and compliance policies as the primary path.

• Interviewer Question: "Our primary commercial LLM API provider suffers an unannounced regional outage. How does your production AI architecture maintain continuity without violating customer data privacy?"
• Candidate Depth: Junior: Returns an error message or retries the failed API call. Senior: Activates circuit breakers that fail over to an approved secondary self-hosted model or cached responses. Principal: Architects multi-cloud fallback routing with unified prompt abstraction layers, verifying that the secondary provider enforces equivalent zero-data-retention (ZDR) agreements and encryption standards.
• Common Trap: Failing over to an unvetted public LLM endpoint that lacks enterprise data processing agreements and security guardrails.

Deploy circuit-breaker proxies (e.g., Resilience4j, Envoy) configured with automated fallback cascades to SOC-2-certified secondary endpoints with matching data-privacy guarantees.

> Axiom: A fallback must enforce the exact same security and data-privacy standards as the primary path.

• Inner Foundation Layer: Measures raw intelligence parameters including perplexity, benchmark accuracy, and raw inference availability.
• Outer Application Architecture: Encompasses five mission-critical subsystems: data ingestion/chunking, vector database latency, identity/ACL filters, tool parsing/serialization, and API gateway timeouts.
• Compounding Failure Modes: Demonstrates that peripheral infrastructure bottlenecks cause system outages even when the underlying foundation model operates flawlessly.

• Interviewer Question: "Our base LLM has a 99.9% uptime SLA and 95% benchmark accuracy, yet customer tickets report our agent fails 10% of the time. How do you diagnose and architect against this disparity?"
• Candidate Depth: Junior: Blames model hallucinations or prompts. Senior: Identifies peripheral points of failure such as vector search timeouts, malformed JSON tool outputs, and context window truncation. Principal: Quantifies compound reliability mathematically (R_(system) = ∏ Rᵢ), showing how five 98%-reliable subsystems multiply to an overall 90% system reliability, and implements end-to-end distributed tracing across all pipeline hops.
• Common Trap: Assuming foundation model intelligence and uptime directly translate into end-to-end application availability.

Instrument end-to-end OpenTelemetry distributed tracing across ingestion, vector retrieval, guardrail filtering, and schema serialization, enforcing component-level SLOs.

> Axiom: A capable foundation model is only one component of an enterprise application; true reliability lives in the surrounding infrastructure.