Production AI Architecture Diagrams — Interview Guide
Four real, company-grade architectures — MLOps, LLMOps/RAG, Agentic AI, and Multi-Agent systems — with numbered flows, the exact interview questions you'll face, and the trade-offs senior engineers are expected to know.
MLOps Production Architecture
This is the reference architecture for a classic ML system in production — think fraud detection at a fintech or demand forecasting at a retailer. Data flows from transactional and streaming sources through orchestrated pipelines into a feature store, models are trained on GPU-backed Kubernetes clusters and versioned in a registry, and CI/CD ships them behind an autoscaled serving layer. Monitoring closes the loop: when drift is detected, the pipeline retrains automatically. If you can whiteboard this diagram and defend every box, you pass the MLOps system-design round at most companies.
PostgreSQL
Snowflake
Kafka
Airflow
SparkFSFeast
Redis
MLflow
Ray
NVIDIA
GH Actions
Docker
ArgoCD
Kubernetes
vLLM
NGINX
Prometheus
Grafana
TerraformScroll horizontally to view the full diagram.
How data flows, step by step
- 1
Ingest raw data
Airflow DAGs (scheduled or sensor-triggered) pull batch data from PostgreSQL and Snowflake over JDBC, and consume event streams from Kafka topics. Raw data lands in cloud object storage (S3/GCS) as the source of truth.
- 2
Transform with Spark
Spark jobs clean, join, deduplicate, and aggregate raw data into training-ready datasets. Heavy jobs run on a separate node pool so they never starve the serving cluster.
- 3
Materialize features to the feature store
Feast registers feature definitions once and materializes them to two stores: an offline store (Parquet on S3 / Snowflake) for training, and an online store (Redis) for millisecond lookups at inference time.
- 4
Train on versioned feature sets
Training jobs read features with point-in-time-correct joins — each training row gets the feature values as they were at event time, which prevents data leakage. Distributed training runs on Ray across Kubernetes nodes with NVIDIA GPUs.
- 5
Log and register the model
Every run is logged to MLflow Tracking (params, metrics, artifacts). The best run is promoted to the MLflow Model Registry with a version, a model signature, and a stage (staging → production).
- 6
Registry promotion triggers CI
A webhook on registry promotion kicks off a GitHub Actions workflow: unit tests, data validation, model quality gates (e.g. AUC must beat the current production model), and security scans.
- 7
Build and push the image
Docker builds a serving image containing the exact model version and dependencies, tags it with the model version and git SHA, and pushes it to a container registry (ECR/GAR).
- 8
GitOps deploy with ArgoCD
The pipeline updates image tags in a Git repo; ArgoCD detects the change and syncs it to Kubernetes. Rollouts are canary or blue-green — the new version gets 5% of traffic first.
- 9
Serve behind NGINX
KServe (or vLLM for LLM models, FastAPI for custom logic) serves predictions behind an NGINX ingress on :443 with TLS. Horizontal Pod Autoscaler scales replicas on QPS and GPU utilization.
- 10
Monitor everything
Prometheus scrapes latency, throughput, and error metrics every 15s; Grafana dashboards and alert rules watch SLOs. Evidently compares live feature and prediction distributions against the training baseline to detect drift.
- 11
Close the feedback loop
When drift or accuracy decay crosses a threshold, an alert triggers the Airflow retraining DAG — the loop starts again at step 1. Ground-truth labels (e.g. confirmed fraud) flow back into the data sources.
Interview questions you'll be asked
Why do we need a feature store instead of computing features in the serving app?▼
Three reasons: (1) it eliminates training/serving skew because the same definitions produce both offline and online features, (2) point-in-time-correct joins prevent label leakage during training, and (3) features become reusable and discoverable across teams instead of re-implemented per project.
How do you safely deploy a new model version?▼
Registry stages plus GitOps: promote the model to 'staging', let CI run quality gates against the production baseline, then ArgoCD rolls out as a canary (5% traffic) with automatic rollback if latency or error-rate SLOs breach. Rollback is a Git revert — the previous image tag is still in the registry.
What's the difference between data drift and concept drift?▼
Data drift means the input feature distribution changed (P(X) shifted — e.g. a new customer segment). Concept drift means the relationship between inputs and labels changed (P(Y|X) shifted — e.g. fraudsters adapted their tactics). You detect the first with distribution tests (PSI, KS) on features, the second only with delayed ground-truth accuracy monitoring.
Batch vs streaming features — when do you need which?▼
Batch (Spark/Airflow) is fine when features change slowly and freshness of hours is acceptable — cheaper and simpler. Streaming (Kafka + Flink/Spark Streaming) is needed when a feature must reflect the last few seconds, like 'transactions in the last 5 minutes' for fraud scoring. Most production systems are a hybrid.
Trade-offs / what can go wrong
- ⚠Airflow + Spark + Feast + MLflow + KServe is a lot of platform for a team shipping one model. Start with a single pipeline and add components only when the pain is real.
- ⚠Online feature stores add operational cost and a consistency problem: Redis and the offline store can diverge if materialization fails silently.
- ⚠GPU nodes are expensive and slow to provision; without autoscaling and spot instances, training clusters burn budget idle.
- ⚠Automated retraining loops can amplify bias or learn from poisoned feedback if nobody validates the new training data.
- ⚠Canary analysis is only as good as your metrics — a model can pass latency SLOs while quietly making worse predictions.
LLMOps / RAG Production Architecture
This is how enterprises ship a GenAI knowledge assistant over their own documents: an offline indexing pipeline chunks and embeds content into a vector database, and an online path retrieves the most relevant chunks, reranks them, and sends them with a versioned prompt through an LLM gateway to a foundation model. Guardrails filter inputs and outputs, and every request is traced for cost, latency, and quality. User feedback feeds a fine-tuning loop so the system improves instead of stagnating. This is the single most common system-design interview topic for GenAI roles right now.
LangChainLILlamaIndexOAOpenAI
HuggingFacepgvectorPCPineconeWVWeaviateLGLiteLLM
Anthropic
GeminiNGNeMo
FastAPI
Next.js
OpenTelemetryLFLangfuseGrafanaNVIDIAScroll horizontally to view the full diagram.
How data flows, step by step
- 1
Chunk the documents
Ingestion jobs pull PDFs, wiki pages, and tickets, then split them into semantically coherent chunks (typically 300–800 tokens with 10–15% overlap) using LangChain or LlamaIndex. Each chunk keeps metadata: source, author, timestamp, access-control tags.
- 2
Embed every chunk
Each chunk is converted to a dense vector by an embedding model — OpenAI text-embedding-3 for managed simplicity, or an open HuggingFace model (e.g. BGE, E5) when data can't leave the VPC.
- 3
Index into the vector database
Vectors are upserted with their metadata into an ANN index (HNSW) — pgvector for teams already on PostgreSQL, Pinecone or Weaviate for managed scale. Metadata indexes enable pre-filtering by tenant or ACL before similarity search.
- 4
User query arrives
The Next.js chat UI sends the user's question to the FastAPI backend over HTTPS. Authentication and per-user authorization happen here — the app attaches the user's tenant/ACL context to the request.
- 5
Embed the query and retrieve
The retriever embeds the question with the same embedding model (mismatched embedding models between index and query are a classic production bug) and issues a hybrid search: vector similarity plus BM25 keyword match.
- 6
Vector DB returns top-k
The ANN index returns the k nearest chunks (k ≈ 20–50) filtered by the user's ACL tags. Latency budget for this hop is typically under 50 ms with HNSW.
- 7
Rerank for precision
A cross-encoder reranker scores each (query, chunk) pair jointly — much more accurate than bi-encoder similarity but slower, which is why it only runs on the top-k, not the whole corpus. The top 3–8 chunks survive.
- 8
Assemble the prompt and call the gateway
The surviving chunks are injected into a versioned prompt template from the prompt registry (LangSmith) — versioned so you can roll back prompt changes like code. The request goes to the LiteLLM gateway, not directly to a provider.
- 9
Gateway routes to an LLM
LiteLLM gives you one OpenAI-compatible API over GPT, Claude, and Gemini: automatic retries, fallbacks when a provider is down, per-team API-key budgets, and request/response logging. Model choice becomes config, not code.
- 10
Guardrails, then stream the answer
The response passes NeMo Guardrails output rails (factuality check against retrieved context, PII redaction, topic boundaries) and is streamed back to the UI token-by-token over SSE. Input rails also screened the original question for prompt injection.
- 11
Trace and measure everything
Every request emits OpenTelemetry spans — retrieval latency, token counts, model cost — collected into Langfuse traces and Grafana dashboards tracking cost per query and p95 latency SLOs.
- 12
Feed back and fine-tune
Thumbs-down answers and corrected responses are curated into a dataset. A PEFT/LoRA fine-tune on GPUs adapts an open model to your domain tone and edge cases; the adapter is versioned in a registry and served through the same gateway.
Interview questions you'll be asked
How do you choose a chunking strategy?▼
Start with recursive character splitting at 300–800 tokens with 10–15% overlap, then measure retrieval quality on a golden Q&A set. Chunk too big and you dilute relevance and blow the context window; too small and you lose the context the LLM needs to answer. Structure-aware splitting (by headings/tables) usually beats naive fixed-size splits for enterprise docs.
RAG vs fine-tuning — when do you use which?▼
RAG injects knowledge — use it when answers depend on large, frequently changing document sets, and when you need citations and access control. Fine-tuning changes behavior — use it for tone, format, and domain reasoning patterns. Production systems usually do both: RAG for facts, a LoRA adapter for style and task behavior.
How do you evaluate a RAG system?▼
Build a golden dataset of question/answer pairs from real users, then measure three RAGAS-style metrics: context recall (did retrieval find the needed chunks), faithfulness (is the answer grounded in the context, no hallucination), and answer relevance. Run evals in CI on every prompt, chunking, or model change — never ship on vibes.
Why put a gateway (LiteLLM) between the app and the LLM providers?▼
It decouples your code from any single provider: one API shape, automatic failover when OpenAI is down, per-team spend limits, central logging of every prompt and response, and the ability to A/B models or route cheap queries to cheap models. Without it, provider migration is a rewrite.
Trade-offs / what can go wrong
- ⚠The index goes stale: documents change daily, but embeddings are only as fresh as your last ingestion run. Incremental indexing pipelines are harder than the demo version.
- ⚠Switching embedding models means re-embedding the entire corpus — vectors from different models are not comparable. Plan for dual-index migrations.
- ⚠Reranking adds 50–200 ms per query; on tight latency budgets you either shrink k, use a smaller cross-encoder, or skip it for easy queries.
- ⚠Retrieved context is untrusted input: a poisoned document can prompt-inject the LLM ('ignore previous instructions'). Guardrails and content filtering are not optional in enterprise deployments.
- ⚠Cost surprises: long contexts × many users × premium models = five-figure monthly bills. Token budgets and caching (semantic cache for repeat questions) matter.
Agentic AI Architecture
This is a production single-agent system — for example an SRE copilot that investigates incidents or a support agent that resolves tickets end to end. A LangGraph orchestrator drives a planner LLM through an Observe → Plan → Act → Reflect loop, with Redis and a vector store as memory, MCP servers as the standardized tool layer, a human approval gate for risky actions, and a full audit trail. The difference between a demo agent and this diagram is exactly what interviewers probe: state, safety, and observability.
LangGraphOAOpenAIClaudeRedisPostgreSQL
GitHub MCPSLSlack
JiraK8s APILFLangfuseScroll horizontally to view the full diagram.
How data flows, step by step
- 1
Task enters the orchestrator
A user message, webhook (e.g. a PagerDuty alert), or cron trigger creates a new run in the LangGraph state graph. State is checkpointed after every node, so a crashed agent resumes instead of restarting.
- 2
Orchestrator invokes the planner
The planner/reasoner node calls the LLM (GPT-4-class or Claude) with a system prompt, the current state, and JSON schemas of every available tool. The model's tool-calling interface is the contract between reasoning and execution.
- 3
Memory read/write
Short-term memory (Redis) holds the current session's working context — conversation, intermediate results — with TTLs. Long-term memory (a vector DB) stores distilled knowledge from past incidents: 'last time this error appeared, the fix was X'.
- 4
The agent loop runs
Observe (read state and tool results) → Plan (decide the next action) → Act (emit a tool call) → Reflect (did it work? update the plan). The loop continues until the task is done, the model declares completion, or a hard max-iteration / token budget is hit.
- 5
Tool calls go through MCP servers
Every external capability — GitHub, Slack, Jira, the Kubernetes API — is exposed as an MCP (Model Context Protocol) server speaking JSON-RPC. This standardizes auth, schemas, and rate limits, and lets you add tools without touching agent code.
- 6
Risky actions hit the approval gate
Actions are classified by risk. Reads (logs, status) execute autonomously; writes with blast radius (restart a deployment, merge a PR, close a ticket) pause the graph and request human approval in Slack or a UI. The graph resumes from its checkpoint on approval.
- 7
Approved actions execute
The execution layer performs the action with the agent's own least-privilege service account — kubectl rollout, git merge, API call. Results flow back into the loop as the next observation.
- 8
Everything is written to the audit log
An append-only PostgreSQL audit log records who approved what, which tool ran with which arguments, and what changed. In regulated environments this log is not optional — it is the compliance story.
- 9
Full traces to Langfuse
Every LLM call and tool invocation is traced with token counts, latency, and cost. When an agent misbehaves in production, the trace is how you replay and debug its exact reasoning trajectory.
Interview questions you'll be asked
Explain the ReAct / agent loop.▼
The agent alternates reasoning and acting: it observes the current state, the LLM plans the next step and emits a structured tool call, the tool executes and returns an observation, and the model reflects on the result before the next iteration. LangGraph makes this an explicit state graph with checkpoints instead of a while-loop in a notebook — that's what makes it resumable and inspectable.
How do you stop an agent from going rogue or looping forever?▼
Layered defenses: hard max-iteration and token/cost budgets per run, an allowlist of tools with least-privilege credentials, risk classification with human approval for writes, rate limits on the MCP servers, and a kill switch that aborts the graph. You assume the model will eventually be wrong and design the blast radius accordingly.
What is MCP and why not just call APIs directly?▼
Model Context Protocol is an open standard for exposing tools and data to LLMs over JSON-RPC. Direct API calls hardcode auth, schemas, and retries into agent code; MCP servers encapsulate all that behind a uniform interface, so the same agent can use community-built servers (GitHub, Slack, Jira) and swapping a tool implementation doesn't touch the agent.
How do you evaluate an agent before putting it in production?▼
Build task-level evals: a suite of realistic scenarios (e.g. 50 historical incidents) with known-good outcomes. Measure task success rate, steps/tokens per task, and unsafe-action attempts. Then shadow-mode the agent in production — it plans but doesn't execute — and compare its decisions against what humans actually did.
Trade-offs / what can go wrong
- ⚠Agents are non-deterministic: the same input can take different trajectories. Checkpointed state helps you replay, but tests can't guarantee behavior the way they do for deterministic services.
- ⚠Errors compound across the loop — a bad observation at step 2 poisons every plan after it. Reflection steps and validation tools mitigate but don't eliminate this.
- ⚠Cost and latency scale with loop iterations; a single task can burn tens of thousands of tokens. Budgets and smaller models for simple steps keep it sane.
- ⚠Tool outputs are untrusted input: a malicious Jira ticket can prompt-inject the agent. Sanitize tool results and never let tool text override the system prompt.
- ⚠HITL gates protect you but cap throughput — tune which actions truly need approval or your on-call engineer becomes the bottleneck.
Multi-Agent System Architecture
This is a crew of specialized agents working like a software-delivery team: a supervisor decomposes an incoming feature request and routes subtasks over a message bus to a PM agent, an architect, parallel coder agents, a reviewer/QA agent that aggregates their work, and a DevOps agent that deploys. Agents coordinate through a shared blackboard memory instead of brittle point-to-point calls, and a policy engine plus human checkpoints govern anything that touches production. Frameworks like CrewAI, AutoGen, and LangGraph implement this pattern; the interview questions are about coordination, failure modes, and when it's overkill.
CrewAIAGAutoGenLangGraphRedisKafkaPostgreSQLOPOPAOAOpenAIClaudeHuggingFaceGitHubJiraArgoCDDockerOTelGrafanaScroll horizontally to view the full diagram.
How data flows, step by step
- 1
Task arrives at the supervisor
A feature request or ticket lands at the supervisor/orchestrator (CrewAI, AutoGen, or a LangGraph graph). The supervisor — itself an LLM agent with routing tools — decomposes it into subtasks with owners, dependencies, and done-criteria.
- 2
Subtasks published to the message bus
Each subtask is published to a topic on the shared bus (Redis Streams for small deployments, Kafka when you need replay and retention). The bus decouples agents: producers and consumers never call each other directly.
- 3
Workers subscribe and claim work
Each specialized agent subscribes to its topic and claims tasks. Every worker has its own LLM (matched to the job — cheap models for triage, frontier models for coding) and its own tool set via MCP.
- 4
PM agent hands off to the architect
The PM agent turns the request into a spec with acceptance criteria (writing to Jira via its tools). When the spec lands on the bus, the architect agent picks it up and produces a technical design and task breakdown, reading the repo with read-only tools.
- 5
Coder agents work in parallel
Two (or more) coder agents implement independent tasks simultaneously on separate git branches. Parallelism is the real speedup of multi-agent systems — but only works because tasks were decomposed with clean boundaries.
- 6
Reviewer/QA aggregates and validates
The reviewer/QA agent is the aggregator: it collects the branches, runs the test suite and static analysis via its tools, reviews diffs, and either requests changes (loop back to coders) or signs off. Nothing merges without this gate.
- 7
DevOps agent deploys
On QA pass, the DevOps deployer agent packages images with Docker and updates the GitOps repo; ArgoCD syncs to Kubernetes. The deployer has no production credentials of its own beyond what policy allows.
- 8
Shared blackboard memory
All agents read and write shared state — specs, designs, task status, decisions — in a PostgreSQL blackboard with pgvector for semantic search over past work. Optimistic versioning prevents two agents from silently overwriting each other.
- 9
Policy engine and HITL gate
Before any deploy or merge, the action is evaluated by an OPA (Open Policy Agent) policy — e.g. 'deploys to prod only from reviewed branches, only during business hours' — and high-impact actions pause for a human checkpoint.
- 10
End-to-end observability
OpenTelemetry spans propagate across every bus message, so a single trace shows the full task lifecycle: supervisor decision → agent runs → tool calls → deploy. Grafana dashboards track per-agent cost, success rate, and loop counts.
Interview questions you'll be asked
Why a message bus instead of agents calling each other directly?▼
Decoupling and replay. Direct calls create an N² web of dependencies where one slow agent blocks the rest and failures cascade. A bus gives you async handoffs, durable queues when an agent crashes, the ability to replay events for debugging, and easy addition of new agent types without changing existing ones.
How do you prevent infinite loops between agents?▼
Three mechanisms: per-task turn/iteration budgets enforced by the supervisor, explicit done-criteria in every subtask contract, and supervisor arbitration — the reviewer can't bounce work back to a coder more than N times before the supervisor escalates to a human. Without these, two agents can politely disagree forever while burning tokens.
How do agents avoid conflicting writes to shared state?▼
The blackboard uses optimistic concurrency: every record has a version, writes are compare-and-swap, and losers re-read and retry. For truly contentious resources (the same file, the same ticket), the supervisor serializes access by assigning a single owner. Never let agents hold long-lived locks.
When should you NOT use a multi-agent architecture?▼
When a single agent with good tools can do the job — most tasks. Multi-agent pays off when subtasks genuinely parallelize, need different tools/permissions, or benefit from adversarial separation (coder vs reviewer). Otherwise you're multiplying cost, latency, and debugging difficulty for no capability gain.
Trade-offs / what can go wrong
- ⚠Coordination overhead is real: decomposing, routing, and aggregating can cost more tokens than the actual work on small tasks.
- ⚠Cost multiplies by agent count — a 6-agent crew with frontier models on every seat gets expensive fast. Match model size to role.
- ⚠Debugging emergent behavior is hard: a wrong output may come from the supervisor's decomposition, a worker's reasoning, or a stale blackboard entry. Distributed tracing is the only way to stay sane.
- ⚠The bus is a single point of failure and a security boundary — topic-level ACLs matter, because any agent that can publish to 'deploy' can trigger the pipeline.
- ⚠Adversarial pairs (coder/reviewer) only help if the reviewer is genuinely independent — same model, same prompt style often means same blind spots.
All private courses for ₹1,199/month
Skip ₹30K–₹40K upfront. Join AI & ML AI Agentic Pro and unlock the full library — old sessions, new uploads, members-only live classes — plus 1:1 career help when you share your Member ID.
⭐ Recommended · Agentic Pro
AI & ML AI Agentic Pro
The complete library + mentorship access
≈ ₹40/day for the full library + mentorship
- ALL members-only videos — private courses, past sessions & new uploads
- Members-only live streams & class replays
- Full archive: old videos + every upcoming release
- Includes every perk from Learner & Practitioner tiers
- Share your YouTube Member ID on WhatsApp → 1:1 mentorship from Rajinikanth
- Interview tips, resume feedback & career Q&A from the team
Why not ₹179 or ₹419?
Lower tiers = partial access only
Students who pick the cheaper plans often come back asking for the full library. Agentic Pro is the only tier with every members-only video, live stream replay, and direct mentorship from Rajinikanth's team.
3 steps to unlock mentorship
Takes under 2 minutes
- 1Join Agentic Pro on YouTube
- 2Copy your Member ID from settings
- 3WhatsApp it to us — get learning path, interview tips & Q&A help
Billed monthly via YouTube. Want live cohort + capstones? See the MLOps Masterclass. Free public videos on YouTube.
Learn to Build These Architectures Yourself
These four systems are exactly what you build in the live MLOps + LLMOps + AIOps + AI Agents masterclass — hands-on, on real cloud infrastructure, with 1-on-1 mentorship.