Machine Learning: Evaluation, Debugging, Inference and Serving, MLOps, and Responsible ML¶
Part 3 of 4 of the Machine Learning reference (plugin
machine-learning), covering §9–§14. Sibling skills:ml-framing-data-and-classical(§0–§3),ml-deep-learning-and-training(§4–§8),ml-reference(§15–§20). Section numbers are shared across the set; a reference written as §N →skillpoints into that sibling skill.Currency: Verified August 2026. See §17 →
ml-referencefor the currency snapshot and what goes stale first.How to read this. Reference, not tutorial. Sections are independent. Three markers: - [DURABLE] — statistics, optimization theory, or a lesson the field has relearned every few years. Does not expire. - [VERSIONED] — framework versions, hardware, library APIs. Moving fast; verify. - [CONTESTED] — practitioners genuinely disagree.
⚠️ GOTCHA boxes mark the mistakes that produce a model that looks great offline and fails in production — which is the characteristic failure of this field.
The three framings that organize everything below: 1. Your data is the model. Architecture choices matter far less than data quality, quantity, and whether your evaluation measures the thing you care about. Most time on a real project goes to data, and most catastrophic failures originate there (§2 →
ml-framing-data-and-classical). 2. The default failure mode is a number that lies. Leakage, distribution shift, a metric that doesn't match the objective, a test set you tuned against. A model that looks too good almost always is — and finding out why is the core skill. 3. Almost everything you'll build should not be a neural network. For tabular data — which is most business data — gradient-boosted trees remain the strongest default (§3.1 →ml-framing-data-and-classical). Reach for deep learning when you have perceptual data, sequences, very large datasets, or a pretrained model to adapt.
§9. Evaluation¶
[DURABLE] This section is where most projects actually fail, and it fails quietly.
9.1 Metrics¶
⚠️ Accuracy is almost always the wrong metric. On a 99:1 imbalanced problem, predicting the majority class gets 99% accuracy and is useless.
| Task | Metrics |
|---|---|
| Binary classification | Precision, recall, F1, PR-AUC (better than ROC-AUC under heavy imbalance), ROC-AUC, MCC, and the confusion matrix — always look at it |
| Multi-class | Macro (treats classes equally) vs. micro (weights by frequency) vs. weighted — say which you used; they disagree substantially |
| Regression | MAE (robust), RMSE (penalizes large errors), MAPE (⚠️ breaks near zero, asymmetric), R² |
| Ranking / retrieval | NDCG, MRR, MAP, recall@k, precision@k |
| Probabilistic | Log loss, Brier score, calibration curves — use these when the probability matters |
| Generative / LLM | Task-specific benchmarks, human eval, LLM-as-judge (⚠️ §9.3), pairwise preference |
[DURABLE] Choose the metric from the decision it informs. If false negatives cost 100× false positives, optimize for that, and tune the decision threshold explicitly — it's a free parameter most people leave at 0.5 for no reason.
9.2 Doing it honestly¶
- One test-set evaluation. Every additional look is leakage (§2.1 →
ml-framing-data-and-classical). - Report variance, not a point estimate. Multiple seeds, confidence intervals, bootstrapping. ⚠️ A great deal of reported "improvement" in ML is within seed noise, and the field's reproducibility problems are substantially this.
- Compare against a strong baseline, tuned as carefully as your method. An untuned baseline is not a comparison.
- Slice your metrics. Aggregate performance hides systematic failure on subgroups — by user segment, geography, device, class, time period. This is both an accuracy practice and a fairness practice (§14.2).
- Evaluate on distribution shift you expect in production: a future time period, a new region, a new source.
- Look at the errors. Sample 50 mistakes and read them. Consistently the highest information-per-minute activity in applied ML, and consistently skipped.
9.3 Evaluating generative models¶
Hard, and getting harder. Benchmark contamination is pervasive — test sets leak into pretraining corpora, so public benchmark scores are systematically optimistic. LLM-as-judge is practical and scalable but carries position bias, verbosity bias, and self-preference bias; mitigate by randomizing order, using multiple judges, and calibrating against human labels on a subset. Build a private eval set from your actual task, hold it out, and version it — this beats any public benchmark for deciding whether your system works.
§10. Debugging Training¶
[DURABLE] A systematic ladder, in order. Skipping steps is what makes debugging take weeks.
- Overfit a single batch. Train on 2–10 examples until loss ≈ 0. If you can't, you have a bug, not a learning-rate problem. This is the single most valuable diagnostic in deep learning and takes two minutes.
- Check shapes and the data. Print them. Visualize an actual batch after
augmentation. ⚠️ Silent broadcasting bugs are endemic — a
(B,1)vs.(B,)mismatch produces a(B,B)tensor and a loss that trains to something meaningless. - Verify the loss at initialization. Cross-entropy on k balanced classes should start
near
ln(k). If it doesn't, your labels, logits, or reduction are wrong. - Check the label pipeline end-to-end. Off-by-one, shuffled labels, wrong mapping.
- Watch gradient norms. Zero → disconnected graph or dead activations. Exploding → clip, lower the LR, check normalization.
- Sweep the learning rate over orders of magnitude before touching anything else.
- Compare train vs. validation curves: both high = underfitting (bigger model, train longer, better features); train low + val high = overfitting (more data, more regularization, augmentation); val better than train = a bug, or dropout/BN train-mode artifacts.
- Turn off tricks — augmentation, scheduler, mixed precision,
torch.compile— and reintroduce them one at a time.
| Symptom | Likely cause |
|---|---|
| Loss is NaN | LR too high; log(0)/div-by-zero; fp16 overflow (use bf16); bad input data |
| Loss flat from step 0 | LR too low; frozen params; disconnected graph; optimizer.zero_grad() misplaced |
| Loss decreases then explodes | LR too high; missing gradient clipping; a bad batch |
| Great train, terrible val | Overfitting, or leakage in reverse (val is harder/different) |
| Great val, terrible production | Distribution shift, or leakage (§2.1 → ml-framing-data-and-classical). Check the leakage list first |
| Works in eager, breaks compiled | Graph breaks, dynamic shapes, or a numerics difference |
| Non-deterministic across runs | Expected — see §12.1. Report variance |
| Slower than expected on GPU | Dataloader-bound (check GPU utilization), small batches, unnecessary syncs, .item() in the loop |
§11. Inference and Serving¶
11.1 The landscape¶
[VERSIONED — this layer moved fast and consolidated in 2025–2026.]
| Engine | Best for |
|---|---|
| vLLM | The practical default. Broadest hardware support (NVIDIA CUDA, AMD ROCm, Intel XPU, Google TPU, and more), one pip install, OpenAI-compatible server immediately |
| SGLang | Long shared prefixes — RAG over a fixed corpus, multi-turn agents, structured decoding. RadixAttention turns prefix overlap into automatic cache hits. Reported ~29% higher throughput than vLLM on H100 in one comparison, with much larger gains on prefix-heavy workloads |
| TensorRT-LLM | Committed to NVIDIA's newest silicon and want the deepest kernel-level optimization — paid for in build complexity |
| llama.cpp / Ollama / MLX | Single-user local inference, CPU and Apple Silicon |
| Triton Inference Server / Ray Serve | General model serving, multi-model, non-LLM |
⚠️ [VERSIONED] Hugging Face TGI moved to maintenance mode (announced December 2025, repository archived read-only 21 March 2026), redirecting users to vLLM, SGLang, llama.cpp, and MLX. If you're on TGI, plan a migration.
[DURABLE] The feature set has converged. Any modern engine gives you continuous (in-flight) batching — new requests join the running batch the moment a slot frees, which is the single biggest throughput unlock over naïve batching — and a paged KV cache (virtual-memory management for the attention K/V store). Choose on hardware breadth, workload shape, and operational complexity, not on feature checklists.
11.2 What actually determines inference cost¶
[DURABLE] LLM inference has two distinct phases with completely different bottlenecks: - Prefill (processing the prompt) — compute-bound, parallel across tokens. - Decode (generating tokens) — memory-bandwidth-bound, one token at a time.
This asymmetry drives everything: it's why disaggregated prefill/decode (routing each phase to different hardware) is a major 2026 architecture, why memory bandwidth matters more than FLOPs for serving, and why batch size helps decode enormously.
The KV cache is usually your real memory constraint, growing linearly with batch size and sequence length. GQA/MQA, paged attention, prefix caching, and KV cache quantization all exist to attack it.
Other levers: quantization (§11.3), speculative decoding (a small draft model proposes, the big model verifies — real latency wins), prefix caching, and structured output constraints.
11.3 Quantization¶
| Format | Use |
|---|---|
| INT8 / W8A8 | Well-established, minimal quality loss |
| INT4 / W4A16 (GPTQ, AWQ) | The common serving choice. ~4× memory reduction, small quality cost |
| FP8 | Native hardware support on Hopper/Blackwell; good quality retention |
| NVFP4 / MXFP4 | 4-bit with Blackwell hardware support; increasingly used |
| GGUF (k-quants) | The llama.cpp ecosystem, CPU and Apple Silicon |
[DURABLE] Quantization-aware training beats post-training quantization on quality and costs more. For most people PTQ with a good calibration set is sufficient. ⚠️ Always evaluate the quantized model on your own task — published "negligible degradation" claims are measured on benchmarks that may have nothing to do with your use case, and degradation is often concentrated in exactly the hard cases you care about.
11.4 The economics¶
[VERSIONED, and the trend is the point]: inference now accounts for roughly two-thirds of all AI compute, up from about one-third in 2023, and cost-per-token has fallen by orders of magnitude — one analysis puts GPT-4-equivalent performance at roughly $0.40 per million tokens in early 2026 versus ~$20 in late 2022. Serving-engine improvements are a large part of it: GPU utilization went from ~30–40% to ~70–80% through continuous batching, paged attention, and speculative decoding.
[DURABLE] The practical consequence: cost-per-token is the KPI, and a 10× inference cost reduction directly enables 10× the users at the same budget. Self-hosting reaches cost parity with commercial APIs at moderate volume — one 2026 study found parity within 1–4 months at ~30M tokens/day — but only if you actually keep the hardware busy.
§12. MLOps and Reproducibility¶
12.1 Reproducibility¶
[DURABLE] Bit-exact reproducibility on GPU is achievable but costly, and most teams
should aim for statistical reproducibility instead: seed everything (Python, NumPy,
framework, dataloader workers), pin all versions and record them, version data and code
together, log the full config, and report variance across seeds rather than a single
run. Full determinism requires deterministic algorithms (torch.use_deterministic_algorithms),
disabling cuDNN benchmarking, and fixed dataloader ordering — and it will slow you down.
Version the data. Model artifacts without the exact training data are not reproducible, and "we retrained and got different numbers" is otherwise unresolvable.
12.2 The pipeline¶
data ingestion → validation → feature engineering → training → EVALUATION GATE
→ registry → deployment (shadow → canary → full) → MONITORING → retraining
Feature stores solve a real problem — training/serving skew, where the features computed at training time differ subtly from those computed at inference. ⚠️ This is one of the most common and hardest-to-find production bugs, and if you compute features in two different codepaths you will eventually have it.
12.3 Monitoring¶
Monitor, in roughly this priority order: prediction distribution (drifts first and cheapest to watch), input feature distributions (PSI, KL divergence), actual performance where labels eventually arrive, latency and throughput, error rates, and business metrics (the only ones that ultimately matter).
[DURABLE] Models degrade. Plan for retraining from day one — trigger it on a schedule, on drift detection, or on performance drop. And keep the ability to roll back to the previous model instantly; a bad model deploy is an outage.
§13. Hardware and Cost¶
[VERSIONED]
| Tier | Notes |
|---|---|
| H100 (80 GB) | ⚠️ The software ecosystem is most mature here — vLLM, SGLang, TensorRT-LLM, PyTorch, JAX. Any optimization technique you read about was probably benchmarked on H100 |
| H200 (141 GB) | Same die as H100 with HBM3e: 141 GB and 4.8 TB/s (+43% bandwidth). Often better per-token cost than 2× H100 for FP16 70B |
| Blackwell (B200/GB200/B300) | 192 GB HBM3e — a 70B FP16 model with real KV headroom, or 405B FP8 on two chips. NVFP4 support. ~3× faster LLM training than the prior generation by NVIDIA's own MLPerf claims |
| Rubin / Vera | The announced next generation, aimed at agentic training and inference |
| RTX PRO 6000 Blackwell (96 GB) | Workstation-class; 4–8 in a rackmount is a standard 2026 self-hosting configuration |
| AMD MI300X / ROCm | Real and improving; NVIDIA remains the default for vLLM, Unsloth, and most production workflows |
| TPU v5e/v6, Trainium/Inferentia, Gaudi 3 | Viable, especially via JAX (TPU) or where you have committed cloud spend |
[DURABLE] For LLM inference, memory capacity and bandwidth dominate the selection decision, not FLOPs (§11.2). For training, interconnect (NVLink, InfiniBand) often matters more than per-chip performance once you're multi-node.
Cost discipline: cloud beats buying unless you keep hardware busy more than roughly 8 hours/day; spot/preemptible instances plus checkpointing are the single biggest training cost lever; profile before scaling up (most jobs are dataloader-bound or badly-batched, not compute-bound); and the cheapest optimization is a smaller model that meets the requirement.
§14. Interpretability, Fairness, Safety¶
14.1 Interpretability¶
Intrinsically interpretable models — linear models, small trees, GAMs — are underrated. [DURABLE] If interpretability is a hard requirement, use an interpretable model rather than explaining a black box.
Post-hoc methods: SHAP (game-theoretic, the practical standard; ⚠️ correlated features make attributions ambiguous), LIME (local surrogates; unstable), permutation importance (⚠️ misleading with correlated features), partial dependence and ICE, attention maps (⚠️ attention is not explanation — a well-known result; attention weights don't reliably indicate what drove the output), and counterfactuals (often the most useful form for an affected person).
⚠️ Feature importance is not causal. A high-importance feature tells you what the model uses, not what drives the outcome. Acting on it as if it were causal is a recurring and expensive error.
14.2 Fairness¶
[DURABLE] The mathematics constrains you: demographic parity, equalized odds, and calibration within groups are provably mutually incompatible except in degenerate cases. You must choose which fairness definition applies to your context — there is no technically neutral option, and pretending otherwise is itself a choice.
Practically: slice every metric by group (§9.2 — this is the same practice as good evaluation), audit training data for representation and historical bias, remember that removing a protected attribute doesn't remove its influence (proxies are everywhere), and document the model's intended use and limitations (model cards, datasheets).
14.3 Security and safety¶
Adversarial examples (small perturbations flip predictions — still largely unsolved), data poisoning, model extraction, membership inference and training data extraction (models memorize; verbatim regurgitation is real), and prompt injection for LLM systems (⚠️ not solved, and any system giving an LLM tools plus untrusted input has this exposure).
⚠️ pickle deserialization is arbitrary code execution. Use safetensors for model
weights. PyTorch's weights_only=True default since 2.6 addresses this, but third-party
checkpoints from unknown sources remain a real supply-chain risk.
Plugin: machine-learning · View SKILL.md on GitHub