AI Makerspace LLM Ops Course Charts the Production AI Toolkit

One of the clearest signals of where an engineering discipline is heading is what practitioners are being taught to build. Few artifacts illustrate this as well as LLM Ops – Large Language Models in Production, the Maven course taught by Dr. Greg Loughnane and Chris "The LLM Wizard" Alexiuk of AI Makerspace. Launched in late 2023 as a four-week live cohort, the course set out to take experienced developers, data scientists, and ML engineers from prototype to production LLM applications — and its curriculum, now retired and folded into a broader AI Engineering Bootcamp, reads today like a fossil record of how quickly the field has moved128.

What the Course Actually Taught

The syllabus was organized around a progression that has since become the default arc for anyone shipping language-model products. Week one covered LLM Ops as a discipline distinct from classical MLOps and ML engineering, prompt iteration and optimization, and a first application built in pure Python. From there it moved through Retrieval Augmented Generation (RAG), the LangChain and LlamaIndex frameworks, caching and versioning of prompts and indexes, evaluation of RAG systems with RAGAS, reranking and advanced retrieval, deployment of open-source models to production endpoints, and finally agents built on Reasoning-Action (ReAct) logic, with LangSmith invoked as an early window into the emerging LLM Ops tooling landscape1.

The framing was explicitly production-minded. Students were told that traditional data science and ML skills were being outpaced by AI, and that it was now possible to build in a single day what recently took months1. The course targeted people who already had ML fundamentals — the instructors themselves warned that it would likely be too advanced for beginners — and third-party roundups of LLMOps courses placed it in exactly that slot: a live, cohort-based option for practitioners who want to prototype and ship production LLM applications with an instructor and peer group rather than working through material alone8.

A distinctive part of the pedagogy was a hack week in which teams had to ship a RAG system containing at minimum caching, evaluation, and visibility components — a compressed rehearsal of the three things production LLM teams spend most of their time arguing about1.

Why This Curriculum Mattered

What makes the course worth studying now is how closely its outline matches the concerns that dominate production AI engineering in 2026. The syllabus's four load-bearing topics — evaluation pipelines, prompt management, monitoring and observability, and agentic infrastructure — have each since ballooned into entire vendor categories and conference tracks.

Evaluation pipelines are the clearest example. The course taught RAGAS for RAG assessment and framed evaluation as a step in improving an application1. Current industry guidance goes well beyond that: pre-deployment testing is now seen as insufficient because models face novel inputs, provider-side model updates, and upstream data changes that degrade quality without any infrastructure-level signal. The 2026 consensus is continuous, in-production evaluation — LLM-as-judge scoring run on a sampled slice of live traffic, typically 10 to 20 percent, with scores written back as trace attributes so quality data can be queried alongside latency and token counts1011.

Prompt management, which the course compressed into versioning, caching, and prompt engineering for retrieval1, has likewise matured. Observability platforms now track prompts as versioned, monitored artifacts with drift detection at the prompt and use-case level, so that degradation in one workflow isn't hidden by aggregate stability12. One alumnus of the AI Makerspace ecosystem credits the course's "build, ship, and share" ethos directly for PromptLab and OpenEvals, open tooling for prompt experimentation and evaluation — evidence that the curriculum seeded not just engineers but tool builders2.

The Observability Layer Grew Up

Perhaps the biggest divergence between the course's moment and now is in monitoring and observability. In the original syllabus, visibility was a hack-week requirement and LangSmith represented "the evolving landscape of LLM Ops" — a single line item1. Today that landscape is crowded and sharply stratified, and the coverage is instructive.

The general trajectory across 2026 reporting is convergence on three ideas. First, OpenTelemetry-based, vendor-neutral telemetry — including GenAI semantic conventions for token counts and model identifiers — has become the substrate for tracing LLM calls, retrieval steps, and tool invocations910. Second, LLM observability is treated as a mandatory layer of the production stack rather than an add-on, tracking hallucination rates, semantic drift, token cost per transaction, and RAG pipeline quality9. Third, and most contentiously, the market has split between tracing-first platforms that bolt on scoring and evaluation-first platforms that treat quality as the primary signal, with curated production traces flowing back into regression test sets1213.

The vendor landscape the course barely gestured at now includes Langfuse, LangSmith, Arize and Phoenix, LangWatch, Helicone, Braintrust, Datadog's LLM monitoring, Comet Opik, and Confident AI, each staking out different positions on the spectrum from gateway-level cost tracking to cross-functional quality review workflows1315. Survey coverage of these tools frequently disagrees with itself — the same platforms are ranked first in one roundup and mid-table in another, with the differences turning on whether the reviewer prizes evaluation depth, agent-trace tooling, or enterprise APM integration121314 — but the disagreement itself confirms that the category is still unsettled and that quality, not latency, is where the differentiation fight is happening.

Agentic Systems Changed the Requirements

The course ended with agents: ReAct loops, agent types, and a comparison of LangChain versus LlamaIndex agent implementations1. That was a reasonable final week in 2023. It is now arguably the load-bearing wall of the entire discipline, because agents broke the assumptions classical observability was built on.

Traditional APM tracked deterministic HTTP paths. A single agentic request today can involve five LLM calls, three tool invocations, two vector lookups, and an inter-agent handoff — each a potential failure point, each invisible to standard monitoring11. The response has been a new set of signal types: intermediate reasoning steps, tool-selection outcomes, handoff traces, per-trace hallucination and faithfulness scoring, replayable execution graphs, and the ability to fork a session from an intermediate step to debug it1114. Security observability — prompt-injection detection, PII scanning, output validation — has shifted from nice-to-have to production requirement, with OWASP's prompt injection vulnerability flagged as the biggest 2026 risk alongside unbounded agent fan-out cost1116.

Even the AI Engineering Bootcamp that absorbed the LLM Ops course reflects this shift: its current syllabus foregrounds agentic RAG, multi-agent applications, agent memory, context engineering across many turns and agents, and "how do we evaluate agents, exactly, and do error analysis, from goals to tools to traces" — a question the original course never had to answer2. One alumnus of the broader program describes the most valuable habit as "reading traces instead of trusting scores," after discovering that a proxy metric had flagged correct answers as failures3. That sentence could serve as an epigraph for the entire field's last three years.

The Reading That Holds Up

The most defensible synthesis across the sources is this: Loughnane and Alexiuk's course was early and directionally right. It identified the correct four pillars — evaluation, prompts, observability, agents — before any of them had mature tooling, and it insisted on shipping rather than lecturing, with a hack week that demanded caching, evaluation, and visibility as non-negotiable components of a RAG system1. Where it has aged is in scope, not in instinct: evaluation has moved from offline RAGAS scoring to continuous online judging on production traffic1011; observability has moved from a single vendor mention to a contested, OpenTelemetry-anchored category913; and agents have moved from a capstone topic to the central problem that defines what LLMOps even means in 202611.

The course itself is now closed to enrollment, its first cohort open-sourced and its lineage continued through the bootcamp successor12. That a four-week class built around LangChain, RAGAS, and ReAct agents is now a historical document is not a criticism — it is the fastest way to measure how far production AI engineering has traveled in under three years.