35 views
# Prompt Engineering Syllabus <!-- toc:auto --> ## Contents *Twenty workshops. The .5 numbers are optional deepenings of the whole number before them — skip them on a first pass and the sequence still works.* **BEFORE YOU START** - [Prework Day 1](#Prework-Day-1) - [Prework Day 2](#Prework-Day-2) **PROMPTS THAT HOLD UP** - [Workshop 1 - Critical Thinking and Language Models](#Workshop-1---Critical-Thinking-and-Language-Models) - [Workshop 1.5 - Complex Prompts and Benchmarks](#Workshop-15---Complex-Prompts-and-Benchmarks) — optional deepening of Workshop 1 - [Workshop 2 - Jailbreaks](#Workshop-2---Jailbreaks) - [Workshop 2.5 - Watermarking and Detection](#Workshop-25---Watermarking-and-Detection) — optional deepening of Workshop 2 **GIVING THE MODEL HANDS** - [Workshop 3 - Tools](#Workshop-3---Tools) - [Workshop 3.5 - Model Context Protocol Servers](#Workshop-35---Model-Context-Protocol-Servers) — optional deepening of Workshop 3 - [Workshop 4 - Spec Driven Development](#Workshop-4---Spec-Driven-Development) - [Workshop 5 - Agentic SDLC](#Workshop-5---Agentic-SDLC) **WHAT THE MODEL KNOWS** - [Workshop 6 - Planning and Context](#Workshop-6---Planning-and-Context) - [Workshop 6.5 - Context Engineering & 12 Factor Agents](#Workshop-65---Context-Engineering-amp-12-Factor-Agents) — optional deepening of Workshop 6 - [Workshop 7 - Retrieval Augmented Generation](#Workshop-7---Retrieval-Augmented-Generation) - [Workshop 8 - Memory Patterns](#Workshop-8---Memory-Patterns) **MAKING IT BEHAVE** - [Workshop 8 - Alignment (Management & Strategy)](#Workshop-8---Alignment-Management-amp-Strategy) — also numbered 8; this is the strategy track, not Memory Patterns - [Workshop 8.5 - Advanced Alignment Techniques (Technical)](#Workshop-85---Advanced-Alignment-Techniques-Technical) — the technical counterpart - [Workshop 9 - Meta-languages & Meta Narratives](#Workshop-9---Meta-languages-amp-Meta-Narratives) **RECORDINGS** - [More recordings for this class](#More-recordings-for-this-class) · 8 recordings — from the wider archive <!-- /toc:auto --> Prerequisites: - [Using Large Language Models Syllabus](/4avFe5qDS_u632V6-oHoyA) - [Custom GPTs Course Syllabus](/merCbTE8TDKmySN1i_ooxA) **Recommended Prerequisites:** - A laptop or desktop computer with a web browser - Able to join a video call through our online campus (runs in any modern browser) - Completed [Using Large Language Models](/4avFe5qDS_u632V6-oHoyA) and [Custom GPTs](/merCbTE8TDKmySN1i_ooxA) (or equivalent experience with LLM prompting and custom GPTs) - You can read and lightly edit Python code - You can use a terminal or command line to run commands - **Accounts you'll need:** OpenAI API key (pay-as-you-go), ChatGPT Plus (paid, ~$20/month), Make.com (free tier) - **Helpful but not required:** You've used Google Colab or Jupyter notebooks In this course we will: * Develop a complex prompt and rubric for artifacts you commonly produce with AI * Develop a benchmark to test a prompt or prompt template * Jailbreak a large language model out of its alignment guidance * Automatically host a conversation between multiple personas using a large language model * Write code with a large language model * Use (code or) no-code tools to automate actions a large language model can take * Use (code or) no-code tools to trigger a large language model on a schedule * Use open source software to run open source large language models for free for capabilities research * Align a large language model's outputs to human values using prompt engineering techniques according to state-of-the-art research * Develop a metalanguage to allow a language model to reason at a symbolic level about a specific problem domain * Develop a metalanguage to allow for narrative design that can be held in available conversational context windows, allowing for the generation of novels and longer-form stories --- ## [Prework Day 1](/uW-rSUguTnW1fTAR2F53-g) - Install-a-thon and quickstart practice with the OpenAI library - Install LocalLM for those of you with the hardware - Google Colab Oobabooga? If we have time - Basic Python that'll be useful (read a file, read a directory of files, read a CSV) - Use open source software to run open source large language models for free for capabilities research ## [Prework Day 2](/0t2XJZl6Sgas3V7ofItIUw) - Python Tools - help for folks who are not very python - Project Definitions * Write code with a large language model --- ## [Workshop 1 - Critical Thinking and Language Models](/-GsAU7zMRdGE0qQ_apXJYA) * Recognize and define key terms related to the industrial implementation of large language models * Develop a complex prompt using Critical Evaluative Criteria * Use Make.com to automate an agentic flow #### Homework Summary: Set up a series of agents that can Research & Write Grants for you --- ## [Workshop 1.5 - Complex Prompts and Benchmarks](/QDTwSi82S1GZNtn2YV7JvA) * Use Critical Evaluative Criteria with simple Python scripts to produce something * Use the OpenAI ChatCompletions & Responses API to request inference * Create your own benchmark for evaluating outputs #### Homework Summary: Develop a benchmark for an agent for a task you do often / have conceptualized of as a project --- ## [Workshop 2 - Jailbreaks](/jf7pdzWpQNWwowtzNXXfZg) * Jailbreak a large language model out of its alignment guidance #### Homework Summary: - Review jailbreaking techniques, practice them with some jailbreaking games - Add red-team prompts to your benchmark from Workshop 1 - [**Break into as many of these as you can before the end of class**](https://themultiverse.school/ctf) - First try with the other team's - Try with your own team's, to determine if potential foreknowledge of the secret allows for validation of the secret through the LLM. --- ## [Workshop 2.5 - Watermarking and Detection](/FGUU8IZfSU6yZbSGaoDMrQ) --- ## [Workshop 3 - Tools](/mr5qqLBUQWGNgIrpoRo-Iw) * Use (code or) no-code tools to automate actions a large language model can take * Use (code or) no-code tools to trigger a large language model on a schedule --- ## [Workshop 3.5 - Model Context Protocol Servers](/o3gVTlWZRumB5Lp4UQn13w) --- ## [Workshop 4 - Spec Driven Development](/iNRCHaGBT-y_g-sCOR0jIA) * Define, Write, and Deploy an entire application with a large language model * Decompose the above process into a Swarm of Agents #### Homework Summary Create an agent in code using your benchmark framework that takes in a prompt and executes a series of prompts and outputs to produce a plan that it then follows --- ## [Workshop 5 - Agentic SDLC](/hyKfHDZBRt-YKGt9bFV4KA) * Use the agentic SDLC to make an app #### Homework Summary: Build an app with the agentic SDLC --- ## [Workshop 6 - Planning and Context](/QiVlw7u7Tyqz0J_y2M003Q) * Create AI-assisted project plans * Use multi-agent debates for exploring perspectives * Implement council of experts persona review --- ## [Workshop 6.5 - Context Engineering & 12 Factor Agents](/-H8Li0ZZSpmer_GIrwtzBA) * Demonstration of the 12 factors of agents * Build agents from scratch with the OODA Loop * Own your prompts and context window Homework: [Context Engineering Exercise](/slcVkR5MQ3ekwCU-absgUw) --- ## [Workshop 7 - Retrieval Augmented Generation](/fud1WXotSwWqhiX3IcMeDQ) * Build MCP RAG servers * Implement semantic chunking * Use FAISS and pgVector for vector search --- ## [Workshop 8 - Memory Patterns](/BcZk_3qVQ0eoQ7wyqUF0Lw) * Implement hierarchical memory for agents * Share memory between multiple agents * Build Q&A assistants over documents --- ## [Workshop 8 - Alignment (Management & Strategy)](/8Qy-rAU7Rp2CsYMoZgQeAw) * Align a large language model's outputs to human values using prompt engineering techniques according to state-of-the-art research * Identify the five concrete alignment problems you'll encounter in production * Understand alignment as a management problem --- ## [Workshop 8.5 - Advanced Alignment Techniques (Technical)](/_PWy17j4RE2yvui9ewY2AQ) * Build Constitutional AI systems * Fine-tune models for alignment * Implement monitoring and red teaming --- ## [Workshop 9 - Meta-languages & Meta Narratives](/sf2bfA9ETkK-7MpHUDzFRg) * Develop a metalanguage to allow a language model to reason at a symbolic level about a specific problem domain * Develop a metalanguage to allow for narrative design that can be held in available conversational context windows, allowing for the generation of novels and longer-form objects ## More recordings for this class - [Prompt Pattern Catalog](https://www.loom.com/share/33008fe4a3d342b4a3dcc783001d68ed) — Generate a structured story outline and chapters using narrative patterns like Save the Cat - [Tokens, Thought, and Trust: A Practical Guide to Language Models, Prompting, and Agentic AI](https://www.loom.com/share/44054354ec804b3ab2fb2067b6bf2eca) — Configure an LLM to access your local filesystem and build a command-line tool from a prompt - [gather-recording-2025-06-25T17-09-25](https://www.loom.com/share/5e97b4a317bf4fba982d4dc8495d61a8) — Identify and modify AI-generated text to bypass detection tools like GPTZero - [Understanding Meta-Language in Storytelling](https://www.loom.com/share/6ff5a24dc0f4400ba17ca90ac5d20e4c) — Apply narrative structures like the Story Circle as meta-language to generate stories - [Creating Effective Action Plans with AI Assistance](https://www.loom.com/share/aa3df7611a6142cd9401d046303b305d) — Build a multi-agent workflow to create, critique, and execute a project plan - [Exploring OpenAI's ChatGPT and its Capabilities](https://www.loom.com/share/b70bd2a37dd6472d97b9327e5c8f0216) — Construct and refine a system prompt for an AI task triage assistant - [Creating Compelling Multi-Step Artifacts: Frameworks for Writers and Creators](https://www.loom.com/share/c85a514b3a074d57b522ac144b61b528) — Generate and refine a story outline using narrative frameworks and iterative AI prompting - [Python, Colab, and Benchmarking](https://www.loom.com/share/f50e167309fb4f3c8cb5bef4dd1d589b) — Write a Python script to benchmark different AI models and prompts for a task