Large Language Models in SE

AI-supported development tools, like Codex, Copilot, ChatGPT, etc., have taken a big role in SE recently. What underpins these tools, how do they work so well, what ethical concerns do they raise, and what can we expect for SE in the AI future?

Learning Outcomes

• a more than passing awareness of how large language models “work” on code
• ability to discuss the (current) tradeoffs of these tools
• analyze the way such tools are evaluated and discern hype from reality
• how to apply these tools for SE problems like refactoring or code comprehension

Topics and slides

These are the submodules I covered in class.

• LLM overview
• Research Opportunities

Readings

1. Hindle et al., On the Naturalness of Software
2. Codex
3. SWE Bench Verified

Exercises

These are done in class. The source code below is a combo of what I typed and what I prepped before hand.

1. Implement Command in code: Command Pattern

Optional Readings and Activities

These readings enrich the material but are not strictly necessary to read.

• LeGoues, Survey of APR
• DeepSeekCoder
• Vaswani et al. Attention is all you need

Helpful tutorials and summaries

• Alammar, The Illustrated Transformer
• Willison, “understanding GPT tokenizers”
• Self-Attention from Scratch