Skip to slide
14 / 39

CH 02 · SWE-bench, Can a Model Fix a Real Bug in Real Code? · 01/06

Read as one page

SWE-bench, Can a Model Fix a Real Bug in Real Code?

SWE-bench hands a model a real GitHub bug and a full codebase, then grades the fix by running the test suite; early top models solved under two percent.

6 slides · 3 min read

Contour map for SWE-bench, Can a Model Fix a Real Bug in Real Code?

In this chapter

  1. 01The problem it solves
  2. 02The core idea
  3. 03What they found
  4. 04Why it mattered
  5. 05The one-sentence takeaway

Paper: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023)

BIG-Bench in the last chapter measured breadth with made-up tasks. SWE-bench takes the opposite tack: one domain, software engineering, but using completely real work pulled straight from open-source projects. It asks a model to do something a professional developer does every day, fix a reported bug in a large existing codebase, and it grades the result the only way that truly counts: by running the tests.

Finished this idea?

Your learning progress stays in this browser.