Release day, 5 p.m. You run git log --oneline v1.2.0..HEAD, see forty lines of "wip", "fix typo", "Merge branch" and "address review comments", and ask an AI to turn them into release notes. It returns six confident bullets. One mentions a caching layer nobody built, because a branch name sounded like it.
The fix isn't a better prompt. It's a pipeline that feeds the model only classified evidence and then checks the draft against the repository. This post builds that pipeline for AI commit messages, changelogs and release notes, with scripts I ran on 2026-10-09 against a throwaway git repo. The repo and commits are invented for the demo; the standards cited were read on their own pages the same day.
What do the standards actually say?
Two short documents define most of what you need.
Conventional Commits 1.0.0 (conventionalcommits.org, read 2026-10-09). A message is type(optional scope): description, then an optional body one blank line later, then optional footers. feat marks a new feature and fix a bug fix; other types such as docs: are allowed but, in the spec's words, have no implicit effect in Semantic Versioning unless they include a breaking change. A breaking change is either a ! immediately before the colon or a BREAKING CHANGE: footer, and the FAQ maps fix to PATCH, feat to MINOR and breaking changes to MAJOR.
Keep a Changelog 1.1.0 (keepachangelog.com, read 2026-10-09). A changelog is "a curated, chronologically ordered list of notable changes for each version". Its guiding principles include: changelogs are for humans, same types of changes grouped, latest version first, release date shown. The types are Added, Changed, Deprecated, Removed, Fixed and Security. It also says using commit-log diffs as a changelog is a bad idea because of the noise.
That last point is the design brief. A changelog is curated. So the work splits into a deterministic step (classify and filter) and a language step (turn evidence into sentences), with a mechanical check on the output.
How do I build a throwaway repo to test this?
Never test a release pipeline on your real history. This script builds a repo with fixed dates, so the hashes below are reproducible:
"""Build a throwaway git repo with fixed dates so hashes are reproducible."""
import os
import shutil
import subprocess
import sys
REPO = sys.argv[1]
if os.path.exists(REPO):
shutil.rmtree(REPO)
os.makedirs(REPO)
ENV = dict(os.environ)
ENV.update({
"GIT_AUTHOR_NAME": "Dev One", "GIT_AUTHOR_EMAIL": "dev1@example.invalid",
"GIT_COMMITTER_NAME": "Dev One", "GIT_COMMITTER_EMAIL": "dev1@example.invalid",
"GIT_CONFIG_GLOBAL": "/dev/null", "GIT_CONFIG_SYSTEM": "/dev/null",
})
counter = {"day": 1}
def git(*args):
subprocess.run(["git", "-C", REPO, *args], env=ENV, check=True, capture_output=True, text=True)
def commit(message, filename, text):
with open(os.path.join(REPO, filename), "a") as fh:
fh.write(text + "\n")
git("add", filename)
stamp = f"2026-09-{counter['day']:02d}T10:00:00+0000"
counter["day"] += 1
ENV["GIT_AUTHOR_DATE"] = stamp
ENV["GIT_COMMITTER_DATE"] = stamp
git("commit", "-q", "-m", message)
git("init", "-q", "-b", "main")
commit("feat: first public release", "core.py", "def get(url): ...")
git("tag", "v1.2.0")
commit("feat(cli): add --timeout flag", "cli.py", "# timeout flag")
commit("fix(parser): handle empty header values (#41)", "parser.py", "# empty headers")
commit("chore: bump dev dependencies", "dev.txt", "pytest")
git("checkout", "-q", "-b", "tidy")
commit("wip", "notes.txt", "scratch")
git("checkout", "-q", "main")
ENV["GIT_AUTHOR_DATE"] = "2026-09-06T10:00:00+0000"
ENV["GIT_COMMITTER_DATE"] = "2026-09-06T10:00:00+0000"
git("merge", "--no-ff", "-q", "-m", "Merge branch 'tidy'", "tidy")
counter["day"] = 7
commit("docs: fix typo in README", "README.md", "typo fixed")
commit("Make retry count configurable", "retry.py", "RETRIES = 3 # now from config")
commit("fix: retry on 502 responses (#44)", "retry.py", "# 502 added")
commit(
"feat(api)!: rename fetch() to get()\n\nBREAKING CHANGE: fetch() is removed; call get() instead.",
"core.py", "# fetch removed",
)
print("repo built at", REPO)
The history since the v1.2.0 tag is deliberately messy: a conventional feat and two fix commits, a chore, a docs commit, a wip commit on a side branch with its merge, one meaningful commit with a non-conventional message ("Make retry count configurable"), and a breaking feat(api)!.
* dd556b9 feat(api)!: rename fetch() to get()
* c2e83a2 fix: retry on 502 responses (#44)
* c0e04b2 Make retry count configurable
* 1a5d051 docs: fix typo in README
* 68f8367 Merge branch 'tidy'
|\
| * 2b25be2 wip
|/
* 45e1170 chore: bump dev dependencies
* aa1d7e6 fix(parser): handle empty header values (#41)
* c883705 feat(cli): add --timeout flag
How do I classify commits before the model sees them?
This script reads git log for a range, classifies each commit by the Conventional Commits header pattern, drops obvious noise, and suggests a version bump. The commit that does not follow the convention is not guessed at; it goes into a "needs-human" bucket.
"""python prep_notes.py <repo> <since-ref>"""
import json
import re
import subprocess
import sys
repo, since = sys.argv[1], sys.argv[2]
SEP, REC = "\x1f", "\x1e"
fmt = SEP.join(["%h", "%p", "%s", "%b"]) + REC
raw = subprocess.run(
["git", "-C", repo, "log", "--reverse", f"--format={fmt}", f"{since}..HEAD"],
check=True, capture_output=True, text=True,
).stdout
HEADER = re.compile(r"^(?P<type>[a-zA-Z]+)(\((?P<scope>[^)]+)\))?(?P<bang>!)?: (?P<desc>.+)$")
NOISE_TYPES = {"chore", "docs", "style", "test", "ci", "build"}
commits = []
for rec in raw.split(REC):
rec = rec.strip("\n")
if not rec:
continue
h, parents, subject, body = rec.split(SEP)
m = HEADER.match(subject)
c = {"hash": h, "subject": subject, "type": None, "scope": None,
"breaking": False, "bucket": "needs-human", "body": body.strip(),
"issues": re.findall(r"#(\d+)", subject)}
if m:
c["type"], c["scope"] = m["type"].lower(), m["scope"]
c["breaking"] = bool(m["bang"]) or "BREAKING CHANGE:" in body or "BREAKING-CHANGE:" in body
if len(parents.split()) > 1:
c["bucket"] = "noise (merge)"
elif subject.strip().lower() in {"wip", "fixup", "tmp"}:
c["bucket"] = "noise (wip)"
elif c["type"] in NOISE_TYPES and not c["breaking"]:
c["bucket"] = f"noise ({c['type']})"
elif c["type"] in {"feat", "fix"}:
c["bucket"] = "include"
commits.append(c)
if any(c["breaking"] for c in commits):
bump = "major"
elif any(c["type"] == "feat" for c in commits):
bump = "minor"
elif any(c["type"] == "fix" for c in commits):
bump = "patch"
else:
bump = "none"
print(f"range: {since}..HEAD, {len(commits)} commits, suggested bump from conventional commits: {bump}")
for c in commits:
flag = " BREAKING" if c["breaking"] else ""
print(f" {c['hash']} {c['bucket']:<16}{flag} {c['subject']}")
packet = [c for c in commits if c["bucket"] in ("include", "needs-human")]
print("\nevidence packet (JSON, this is what goes into the prompt):")
print(json.dumps([{k: c[k] for k in ("hash", "subject", "breaking", "bucket", "body") if c[k] != ""} for c in packet], indent=1))
Output:
range: v1.2.0..HEAD, 9 commits, suggested bump from conventional commits: major
c883705 include feat(cli): add --timeout flag
aa1d7e6 include fix(parser): handle empty header values (#41)
45e1170 noise (chore) chore: bump dev dependencies
2b25be2 noise (wip) wip
68f8367 noise (merge) Merge branch 'tidy'
1a5d051 noise (docs) docs: fix typo in README
c0e04b2 needs-human Make retry count configurable
c2e83a2 include fix: retry on 502 responses (#44)
dd556b9 include BREAKING feat(api)!: rename fetch() to get()
Nine commits become five pieces of evidence. The suggested bump is major, because of the !, so the next version is 2.0.0, not the 1.3.0 a feature-count instinct would suggest. The script also pulled the BREAKING CHANGE: footer text into the packet, which is the sentence users need most.
Two limits are worth saying out loud. A chore: or docs: commit can still matter to users (a docs commit that fixes a wrong install command, say), and this script drops it; if your project cares, narrow the noise rules. And the script trusts the commit header, so a fix: that actually changes behaviour keeps its label whether or not it should.
What prompt turns the evidence into a changelog?
Feed the model the packet, not the log. This is an untested template; I wrote it for this post and did not run it on a live model.
You are drafting a changelog section in the Keep a Changelog format.
Rules:
- Use only the commits in the evidence JSON below. Every bullet must end
with the short hash(es) of the commit(s) it comes from, in parentheses.
- Do not add numbers, versions, performance claims or features that are
not stated in a commit subject or body.
- Group under: Added, Changed, Deprecated, Removed, Fixed, Security.
Omit empty groups.
- For any commit with "breaking": true, write a bullet starting with
"BREAKING:" that says what users must change, using only the commit body.
- For commits with bucket "needs-human", do not guess. Put each in a list
called QUESTIONS at the end with one question for the maintainer.
- Heading: ## [<VERSION>] - <DATE>. Use exactly the values I give.
VERSION: 2.0.0
DATE: 2026-10-09
EVIDENCE:
<paste JSON>
The needs-human rule matters. "Make retry count configurable" is ambiguous: is the count now read from a file, an environment variable, a function argument? The honest output is a question, not a guess. A maintainer answers it in ten seconds and the draft improves.
How do I check the draft for invented claims?
A second script compares the draft with the repository. It cannot judge writing, but it catches the failures that embarrass you: a bullet with no commit behind it, a hash from the wrong range, a number that appears in no cited commit, a missing breaking change, a wrong version heading.
"""python check_notes.py <repo> <since-ref> <draft.md> <expected-version>"""
import re
import subprocess
import sys
repo, since, draft_path, expected = sys.argv[1:5]
SEP, REC = "\x1f", "\x1e"
raw = subprocess.run(
["git", "-C", repo, "log", f"--format=%h{SEP}%s{SEP}%b{REC}", f"{since}..HEAD"],
check=True, capture_output=True, text=True,
).stdout
commits = {}
for rec in raw.split(REC):
rec = rec.strip("\n")
if rec:
h, subject, body = rec.split(SEP)
commits[h] = subject + "\n" + body
HASH = r"\b(?=[0-9a-f]*\d)[0-9a-f]{7}\b" # 7 hex chars with at least one digit
breaking = [h for h, t in commits.items() if re.match(r"^\w+(\([^)]*\))?!:", t) or "BREAKING CHANGE:" in t]
draft = open(draft_path).read()
problems = []
heading = re.search(r"^## \[?(\d+\.\d+\.\d+)", draft, re.M)
if not heading:
problems.append("no '## [x.y.z]' heading found")
elif heading.group(1) != expected:
problems.append(f"heading says {heading.group(1)} but expected {expected}")
cited_all = set()
for line in draft.splitlines():
if not line.startswith("- "):
continue
hashes = re.findall(HASH, line)
if not hashes:
problems.append(f"bullet cites no commit: {line[:70]}")
continue
cited_all.update(hashes)
for h in hashes:
if h not in commits:
problems.append(f"cites {h}, which is not in {since}..HEAD: {line[:60]}")
evidence = " ".join(commits.get(h, "") for h in hashes)
text = re.sub(HASH, "", line)
evidence_numbers = set(re.findall(r"\d+(?:\.\d+)?", evidence))
for num in re.findall(r"\d+(?:\.\d+)?", text):
if num not in evidence_numbers:
problems.append(f"number '{num}' appears in no cited commit: {line[:60]}")
for h in breaking:
if h not in cited_all:
problems.append(f"breaking commit {h} is not mentioned")
if problems:
print(f"{len(problems)} problem(s) in {draft_path}")
for p in problems:
print(" -", p)
sys.exit(1)
print(f"ok: {draft_path} ({len(cited_all)} commits cited, {len(breaking)} breaking covered)")
I wrote two drafts by hand to test it. They are illustrative, not model output. The first mimics the usual failure modes: a heading that ignores the breaking change, a made-up default of 30 seconds, a cached-response feature with a hash that isn't in the range, an invented "up to 5 times", an uncited vague bullet, and no mention of the rename.
6 problem(s) in draft_bad.md
- heading says 1.3.0 but expected 2.0.0
- number '30' appears in no cited commit: - `--timeout` flag for the command line tool, default 30 sec
- cites 9f3e2a1, which is not in v1.2.0..HEAD: - Response caching for repeated requests (9f3e2a1)
- number '5' appears in no cited commit: - Requests are retried after a 502 response, up to 5 times (
- bullet cites no commit: - Much faster retries
- breaking commit dd556b9 is not mentioned
The second draft keeps to the evidence: the breaking rename first, the retry count bullet from the human-reviewed commit, and the rest cited by hash.
ok: draft_good.md (5 commits cited, 1 breaking covered)
A first version of the number check used a plain substring test and wrongly accepted "5" because it appears inside "502". Matching whole numbers fixed it. If you build a checker like this, test it against a deliberately bad draft, as above, or you won't know it works.
What it still misses: wrong grouping (a fix listed under Added), a true hash attached to a misleading sentence, and tone. Read the draft yourself.
How do I get a good commit message from a diff?
Treat it like the changelog: constrain, then verify. Untested template:
Write a commit message for this staged diff.
Format: type(scope): subject, subject at most 50 characters, imperative
mood. Then a blank line and a body of at most 3 short lines explaining
WHY, using only what the diff shows. Types: feat, fix, refactor, docs,
test, chore. Add "!" and a "BREAKING CHANGE:" footer only if the diff
removes or renames public behaviour. If the diff mixes unrelated changes,
say so and suggest how to split the commit instead of writing one message.
<paste output of git diff --staged>
The "split the commit" line is the most useful one. A diff that touches a rename, a bug fix and a dependency bump should become three commits, and no summary sentence fixes that. For the editor workflow and Git's own subject-line guidance, see dictating commit messages and code reviews, which covers the 50-character convention. Before you send any diff to a hosted model, read what not to paste into ChatGPT: fixtures and config files are where secrets hide.
How do I write release notes from the changelog?
Release notes answer a different question from the changelog: "do I need to do anything, and is it worth upgrading?" Start from the verified changelog, not from git history:
Write release notes for version 2.0.0 for users of this library, from
the changelog below. Structure: one-sentence summary; "Before you
upgrade" (breaking changes, each with a before/after code line taken
only from the changelog text); "What's new" (at most 3 bullets);
"Fixes" (one line). Do not add information that is not in the
changelog. If a breaking change lacks a migration hint, write
"MISSING MIGRATION HINT" and stop.
<paste the checked changelog section>
The final instruction forces the gap into the open. In the demo repo the breaking commit body says "call get() instead", which is enough for a one-line migration; plenty of real breaking commits say nothing about how to migrate, and the model should not fill that in from imagination.
Release checklist for AI-assisted notes
- Tag the previous release; make sure
since..HEADis the range you mean. - Run the classifier. Resolve every
needs-humancommit yourself. - Decide the version from the suggested bump, then confirm it by hand.
- Draft with the constrained prompt; paste in the version and date yourself.
- Run the checker on the draft. Fix problems by editing the draft or the evidence, never by loosening the checker.
- Read the draft once as a user: would you know what to change before upgrading?
- Derive release notes from the checked changelog.
If your CI already runs AI on pull requests, the same discipline applies there; see the AI code review GitHub Actions template. And if you want tests that would notice a silent change before it reaches a changelog, mutation-check your AI-written tests.
What this does not solve
Garbage in, garbage out still applies: if a team writes "fix stuff" for every commit, no pipeline can recover meaning that was never recorded. The honest fix is upstream, with PR titles or commit templates, not downstream with a smarter model. Squash-merge workflows change the picture too, since the squashed message becomes the only evidence. Test the scripts on a copy of your own history before trusting them; this demo covers one small repo with nine commits.
