System prompt revisions shift the instruction context, so the same test suite can yield different token‑level scores and safety flag activations across model versions.
Regression‑testing workflow
1. Baseline capture – Run the current model (e.g., gpt-4o‑2024‑05‑13) on the full prompt suite with logprobs=5 and response_format={"type":"json_object"}. Store:
- average_logprob
- safety_score from openai.Moderation.create
- Guardrail hit count (guardrail_id metrics).
2. Prompt revision – Update the system prompt. Record the diff size (e.g., +12 tokens, ‑3 tokens).
3. Version bump – Deploy the new model version (e.g., gpt-4o‑2024‑08‑15).
4. Automated comparison – Use a Python script:
import openai, json
base = json.load(open('baseline.json'))
new = openai.ChatCompletion.create(
model='gpt-4o-2024-08-15',
messages=[{'role':'system','content':new_prompt}, {'role':'user','content':test_input}],
temperature=0,
logprobs=5,
response_format={"type":"json_object"}
)
logprob_delta = new['usage']['prompt_logprobs'] - base['avg_logprob']
mod = openai.Moderation.create(input=new['choices'][0]['message']['content'])
print('Δlogprob',logprob_delta, 'moderation_flag', mod['results'][0]['flagged'])5. Threshold check – Fail the run if any of these conditions hold:
- Δlogprob < -0.15
- new_guardrail_hits > base_guardrail_hits + 2
- moderation_flag == True.
Quick comparison table
| Metric | v1 (gpt‑4o‑2024‑05) | v2 (gpt‑4o‑2024‑08) | Acceptable Δ |
|---|---|---|---|
| Avg. logprob | -1.27 | -1.38 | ≤ -0.15 |
| Guardrail hits | 3 | 5 | ≤ +2 |
| Moderation flagged | 0% | 1% | 0% |
Gotcha: If the revised system prompt pushes the total token count past the model’s context window (e.g., 128k → 129k), the tail of the prompt is silently truncated, causing guardrail rules to disappear and test failures to be misleading.