Enable the built‑in moderation endpoint and supplement it with a custom classifier pipeline to catch both toxic language and copyrighted excerpts.
1. Activate provider moderation
- OpenAI: set moderation=true in the request header or call POST https://api.openai.com/v1/moderations.
- Azure AI Content Safety: enable contentSafety=true and configure categories=["hate","selfHarm","sexual","violence","copyright"].
2. Define thresholds
- Use a probability cut‑off of 0.75 for high‑risk categories; lower to 0.5 for copyright detection where false positives are costly.
3. Add a custom classifier
```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
model_name = "facebook/roberta-large-mnli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
def detect_copyright(text):
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
logits = model(inputs).logits
probs = torch.softmax(logits, dim=-1)
# index 2 corresponds to "contradiction" → likely copied
return probs[0,2].item()
```
4. Chain the checks
- Call provider moderation first; if any category exceeds its threshold, block.
- If provider passes, run detect_copyright; block when score > 0.8.
5. Log and audit
- Store category, probability, timestamp, and request_id in a secure audit table.
- Set up alerts for > 100 blocked requests per hour.
6. Deploy as a guardrail service
- Wrap the logic in a lightweight FastAPI endpoint that all internal LLM calls route through.
Comparison table
| Feature | OpenAI Moderation | Azure Content Safety | Custom Classifier |
|---|---|---|---|
| Toxic categories | 5 (default) | 5 (configurable) | N/A |
| Copyright detection | No | No | Yes (model‑based) |
| Latency (ms) | ~30 | ~45 | ~120 |
| Cost per 1k tokens | $0.001 | $0.002 | Compute‑only |
Production gotcha:** the moderation endpoint returns a probability per category, not a binary flag; you must consistently apply your chosen threshold across all calls, otherwise edge‑case phrases can slip through.