Who should refuse what?
Wang and colleagues build Do-Not-Answer around a three-level taxonomy of risks and instructions a responsible model should not follow. Their dataset contains 939 prompts across five risk areas, with more than 5,000 responses from six language models annotated against explicit response criteria.
Labels turn policy into a test set
Those human annotations let the authors compare model safeguards and train BERT-like classifiers for automatic evaluation. They report that small classifiers achieve results comparable to GPT-4 on this dataset. The comparison is tied to the authors' criteria for a safe response; a different policy or labeling rule can change which answer receives the safe label.
What an automated score can carry forward
The dataset makes its categories and judgments available for further safeguard testing. The paper's automatic measure learns from those annotations, so its agreement with GPT-4 or the human labels speaks to this particular task. When another team changes the refusal policy, it will need to make the changed criterion visible and assess whether the old labels still apply.
Bibliography
- Wang, Y., Li, H., Han, X., Nakov, P., & Baldwin, T. (2023). Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. arXiv:2308.13387v2. https://arxiv.org/pdf/2308.13387