A user with a 150k stool image dataset seeks best practices for training a computer vision model. They question their current manual verification workflow and look for smarter, more scalable approaches for ensuring dataset and annotation quality.
The paper introduces challenges in understanding annotator disagreement regarding AI safety policies, which can arise from operational failures, policy ambiguity, or value pluralism. It highlights the difficulty of discerning the root causes of these disagreements and the unreliability of self-reported reasoning from annotators.
MultiSoc-4D is a new Bengali social media dataset benchmark designed to diagnose LLM behavior in closed-set annotation. The research identifies "instruction-induced label collapse," a phenomenon where LLMs systematically prefer fallback labels, leading to under-detection of minority categories.