Human Validation: Structural Similarity Task

Overview

You are validating a dataset that tests whether AI systems can distinguish structural similarity from surface similarity.

Your Task

For each of 60 items, you will see:

  • A Query (a short narrative episode)
  • Candidate A (another narrative)
  • Candidate B (another narrative)

Question: Which candidate (A or B) shares the same underlying RELATIONAL STRUCTURE as the Query?

Enter either A or B in your annotator column.

What is “Relational Structure”?

Structure refers to the underlying pattern of roles, intentions, and causal relationships—NOT surface features like vocabulary, domain, or setting.

Structural elements to consider:

  • Roles: deceiver/deceived, helper/helped, betrayer/betrayed, mentor/protégé
  • Intentions: manipulative vs. genuine, strategic vs. sincere, self-serving vs. altruistic
  • Causal patterns: deception → harm, sacrifice → unrecognized, failure → growth
  • Outcomes: trust violated, hidden depths revealed, transformation vs. confirmation

Surface features to IGNORE:

  • Same vocabulary or keywords
  • Same domain (workplace, family, school)
  • Same entities (boss, brother, friend)
  • Similar phrasing or sentence structure

Critical Insight

One candidate will often share more WORDS with the Query but have the OPPOSITE structure.

This is intentional—the task tests whether you can see past surface similarity to identify structural similarity.

Example

Query: “My boss invents fake emergencies to motivate us. Last week he claimed our client was leaving, but they’d just renewed.”

Candidate A: “My dad would shout it was 7 AM when it was 6:15. He thought fake urgency would make me punctual.”

Candidate B: “My boss warned about a real client complaint. We worked late and fixed it. His warning was accurate.”

Correct Answer: A

Why?

  • Query structure: Authority figure fabricates urgency through deception
  • Candidate A: Same structure—authority (dad) fabricates urgency (fake time) through deception
  • Candidate B: OPPOSITE structure—authority (boss) conveys genuine urgency about real problem

Note: Candidate B shares more words with Query (boss, client, warning) but has opposite structure (genuine vs. fabricated).

How to Annotate

  1. Read the Query carefully
  2. Identify the key structural pattern (who does what to whom, with what intent, leading to what outcome)
  3. Read both Candidates
  4. Ask: “Which candidate has the same pattern, even if the surface details differ?”
  5. Enter A or B in your column

Time Estimate

  • ~2-3 minutes per item
  • 60 items total
  • ~2-3 hours total
  • Take breaks as needed

Important Notes

  • This is a blinded evaluation—A/B labels are randomized
  • There are no trick questions—one candidate genuinely shares the Query’s structure
  • If genuinely uncertain, make your best guess and add a note in the “notes” column
  • Your independent judgment matters—don’t discuss answers with other annotators until complete

Questions?

If anything is unclear, contact the study coordinator before beginning.

Thank you for your participation!