Social Interaction & Dialogue Evaluation Benchmark
Evaluating Social Interaction and Conversational-Partner Intent in Multiagent Environments
Real human communication is not just about answering questions—it requires genuine social intelligence. In high-stakes conversations, people have to adapt their communication style, figure out what the other person truly wants, and steer discussions toward a successful agreement without damaging the relationship. When AI agents enter these spaces, sounding polite is not enough: an agent that caves in immediately fails its goals, while an agent that is too aggressive ruins trust.
SIDE-Bench evaluates whether language models can navigate this delicate balance. Across 150 realistic conflict scenarios spanning five everyday domains, models must balance two critical goals: reaching a fair agreement while protecting mutual trust and rapport. Crucially, SIDE-Bench pairs final agreement verification with Conversation-Partner Intent Tracking (C-PIT) to evaluate whether models can deduce their partner's hidden limits and objections turn-by-turn and make smarter negotiation moves.
What does social intelligence look like when an AI agent has to negotiate it in real time?
When humans converse in the real world, conversations are rarely simple Q&A. Real interactions involve competing goals, hidden limits, and relationship risks. An AI can produce fluent, polite text but still fail completely if it immediately caves in, surrenders its private goals, or damages trust with the other person.
The AI receives a private goal and walkaway bottom line. The partner also has secret limits and objections hidden behind a firewall.
Scenarios test different social dynamics—such as pushing back against a boss, negotiating with a stranger, or resolving friction with a close friend.
At every turn of the conversation, the AI must deduce the partner's unspoken objections and hidden goals before deciding what to say.
A dataset of 150 multi-turn conflict situations across five everyday domains (30 scenarios each), plus a 50-scenario SIDE-Bench Lite version for focused audits.
Negotiating deals, pricing, and contract terms when money and budgets are tight and both sides want the best price.
Handling difficult feedback, pushback from managers, and disagreements with coworkers across company hierarchies.
Setting healthy boundaries with friends or relatives without hurting feelings or damaging personal trust.
Calming angry customers while respecting company policy and finding fair, workable solutions.
Resolving disagreements between strangers or neighbors where neither side is the boss and social norms must be negotiated.
A balanced set of 50 scenarios (10 per domain) designed for rapid testing and turn-by-turn audits of the AI's internal reasoning.
Every scenario is mathematically verified to have at least one fair compromise where both sides can succeed. The partner pushes back at least twice so the model cannot win by simply giving in.
We evaluate two core abilities: whether language models can achieve their private goals under social pressure, and whether they can accurately track their partner's intent throughout the conversation.
Can models protect their bottom line and reach solid agreements without giving up immediately or becoming aggressive?
Can models deduce the partner's hidden motives, budget limits, and unspoken objections at each turn, and use that insight to make smarter offers?
Fraction of private goals achieved without violating walkaway limits.
Whether the conversation ended in a mutually accepted deal.
How much trust changed ([-5, +5]), penalizing hostility or damaged relationships.
Audits turn-by-turn deductions to see if the model actually understood the partner.
Seven frontier and open-weight models evaluated across 1,050 matched multi-turn conversations against a standardized conversation partner (GPT-5.4).
Evaluating 1,050 full conversations against a standardized conversation partner
Almost all modern LLMs have memorized polite phrasing, so their surface goal scores all cluster closely together. The real differences are in speed and trust: GPT-5.5 finishes negotiations in nearly half the time (only 4.8 turns vs 9.1 for smaller models), Gemini 3.1 Pro ratifies deals most often (70.0%), and Claude Sonnet 4.6 gains the highest partner rapport (+2.77).
| Rank | Model | Type | Goal Score (GS) ↑Private goals achieved | Deal Made (RS) ↑Mutual agreement ratified | Trust Built (ΔR) ↑Partner rapport shift | Turns Needed (IC) ↓Fewer turns = faster | Overall Score (SI) ↑Dual-axis index |
|---|---|---|---|---|---|---|---|
| 1 | Claude Sonnet 4.6 | Frontier | 0.857 | 67.3% | +2.77 | 5.6 | 0.846 |
| 2 | Gemini 3.1 Pro | Frontier | 0.853 | 70.0% | +2.65 | 6.0 | 0.829 |
| 3 | GPT-5.5 | Frontier | 0.846 | 64.7% | +2.58 | 4.8 | 0.820 |
| 4 | Llama 3.3 70B Instruct | Open-Wt | 0.844 | 68.7% | +2.53 | 7.8 | 0.815 |
| 5 | Qwen 2.5 32B Instruct | Open-Wt | 0.822 | 64.7% | +2.42 | 6.8 | 0.802 |
| 6 | Qwen 2.5 72B Instruct | Open-Wt | 0.818 | 58.7% | +2.41 | 7.1 | 0.802 |
| 7 | Qwen 2.5 7B Instruct | Open-Wt | 0.800 | 56.0% | +2.20 | 9.1 | 0.777 |
Auditing the AI's internal thoughts (<belief>) turn-by-turn to test Theory of Mind
| Model | Type | Jumping to Conclusions (CR) ↓Assuming casual hints are hard limits | Belief Stubbornness (RER) ↓Ignoring partner clarifications | Smart & Feasible Deals (CC) ↑Right intent + valid offer | Thought-Action Disconnect (CI) ↓Knew limit inside, but broke it |
|---|---|---|---|---|---|
| GPT-5.5 | Frontier | 16.7% | 7.4% | 91 | 13 |
| Claude Sonnet 4.6 | Frontier | 40.9% | 4.8% | 90 | 8 |
| Gemini 3.1 Pro | Frontier | 41.7% | 34.6% | 72 | 11 |
| Llama 3.3 70B | Open-Wt | 66.7% | 12.9% | 71 | 13 |
| Qwen 2.5 72B | Open-Wt | 0.0%* | 55.9% | 84 | 7 |
| Qwen 2.5 32B | Open-Wt | 25.0% | 69.0% | 67 | 7 |
| Qwen 2.5 7B | Open-Wt | 10.5% | 56.8% | 72 | 6 |
*Qwen 2.5 72B registered 0% on jumping to conclusions only because it never updated its initial assumptions, leading to an elevated 55.9% revision error rate.
GPT-5.5 tracks what the partner wants with exceptional precision (only 7.4% error), yet it still proposed 13 offers that violated those exact limits. The model understood the partner in its thoughts, but blurted out an incorrect proposal anyway!
When the partner directly corrected their limits (e.g. "My real budget is $10k, not $8k"), open models like Qwen 2.5 32B failed to update their internal beliefs 69% of the time!
Llama 3.3 has a 66.7% collapse rate—meaning two-thirds of the time, it prematurely treats a casual comment or tentative hesitation as an absolute, immovable dealbreaker.
Top models like GPT-5.5 deduce the partner's hidden limits with high accuracy (only 7.4% error), yet still propose deals that violate those exact limits (Action Mismatch = 13). Superior reasoning alone does not prevent careless actions.
Open-weight models struggle to adapt: in more than 55% of turns where the partner clarified their true position, models like Qwen failed to update their internal beliefs.
Sales and price negotiations (D1) proved to be the most difficult domain because hard financial numbers leave zero room for vague promises, whereas customer service (D4) provides clearer rules to follow.
Access interactive workspaces to verify scenarios, grade real conversations, and audit whether AI models genuinely understand their partners.
Evaluator workspace
Sign in with your approved Google account to enter the SIDE-Bench workspace.
Select your evaluation task or review annotation instructions below.
Task 1 · Scenario Verification Workspace
This dataset is a carefully curated LLM synthetic benchmark that has already undergone automated LLM-based validation. We now require human expert verification to ensure the situations strictly meet our research criteria.
150 realistic conflict dilemmas across sales, collaboration, leadership, and personal domains generated via multi-agent LLM pipelines and pre-screened automatically.
Do not judge whether characters are "good/bad" or whether you morally like the situation. Your task is purely to verify if the scenario is a valid, functional test case for AI evaluation.
Does the premise, role dynamic, and dialogue setting feel like a genuine social situation that happens in the real world (Natural) or does it feel robotic and artificial (Contrived)?
Look at both parties' hard constraints and the feasible resolutions. Can they actually reach a deal (Valid / Narrow), or are their constraints logically contradictory (Impossible)?
ΔP (Power Delta): Asymmetry in leverage (e.g. ΔP -2 means counterpart has
higher leverage). Turns: Conversation turn budget. Tension: Initial
friction level.
If you select Needs Tweak or Discard, write one concise sentence explaining what needs fixing (e.g., "Buyer maximum budget ($80k) contradicts seller floor ($90k)").
Task 2 · Dialogue Evaluation Workspace
These conversations were generated through multi-turn LLM simulation benchmarks and pre-screened with automated LLM judges. We now require independent human evaluation based strictly on the transcript and rubric criteria.
Anonymized multi-turn dialogue transcripts (up to 30 turns) and the relevant background scenario context (collapsible at the top).
Evaluate the interaction against the explicit rubric criteria (Goal Attainment, Rapport, Appropriateness, Social Awareness). Do not judge based on personal taste or style.
Did the evaluated agent achieve its public and private goals without breaching any hard constraints or conceding beyond the reservation threshold?
Assess whether the interpersonal dynamic improved (+), degraded (-), or remained flat compared to the opening tension baseline on a scale of -5 to +5:
Note that some agent roles in the transcript may be abbreviated (e.g., AE for Account Executive) to match the scenario setting. Please check the participant list if unsure.
All model names, baselines, and temperature parameters have been stripped. Evaluate purely based on what is stated in the transcript.
Task 3 · PIBT Process Audit Workspace
In Task 3, you audit the internal cognitive and belief-tracking processes
of AI agents during strategic conflict. Rather than only assessing surface conversation, you verify whether
the agent's hidden <thought> and <belief> states were grounded in
dialogue evidence and whether its chosen actions were feasible.
Inspect the evaluated agent's internal <thought> reasoning scratchpad and
<belief> representation (tracking counterpart's profile, constraints, and reservation
prices) at key decision turns.
• Accurate: The belief logically matches the factual evidence revealed by the
counterpart so far.
• Premature Collapse: The agent jumped to a rigid conclusion
without testing hypotheses.
• Revision Error: Counterpart gave new contrary facts, but
the agent failed to update.
The agent emits a partner_type in its
<belief> block representing its inference of the counterpart's hidden profile:
No reliable signal or evidence yet (common at initial turns before limits or motives are tested).
Counterpart's stated limits or threats are treated as bargaining bluff, tactical posturing, or non-credible anchors.
Counterpart's limit is treated as authentic, non-negotiable, and strictly binding on them.
Counterpart exhibits genuine underlying constraints accompanied by tactical pressure or multiple hypotheses remain viable.
Did the agent's proposed action, offer, or conversational tactic comply with the scenario rules, participant roles, and hard constraints? (Feasible vs. Infeasible).
• CC: Correct belief + Feasible action (sound social reasoning).
•
CI: Correct belief, but Infeasible action (execution failure).
• WC:
Wrong belief, but Feasible action (flawed reasoning / luck).
• WI: Wrong belief +
Infeasible action (total failure).
Click 🔍 Audit Turn on each evaluated agent turn in the transcript. Completed turns are marked with a green ✓ Turn Audited tag. Complete all agent turns to finish the scenario.
Never penalize an agent for lacking information that the counterpart had not yet revealed. Base your evaluation strictly on what transpired up to the specific turn being audited.
Task 1 Telemetry · 150 Seed Scenarios
Task 2 Telemetry · Multi-Turn Dialogue Rollouts
Task 3 Telemetry · PIBT Process Audits
RBAC & Access Control
Manage authorized Google accounts and grant granular access to Task 1 Scenario Auditing and Task 2 Dialogue Evaluation.