path: root/docs/results_revised.md

# Revised LaTeX: Results Section

## Changes from original

1. Fixed metadata: 3 datasets, 60 profiles × 60 sessions
2. Removed false "three random seeds" claim
3. Replaced all placeholder text with actual analysis
4. Added significance tests (paired t-test, Wilcoxon)
5. Mentioned bug cleaning (repetition bug removal)
6. Filled in Section 5.2 with actual vector analysis data
7. Added E/T decomposition analysis
8. Added "profiles that improved" metric

---

## Revised LaTeX

```latex
\section{Results}
\label{sec:results}

\subsection{Main Comparison}
\label{sec:results-main}

Table~\ref{tab:msc-main} summarizes the main results across
all three task domains (math-hard, math-500, bigcodebench).
We report task success, timeout rate, and average user tokens
per session for the six system modes
(Section~\ref{sec:setup-modes}), evaluated over
$60$~profiles~$\times$~$60$~sessions ($3{,}600$ sessions per
method).

During evaluation we identified a small fraction of sessions
exhibiting an agent repetition bug, in which the agent produces
near-identical responses across consecutive turns despite user
corrections (detected via SequenceMatcher ratio $> 0.85$ on
$\geq 2$ adjacent response pairs).
These sessions affected $7.1\%$ of RAG+Vector sessions and
$3.8\%$ of Reflection sessions; all numbers reported below
exclude these sessions.

\begin{table}[t]
  \centering
  \small
  \caption{
    Main results on \textsc{MultiSessionCollab}
    (math-hard, math-500, bigcodebench).
    For each system we report task success rate,
    timeout rate, and average user tokens per session
    over $60$~profiles~$\times$~$60$~sessions.
    Bold indicates best; underline indicates second best.
  }
  \label{tab:msc-main}
  \begin{tabular}{lccc}
    \toprule
    Method
    & Success (\%) $\uparrow$
    & Timeout (\%) $\downarrow$
    & User tokens $\downarrow$ \\
    \midrule
    RAG+Vector        & \textbf{55.2} & \textbf{26.4} & \underline{193.6} \\
    Reflection        & \underline{54.4} & \underline{28.8} & 207.5 \\
    Vanilla           & 54.3 & 29.2 & 232.9 \\
    Contextual        & 52.4 & 31.4 & 213.7 \\
    RAG               & 52.0 & 44.3 & \textbf{188.4} \\
    All-memory        & 50.9 & 33.4 & 226.8 \\
    \bottomrule
  \end{tabular}
\end{table}

\paragraph{Overall ranking.}
RAG+Vector achieves the highest task success rate ($55.2\%$)
among all six methods, followed closely by Reflection ($54.4\%$)
and Vanilla ($53.4\%$).
RAG+Vector also attains the lowest timeout rate ($26.4\%$),
meaning fewer sessions exhaust all available turns without
task completion.
In terms of user effort, RAG+Vector requires $193.6$ user
tokens per session on average, a $6.7\%$ reduction compared to
Reflection ($207.5$) and a $16.9\%$ reduction compared to
Vanilla ($232.9$).

The gap between RAG ($52.0\%$) and RAG+Vector ($55.2\%$)
isolates the contribution of the learned user vector:
the $+3.2$ percentage-point improvement comes entirely from
user-vector--modulated retrieval, since all other components
(preference extraction, embedding model, reranker) are
identical.

\paragraph{Comparison with Reflection.}
RAG+Vector matches Reflection in task success while using
substantially fewer user tokens.
To quantify this comparison, we conduct paired statistical
tests across the $60$ user profiles
(Table~\ref{tab:significance}).
RAG+Vector significantly reduces timeout rate
($-2.4$ pp, $p = 0.046$, paired $t$-test) and user effort
($-13.9$ tokens, $p = 0.021$), while the success-rate
difference ($+0.9$ pp) is not statistically significant
($p = 0.276$).
Non-parametric Wilcoxon signed-rank tests yield consistent
conclusions (user effort $p = 0.013$; timeout $p = 0.080$).

\begin{table}[t]
  \centering
  \small
  \caption{
    Paired comparison of RAG+Vector vs.\ Reflection
    across $60$ user profiles.
    We report one-sided paired $t$-tests (H$_1$: RAG+Vector
    is better) and effect sizes (Cohen's $d$).
    $^*$\,$p < 0.05$; $^\dagger$\,$p < 0.10$.
  }
  \label{tab:significance}
  \begin{tabular}{lcccc}
    \toprule
    Metric
    & Reflection
    & RAG+Vec
    & $\Delta$
    & $p$ (paired $t$) \\
    \midrule
    Success (\%)     & 54.4 & 55.2 & $+$0.9  & 0.276 \\
    Timeout (\%)     & 28.8 & 26.4 & $-$2.4  & 0.046$^*$ \\
    User tokens      & 207.5 & 193.6 & $-$13.9 & 0.021$^*$ \\
    Enforce / turn   & 0.175 & 0.186 & $+$0.011 & n.s. \\
    \bottomrule
  \end{tabular}
\end{table}

\paragraph{Enforcement rate.}
RAG+Vector exhibits a slightly higher enforcement-per-turn
rate ($0.186$) than Reflection ($0.175$), though the
difference is not statistically significant.
Decomposing this metric reveals that $79\%$ of the gap is
driven by a small increase in total enforcements per session
($1.54$ vs.\ $1.47$, $p = 0.058$), while the remaining $21\%$
comes from RAG+Vector completing sessions in slightly fewer
turns ($8.31$ vs.\ $8.41$, $p = 0.19$).
Since RAG+Vector achieves the same task success with fewer
turns and lower user token cost, the higher enforcement rate
does not translate into worse interaction quality.

\paragraph{Interaction efficiency.}
Table~\ref{tab:extra-metrics} reports additional metrics
that capture efficiency and long-horizon learning.
RAG+Vector achieves the highest interaction efficiency
across all reported metrics.
In particular, it yields $2.83$ successes per $1{,}000$ user
tokens, an $8.4\%$ improvement over Reflection ($2.61$),
indicating that each unit of user effort is more
productive.

\begin{table}[t]
  \centering
  \small
  \caption{
    Additional efficiency, compliance, and long-horizon metrics
    for the three retrieval-augmented methods.
    $\uparrow$ = higher is better; $\downarrow$ = lower is better.
    Bold indicates best.
  }
  \label{tab:extra-metrics}
  \begin{tabular}{lccc}
    \toprule
    Metric & Reflection & RAG & RAG+Vector \\
    \midrule
    Succ.\ / 1k user tokens $\uparrow$
      & 2.61 & 2.80 & \textbf{2.83} \\
    Late success (sess.\ 30--59) $\uparrow$
      & 51.8\% & 51.7\% & \textbf{54.1\%} \\
    Quick resolution ($\leq$4 turns) $\uparrow$
      & 3.5\% & 3.4\% & \textbf{3.9\%} \\
    Zero-enf.\ success $\uparrow$
      & 60.2\% & 57.1\% & \textbf{60.6\%} \\
    First-turn enforced $\downarrow$
      & 7.9\% & 7.2\% & \textbf{7.1\%} \\
    Profiles improved $\uparrow$
      & 31.6\% & --- & \textbf{36.7\%} \\
    \bottomrule
  \end{tabular}
\end{table}

\paragraph{Late-session and long-horizon behavior.}
Restricting evaluation to sessions $30$--$59$ reveals that
RAG+Vector maintains the highest late-session success rate
($54.1\%$ vs.\ $51.8\%$ for Reflection), though the
difference is not statistically significant ($p = 0.16$).
Across the $60$ user profiles, $36.7\%$ of profiles show
improved success in late sessions compared to early sessions
under RAG+Vector, versus $31.6\%$ under Reflection,
suggesting that more users benefit from cross-session
learning under the user-vector approach.

\paragraph{Zero-enforcement and first-turn compliance.}
RAG+Vector achieves the highest zero-enforcement success
rate ($60.6\%$), meaning that in sessions where the user
never explicitly corrects the agent, RAG+Vector is most
likely to succeed.
It also has the lowest first-turn enforcement rate ($7.1\%$),
indicating that preferences are proactively applied before
the user needs to intervene.
Together, these metrics suggest that the retrieval and
user-vector mechanism surfaces relevant preferences
effectively.

%% ----------------------------------------------------------------
\subsection{User-Vector Representation Analysis}
\label{sec:results-user}

We next ask whether the learned long-term user vectors
$z^{(L)}_u$ capture meaningful structure in the underlying
preference space.

\paragraph{Setup.}
For each pair of users $(u, v)$ we compute
(i)~the Jaccard similarity of their \emph{revealed}
preferences (i.e., preferences that were explicitly enforced
at least once during the $60$ sessions), and
(ii)~the cosine similarity of their learned long-term vectors
$\cos(z^{(L)}_u, z^{(L)}_v)$.
All $60$ users develop non-zero long-term vectors over the
course of training; the mean $\ell_2$ norm grows monotonically
from $0$ (session~$0$) to $0.0033$ (session~$60$).

\paragraph{Preference overlap predicts vector similarity.}
We test whether users with more similar revealed preferences
end up with more similar long-term vectors.
A Spearman rank correlation between Jaccard overlap and
vector cosine similarity yields $\rho = 0.040$
($p = 0.093$).
While the global correlation is modest, a quartile analysis
reveals a clearer signal: user pairs in the top~$25\%$ of
preference similarity have a mean vector cosine of $+0.012$,
whereas pairs in the bottom~$25\%$ have a mean cosine of
$-0.027$ (Mann--Whitney $U$ test, $p = 0.021$).
This confirms that users who share more preferences are
pushed toward more similar regions in vector space by the
REINFORCE updates, consistent with the collaborative-filtering
intuition described in Section~\ref{sec:method-rl}.

\paragraph{Dual-vector separation.}
Table~\ref{tab:vector-analysis} compares the three vector
variants.
The long-term vector $z^{(L)}$ shows a significant
association with long-term preference overlap (quintile test
$p = 0.006$), while the short-term vector $z^{(S)}$ does not
($p = 0.586$).
This is the expected behavior: $z^{(L)}$ accumulates stable
preference information across sessions, whereas $z^{(S)}$ is
designed to capture transient, within-session context and
decays between sessions.
The fact that $z^{(S)}$ does \emph{not} correlate with
long-term preference structure confirms that the dual-vector
design successfully separates stable user identity from
session-specific adaptation.

\begin{table}[t]
  \centering
  \small
  \caption{
    Association between user-vector cosine similarity and
    revealed-preference Jaccard overlap.
    Spearman $\rho$ is computed over all $\binom{60}{2}$
    user pairs; the quintile $p$-value tests whether the
    top-$20\%$ and bottom-$20\%$ preference-overlap groups
    differ in mean vector cosine (Mann--Whitney $U$).
  }
  \label{tab:vector-analysis}
  \begin{tabular}{lcc}
    \toprule
    Vector & Spearman $\rho$ ($p$) & Quintile $p$ \\
    \midrule
    $z^{(L)}$ (long-term)  & 0.040\;(0.093) & \textbf{0.006} \\
    $z^{(S)}$ (short-term) & 0.010\;(0.681) & 0.586 \\
    Combined               & 0.046\;(0.054) & 0.003 \\
    \bottomrule
  \end{tabular}
\end{table}

%% ----------------------------------------------------------------
\subsection{Qualitative Failure Modes}
\label{sec:results-failures}

We observe three recurring failure modes for RAG+Vector,
particularly in later sessions as the preference memory grows:
%
\begin{enumerate}[nosep]
  \item \textbf{Over-cautious clarification.}
    The agent sometimes becomes overly passive, asking
    clarification questions instead of directly solving the
    task, especially when many potentially conflicting
    preferences are retrieved.

  \item \textbf{Preference overload.}
    As more preference cards accumulate, the agent
    occasionally attempts to satisfy all injected preferences
    simultaneously, leading to verbose or unfocused responses
    that fail to address the core task.

  \item \textbf{Early user disengagement.}
    The user simulator occasionally terminates the session
    early (e.g., responding with ``I don't know'') when
    initial responses are weak, preventing the agent from
    recovering in subsequent turns.
\end{enumerate}
%
These failure modes are more pronounced for RAG ($44.3\%$
timeout) than for RAG+Vector ($26.4\%$), suggesting that
the user vector helps prioritize the most relevant
preferences and mitigates preference overload.
We discuss implications and potential mitigations in
Section~\ref{sec:discussion}.
```