Analysis
The Analysis menu opens a hub of analyses. The entries are always listed (so every method is discoverable, even before the survey has the matching question type): Contingent valuation, Discrete Choice Analysis, Interactive Dashboard, MaxDiff Analysis, Q-Model Analysis and Attention Sensitivity. Contingent valuation, Discrete Choice Analysis and Interactive Dashboard are summarised below; their full documentation lives in the corresponding subsection of Section 9 (the same screens are also reachable from there). MaxDiff Analysis turns best/worst choices into a ranking and an aggregate logit (see the MaxDiff Analysis screen in Section 3.34); Q-Model Analysis opens the Q methodology dashboard documented in Section 3.21. If a survey has no question of the relevant type, opening that analysis just shows a short notice; Q-Model lists the survey's Q-Model questions and opens the chosen one's dashboard (with a single Q-Model question it opens directly).
10.1 Contingent valuation¶
tickStat's contingent valuation model. Estimates the demand curve and willingness-to-pay from a binary choice question (yes/no at a randomized price), reporting optimal price, mean WTP and maximum expected revenue. See Section 9.6 for the parent Choice Analysis when the experiment is multi-attribute rather than a single price prompt.
10.3 Interactive Dashboard¶
Cross-filtering charts and tables built from the survey responses. See Section 9.8.
10.5 Attention Sensitivity¶
The Attention Index is produced by a commercial large language model, so a natural question — and one reviewers ask — is whether the index depends on the specific model version used. Attention Sensitivity answers it empirically: it replays a sample of the survey's existing attention ratings under a different LLM configuration and compares the new ratings against the stored baseline.
Nothing about the original data is altered. The screen reads the stored ratings and writes its results to separate tables, so the dataset behind a published analysis stays exactly as it was.
Baseline panel. Shows the configuration that produced this survey's ratings and splits the existing ratings by reasoning regime.
The configuration shown is the survey's own, not the platform's current setting. Each survey's LLM configuration — model, temperature, Top-P, Top-K, seed and region — is frozen the first time the survey is rated, and every later rating of that survey uses it. This means a platform update, or a change to a shared setting, cannot alter the instrument measuring a survey that is already in the field; and months later you can still say exactly what produced a given dataset, which is what a control run has to reproduce. The panel notes whether the configuration was captured live or reconstructed for a survey rated before this was recorded, and if more than one model appears it says so, because ratings from different models are not strictly comparable.
Freezing does not restrict experimentation: it only sets the starting values of the New run form, so a control run is one click away and correct by construction. Everything in the form remains editable.
The New run form is pre-filled with the survey's own baseline, so queueing a control run takes one click. Modern Gemini models may perform internal reasoning before answering; those reasoning tokens are billed as output but are not part of the visible response. Because providers can enable or change this behaviour without the model name changing, a survey fielded across such a change ends up with two regimes in the same dataset. The panel makes that visible: number of ratings, share, mean rating and date range for each regime.
New run. Set a label and the configuration to test — model, temperature, Top-P, Top-K, seed and a Reasoning setting.
Reasoning deserves a word, because it is the least obvious control. Before answering, a modern model can write itself a draft: intermediate text you never see, but which is billed as output at the output rate. There are three choices. On lets the model decide how much to spend, which is the normal behaviour. Off skips that step entirely — the model answers directly, and no reasoning tokens are produced at all, not merely hidden. The ceilings let it reason but no further than a set number of tokens, and exist only to cap cost. The panel shows how many reasoning tokens the survey actually averaged, so a ceiling can be chosen against a real figure instead of guessed.
The Model list offers the text models that can perform the rating task, grouped by role: the baseline itself, newer models, lighter and cheaper ones, and a higher-quality preview option. Image and video generation models are deliberately excluded — they cannot rate attention. Because providers release models continually, an Other option lets you type any model id directly, so the screen never becomes a bottleneck when something newer appears.
Then choose the sample:
| Sample strategy | What it draws |
|---|---|
| Stratified by rating | Proportional across rating values, so the extremes are represented rather than swamped by the most common scores |
| Only "no reasoning" records | Ratings produced before the provider enabled reasoning |
| Only "reasoning" records | Ratings produced after it |
| Random across all | A plain random sample |
| Complete respondents (every screen), stratified by index | Every rated screen of N complete respondents (status Done, with a stored index), drawn proportionally across attention-index bands (below 6, 6–7, 7–8, 8–9, 9 and above, at least one per band) with the sample seed. Here the sample size counts respondents, not screens; once the sample is drawn the run shows both. |
The sample seed makes the draw reproducible: the same seed always selects the same records. It defaults to the survey's own seed, so every run on a survey replays the same records unless you change it — which means two configurations are compared over an identical sample rather than over two different draws. Change it only when you deliberately want a fresh sample, for instance to check that a result holds on different records.
Always run the control first. A run using the baseline configuration must reproduce the ratings already stored. If it does not, the harness itself is introducing variation and no later comparison means anything. Start with a sample of 10 to confirm the pipeline before committing to a larger run.
Runs. Each run is queued and processed in the background, one at a time, so it never competes with live rating of incoming responses. The table shows each run's number (the stable identifier to cite in a paper or report) and its design read left to right: Origin records — which stored baseline ratings the sample was drawn from (rated with reasoning, rated without reasoning, or all ratings stratified/random) — followed by Replayed with, the reasoning setting they were re-rated under, then the sampling parameters, progress and status. Click a finished run to see its comparison.
A queued or running job can be stopped with its Cancel button. Cancelling a job that has not started costs nothing. Cancelling one that is already running stops it at the next record boundary — the call already in flight still completes, so at most one further rating is paid for — and everything it had replayed up to that point is kept and can still be compared. While it is stopping, the status reads Cancelling…; it becomes Cancelled once the job has actually finished, and only then does the next queued job start.
Who can launch and cancel runs. Queueing and cancelling are restricted to tickStat staff, because every run calls the LLM provider at a real cost and draws on the same quota as the live rating of incoming responses. Researchers see the whole screen — baseline configuration, reasoning regimes, the runs table and every comparison — but without the New run form. If you need a sensitivity run for your study, ask us and we will queue it.
Comparison. Reports exact agreement, mean absolute difference, Spearman rank correlation (the index is ordinal, so a rank measure is the appropriate one), the mean rating before and after, the token counts of the baseline against the replay (input, visible output and reasoning, reported in tokens rather than money because prices differ by model and change over time), and a confusion matrix of baseline rating against new rating — which reveals changes in the shape of the distribution that a difference of means would hide.
Respondent level. For a run drawn with Complete respondents, a second block reports the comparison at the level a study actually uses: each respondent's index (the mean of their replayed screens) against the stored one. It shows the Spearman and Pearson correlations between the two indices, the mean and maximum change per respondent, the low-attention share before and after, and — the figure that decides — cut-off crossings: how many respondents change side of the exclusion cut (default 7, editable) when their whole session is rated again. This is the test–retest reliability of the classification, which the screen-level strategies cannot give: one rating may move by a point and the respondent's mean of thirty barely move at all. Respondents with any failed replay are left out of the figures and counted separately.
The figure that should drive the decision is flag changes: with a configurable low-attention threshold, how many respondents cross the exclusion cut-off. A rating drifting from 8.1 to 8.2 changes nothing in practice; one that crosses the threshold changes who is excluded from your analysis. Two configurations can differ noticeably in mean rating and still classify every respondent identically — in which case they are interchangeable for your purposes.
A note on changing configuration mid-study. Any change of model or reasoning setting alters the measuring instrument. Ratings produced under different configurations are not strictly comparable, so decide the configuration at the start of a study, validate it once, and keep it fixed for the whole fieldwork.
Discrete Choice Analysis¶
Discrete Choice Analysis has its own dedicated page: see Discrete Choice Analysis for the full documentation.
Discrete Choice Simulator¶
Discrete Choice Simulator has its own dedicated page: see Discrete Choice Simulator for the full documentation.