What we measure, and what we still need to learn

What the tasks measure (§03)

Applied AI literacy means using AI to do useful work while keeping your own judgment. Foray tests four skills:

Each track has limits. Following NIST AI 800-2, we state what each score measures. T2 directly measures accuracy at spotting differences. T3 directly tests whether you catch planted errors, but its rubric score is an indirect measure of reasoning. T1 and T4 use work quality as a proxy, or indirect measure, of creative skill. Human comparisons are the reference for those tracks, not model scores. The combined score is not yet validated as a predictor of real-world outcomes. Testing that needs long-term follow-up, planned for 2028.

What the early results can tell us (§09)

The 2026 calibration group had just 45 people. It helped develop questions, not certify anyone. We use the Rasch model to estimate question difficulty and flag questions that fit poorly. Person-ability logits are not reported as scores. More complex models, called 2PL and 3PL, need larger samples. We defer pass thresholds and competence certification until the pooled sample exceeds 250. Until then, percentiles and bands describe relative performance, not certified competence.

• Shared questions link results. The design includes 18 published AICOS-SV questions, tested on 514 adults. They take about eight minutes and provide an outside comparison group.
• The combined score uses a bell-shaped scale. Ranks become percentiles, then scores with a mean of 50 and a standard deviation of 15. This does not mean raw performance follows a bell curve. Exports keep the raw distribution separately.
• Agreement between judges needs checking. The plan uses Krippendorff's alpha for rubric scores, with .80 as the target, and ICC with 95% confidence intervals to measure agreement. Separate judging panels are compared for paired work. T3 also uses weighted kappa, a measure of rating agreement, against human judges' agreement.
• Experts set the planned pass standard. Modified Angoff asks 8 to 10 experts, over two rounds, what a minimally competent person should achieve. The early sample is too small for methods that need stable estimates of each question's difficulty or comparisons between groups of test takers.
• Comparisons across years need stable questions. The design keeps a secure set of questions to link annual versions. Once set, the competence standard stays fixed, though the raw score needed may change. Comparing results on public and secure questions helps check whether prior exposure is affecting scores.

How judging is checked (§10)

Model judges can make mistakes. The judging plan follows NIST AI 800-2, Practices for Automated Benchmark Evaluations. It calls for exact model and prompt versions, results for each question, and uncertainty estimates. Reports must separate what the evidence shows from what we infer.

Published studies show these risks. The figures below are not measurements of Foray judges:

The planned judging process:

1. Fix the rules. Save the rubric and prompts as a version. Any change creates a new version.
2. Use several judges. Three model families follow a detailed rubric, explain ratings, and compare work in both orders.
3. Check the evidence. Judges must quote the work. A quote that cannot be verified invalidates the judgment.
4. Compare with humans. Use human-rated examples to adjust model ratings. A published essay study improved rating agreement from about 0.5 to 0.71 using roughly 200 examples; this is not a Foray result.
5. Account for errors. Adjust estimates for measured judge errors and report confidence intervals, rather than treating raw model ratings as truth.
6. Ask humans when needed. Send disagreements and uncertain cases to reviewers. Publish the referral rules and use known-error examples to check reviewer attention.

Before the first scored cohort, the plan requires published tests of attempts to manipulate the judges, including attack success rates, and tests of how formatting changes ratings. These requirements are not evidence that the checks have already passed.

What is stored, and who calls the model

On the hosted site, the Foray service stores your activity during a run, your answers, and any site you publish. If you connect a model, the service holds its key against your account. Your browser receives only a 12-character fingerprint to identify the connection, never the key.

Tasks that support offline practice can use a simulator and label it on screen. A simulator is not a live model, and its results are not an official score.

The GitHub Pages demo has no Foray exam service or account sign-in. You can use the capped shared model or connect a model running on your own machine. Inputs sent to a connected model leave this page, so do not include private information.

How scores can be checked later (§14)

Changing next year's rules must not erase this year's results. The goal is simple: recompute the same score from the same stored inputs. The release design has four rules:

• Release fixed versions. The design calls for signed instrument packages loaded by a hash of their contents, not a changeable name.
• Changed questions get new IDs. Each ID is a hash of the question content. Editing a question creates a new item, so changes cannot silently affect comparisons.
• Save the judging instructions. The prompt is part of rubric_version. Changing it creates a new version.
• Keep translations with each question. Record how they were translated. Comparable scores across languages need evidence.

score() uses no network, clock, or randomness. It calculates a score from stored answers and judgments. Re-scoring is reproducible; re-judging is not. A model may give a different judgment when asked again, even at temperature 0. We save its original judgment as an input rather than call it again to check a score. Automated tests block common sources of changing results and compare scores with fixed examples. These tests are not a sandbox or a proof of purity. They can miss references captured earlier, file imports, or environment reads. New scores link to the old records through superseded_by, without overwriting the history.

There is also a runtime limit. Tests verify byte-identical results on the same JavaScript runtime, not across engine versions. Score records do not yet store that version. Some text-processing rules can change with the engine. Recording the runtime and replaying old scores on it is still needed for a stronger long-term guarantee.

You can run checks of the scoring code and content IDs in your browser on the validation page. They check code consistency, not whether a score predicts real-world skill.