Sara Ghorayeb

Evals/Area 5 of 5/Hands-on LLM work/P6

Few-shot designs

When does a few-shot prompt earn its cost? Training notes to records, with a rule baseline that the prompt has to beat.

Headline result

22of24

notes are parsed correctly by 30 lines of keyword rules, so that is the bar for the prompt

The result tables
Data
40 training notes, written and labeled for this project
Sample
24 eval notes, 16 notes in the example pool
Model calls
None. The model run is open
Checks in CI
Tests, the baseline, the example selection
Repository
few-shot

The question#

When does a few-shot prompt earn its cost? A prompt with a few labeled examples is worth a model call only if it beats the cheap alternative. So the cheap alternative has to be built and measured first.

I log training in free text: "Turbo 45', recovery" or "Half marathon race, 21.1 km in 1h52". A training load chart needs a record with four fields: sport, minutes, kilometers, intensity.

What I built#

No model has run on this yet. The parts below are the ones that need no model: the labeled data, the baseline, the prompt, the example selection and the scoring.

  • Labeled data. 40 notes with their records: 16 in the example pool and 24 for the eval. No eval note is in the example pool, and a test checks it.
  • A baseline. 30 lines of keywords and patterns.
  • The prompt. It shows k solved examples and then the new note.
  • Four strategies that choose the k examples for each note.
  • A strict scorer. The reply must be one JSON object with exactly four fields and values from fixed sets. Text around the object, an extra field or 45.5 minutes makes the reply invalid, and an invalid reply counts as wrong on every field.
  • The model is a function. evaluate.few_shot takes any function from prompt to reply. Tests pass a stub. The command line takes any shell command that reads the prompt on standard input.

The prompt, with two examples#

$ training-notes prompt "Swam 1.9 km in 38 min, race effort" -k 2
...
Note: 10 km tempo run in 48 min
Record: {"sport": "run", "duration_min": 48, "distance_km": 10.0, "intensity": "moderate"}

Note: Sportive, 100 km in 4h10, hard day
Record: {"sport": "bike", "duration_min": 250, "distance_km": 100.0, "intensity": "hard"}

Note: Swam 1.9 km in 38 min, race effort
Record:

Result#

The rules get 22 of 24 notes right#

So the bar for the model is 22 of 24, and the place where it can win is notes that need reading, not matching.

The baseline on the 24 eval notesA record counts as right when all four fields are right.
Parser Whole record sport duration_min distance_km intensity
Rules, no model 22 of 24 24 of 24 23 of 24 23 of 24 24 of 24
Few-shot model not run
whole record rightwrong
The two notes that the rules get wrongBoth hold two numbers of one kind.
Note What the rules read What is right
"Threshold run, 3 x 10 min, 55 min all in" The first time, 10 The total is 55
"400 m repeats in the pool, hard, 1h" 400 m as the distance 400 m is the length of one repeat

Which examples go in the prompt#

This measure counts how often the prompt shows an example of the right sport. It does not say whether the answer is right.

How often the prompt shows an example of the sport that the note is aboutCounts of the 24 eval notes. The pool has 16 labeled notes, and each strategy chooses k of them for each note. This table needs no model.
Strategy k = 2 k = 4 k = 8
first_k: the first k of the pool 11 of 24 21 of 24 24 of 24
random_k: a seeded sample per note 13 of 24 18 of 24 24 of 24
similar_k: the most shared words 16 of 24 20 of 24 24 of 24
one_per_sport: each sport before a repeat 11 of 24 21 of 24 24 of 24
  • With 2 examples, choosing by shared words shows the right sport most often.
  • Shared words are not shared meaning. For the swim note above, similar_k picks a run and a bike ride, because they share "km", "in" and "min". They show the right format and the wrong sport.
  • With 4, covering the sports does as well as similarity, and the prompt is the same for every note, which a provider can cache.
  • With 8, the choice no longer matters for this measure. The prompt grows from about 970 to about 1,750 characters.

Whether showing the right sport changes the answer is the question for the model run.

Reproduce it#

The commands are the ones in the README. The last one is the model run, which takes any command that reads a prompt on standard input.

python -m pip install -e ".[test]"
python -m pytest -q

training-notes baseline
training-notes selection -k 4
training-notes prompt "Easy hour on the bike" --strategy one_per_sport -k 5
training-notes model --command "<a command that reads a prompt on stdin and prints the reply>" --strategy similar_k -k 4

Limits#

  • No model has run. The page shows the bar that a prompt has to beat, and no evidence yet that a prompt beats it.
  • The notes, the labels and the rules have one source. All 40 notes were written for this project. Notes from other people will be harder, and the baseline will score lower on them.
  • 24 eval notes give wide intervals. One note is 4 points.
  • The schema has one session per note. A note with two sessions, such as a bike ride followed by a run, has no record in this schema.
  • The selection table measures the prompt. It counts examples of the right sport, and it does not measure answers.