Evals/Area 5 of 5/Hands-on LLM work/P6
Few-shot designs
When does a few-shot prompt earn its cost? Training notes to records, with a rule baseline that the prompt has to beat.
Headline result
22of24
notes are parsed correctly by 30 lines of keyword rules, so that is the bar for the prompt
The result tables- Data
- 40 training notes, written and labeled for this project
- Sample
- 24 eval notes, 16 notes in the example pool
- Model calls
- None. The model run is open
- Checks in CI
- Tests, the baseline, the example selection
- Repository
- few-shot
The question#
When does a few-shot prompt earn its cost? A prompt with a few labeled examples is worth a model call only if it beats the cheap alternative. So the cheap alternative has to be built and measured first.
I log training in free text: "Turbo 45', recovery" or "Half marathon race, 21.1 km in 1h52". A training load chart needs a record with four fields: sport, minutes, kilometers, intensity.
What I built#
No model has run on this yet. The parts below are the ones that need no model: the labeled data, the baseline, the prompt, the example selection and the scoring.
- Labeled data. 40 notes with their records: 16 in the example pool and 24 for the eval. No eval note is in the example pool, and a test checks it.
- A baseline. 30 lines of keywords and patterns.
- The prompt. It shows k solved examples and then the new note.
- Four strategies that choose the k examples for each note.
- A strict scorer. The reply must be one JSON object with exactly four fields and values from fixed sets. Text around the object, an extra field or
45.5minutes makes the reply invalid, and an invalid reply counts as wrong on every field. - The model is a function.
evaluate.few_shottakes any function from prompt to reply. Tests pass a stub. The command line takes any shell command that reads the prompt on standard input.
The prompt, with two examples#
$ training-notes prompt "Swam 1.9 km in 38 min, race effort" -k 2
...
Note: 10 km tempo run in 48 min
Record: {"sport": "run", "duration_min": 48, "distance_km": 10.0, "intensity": "moderate"}
Note: Sportive, 100 km in 4h10, hard day
Record: {"sport": "bike", "duration_min": 250, "distance_km": 100.0, "intensity": "hard"}
Note: Swam 1.9 km in 38 min, race effort
Record:
Result#
The rules get 22 of 24 notes right#
So the bar for the model is 22 of 24, and the place where it can win is notes that need reading, not matching.
| Parser | Whole record | sport | duration_min | distance_km | intensity |
|---|---|---|---|---|---|
| Rules, no model | 22 of 24 | 24 of 24 | 23 of 24 | 23 of 24 | 24 of 24 |
| Few-shot model | not run |
| Note | What the rules read | What is right |
|---|---|---|
| "Threshold run, 3 x 10 min, 55 min all in" | The first time, 10 | The total is 55 |
| "400 m repeats in the pool, hard, 1h" | 400 m as the distance | 400 m is the length of one repeat |
Which examples go in the prompt#
This measure counts how often the prompt shows an example of the right sport. It does not say whether the answer is right.
| Strategy | k = 2 | k = 4 | k = 8 |
|---|---|---|---|
first_k: the first k of the pool |
|||
random_k: a seeded sample per note |
|||
similar_k: the most shared words |
|||
one_per_sport: each sport before a repeat |
- With 2 examples, choosing by shared words shows the right sport most often.
- Shared words are not shared meaning. For the swim note above,
similar_kpicks a run and a bike ride, because they share "km", "in" and "min". They show the right format and the wrong sport. - With 4, covering the sports does as well as similarity, and the prompt is the same for every note, which a provider can cache.
- With 8, the choice no longer matters for this measure. The prompt grows from about 970 to about 1,750 characters.
Whether showing the right sport changes the answer is the question for the model run.
Reproduce it#
The commands are the ones in the README. The last one is the model run, which takes any command that reads a prompt on standard input.
python -m pip install -e ".[test]"
python -m pytest -q
training-notes baseline
training-notes selection -k 4
training-notes prompt "Easy hour on the bike" --strategy one_per_sport -k 5
training-notes model --command "<a command that reads a prompt on stdin and prints the reply>" --strategy similar_k -k 4Limits#
- No model has run. The page shows the bar that a prompt has to beat, and no evidence yet that a prompt beats it.
- The notes, the labels and the rules have one source. All 40 notes were written for this project. Notes from other people will be harder, and the baseline will score lower on them.
- 24 eval notes give wide intervals. One note is 4 points.
- The schema has one session per note. A note with two sessions, such as a bike ride followed by a run, has no record in this schema.
- The selection table measures the prompt. It counts examples of the right sport, and it does not measure answers.