All learning paths Learn See Try Make Measure it before you change it What this lesson is about Write down how it does on fifty of your own examples. That number is the only thing your change can be judged against.
You are about to reword the instruction you give a small model so its Hindi replies improve. What comes first?
A Reword it, read five replies, and write the score down once it is clearly better. B Write down how it does on fifty of your own examples before you change it. C Score it on the fifty messages you used to write the instruction, since those are ready.
Check my recall Why this matters Hugging Face's guidance warns against testing on the same examples used to build the thing. The score looks great and means nothing. A number with no date and no version compares with nothing.
You can measure a model on your own examples and record the run so a later one compares.
Hugging Face's guidance warns against testing on the same examples used to build the thing. The score looks great and means nothing. A number with no date and no version compares with nothing.
Write down how it does on fifty of your own examples before you change it. Include the hard cases you expect it to fail, or the score flatters it. Record the date, the model version and the inputs beside the score. Task You are about to change the instruction you give a small model, to improve its Hindi replies.
Weak approach Changing the instruction, reading five replies, and deciding the Hindi is better now. Nobody wrote down what the old instruction scored, so better has nothing to stand against.
Stronger approach Scoring fifty of your own messages first, writing the date and version, and only then changing one thing.
Why the stronger approach works Without the first score, better is a feeling. With it, the change is a number you can show somebody.
Try a changed situation A friend wants to prove her new tuition method works. She plans to test it on the same ten questions she taught from. What do you tell her?
Make something yourself Make a first measurement of one model. Write fifty of your own inputs, the answer you expected, and what it gave. Add your score, the date and the version.
Lesson 4 of 8 on Open models on a small budget next Decide when tuning is worth it Part 1 ends in a work sample you can send The rules this work sample is read against
Read the primary or official source: Hugging Face Evaluate · considerations for model evaluation