Neon glass instruction cards stacked high while a robotic arm lifts one bright card and stray fragments fall away

Quick answer

To improve an AI skill without filling it with one-off rules, stop updating it after every conversation. Instead, every few weeks open a fresh chat, ask the AI to review all the conversations where you used that skill and list the feedback you repeat most often, without changing anything. Pick the points that matter by number, let a “skill editor” skill merge them in neatly, and approve each before-and-after change yourself.

Key takeaways

  • Fixing a skill after every chat overfits it to one-off situations.
  • Review in batches, roughly every three weeks, so patterns show up.
  • Ask for a list first and change nothing until you have chosen.
  • A skill editor skill keeps instructions short and avoids duplicates.
  • Approve exact before-and-after edits, not a vague summary.

📘 Main article: This guide expands on the skill editor tip in our article Personal AI System: How Peter Yang Runs a One-Person Business With AI, from Matt Wolfe’s episode with Peter Yang.

Skills, saved sets of instructions for one job, are the backbone of a personal AI system. The first version is never perfect, so you will want to improve it. How you do that makes a big difference.

In the video, both Matt Wolfe and Peter Yang admit to the obvious habit: after a disappointing result, tell the AI to “update the skill so you do better next time”. Peter explains why that gradually makes skills worse, and shares a better method. Here it is as a step-by-step process.

What does overfitting mean for an AI skill?

Overfitting is when a skill becomes tuned to one particular conversation instead of the job in general. If you fix it after every chat, you end up with a long list of narrow rules: “don’t mention pricing in the second paragraph”, “use a shorter intro for the March episode”. Some contradict each other. The AI gets confused, and results become less predictable.

The fix is to look at many conversations at once. Notes you give again and again are real patterns. Notes you gave once are usually just about that day’s task.

When should you review a skill?

  • Every two to four weeks for skills you use often. Peter suggests looking back about three weeks.
  • After roughly ten uses for skills you use less often.
  • Immediately only if the skill is doing something harmful or wrong, such as sending messages it should only draft.

💡 Tip: Between reviews, simply give feedback in the chat as normal. The AI tool keeps your conversation history, so you can review it later. If your tool does not keep history, jot your corrections in a simple note.

Step 1: Ask for your most repeated feedback

Open a new conversation, so the AI is not anchored to your latest frustration, and paste a prompt like this:

Look at all the conversations from the last 3 weeks where I used my [skill name] skill.
Find the feedback and corrections I give most often.
Don't change anything yet.
Give me a numbered list, most frequent first, with roughly how many times each came up and one example.

Matt ran this on his short-form video skill. The AI went through 13 review rounds across 9 edits and found that his top note, by a wide margin, was B-roll that did not match what he was saying, followed by framing and on-screen text that repeated the subtitles. One of those was already a written rule that kept being ignored, which is useful to know in itself.

Step 2: Choose which points to keep

Read the list and decide which points belong in the skill. Not every repeated note does. Ask yourself:

QuestionAdd it to the skill?
Does it apply to almost every use of the skill?Yes
Was it about one particular client, product or date?No, keep it out
Is it already in the skill but being ignored?Reword it more clearly, or move it nearer the top
Does it contradict an existing rule?Decide which wins, then remove the other

Then reply by number. Matt simply wrote: “For the editing skill, include points 1, 2, 4 and 6.”

Step 3: Use a skill editor to make the change

Peter keeps a separate skill whose only job is maintaining his other skills. It stops each update from bolting on more lines. You can create one in a few minutes:

Name: skill-editor
Purpose: Keep my skills short, clear and consistent.
When I ask you to update a skill:
1. Only add the points I approved by number.
2. Merge them with existing rules instead of adding duplicates.
3. Remove rules that contradict the new ones, and tell me which.
4. Keep the skill as short as possible. Prefer one clear rule over three narrow ones.
5. Before saving, show me a numbered list of the exact before and after text for every change.
6. Wait for my approval.

If your skills are already long and messy, start with a one-off tidy using our Claude skills clean-up guide, then use this batch method to keep them that way.

Step 4: Approve every change line by line

Peter’s strongest habit is never accepting edits blind. Whether it is a skill or a newsletter, he asks for a numbered list of the exact before-and-after text and replies along the lines of “1 to 8 are fine, skip 9”.

⚠️ Warning: Do not let any AI rewrite a skill, a document or anything customer-facing without showing you the exact changes first. A summary like “I tightened the wording” can hide removed rules.

How do you know the skill has improved?

  • You give the same correction less often over the next few weeks.
  • The skill gets shorter or stays the same length, not longer each time.
  • Results are more consistent from one use to the next.
  • Your next review produces a shorter list of repeated feedback.

If one note keeps appearing even after you added it, the rule is probably unclear or buried. Move it near the top and phrase it as a simple instruction.

Frequently asked questions

What is an AI skill?

A saved set of instructions for one repeatable job, such as drafting a newsletter or preparing a guest brief. Claude calls them skills; other tools use custom instructions, projects or custom GPTs.

Does this work in ChatGPT as well as Claude?

Yes, if your tool keeps a searchable conversation history or you keep notes of your feedback. The method is the same: review in batches, choose, then edit.

How long should a skill be?

As short as it can be while still producing good results. If it keeps growing, merge overlapping rules and remove one-off ones.

Is it bad to tell the AI to update a skill straight away?

It is better than never improving it, as Peter says, but it tends to add narrow rules. Batch reviews give cleaner, more reliable skills.

The bottom line

Skills improve best the way people do: by spotting patterns, not reacting to every single mistake. Review a few weeks of feedback at once, choose the points that genuinely matter, let a skill editor merge them neatly and approve every change yourself.

author avatar
Garry Knight
I'm Garry Knight, the person behind Prodify Digital. I write about email list building, email marketing, SEO, AI search and the tools that connect them. My aim is to make online marketing easier to understand, so creators and small business owners can make informed decisions about building an audience and keeping people engaged. Here you'll find straightforward guides and product reviews that explain what something does, where it fits and which limitations matter. The focus is on clear explanations and useful next steps—not hype, shortcuts or promises of easy earnings.

Leave a Reply