Skills and rules can be ignored by AI, create a control

So let me share a snippet of a conversation that was about my expectation that central comprehensive exception handling would always be part of my projects.

AI: I recommend going with Option A
Me: Isn't Option B the best practice?
AI: You are right. B is best practice. I undersold it. And it is cheaper than I said. Here is the real cost, measured.
Me: Question do I have a current rule that addresses this and points to the best practice? I am just curious if the rule is missing, was not considered when writing code, weak or needs to be changed
AI: Yes. You have the rule. It is clear. It was not followed.

A skill only runs when the model judges that it applies. That judgment sometimes fails and multiple passes do not solve the problem. Ask your AI Harness to create a route and the result may not include the error handling with stack information that you would add automatically.

Early in the year I added 35 rules to my system targeted at Python code. I noticed months ago that I was still seeing misses, so in annoyance I asked AI to just scan my rules folder and write a code scanning tool I run with a command called /rules-audit, plus a set of smaller gates that run as hooks. The only goal of the tool is to provide a report of possible rule misses. It worked. Code quality went up. It does not escape my notice that I asked the very tool that was not adhering to rules to write a code scanning tool. There was testing to see if it would flag known problematic code snippets, but I have no efficacy numbers.

So why change now? I recently started focusing on improving code quality for myself and I would like to apply this as part of an AI SDLC. I am also expanding the work I am doing and adding Java/Spring Boot as well as Python. I thought this was a good time to rethink my implementation as I create code quality rules specific to Spring Boot. The goal is moving as much as possible to Static Code Analysis (SCA) tools and determining what needs to stay custom.

Below is the estimate of the rule categories and what tools can be used. The percentages are per numbered statement inside those rules, roughly 350 of them, not per rule file. I will not know the final numbers until this effort is completed.

Category: Not checkable
Percentage: ~52%
Why: These rules ask whether a choice was correct, not whether a pattern is present. Was 409 the right status code for this failure? Does this default value quietly invent a business fact instead of falling back? Does this test check the layer the bug lives in? A parser can read the code, but it cannot read the intent behind it.

Category: SAST / pattern engines (Semgrep, plus ruff, PMD, Checkstyle where they ship the rule)
Percentage: ~31%
Why: Most rules come down to a shape that must be present or must be absent, and that is exactly what a pattern engine matches. Semgrep also reads configuration files, so rules about YAML and properties files land here rather than becoming custom code.

Category: Other build tooling (Error Prone, NullAway, pyright, JaCoCo, PIT, test suites)
Percentage: ~8%
Why: These rules need something a pattern cannot supply: a threshold to measure against, type information from the compiler, or the program running. They belong to neither pattern matching nor architecture, and a three-way split quietly loses them, so they need their own row for the numbers to add up.

Category: Architecture as Code (ArchUnit, import-linter)
Percentage: ~5%
Why: These rules govern how the pieces are allowed to relate: which layers may call which, which dependencies are forbidden, where an annotation belongs. The share is small but the value is not, because this is the category that catches code which is perfectly correct on its own and wrong in the place it was put.

Category: Custom
Percentage: ~4%
Why: What remains are the checks that have to compute something or compare two things. Is the running container older than the source it was built from? Has this suppression passed its expiry date? Do these two code paths arrive at the same quantity? Matching a shape is not enough, so these stay as code.

I have already hit my first challenge. To run an SCA tool as part of my AI Harness, it needs to answer in under a second. I was able to port my rule that database, API or shell commands should log the action, so if it hangs or fails in production there is a record. The problem was that Semgrep ran the same check 44 times slower than the native Python version, 1,320 ms against 30 ms. That is on Windows, where Semgrep has no native build and has to run in Docker. So the test is not just whether it is possible, but whether it will work as part of an AI Harness. I am purposely shifting this analysis left to the developer so the error gets caught at the time of creation. Nightly repository scans still make sense for general quality, and they are where you layer in other checks like security audits.