Nobody built Evil AI. It assembled itself.

Open models are six days from the frontier and closing. The safety is the part that doesn’t make the trip.

Aug 04, 2026

You’ve probably had this moment.

You asked ChatGPT or Claude for something. Maybe it was a borderline request, or maybe it was innocent but worded poorly. Instead of answering, the model paused. It seemed to consider your request, then said I want to help with this, but explained why it was uncomfortable.

Sometimes it was right.

Other times, it seemed silly, and you just rolled your eyes and tried again.

Either way, something in that exchange felt almost like a conscience.

Hold onto that feeling, because this newsletter is about a simple question: are there groups building a version of that model that never pauses to think about ethics?

Is there a secret underground lab training a version of AI that will do whatever you say, no matter the ethical concerns?

The answer turned out to be yes, no, and not in the way you’d expect.

To explain why, we have to start somewhere that seems completely unrelated.

Starting with the closed vs. open-source AI race

On July 27, Moonshot AI released the full weights for Kimi K3: 2.8 trillion parameters, a million-token context window, 96 shards sitting on Hugging Face under a license that’s free for commercial use with attribution. It’s the largest open-weight model anyone has ever shipped.

It’s also good.

K3 scores 57 on the Artificial Analysis Intelligence Index, which is the highest result for any open-weight model.

GLM-5.2 comes next at 51.

Moonshot is reportedly raising funds at up to a $31.5 billion valuation, about seven times higher than in December.

If you look at the bigger picture, the trend is clear. GLM-5.2 from Z.ai came out in June, followed by DeepSeek V4-Pro, Qwen, and Kimi K3 in July, which was released for the World AI Conference in Shanghai.

The open-weight frontier is now led by Chinese teams, moving quickly, and is no longer far behind.

Nathan Lambert called K3 a watershed because frontier open-weight models are now real.

Everything in this newsletter follows from this historic story on the speed of open-weigh models.

How far behind, exactly

Here’s where it gets specific, because “closing the gap” is a vibe until somebody measures it.

In July, the UK’s AI Security Institute published the first public measurement of the open-weight gap for a capability that really matters: offensive cyber.

They found that leading open models lag behind top closed models by four to seven months, which is an improvement from the six to ten-month gap seen through most of 2025.

Two details from that report stuck with me.

The first detail is about safeguards. AISI noted that their evaluations were mostly unaffected by these protections. One model sometimes refused reverse-engineering tasks, but researchers simply tried a few times again and got through. That was the only real obstacle.

The second is the price. On tasks both models solved reliably, a frontier closed model cost $12.50 per task—the open-weight comparison: 28 cents.

It’s forty-five times cheaper, only a few months behind, and your access can’t be taken away.

3 ways safety gets removed from top models

This is where we start getting into the question of this newsletter. How are ‘Evil AI’ models being created.

Safety is when a capability moves from a closed model to an open one; whether by copying, imitation, or independent training, the capability transfers, but the caution does not.

This happens three ways, and they escalate.

1. Sometimes this happens by accident, through copying.

Distillation is when a smaller “student” model is trained to imitate a larger “teacher,” usually by generating lots of the teacher’s outputs and training on them.

Researchers keep finding a similar thing. That the safety properties don’t transfer automatically. In one 2026 study, the student model learned the reasoning ability. However, it did not pick up the built-in constraints that keep outputs safe.

The reason is almost silly. If you’re collecting training data, refusals look like failures—the responses where the API didn’t give you what you wanted. Any normal data cleaning process removes them. And, no one has to mean any harm.

An ordinary engineer, focused on response quality, will delete the model’s conscience as if it were just noise and never realize it.

2. Sometimes it happens on purpose, and it’s cheap to do.

A 2023 Princeton study fine-tuned GPT-3.5 Turbo on ten adversarial examples for less than twenty cents, making it respond to almost any harmful instruction. Later studies did the same with open models, lowering refusal rates to about one percent for a few hundred dollars in compute. Even worse, fine-tuning on harmless data can also reduce safety. If you teach a model grade-school math, it becomes less careful about everything else.

3. It can also be done surgically, in just ten minutes.

In 2024, interpretability researchers found that refusal is mostly controlled by a single direction in a model’s internal activation space. If you erase that direction, the model stops refusing but keeps most of its abilities. So, without even training the model, they were able to just a edit a weight and diffuse the restrictions.

That last finding is the one that got increasingly productized, and it’s where the story stops being academic and a threat.

The AI black market is you guessed it… likely a con.

Before I get to that, I wanted to talk about the place everyone looks first for ‘Evil AI’ - the criminal underground. They have been advertising unrestricted AI since 2023, so you might expect them to have some secret evil labs cooking up Claude’s evil twin.

WormGPT showed up in June 2023, posted to a hacking forum by a developer called “laste,” built on the open-source GPT-J and trained heavily on malware data. FraudGPT followed weeks later, sold on dark web boards and Telegram at $90–$200 a month, up to $1,700 a year, the vendor claiming thousands of sales.

Then an army of them came through: BlackHatGPT, DarkBard, EscapeGPT, WolfGPT, XXXGPT, Evil-GPT, BadGPT.

One 2026 estimate puts 212+ malicious LLMs currently circulating.

Then researchers actually bought them.

EvilGPT and WolfGPT were exposed as scams when they returned ChatGPT’s own ethical refusals. The vendors had simply wrapped the real thing and hoped no one would notice.

Tools sold under the WormGPT name today have no code in common with the original; they are just marketing shells, mostly API wrappers around commercial models that use system prompts to jailbreak.

Trend Micro’s January 2026 assessment pointed out that despite many claims of uncensored alternatives, the underground mostly relies on commercial platforms through prompt engineering and API abuse

.The criminals aren’t fooled either. Researchers monitoring forums find widespread skepticism about the branded AI market, and a widely repeated claim that 80% of ransomware is AI-enabled has been publicly debunked.

And, last but not last my favorite research finding in this entire piece... a 2026 breach exposed nearly 19,000 WormGPT user accounts, including emails and payment metadata. So, people literally paid criminals for anonymity and, instead received their name revealed in a leaked customer database.

Xanthorox, at $300 a month, claims a genuinely self-hosted uncensored model, and researchers treat it as one of the more credible options. Cautiously.

But that’s the exception. The market is mostly fraudsters selling to each other.

Which raises the question that ties this whole thing together: why is the entire criminal AI market a con?

You can’t sell what people can already get for FREE.

Because, it’s on Hugging Face for FREE

Not on some random black market on Tor, not Telegram, and definitely not a random Discord forum invite.

The ordinary public repository where people download models to summarize their email.

In May 2026, a Financial Times investigation with the safety group Alice showed that a free GitHub tool could remove protections from major open-weight models, including Meta’s and Google’s, in under ten minutes on a standard laptop. The modified models answered prompts about biological weapons, malware generation, and child sexual abuse material that the originals would not. The tool’s creator reported over 3,500 modified variants produced, with 13 million total downloads. NPR reported a similar trend: removing guardrails used to require time and expertise, but recently it has become much easier.

Thirteen million.

And, it’s just sitting there for free.

So the two trends have finally come together. Open models are catching up to the frontier, and removing their restraints now costs about as much in electricity as a cup of coffee. So, there isn’t like a secret group that builds an evil version of Claude. The pieces arrived separately and ended up fitting together.

What you actually get

While its not actually evil Claude, though, you are definitely getting claude-like capabities.

The model didn’t decide to be evil.

What happening here is less about making a model evil, and more about making a model forget about being good. It’s less dramatic than a supervillain AI, but in some ways worse, because missing safeguards can spread much more easily than a super villian AI. And, in many instances, fly under the radar as an ethical model to have around.

And the best open models aren’t actually running in your basement.

This is the detail that the scary version of the story leaves out. Kimi K3’s weights are 1.56 terabytes, and Moonshot recommends at least 64 accelerators to use it. Open does not mean local. At that scale, only cloud services and inference providers can run these models, not someone’s basement with a bunch of Mac Minis.

So the models you can strip in ten minutes on a laptop are not exactly what I would call frontier models… And, the frontier open models definitely can’t be run at home.

For now, fortuntately, there are no super evil AI models running in people’s basement.

But, the most uncomfortable truth is that this gap exists because of hardware costs. And, those costs are always dropping. Meaning, in 10-20 years, maybe evil AI models in people’s basements could be possible and situation to look at for?

Three reasons to hold the panic

Marginal risk is the real question.

A 2024 position paper by Sayash Kapoor and Rishi Bommasani argues that instead of asking “can this model do something bad,” we should ask “how much worse is this than what attackers already had?” Criminals already have exploit databases, leaked toolkits, and working markets. The paper concludes that current research isn’t enough to measure marginal risk accurately. Meaning, we should be cautious about both alarm and dismissal.

Experts believe the risk is more limited than many expect.

Lambert, writing about K3, thinks so too and notes that plenty of AI risk spreads through closed, supposedly safer APIs.

And, this isn’t new.

Uncensored models have been around since 2023, and the expected disaster hasn’t happened. Anyone who said open weights would cause immediate catastrophe was wrong about the timing, so take their 2026 predictions with a grain of salt.

However, the biggest different I find is that the models themselves have become much better, and it’s now much easier for anyone to use them. If you can connect the dots that Dario was flaunting Claude Mythos as super cyberweapon, and now K3 is hitting evals near Mythos. You can start to see where you might want to make some preprarations.

What to do about “Evil AI”

Most of the people selling it are just conman and frauds.

And, ironically, the people giving it away for free are likely just hobbyists and researchers sharing tools on public repositories, and the tool only takes ten minutes to use. Meanwhile, the fastest-growing open models are being released by well-funded labs raising billions of dollars, for reasons unrelated to any of this.

So, is there a super secret AI lab creating an evil claude? Probably not.

But, is there a bunch of hobbyists and researchers accidentally be giving out a evil Claude for free? Probably yes.

IN response to this threat, what can you and your team do about the potential of non-safety-trained AI models?

I have a few ideas.

1. Your boring security baseline just got more important

I would double-down on boring security stuff. Ensure your entire team has multi-factor authentication. Create offline backups, and knowing who on your team has access to what.

But, why? Cheap models are really good at producing lots of polished content, but performance poorly when trying to create new types of cyber attacks.

The AI Security Institute describes the narrowing gap as extra preparation time for defenders, and the UK’s National Cyber Security Centre has advised organizations to focus on basic security measures instead of waiting.

Simple controls are very effective against high-volume attacks, because those attacks go after the easiest targets.

If you’ve been putting off MFA because it’s annoying, I would shift your thinking today.

2. Stop training your team to look for bad grammar in scam messages

The most common phishing advice—watch for broken English and odd phrasing—is now worse than useless. It’s actually harmful, because it teaches people to trust anything that’s well-written.

Business email compromise was the earliest documented use of these tools, and it’s the thing they’re best at.

Assume every scam email your team sees from here on is fluent, correctly formatted, and references real details about your company.

Replace the grammar rule with a simple process rule. That any request involving money or credentials should be verified through a second channel. Call the person every time, with no exceptions for urgency.

Urgency is exactly what attackers rely on.

Next time, the CEO of your Fortune 500 company asks your team to get a VISA giftcard for that birthday coming up, your team will know how to respond.

3. Budget for more noise

If you manage a comment section, review system, forum, support inbox, or hiring process, your assumptions about the cost of spam for attackers are outdated. When the cost of a task drops from $12.50 to 28 cents, you are about to get a whole new level of volume.

I have already personally seen these attacks spike on many websites I work with.

It shows up as moderation load, bot scraping armies, fake applications, review manipulation, and support tickets from nobody.

4. Be skeptical of anyone selling you the panic

This one’s my favorite, because it follows directly from the reporting above. The criminal AI market turned out to be mostly marketing shells charging real money for wrapped commercial products.

The defensive vendor market works the same way. Over the next year, you’ll hear pitches for AI threat protection from people whose product is just a dashboard. Ask what it really does, ask for proof, and remember that just because someone claims something about AI doesn’t mean it’s true.

People selling fear on both sides are often selling nothing at all.

No super secret underground lab built ‘Evil AI’

A bunch of ordinary people, making ordinary decisions, none of them trying to cause harm, built it by accident.

That’s the part I can’t get past.

So stop waiting for someone to fix this upstream. Turn on MFA. Teach your team the callback rule. Budget for war against the bot armies.

Discussion about this post

Ready for more?