Holoalphabetic Sentences: Substack's Latest Purveyor of Truth

All models are wrong, but is this one useful?

I’m pleasantly shocked that this once sleepy platform is now buzzing with conversation about no other than AI—¹ due to Substack’s Co-founder & CEO announcing an AI writing detection feature powered by Pangram²:

Against Claudefishing

While I personally find this fascinating as a data quality nerd, I would be remiss to call out the wider context of such a decision. Who is the arbiter of truth in a world where written “prose” has been commoditized to a series of API calls? Such a question dêlves³ into philosophy and defining ethical AI use, which I feel requires perspectives from multiple disciplines. Thus, I share my views from a data quality lens and hope others can add to the conversation.

What is Pangram?

Thankfully, Substack didn’t try rolling out an internal feature for AI detection (i.e., the “trust us, bro” approach). While this doesn’t preclude scrutiny, having a third-party with open-source research⁴⁵ behind their engine is a step in the right direction. According to Pangram’s article⁶ linked in Substack’s docs:

Pangram is a statistical model that makes predictions based on text alone, so its accuracy must also be tested statistically. That is what we do. We publish internal benchmarks for every Pangram model we release, as well as every time an AI lab updates their LLMs. We have also been tested and verified by independent third parties, such as teams at the University of Chicago and the University of Maryland.

How does Pangram work?

Thus, it comes down to the central question of data science: to what degree can we trust the results of a statistical and or machine learning model, and is this result useful to the business?

What Makes a Model Useful?

In my own data career, I have developed classifiers that made it into customer-facing products or powered critical business workflows. None of them had 100% accuracy.

Accuracy improvements require exponential effort as you get closer to 100%, and thus businesses need to make tradeoffs. For example, say you had a vendetta against Big Toothpaste, and you wanted to validate if “9 out of 10 dentists” truly recommend a particular brand. If you were fully committed to the cause, you would spend substantial amounts of money and resources on surveying every dentist in the world on their recommendations. And then reality sets in… you listen to the statisticians and use sampling methods as a reasonable and sane tradeoff. This is the crux of data quality.

My favorite piece of research in the data quality space, Beyond Accuracy: What Data Quality Means to Data Consumers by Drs. Richard Wang and Diane Strong, defined this as being “…fit for use by data consumers.” To me, this definition from the 90s is perfect, and has been my guiding mental model for evaluating AI products.

Thus, as a consumer of Pangram’s AI detection outputs via Substack’s platform, is the underlying data fit for use? For me personally, it absolutely is, as I’m so desperately tired of the AI drivel masquerading as novel insight on posts and comments throughout the internet. With that said, I recognize I’m being presented information that could be wrong, and thus I use my judgment.

With that said, my curiosity still gets the best of me…

Is Pangram Accurate On This Article?

I’m going to apologize upfront… The second half of this article will be nothing but pure AI slop. BUT FOR SCIENCE!

This section and every section before in this article was written by my own human hands with intermittent spell check from Grammarly. How would the Pangram AI detection model grade this article if the section below was:

1. A summary of the Pangram 3.3 model card written fully by Anthropic’s Fabel 5.

2. Was the same character length as the previous half of the article (3,937 characters).

I personally expect it to either show 50/50 human and AI-generated, or 50% AI-generated and a mix of human-AI-assisted (i.e., Grammarly). I’ll make sure to add an edit after I publish and share a screenshot of the results!

Edit:

---

THE SECTION BELOW IS AI-GENERATED

# Pangram 3.3 Model Card: A Summary

Pangram Labs has released Pangram 3.3, the latest version of its AI-generated-text detector, and the accompanying model card offers an unusually candid look at what changed, what improved, and where the model still struggles. Released on May 13, 2026 as the successor to Pangram 3.2, the model is built on the company’s EditLens architecture, presented at ICLR 2026, which frames detection not as a binary human-or-AI question but as a measure of how much AI intervention a text contains.

## What’s New

Pangram 3.3 was optimized for a lower false negative rate on the newest generation of large language models, including Claude 4.7 and GPT 5.4 and later. Internal evaluations show a 3x improvement in detecting GPT-5.5 Pro text and more than a 4x improvement on Claude Opus 4.7 compared to Pangram 3.2.

The model also makes significant gains against evasion attempts. It catches twice as many commercially humanized texts as its predecessor and shows a 3x improvement on Pangram’s internal adversarial dataset, which consists of LLM outputs where users explicitly prompted the model to evade detection.

Long-form content is another focus. Pangram 3.2 occasionally classified AI-generated documents over 2,000 words as “mixed,” mislabeling later segments as human. Version 3.3 significantly reduces this error, classifying long synthetic texts as fully AI far more consistently.

## Fewer False Positives

Pangram states it will never release a model that improves recall at the cost of misclassifying more human writing. True to that policy, 3.3 reduces the false positive rate on non-native English (ESL) writing while the overall false positive rate actually decreased, thanks partly to improvements in challenging domains like poetry. The card cites a 0.01% false positive rate in common use cases such as creative writing.

## How It Works

The EditLens architecture uses bucket-based classification: a transformer emits logits across levels of AI pervasiveness, decoded into four categories — Human-Written, Lightly AI Assisted, Moderately AI Assisted, and Fully AI-Generated. These collapse into a single ai_assistance_score between 0 and 1, which is now normalized to be more easily interpretable.

Training data and supported languages are unchanged from 3.2: a human corpus of long-form prose spanning essays, creative writing, reviews, books, Wikipedia, news, scientific papers, and web text, with support for 22 languages. The minimum input length remains 50 words, giving the model enough context to make a trustworthy prediction.

## Evaluation Approach

The card distinguishes three evaluation types: in-domain test sets, out-of-domain evaluation on completely held-out sources and domains to measure generalization, and external benchmarks — which Pangram cautions “should not be trusted as a current measure once released, as benchmarks can be trivially trained on.”

## Limitations

Pangram 3.3 is intended for long-form writing samples in complete sentences. Bullet point lists, technical manuals, tables of contents, reference sections, templated writing, and dense mathematical equations are more susceptible to false positives. For best results, Pangram recommends removing human-written instructions, headers, footers, and other extraneous formatting before checking a text. One honest disclosure: 3.3 shows a slight uptick in false positives on human text passed through Google Translate, an issue the team is actively working on.

## Version History

Pangram 3.3.1 (May 15, 2026) kept the same underlying model but improved the segmentation algorithm for documents over 450 words. Pangram 3.3.2 (May 18, 2026) was a small bugfix release affecting fewer than 3% of predictions.

## Bottom Line

Pangram 3.3 is a meaningful step forward in AI text detection: better recall on frontier models, stronger resistance to humanizers, and fewer false positives — all with transparency about its remaining weaknesses.

1. Artisan em dash.

2. To Substack’s benefit, all notes and writing before this announcement are not eligible for AI review.

3. Artisan “delve.”

4. Their 2025 research, EditLens: Quantifying the Extent of AI Editing in Text, where they provide the models and datasets via this GitHub repo.

5. Technical Report on the Pangram AI-Generated Text Classifier

6. For a more technical breakdown, Pangram provides a model card and additional articles.