Yarrow Bouchard 🔸

1483 karmaJoined Canadastrangecosmos.substack.com

Bio

Pronouns: she/her or they/them. 

Parody of Stewart Brand’s whole Earth button.

I got interested in effective altruism back before it was called effective altruism, back before Giving What We Can had a website. Later on, I got involved in my university EA group and helped run it for a few years. Now I’m trying to figure out where effective altruism can fit into my life these days and what it means to me.

I write on Substack, and used to write on Medium.

Sequences
2

Criticism of specific accounts of imminent AGI
Skepticism about near-term AGI

Comments
720

Topic contributions
13

Here’s a longer and more specific explanation of why the revenue growth thing makes no sense.

In 2023, Anthropic's annualized revenue reached $100 million. Now, in 2026, Anthropic recently announced its annualized revenue has rapidly grown to $30 billion. This is an increase of 300x over 3 years. If Anthropic were to grow revenue at the same rate for another 3 years, in 2029, it would hit $9 trillion in annualized revenue. After 6 years, in 2032, it would hit $2.7 quadrillion in annualized revenue.

For comparison, gross world product (the combined GDP of all countries) is only $111 trillion. For Anthropic to sustain this growth rate for 6 years, the world economy would need to grow more than 24x. Otherwise, it’s impossible.

Gross world product grew at an average rate of 3.2% per year from 2010 to 2019, according to the UN. It averaged 3.8% per year over the 20 years from around 1993 to 2023, according to the IMF. This growth rate would need to average 54% over the next 6 years for gross world product to grow from $111 trillion now to $2.7 quadrillion in 2032. Obviously, economists are not forecasting this. 

This is another illustration of how extrapolating statistical trends forward indefinitely doesn’t make sense and doesn’t work. Anthropic’s revenue growth and global economic growth are both statistical trends. Extrapolating them both forward leads to a logical contradiction — we get wildly different numbers for gross world product in 2032. We have to make a deeper analytical decision about what we think will happen, rather than just extrapolate trends forward. Just extrapolating trends doesn’t get you anywhere.

Everyone intuitively understands why simple extrapolation would be problematic in many, many, many cases. Jersey Mike’s Subs grew revenue from $2.68 billion in 2022 to $4.2 billion in 2025. That’s an annual growth rate of 15%. If this rate of growth continued, in 30 years its revenue would reach $36.7 trillion, exceeding the current GDP of the United States. Is Jersey Mike’s going to grow larger than the current United States economy in 30 years? Obviously not. These are the sort of absurd results you get if you just extrapolate statistical trends forward indefinitely. 

People have a causal hypothesis for why Anthropic will grow revenue into the quadrillions within a decade: artificial general intelligence! But the flaw is in using Anthropic’s revenue growth rate as evidence for that hypothesis. For instance, let’s say I want to argue Jersey Mike’s Subs will invent artificial general intelligence within the next 30 years. I can point to Jersey Mike’s revenue growth and say, see, if we extrapolate the current trend, Jersey Mike’s revenue will grow larger than the current U.S. economy within 30 years, and larger than the current world economy within 40 years. The only thing that could explain this trend, I argue, is if Jersey Mike’s invents AGI. Therefore, this trend is evidence that Jersey Mike’s will invent AGI.  

Is Jersey Mike’s Subs 15% annual growth rate evidence that it will invent AGI? No! Of course not! Obviously the trend won’t continue that long, and obviously the outcome you get by extrapolating the trend that long won’t be the real outcome. It’s not that AI companies are somehow special in that if you follow statistical trends through their indefinite continuation, you get radical results. You get radical results if you do that for fast food restaurants! This is not a feature of AI companies, it’s a feature of blind extrapolation!

If you believe Anthropic (or OpenAI, or Google DeepMind) will invent AGI within the next decade, and if you want to try to convince skeptics of this — including the many skeptical economists, financial analysts, and professional investors, who have no problem understanding companies’ revenue growth — it will not do to extrapolate revenue growth until it gets to an amount that would be impossible without AGI and then work backwards from that to the conclusion AGI will be invented. Equally possible, and more likely, AGI will just not be invented and revenue won’t grow that much. You have to argue for your causal hypothesis — that Anthropic (or another company) will invent AGI soon — using other kinds of evidence. Extrapolating revenue growth isn’t evidence. Not for Anthropic, and not for Jersey Mike’s. 

Can't because I've received an indefinite, possibly permanent soft ban (severe rate limit of 2 comments per week) since January from the EA Forum mods with a warning that it may escalate to a full ban.

It's not really about something I can change — or at least that I'm willing to change — because it's ultimately an editorial decision by the mods about what kind of conversations, content, and disagreement they want on the EA Forum. In my understanding (which could be wrong), they want the EA Forum to be more of a collaborative space where contributors are uplifted, and not have too much emphasis on debate or criticism. And/or they just fundamentally disagree with what I think and the way I think (e.g., they think that what I take to be the cruxes of disagreements in many cases is not what's really the crux), to such an extent they don't think it's productive to discuss on the EA Forum. So, an editorial decision.[1]

Anyway, for people who do want to engage in these sorts of discussions, Substack is a great place for it. And if not Substack, where else do people have these sort of discussions these days? Uhh, Reddit? 

  1. ^

    (Really long footnote for people who want a longer explanation.) Also, in many cases the users agree with the mods on this, and that's part of why the mods made this decision. A lot of users don't want to engage with too much criticism or debate, and tend to downvote posts or comments that are negative or critical rather than collaborative and uplifting. Which is a perspective I can certainly understand and empathize with. (Although, in my opinion, it also contributes to a filter bubble/echo chamber-type effect.)

    This is all in my understanding and partially me reading between the lines, since I did not fully understand the explanation the mods gave me and found parts of it really confusing. This is me trying my best to reconstruct what they meant by what they said. I could be getting it wrong. I don't really know.

    The mods also mentioned civility as one concern, but I never really got to the bottom of that with them. I got the impression this was a much less important consideration than the other things described above. The mods seem pretty reluctant to moderate for civility on the EA Forum in general, and mostly just let things slide. So, that seemed like a minor concern to them relative to the other stuff. (Again, this is all in my understanding and my attempt to make sense of an explanation I found difficult to follow. So, I could be wrong.)

I love this quote from a TED Talk given by the physicist David Deutsch:

So, in science, two false approaches blight progress. One's well-known: untestable theories. But the more important one is explanationless theories. Whenever you're told that some existing statistical trend will continue but you aren't given a hard-to-vary account of what causes that trend, you're being told a wizard did it.

What’s the casual explanation for why AI revenue is happening? What’s the casual explanation for why it will supposedly continue sustainably, long-term? What theory or hypothesis explains the data, and would justify extrapolating the current statistical trend far into the future?

There are over 50,000 publicly traded companies on the global stock market. The NASDAQ has existed for 55 years; the New York Stock Exchange for 234 years. We have lots of data on stock prices.

Here’s an important fact about stocks: you can’t just look at the price graph for a stock over the last year, or the last three years, extrapolate this price movement forward for the next three years, and buy the stock (or short it) on that basis. You can’t just extrapolate the trend long-term. It doesn’t work.

You need a causal explanation, an investment thesis, about why the stock will go up or down, and by how much. What’s your thesis? And what’s the evidence for that thesis? What’s the analysis behind it?

From what I can tell, the vast majority of economists, professional investors, and financial analysts do not believe that LLMs will lead to AGI or transformative AI within the next decade. Yet they’re well-aware of the revenue growth of AI companies over the last three years. Many professional investors — maybe about half — think AI is in a bubble. What gives? Are they not seeing these graphs? Are they not looking at the graphs hard enough? If the statistical trend of AI revenue growth over the last three years is going to continue for the next five to ten years or more, how could they think this? It’s a real mystery!

If you think trends always continue!

(Much more on this topic here.)

My three most recent posts on Substack are relevant to effective altruism:

I can’t discuss them on the EA Forum, but I’m happy to do so on Substack.

The blog post by the Australian AI safety organization says, “We apply METR’s time-horizon methodology…” How would this address the criticisms raised of METR’s methodology?

At a glance, the FutureTech pre-print makes some interesting choices, e.g., task quality is only scored up to above-average and above-average gets a perfect score, and acknowledges some of the limitations with their methodology, e.g., all tasks used for this experiment must contain all relevant information in the LLM prompt. (Is that realistic for most work tasks?) I wonder if this pre-print will be submitted for publication in a journal? FutureTech seems to be one of those weird MIT hybrids between an academic research group and a management consultancy. I’m not sure if they’ve ever published a peer-reviewed paper. 

[Edit on 2026-05-14 at 18:56 UTC: After reading Peter Slattery’s comment below, I spent a few more minutes looking into it, and I’m still not sure what FutureTech is or what kind of stuff they publish. If someone knows and can explain it, that would be helpful. I could spend more time and get to the bottom of it, but I don’t want to spend more time on it right now.

Please also note the EA Forum team has limited my ability to reply to comments, so I can’t reply further. But if you want to continue the discussion, I’m reachable here.]

Someone could take the time to do a deep dive into the FutureTech pre-print and write a review, but I wonder if that’s a good use of anyone’s time? Is there a reason to think this group publishes high-quality research that is worth getting into?

If someone thinks it’s worthwhile, and they also think the pre-print is unlikely to be submitted for peer review, one option would be to ask the EA organization called The Unjournal to commission a review by an external expert. 

Hi Kouadio. Just want to let you know that your comments don't have paragraph breaks between the paragraphs. Maybe you are copying and pasting from another app and the formatting is getting messed up? I'm just saying this because the text looks like it's all in one big block and that makes it harder to read. I want to make sure you get a fair shot at saying what you want to say, and fixing this formatting issue will make people more likely to read your comments.

The AI revenue growth we've seen so far is compatible with several different explanations, including an AI investment bubble and narrow AI applications that are economically useful but will not lead to AGI anytime soon. Professional investors and financial analysts are generally split between these two camps. Only a small minority believe in near-term AGI.

Some criticisms of the famous METR time horizons graph:

  • As you mentioned, some of the problems and limitations of the METR time horizons graph are sometimes (but not always) clearly disclosed by METR employees, including the CEO of METR. However, note the wide difference between the caveated description of what the graph says and the interpretation of the graph as a strong indicator of rapid, exponential improvement in general AI capabilities.
  • Gary Marcus, a cognitive scientist and AI researcher, and Ernest Davis, a computer scientist and AAAI fellow, co-authored a blog post on the METR graph that looks at how the graph was made and concludes that “attempting to use the graph to make predictions about the capacities of future AI is misguided”.
  • Nathan Witkin, a research writer at NYU Stern’s Tech and Society Lab, published a detailed breakdown of some of the problems with METR’s methodology. He concludes that it’s “impossible to draw meaningful conclusions from METR’s Long Tasks benchmark” and that the METR graph “contains far too many compounding errors to excuse”. Witkin calls out a specific tweet from METR, which presents the METR graph in the broad, uncaveated way that it’s often interpreted by believers in near-term AGI. He calls the tweet “an uncontroversial example of misleading science communication”. In a response to a comment on that post asking how much we should update our views based on the METR graph, Witkin responded, "to be very clear I am in fact claiming that the proper update is zero."

I'm just summarizing the conclusions here, not the substance of the critiques. I recommend that people go and read the critiques to how the authors reach these conclusions.

I guess the point of the expert survey you cited was to explain that it does not support the idea of near-term AGI, right? I was confused because the title and introduction strongly states that the evidence has turned in favour of near-term AGI, but then you say that 2 out of the 4 pieces of evidence you cite do not support the idea of near-term AGI. I think you're just trying to do a general survey of the evidence, both the convincing and unconvincing evidence, right?

I agree that Bio Anchors is also not convincing evidence of anything, for the reasons explained here.

Something I changed my mind about after looking into both the AI Impacts survey and the Forecasting Researching Institute's LEAP survey (as I wrote about here) is that survey results seem to be super sensitive to survey design, even choices in survey design that seem small to the designers, and that they don't anticipate having an impact. I'm not sure these kinds of surveys really matter that much, anyway, but I'm at least more interested in surveys where the designers are careful about these factors that can bias the results. The effects are not small, either. In one case, the result was 750,000 times higher or lower depending on how the question was posed.

Overall, this post is a bit weird because the title and intro make a super strong claim — the tables have turned! — but then the body doesn't cash the cheque that the title and intro write. The new evidence that has turned the tables on AI skepticism is just AI revenue and the METR graph? So, if you agree that the METR graph has been debunked at this point, then it's just AI revenue. And what does AI revenue really show? Can narrow AI not make a lot of money? Are you really prepared to defend that claim? Have at it!

Maybe the claim is something really specific, which is that if you take AI revenue growth over the last 3 years and extrapolate the same rate of growth indefinitely, you end up with some ridiculously large number, and for that number to be true, we would need to have something like AGI. But you can't just take any trend and extrapolate it indefinitely. You need to have some explanation of what's causing the trend and whether it will continue or not. When you step on the accelerator of your car, extrapolating that trend forward indefinitely means you'll eventually exceed the speed of light. But we don't just extrapolate things forward, we think about cause and effect.

You could look at all sorts of industries (like SaaS) or companies (like Tesla) during a few years when growth is super fast, extrapolate that forever, and conclude that one day they will account for 100% of gross world product and take over the entire world economy. But we assume this won't happen because we understand what will prevent this from happening, and we also don't know about anything that would cause it to happen. So, will AI revenue increase until the Singularity happens? That depends on the technology. So, what will happen with the technology? Now we're back to square one! Looking at a chart of AI revenue doesn't settle anything. Will the chart go asymptotic into AI heaven? Or will it level out, or even crash? The answer to that question is not in the chart. It's in the world.

Extrapolation of past trends with no causal explanation of why the trend will continue is not empiricism! It is mysticism! It amounts to saying: we don't know what's happening or why or how, but, somehow, we know what will happen. This is not science. This is not financial analysis. This is not anything.

A facetious graph from The Economist extrapolating when the first 14-bladed razor will arrive:

My own facetious graph:

(Why do you expect this trend not to continue?)

This is a beautifully written comment, and succinct, and funny, and true.

I would give EA much more grace if its self-image was the same as what I presume the Big Garden Birdwatch's self-image is. Part of what gets me tilted out of my mind about the EA community is when people express this almost messianic Chosen Ones self-image — which ties into the pseudo-religious aspect you mentioned.

The high-impact, low-probability logic of existential risk is hypnotically alluring. If a 1 in 1 quintillion chance of reducing existential risk is equivalent to 100 human lives, what does that imply in terms of your moral responsibility when discussing existential risk? If you have things to say that could cast doubt on existential risk arguments, should you self-censor and hold your tongue? If you speak out and you're wrong, it could be the moral equivalent of killing 100 people. Would it be okay to lie? To exaggerate? Why not? Wouldn't you lie or exaggerate to save 100 lives? If the Nazis knocked at your door, wouldn't you lie to save Anne Frank in the attic?

I don't think many people are actually outright lying when it comes to existential risk. But I do think people are self-censoring when it comes to criticism, and I do think people are willing to make excuses for really low-quality products like AI 2027 or 80,000 Hours' video on it because anything that builds momentum for existential risk fear is plausibly extremely high in expected value.

Much could be said in response to this comment. Probably the most direct and succinct response is my post “Unsolved research problems on the road to AGI”.

Largely for the reasons explained in that post, I think AGI is much less than 0.01% likely in the next decade.

Load more