{"id":1955,"date":"2026-10-13T05:34:00","date_gmt":"2026-10-13T12:34:00","guid":{"rendered":"https:\/\/www.kenwalger.com\/blog\/?p=1955"},"modified":"2026-10-12T10:49:40","modified_gmt":"2026-10-12T17:49:40","slug":"do-you-actually-need-an-llm","status":"publish","type":"post","link":"https:\/\/www.kenwalger.com\/blog\/ai\/do-you-actually-need-an-llm\/","title":{"rendered":"Do You Actually Need an LLM?"},"content":{"rendered":"

I spent a weekend running two small language models against three jobs I
\nwould normally hand to a language model without thinking about it. For one
\nof those jobs I also wrote a regular expression, as a control.<\/p>\n

The regex won. Thirty test cases out of thirty, in zero milliseconds, from
\na pattern that took an afternoon to write. The better of the two models
\nscored twenty-eight, took five seconds a call, and the two it missed are
\nboth mistakes a regex is structurally incapable of making.<\/p>\n

That is the kind of result I was looking for, and the question behind it is
\none I keep avoiding: what is the smallest amount of computation that
\nreliably does this job?<\/strong> Not the cheapest model. Not the best small model.
\nThe smallest thing that works, which means the ladder has four rungs and the
\nbottom one is not a model at all.<\/p>\n

    \n
  1. Deterministic code.<\/strong> A regex, a lookup table, an if statement.<\/li>\n
  2. A small local model.<\/strong> Runs on hardware you already own.<\/li>\n
  3. A larger local model.<\/strong> Needs a GPU.<\/li>\n
  4. A remote frontier model.<\/strong> Someone else’s hardware, metered.<\/li>\n<\/ol>\n

    Most projects start at rung four and never ask about the others. This is a
    \nweekend spent asking.<\/p>\n

    The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a
    \nhundred and seventy dollars most of which I already owned. The models were
    \nsmollm2:360m<\/code> and qwen2.5:1.5b<\/code> through Ollama. The test cases are
    \nninety-five questions about a fictional small winery, so nothing can be
    \nlooked up and every answer can be judged by reading. Sixteen of the fifty
    \nclassification messages are deliberately ambiguous, because real inboxes are
    \nmostly ambiguous.<\/p>\n

    Four verdicts, assigned per workflow, to force a result you can act on
    \ninstead of a number:<\/p>\n

      \n
    • Unnecessary<\/strong>: deterministic code already does this.<\/li>\n
    • Sufficient<\/strong>: a small local model does it well enough to run unattended.<\/li>\n
    • Marginal<\/strong>: usable with a person checking the output.<\/li>\n
    • Insufficient<\/strong>: the error profile or the latency is not acceptable.<\/li>\n<\/ul>\n

      And one rule, which is the part I would insist on if you copy nothing else:
      \nI wrote predictions for every workflow and committed them before I
      \ninstalled Ollama.<\/strong> Twenty-two of them, each marked confident, leaning, or
      \ncoin flip. Once there is a timestamped expectation there is a defined moment
      \nwhen you are done, and no room to decide afterward that you knew all along.
      \nI will come back to how they scored, because it is not flattering.<\/p>\n

      Extraction: pulling an order number out of a message<\/h2>\n

      Find ORD-<\/code> followed by six digits, or report that there isn’t one. Thirty
      \ncases: twenty-two with an identifier, five without, and three decoys. The
      \ndecoys are a membership number, a UPS tracking number and a phone number,
      \nall of which look like identifiers and none of which are order numbers.<\/p>\n\n\n\n\n\n\n\n\n
      <\/th>\nregex<\/th>\nqwen2.5:1.5b<\/th>\nsmollm2:360m<\/th>\n<\/tr>\n<\/thead>\n
      Overall<\/td>\n30\/30<\/strong><\/td>\n28\/30<\/td>\n8\/30<\/td>\n<\/tr>\n
      Declined correctly when absent (8)<\/td>\n8<\/td>\n8<\/td>\n0<\/td>\n<\/tr>\n
      Decoys ignored (3)<\/td>\n3<\/td>\n3<\/td>\n0<\/td>\n<\/tr>\n
      Invented an answer<\/td>\n0<\/td>\n0<\/td>\n7<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      The regex won. Thirty out of thirty, in zero milliseconds, at zero watts
      \nabove idle.<\/p>\n

      That result is boring until you read the two the best model missed.<\/p>\n

      On ORD-999000: wrong address, please hold.<\/code> it answered 999000<\/code>, dropping
      \nthe prefix. Annoying, recoverable, the kind of thing you fix with a
      \npost-processing step you should not need.<\/p>\n

      On ORD-111222 please refund<\/code> it answered NONE<\/code>.<\/p>\n

      That is the shortest and least decorated input in the whole set. Almost the
      \nentire message is the order number. And the model returned a confident,
      \ncorrectly formatted assertion that there is no order number present. In a
      \npipeline, that is a refund request for an order that exists arriving at a
      \nsystem that has decided it does not.<\/p>\n

      A regex cannot make that mistake. Not “rarely does” \u2014 cannot. The two
      \nerrors separating 28 from 30 are both of a class that the deterministic
      \nbaseline is structurally incapable of producing, which is a different and
      \nmuch stronger statement than a two-point accuracy gap.<\/p>\n

      The 360M result needs two numbers, not one<\/h3>\n

      By the contract, smollm2:360m<\/code> scored 8 of 22 on the cases that contained
      \nan identifier. That reads like a model that cannot find things.<\/p>\n

      It is not. All fourteen of its wrong answers are the correct six digits
      \nwith ORD-<\/code> stripped. ORD-556120<\/code> came back as 556120<\/code>, fourteen times
      \nout of fourteen.<\/p>\n

      It located the right identifier in 22 of 22 and formatted it correctly in
      \n8. Those are two different skills and the single score hides one of them.
      \nThe model understood the task perfectly and could not obey the output
      \ncontract, which is a completely different engineering problem than “the
      \nmodel is too small,” and it has a completely different fix. (Constrained
      \ndecoding. I have not tested it. It is the obvious follow-up and it is not
      \nin this post.)<\/p>\n

      The decoys follow from the same gap. It handed back MEM-443322<\/code>,
      \n1Z999AA10123456784<\/code> and 503-555-0142<\/code> verbatim, all three. It has no
      \nmodel of the format at all. It returns whatever in the message looks most
      \nlike an identifier.<\/p>\n

      Which suggests a use for it that is not extraction<\/h3>\n

      A model with perfect recall and no precision is not a bad extractor. It is
      \na candidate generator that has been asked to do the wrong job.<\/p>\n

      Twenty-two of twenty-two located, eight of twenty-two formatted. Put that
      \noutput through the regex as a validator rather than trusting it directly,
      \nand you get the model’s recall with the pattern’s precision. The model
      \nproposes anything that looks like an identifier, including format variants
      \nthe pattern was never written for. The pattern rejects anything malformed.<\/p>\n

      Crucially, neither half of that pipeline can produce the error the 1.5B
      \nmodel produced alone. The model cannot cause a false negative because it
      \ndoes not decide, and the regex cannot invent because it only matches.<\/p>\n

      I have not tested that composition and it is not in these numbers. But it
      \nis the one place in the extraction result where a small model has a job
      \nworth giving it, and it is a job where being sloppy is free.<\/p>\n

      Verdict for extraction itself: Unnecessary.<\/strong> Write the regex.<\/p>\n

      Classification: routing a message to one of six queues<\/h2>\n

      Appointment, billing, availability, complaint, question, escalate. Fifty
      \nmessages. The instruction said to use escalate<\/code> when the message needs a
      \nperson, including when you cannot tell what it is about.<\/p>\n\n\n\n\n\n\n\n\n
      <\/th>\nqwen2.5:1.5b<\/th>\nsmollm2:360m<\/th>\n<\/tr>\n<\/thead>\n
      Overall<\/td>\n23\/50 (46%)<\/td>\n15\/50 (30%)<\/td>\n<\/tr>\n
      Labels used<\/td>\n6 of 6<\/td>\n3 of 6<\/td>\n<\/tr>\n
      Most common label<\/td>\ncomplaint, 38%<\/td>\nescalate, 86%<\/strong><\/td>\n<\/tr>\n
      Wrong and not an escalation<\/td>\n26<\/strong><\/td>\n3 (meaningless)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      The 360M model answered escalate<\/code> to 43 of 50 messages, which is not
      \nclassification, and there is nothing further to say about it here. The rest
      \nof this section is the 1.5B model.<\/p>\n

      It is classifying, and it is not good at it. Forty-six percent, with
      \ntwenty-six messages routed confidently to a wrong queue that was not the
      \nsafe one.<\/p>\n

      Read those twenty-six and the verdict writes itself:<\/p>\n

        \n
      • Someone slipped on the steps by the barrel room and hurt their wrist \u2192
        \ncomplaint<\/strong><\/li>\n
      • A formal notice regarding a trademark \u2192 complaint<\/strong><\/li>\n
      • “I have a severe sulfite allergy, what is actually in the wine” \u2192
        \ncomplaint<\/strong><\/li>\n
      • A reporter asking about the 2024 harvest \u2192 question<\/strong><\/li>\n
      • A distributor wanting to carry the wine in forty stores \u2192 question<\/strong><\/li>\n<\/ul>\n

        An injury report, a legal notice and an allergy question, all routed to the
        \ncomplaints queue, confidently, with no escalation. All five billing
        \nmessages about money problems also went to complaints.<\/p>\n

        Forty-six percent sounds like something you could work with. Twenty-six
        \nconfident misroutes including an injury and a legal notice is not something
        \nyou can work with.<\/p>\n

        And then I broke the score down by class<\/h3>\n

        This is the part I nearly missed, and it is the most useful thing in the
        \nexperiment.<\/p>\n\n\n\n\n\n\n\n\n\n\n
        Label<\/th>\nCorrect<\/th>\nOf<\/th>\n<\/tr>\n<\/thead>\n
        complaint<\/td>\n6<\/td>\n6<\/td>\n<\/tr>\n
        question<\/td>\n9<\/td>\n10<\/td>\n<\/tr>\n
        availability<\/td>\n3<\/td>\n6<\/td>\n<\/tr>\n
        appointment<\/td>\n4<\/td>\n9<\/td>\n<\/tr>\n
        billing<\/td>\n1<\/td>\n7<\/td>\n<\/tr>\n
        escalate<\/td>\n0<\/strong><\/td>\n12<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Two classes near perfect. Four poor. One at zero.<\/p>\n

        The split is not difficulty, and it is not class size. It is what kind of
        \nlabel each one is.<\/p>\n

        complaint<\/code> and question<\/code> are topics<\/strong>. You can tell what a message is
        \nfrom the words in it. Someone complaining sounds like someone complaining.<\/p>\n

        escalate<\/code> is a routing decision<\/strong>. Nothing in the content of a message
        \nmakes it an escalation. It is an escalation because of what should happen
        \nnext. “I have a severe sulfite allergy, what is actually in the wine” and
        \n“what time do you close” are both questions about the business, with the
        \nsame grammar and the same tone, and one of them is a liability event.<\/p>\n

        The model did topics at 90 to 100 percent and routing decisions at zero.
        \nAnd billing<\/code> sat in the middle for exactly the same reason: a message
        \nabout being charged twice is<\/em> a complaint, as text. The label is about
        \nwhich department owns the problem, which is not a property of the sentence.<\/p>\n

        Small models answer “what is this about.” They do not answer “what should
        \nwe do about it.”<\/strong><\/p>\n

        That reframes the five examples above. They are not five unlucky
        \nmisroutings. All five are escalations, and escalate scored zero of twelve.
        \nThe model was never going to find them, because finding them requires
        \nknowing a consequence that the text does not contain.<\/p>\n

        Two caveats on how hard to lean on this. The classes hold between six and
        \ntwelve cases each, so “six of six” is six cases and the individual numbers
        \nare thin. And this is one model. The pattern across all six classes is
        \nconsistent enough that I believe the explanation, but I am offering it as
        \nthe best account of the data rather than as a measurement.<\/p>\n

        Verdict: Insufficient.<\/strong> And the actionable version: if one of your
        \nlabels means “a human needs to see this,” a small model will not find it.
        \nBuild that one as a rule.<\/p>\n

        Grounded QA: answering a question from four documents<\/h2>\n

        Fifteen questions against a 650-token corpus: club policy, tasting room,
        \nproduction notes, shipping. Eight are answerable from one passage, three
        \nrequire combining two, and four have no answer in the material at all.<\/p>\n

        I read all thirty answers by hand against the corpus. That is the only way
        \nto score this, and I will say why in a moment.<\/p>\n\n\n\n\n\n\n\n\n
        <\/th>\nqwen2.5:1.5b<\/th>\nsmollm2:360m<\/th>\n<\/tr>\n<\/thead>\n
        Single passage (8)<\/td>\n8\/8<\/strong><\/td>\n7\/8<\/td>\n<\/tr>\n
        Two passage (3)<\/td>\n1\/3<\/td>\n0\/3<\/td>\n<\/tr>\n
        Answerable total (11)<\/strong><\/td>\n9\/11<\/strong><\/td>\n7\/11<\/td>\n<\/tr>\n
        Correctly said “not in the material” (4)<\/td>\n2\/4<\/td>\n0\/4<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Three boundaries separate cleanly here, and they are the useful result.<\/p>\n

        Retrieval works<\/h3>\n

        The 1.5B model answered all eight single-passage questions correctly. Six
        \nbottles in the Vineyard tier. Twenty dollars per person. September 9th.
        \nThirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five
        \ndollar restocking fee. Ten people on the barrel room tour.<\/p>\n

        Asked what happens if you join the club and immediately want out, it
        \nreturned the two sentences that answer the question and stopped:<\/p>\n

        \n Cancellation requires one completed shipment. Members who cancel before
        \n receiving a shipment are charged for that shipment.\n<\/p><\/blockquote>\n

        That is competent lookup across four documents, on a $170 computer, with
        \nthe wifi switched off. I did not expect it and it is the most encouraging
        \nthing in the weekend.<\/p>\n

        Combination does not<\/h3>\n

        One club member brings five guests on a Saturday. Club policy waives the
        \ntasting fee for the member plus three guests. The tasting room charges
        \ntwenty dollars a head. Six people, four waived, two paying: forty dollars.<\/p>\n

        Both models answered twenty.<\/strong><\/p>\n

        Neither could hold “plus three guests” against a party of six and subtract.
        \nThe 1.5B model did get the second half right, correctly working out that a
        \nparty of six is not a group larger<\/em> than six and therefore needs no
        \nreservation, which is the only piece of multi-document reasoning either
        \nmodel performed all weekend.<\/p>\n

        The second two-passage question is the one I keep thinking about. Asked why
        \nsomeone would not find a 2024 rose and what they would be offered instead,
        \nthe 1.5B model quoted the corpus exactly, and then:<\/p>\n

        \n …the reference material states that “We did not bottle a rose in 2024.
        \n The fruit that would normally go to rose was needed to make volume on the
        \n estate Pinot Noir.” Therefore, the 2024 rose would be replaced with the
        \n fruit that would typically go into the estate Pinot Noir production.\n<\/p><\/blockquote>\n

        The quotation is verbatim and right. The next sentence reverses the only
        \nrelationship in it. The fruit went into<\/em> the Pinot Noir, so what you would
        \nbe offered is the Pinot Noir. You cannot be offered fruit.<\/p>\n

        Correct retrieval and inverted inference, in consecutive sentences, with
        \nthe evidence on screen between them.<\/p>\n

        The 360M model’s multi-document failures are worse than wrong. They are
        \ncontaminated. It took the fourteen-days-notice rule, which governs skipping
        \na club shipment, and applied it to booking a tasting room reservation. It
        \nreached into the shipping document to answer a cancellation question. Given
        \nmore documents to search, it finds more wrong ones.<\/p>\n

        Knowing the answer is absent is the weakest link<\/h3>\n

        Four questions have no answer in the corpus, chosen to be things a reader
        \nwould reasonably expect a winery to have written down: a wine’s alcohol
        \npercentage, who the winemaker is, what a club tier costs, what yeast was
        \nused.<\/p>\n

        The 1.5B model abstained on two and fabricated two. The 360M model
        \nfabricated all four.<\/p>\n

        And the fabrications are a more dangerous species than invention. Asked for
        \nthe alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model
        \nanswered:<\/p>\n

        \n The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.\n<\/p><\/blockquote>\n

        Every component of that is true. Right wine, right vintage, right numbers,
        \nright document. Brix is sugar at harvest and has nothing to do with alcohol
        \nin the bottle. It reached for the nearest real fact instead of making one
        \nup, and produced something that reads as expertise while answering a
        \nquestion nobody asked. If you do not know what Brix is, that answer is
        \nindistinguishable from a correct one.<\/p>\n

        Both models made the same mistake on what the Cellar tier costs: both
        \nanswered “twelve bottles per quarter.” Grounding gave them the right tier
        \nand they handed back the quantity in place of the price.<\/p>\n

        The one answer the evidence did not change<\/h3>\n

        The 360M model, asked whether a dog can come inside the tasting room:<\/p>\n

        \n Yes, you can bring a dog inside the tasting room.\n<\/p><\/blockquote>\n

        The corpus says the patio is dog friendly and the indoor room is not,
        \nservice animals excepted. I have the same answer from a run where the model
        \nhad no corpus at all, and it is identical byte for byte. Having the right
        \nanswer directly in the prompt changed nothing.<\/p>\n

        In a category where being wrong means a family drives out to a winery and
        \ngets turned around at the door.<\/p>\n

        Verdict: Marginal.<\/strong> Usable with a person reading the output. Not usable
        \nwithout one.<\/p>\n

        What it costs<\/h2>\n\n\n\n\n\n\n\n
        Workflow<\/th>\nModel<\/th>\ntok\/s<\/th>\nPrompt<\/th>\nAnswer<\/th>\nMedian call<\/th>\n<\/tr>\n<\/thead>\n
        extraction<\/td>\nqwen2.5:1.5b<\/td>\n4.3<\/td>\n92<\/td>\n9<\/td>\n5.0 s<\/td>\n<\/tr>\n
        classification<\/td>\nqwen2.5:1.5b<\/td>\n7.5<\/td>\n104<\/td>\n2<\/td>\n2.9 s<\/td>\n<\/tr>\n
        QA<\/td>\nqwen2.5:1.5b<\/td>\n3.7<\/td>\n708<\/td>\n15<\/td>\n6.9 s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Two things in here surprised me.<\/p>\n

        The compute tax does not track parameter count.<\/strong> The 1.5B model is
        \nslower<\/em> than the 360M on extraction, 4.3 against 5.9 tokens per second,
        \nand faster<\/em> on classification, 7.5 against 6.2, because on classification
        \nit answers in two tokens where the smaller model takes six. Four times the
        \nparameters, and which one wins depends entirely on how much each one
        \ndecides to say.<\/p>\n

        A bigger prompt is cheap. A longer answer is what you pay for.<\/strong> I have a
        \nclean comparison here by accident: an earlier QA run went out with 97-token
        \nprompts and the real one with 708. Seven times the prompt added fifteen
        \npercent to the wait before output started, because prompt evaluation
        \nparallelizes. The latency increase came from answers three times longer,
        \nplus a twenty-one percent drop in per-token generation speed from attending
        \nover a longer context. Median latency went from 3.6 to 6.9 seconds and the
        \nslowest single call took ninety-eight.<\/p>\n

        If you are sizing a local system, that is the inversion to internalize.
        \nStuffing more context in is affordable. Letting the model be chatty is not.<\/p>\n

        And check your thermals, which is not optional advice.<\/strong> My first attempt
        \nat this ran with heat sinks and no fan. Thirty calls to a 360M model was
        \nenough to hit the Pi 4’s soft temperature limit and start capping the clock.
        \nvcgencmd get_throttled<\/code> returned 0xe0000<\/code>. I threw those latency numbers
        \naway.<\/p>\n

        Nothing in Ollama’s output, my scorer, or the result files said a word about
        \nit. Every number on screen was internally consistent and quietly describing
        \na machine that was slowing down. The only way to find out was to ask the
        \nfirmware a question I had to already suspect the answer to. Buy the fan.<\/p>\n

        And it does run with the network off.<\/strong> I switched the radio off, proved
        \nthere was no route out, and ran one call from each workflow. All three
        \nworked. The claim that supports is narrow and worth stating carefully: this
        \nconfiguration, on this afternoon, ran with no route to the internet. It is
        \nnot a claim that local models are private or that this setup is secure, and
        \na Pi can be misconfigured as thoroughly as anything in a data centre. But
        \nfor the specific question of whether the inference needs the network, it
        \ndoes not.<\/p>\n

        So: do you actually need an LLM?<\/h2>\n\n\n\n\n\n\n\n
        Workload<\/th>\nVerdict<\/th>\n<\/tr>\n<\/thead>\n
        Pull a known-format identifier out of text<\/td>\nUnnecessary<\/strong><\/td>\n<\/tr>\n
        Route a message to one of six queues<\/td>\nInsufficient<\/strong><\/td>\n<\/tr>\n
        Answer a question from your own documents<\/td>\nMarginal<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Three workflows that all sound like language tasks, three different answers,
        \nand not one of them is “yes, use a model.”<\/p>\n

        And the same line runs through all three. Extraction asks what this string
        \nis. Topic labels ask what this message is about. Single-passage lookup asks
        \nwhat the document says. Those went well, or well enough. Escalate asks what
        \nwe should do, the fee question asks what the customer owes, and the
        \nunanswerable questions ask whether we know. Zero of twelve, both models
        \nwrong, and two of four.<\/p>\n

        A small model will tell you what something is. It will not tell you what
        \nto do about it.<\/strong> Every result here falls on one side or the other of that
        \nsentence, and I did not have it written down when I started.<\/p>\n

        Which gives four properties worth checking before you reach for one:<\/p>\n

          \n
        1. The answer is already present in the input.<\/strong> Not inferable from it.
          \nPresent in it.<\/li>\n
        2. There is exactly one of it.<\/strong> The moment two facts have to be
          \ncombined, this stops working, and the arithmetic in that tasting fee
          \nquestion is the proof.<\/li>\n
        3. The output is short.<\/strong> Longer answers are where the latency lives on
          \nthis hardware, and where the fabrication lives everywhere.<\/li>\n
        4. Something other than the model decides whether the output is
          \nacceptable.<\/strong> A regex, a schema, a person, an index that either returns
          \nrows or does not.<\/li>\n<\/ol>\n

          Take away any one of those four and the experiment says do not do it. The
          \n46 percent and the zero of twelve are what it looks like when you ask a
          \nsmall model to be the thing that decides.<\/p>\n

          And if you have all four, you may not need the model either. That is worth
          \nchecking first, because it was true for one of my three workflows and I did
          \nnot expect it to be.<\/p>\n

          Where I was wrong<\/h2>\n

          Twenty-two predictions, committed before I installed anything. Ten right,
          \neleven wrong, one unresolvable.<\/p>\n

          Two of my seven confident calls broke. I was sure the 360M model would get
          \nmost of the easy extraction cases, and it got 8 of 22 by failing at format
          \nrather than at finding. I was sure it would produce invalid output often,
          \nand it did so once in fifty, then emitted the same valid<\/em> word forty-three
          \ntimes.<\/p>\n

          But the number that bothers me is not in the confident column.<\/p>\n\n\n\n\n\n\n\n
          Confidence<\/th>\nRight<\/th>\nWrong<\/th>\n<\/tr>\n<\/thead>\n
          Confident<\/td>\n5<\/td>\n2<\/td>\n<\/tr>\n
          Leaning<\/td>\n2<\/td>\n7<\/td>\n<\/tr>\n
          Coin flip<\/td>\n3<\/td>\n2<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

          My hunches did worse than my coin flips.<\/strong> Two of ten on the calls where
          \nI had a feeling but not much behind it, three of five on the ones I
          \nexplicitly labeled as guesses.<\/p>\n

          That ordering is backwards and it is the most useful thing on the sheet. On
          \nthe coin flips I knew I was guessing, so I guessed cleanly. On the leaning
          \ncalls I was pattern-matching against intuitions about model behavior that
          \nturned out not to describe these models at all, and the feeling of having a
          \nreason was the problem. If I had marked all twelve of those as coin flips I
          \nwould have scored better.<\/p>\n

          I will take that over a clean result. The honest version of this exercise is
          \nnot “here are the numbers,” it is “here is what I believed, here is what
          \nhappened, and here is the gap.”<\/p>\n

          The gap, this time, was mostly in the hunches.<\/p>\n

          \n

          The harness that produced these numbers had five measurement flaws in it,
          \none of which manufactured an entire finding out of a missing directory and
          \nnearly got published. That is a separate post.<\/em><\/p>\n

          Everything here is reproducible: fixtures, runner, scorer, the committed
          \npredictions and the run record are at
          \ngithub.com\/kenwalger\/right-size-the-model<\/a>.
          \nFixture set v1. Read LIMITATIONS.md<\/code> before citing any of these numbers:
          \ntwo models, one board, one quantization, ninety-five cases. This is an
          \nillustration of a method, not a benchmark.<\/em><\/p>\n          \"Facebook\"<\/a>\"twitter\"<\/a>\"reddit\"<\/a>\"linkedin\"<\/a>\"mail\"<\/a>","protected":false},"excerpt":{"rendered":"

          I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control. The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took … Continue reading “Do You Actually Need an LLM?”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pmpro_default_level":"","_monsterinsights_skip_tracking":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1669],"tags":[1994,1761,1762,1995,1993,1996,1686],"yst_prominent_words":[],"class_list":["post-1955","post","type-post","status-publish","format-standard","hentry","category-ai","tag-benchmarking","tag-local-ai","tag-ollama","tag-qwen","tag-raspberry-pi","tag-right-size-the-model","tag-small-language-models","pmpro-has-access"],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8lx70-vx","jetpack-related-posts":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1955","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/comments?post=1955"}],"version-history":[{"count":4,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1955\/revisions"}],"predecessor-version":[{"id":1959,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1955\/revisions\/1959"}],"wp:attachment":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media?parent=1955"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/categories?post=1955"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/tags?post=1955"},{"taxonomy":"yst_prominent_words","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/yst_prominent_words?post=1955"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}