How keeping LLM responses short can boost your AI plans

As anyone who’s ever tried to ask an over-caffeinated coworker about their weekend knows, a simple, open-ended question can sometimes lead to the most unexpectedly lengthy responses.

And when the respondent is a large language model (LLM), lengthy responses can translate into serious budget issues, especially if an IT team is querying their AI tools multiple times per day.

For example, a software developer might ask their LLM of choice to help build an internal chat app from scratch, which could lead to excessively long outputs as the AI attempts to solve several massive architectural challenges. And since most AI tools charge by the token, epic-length outputs (and inputs) can quickly strain an AI budget.

(A token is the smallest data processed by a large language model. One token corresponds to roughly 4 English-language text characters, according to OpenAI, or roughly 100 tokens is equal to 75 words. GitHub in April announced metered, per-token pricing; ChatGPT’s API usage is charged by the number of tokens used; and Anthropic’s models have token-based cost structures.)

As AI costs rise, we spoke with IT pros on how they’re keeping outputs in line.

Cave us. As reported by 404 Media, developers at companies like OpenAI, Nvidia, and GitHub are using output-reduction tools—including a “Caveman plugin” designed to restrict large language model (LLM) responses to bare essentials—to address high costs and high token usage. For example, during a programming conversation, it might return an output like, “Authenticate API. Include API key in Authorization header.”

Julius Brussee, the plugin’s creator, told 404 that he made Caveman in early April: “I was using Claude Code heavily and noticed a lot of my token spend was going to unnecessary prose: pleasantries, hedging, transitions, and chatty language that does not really matter inside an agent loop.”

At his AI engineering firm Krazimo, CTO and co-founder Mridul Nagpal suggested that “how brief the output should be” is a regular design decision for his company, which ships customized LLMs to clients.

A list of rules can steer an LLM to desired responses. In addition to the caveman technique, Nagpal also prompts models to follow simplicity guidelines inspired by researcher Andrej Karpathy’s observations on LLM coding pitfalls and overcomplication.

Another skill called “Grill Me” acts as a relentless interviewer and scope definer; a bot that clearly understands an objective has a better chance of success when you omit irrelevant details. The skill guidelines, Nagpal said, can be hooked into IDEs and enforced on coding platforms like Codex, Claude Code, or Cursor.

Be brief. When Brady Lewis, senior director of AI innovation at marketing consultancy Marketri, first gave his team AI tools, many hit their usage limits quickly—too quickly. Lewis has built output-restricting automations for tasks that just require data and not lengthy explanations, he said, like a regular, repetitive report.

For a task like delivering a website, however, Lewis wants the extra information and suggestions.

“If it’s a critical task, I would be much less concerned about limiting my input and output, and more concerned with making sure I was giving it enough context and getting the output that contains all of the critical details that I need,” he said.

Input/output. Guy Bourgault, head of agentic services at tech-services company Concentrix, believes a good, focused output begins with a good, focused input. Tools like Prompt Cowboy, for example, offer AI input suggestions to “avoid generic or robotic AI output that’s blah,” according to Prompt Cowboy’s FAQ page, and such functions can be integrated into enterprise tools.

In addition to his hooks, Nagpal also applies automated restrictions for certain open-ended queries. For example, if an analyst at an insurance company pastes a full claim into an AI assistant and asks, “Does this violate any agreements?” Nagpal’s team aims to understand the need behind that question and, if needed, rebuild that interaction as an automation. Maybe an automation extracts key fields from a claim into a structured template and gathers applicable, preconfigured policies and regulations, under strict token and length limits; that way, you don’t get a bot dissertation for a yes/no question.

“We literally have guardrails that [say], ‘By the end of this step, the token count cannot be more than 2,000 tokens,’ for example,” Nagpal said.

Lewis, however, advises IT pros to not just blindly cut inputs and outputs. Classification tasks can have short outputs, he thinks, but employers should be freer on the reins when it comes to strategic questions.

You want to give the AI the opportunity to provide suggestions, he said.

“If you run everything through that sort of setup, where you’re not giving any flexibility to the AI tools, you’re going to get very deterministic, same-as-everybody-else [responses],” Lewis told us.