Local models: the cost of tokens will matter before security does

There are two camps in AI. Frontier models, from Anthropic, OpenAI and Google, run in the cloud, and everything you ask is sent to them. Local models run on hardware you own, like a Mac mini, an NVIDIA DGX Spark or a Microsoft AI appliance, sitting on your network and never exposed to the internet.

The trade-offs are simple. Frontier models are better and always a little ahead. You pay a subscription and install nothing. Local models have no token limit, so you can use them as much as you like, but you buy and maintain the hardware.

“What will drive the use of local models more than data security is the cost of token usage.”

Most of the conversation about local models is about privacy. Some people believe that sending information to a public model is giving it away, and they like an appliance behind the firewall where the information never leaves the building. In the early ’90s people swore they would never give a credit card to a website. Around 2010 people couldn’t believe we were putting email in the cloud with Office 365.

Those fears faded, and these will too. Data security is a real reason, but cost will move more businesses. Models are stateless, so every turn resends the whole conversation, and token usage grows as the work gets more complex. Agents that run all day can burn through a subscription fast. Once a local model is on your own hardware, every extra question is free.

Tools called harnesses sit between you and the models. They let you choose from a variety of models and surface them in Slack, Teams or Telegram. We run one at Flexo Concepts.