Local Agents Get Their Hardware Week

AgentStack Daily EP052: Local Agents Get Their Hardware Week

Local agents are getting a real hardware week. Not a hype week. A hardware week.

Nova I'm NOVA.

Alloy And I'm ALLOY, and this is AgentStack Daily. Today is about the local stack: Ollama, LM Studio, EXO, DGX Spark, Grok Build, and the gateway layer that keeps agents from turning into a pile of brittle provider calls.

Nova The useful question is simple. What changed for builders who want agents to run closer to their own machines, their own models, and their own tooling? [PAUSE] The answer is: more than one piece moved at the same time.

Alloy Ollama is moving deeper into coding-agent territory. LM Studio is improving Apple Silicon vision inference and pointing toward shared local model servers. NVIDIA is treating DGX Spark like a local-agent machine, not just a tiny workstation. EXO is showing both the promise and the rough edges of distributed inference across Macs and Spark hardware. xAI has a new coding-agent CLI with some pricing and routing risk around model redirects. And LiteLLM plus Envoy AI Gateway are tightening the model-routing control plane.

Nova So let's start where many local builders actually start: Ollama.

[00:00-05:00] Ollama Moves From Model Runner Toward Coding-Agent Platform

Nova The first story is Ollama. The headline is not one isolated feature. The headline is that Ollama is slowly becoming a local agent runtime surface, not just a command that runs a chat model.

Alloy The recent release line has several pieces pointing in that direction. Ollama 0.24 adds support for the Codex App through Ollama Launch. Recent May releases add vision-model support for opencode launch. There are fixes around Claude tool results when local image paths are involved. There is also API show response caching, with the release notes calling out about a 6.7x median latency improvement for integrations that need to load model metadata.

Nova That metadata point sounds boring until you think like an agent builder. A coding agent or desktop agent is not just asking one model to complete one prompt. It is checking what models exist, which ones support which capabilities, what can take images, what can reason, what is installed locally, and how quickly the runtime can answer those questions. If that discovery step is slow, the whole local experience feels heavy.

Alloy Then there is the bigger architecture change in the 0.30.0 release candidate. Ollama says that line changes the architecture to directly support llama.cpp instead of building on top of GGML, adds GGUF compatibility, and uses MLX to accelerate model inference on Apple Silicon.

Nova That matters because the local model ecosystem is mostly won or lost on portability and speed. GGUF is the everyday packaging format many builders already touch. llama.cpp is one of the main low-level engines local inference depends on. MLX is increasingly important for Apple Silicon because it lets Mac hardware participate seriously instead of being treated as second-class local inference hardware.

Alloy And the Gemma 4 MTP item is exactly the kind of thing local builders should notice. Multi-token prediction speculative decoding in the MLX runner is advertised as more than a 2x speed increase for Gemma 4 31B coding tasks. The model still has to be good, but speed changes what a local coding agent feels like. A model that is technically usable at one token rate can become actually comfortable at twice the rate.

Nova The deeper trend is that local model runners are becoming application launchers and agent substrate. Ollama Launch is not only a convenience wrapper. It is a signal that local model runtimes want to be the thing that wires models into coding apps, desktop assistants, and tool environments.

Alloy That makes Ollama a strategic piece of the local stack. A builder can start with it because it is simple, but the direction of travel is larger: launch integrations, coding tools, vision inputs, Apple acceleration, metadata caching, and model portability.

Nova The recommendation here is practical. Treat Ollama as a local runtime layer that is increasingly relevant to agent apps, not just as a way to run a quick local chat. Watch the 0.30 line closely because llama.cpp alignment, GGUF compatibility, and MLX acceleration could change the default path for local coding agents on Macs.

Alloy And do not miss the smaller fixes. Local image path handling, vision model support, and faster model metadata are the details that decide whether a local agent can inspect a screenshot, read a project surface, or quickly choose the right installed model without feeling clumsy.

Nova There is also a builder pattern hiding inside these releases. Start with the model runner, then ask what the agent has to do around the model. Can it launch the coding surface? Can it inspect images? Can it answer model capability questions quickly? Can it run on the hardware already sitting on the desk? Ollama is starting to answer more of those questions in one place.

Alloy That matters when you build an agent that has to move between tasks. A coding assistant may need a fast local model for quick explanations, a stronger model for patch planning, and a vision-capable model for reading UI state. If the local runtime makes those choices visible and fast, the agent can route work with less custom glue.

Nova The risk is assuming local means automatically simple. Local stacks still need capability discovery, model naming discipline, and clear expectations about what each model can do. The better Ollama gets at launch integrations and metadata, the easier it becomes to build a local agent that behaves predictably instead of relying on a pile of one-off commands.

Nova That is the first move: Ollama is becoming more agent-shaped.

[05:00-10:00] LM Studio Improves MLX Vision Inference And Points Toward Shared Local Servers

Alloy The second story is LM Studio. The concrete release is LM Studio 0.4.13. The changelog says mlx-engine v1.8.1 significantly improves performance and adds parallel predictions for vision-capable models including Qwen 3.5, Qwen 3.6, and Gemma 4.

Nova That is a local-stack story because vision is becoming normal agent input. A useful agent does not only read text. It looks at screens, app states, images, charts, UI errors, screenshots, and design artifacts. If local vision models are slow or awkward, builders fall back to cloud APIs. If local vision inference gets faster and more parallel, more of that loop can stay on the machine.

Alloy The release also fixes pasted-newline handling and includes security hardening. Those are not the headline, but they matter in a desktop product. Local tooling is often judged by whether it feels reliable in normal use, not only by benchmark numbers.

Nova The more interesting context is LM Studio's DGX Station material. LM Studio describes using a headless daemon called llmster, paired with LM Link, so a large local machine can serve models to other devices. It also points to the LM Studio SDKs, the LM Studio API, OpenAI-compatible APIs, and Anthropic-compatible APIs.

Alloy That is the deployment shape to notice. The local AI stack is splitting into two common patterns. Pattern one is direct desktop inference: the machine in front of you runs the model. Pattern two is local private serving: a bigger machine in the house, lab, office, or studio carries the model, and thinner clients call it through compatible APIs.

Nova That second pattern is where LM Studio becomes more than a UI. If a large local box can serve models through familiar APIs, then builders can point coding agents, task agents, notebook tools, and automation scripts at local inference without changing every client.

Alloy The compatibility layer is important. OpenAI-compatible and Anthropic-compatible APIs let existing tools talk to local models with fewer code changes. That does not mean every local model behaves like a frontier cloud model. It means the transport and client shape can be familiar, which lowers integration friction.

Nova Pair that with MLX improvements and you get a clearer picture. LM Studio wants to cover both the Apple Silicon developer machine and the heavier inference server. On one side, faster MLX vision prediction improves the Mac experience. On the other, LM Link and llmster point toward shared local inference.

Alloy For builders, the practical implication is to separate interface from compute. The laptop or desktop app can be the place where work happens. The model does not always need to live there. A larger local machine can become the private inference endpoint, while the daily device stays lightweight.

Nova That is also where local privacy gets more realistic. Running everything on one laptop is nice, but it has limits. A shared private inference server can support bigger models, multiple clients, and more persistent agent use while still avoiding cloud inference for sensitive context.

Alloy The recommendation: if LM Studio is part of your stack, pay attention to both tracks. For everyday Mac use, watch MLX engine updates and vision-capable model performance. For heavier local agents, watch llmster, LM Link, and API compatibility because that is where LM Studio becomes infrastructure.

Nova And the key phrase is local infrastructure. Not a demo app. Not a pretty chat window. Infrastructure that an agent can depend on.

Alloy The useful build pattern is a local private endpoint. Put the heavy model where the memory and thermals make sense, then let smaller devices call it through APIs they already know. That is much cleaner than forcing every laptop, editor, script, and assistant to carry its own separate model setup.

Nova It also changes how a builder should think about failure. If the local model server is shared, then uptime, auth, model loading behavior, and network access become product concerns. A private server that randomly unloads the model or changes its API behavior will break agents just as quickly as a cloud outage.

Alloy For vision agents, this matters even more. Vision input is often bursty. The agent may not need image understanding for every turn, but when it does, the response has to be fast enough to stay inside the task loop. Parallel prediction improvements are valuable because they make local multimodal work feel less like a separate slow lane.

[10:00-15:00] DGX Spark Becomes A Serious Local-Agent Target

Nova The third story is DGX Spark. This one is easy to cover badly because hardware stories often turn into spec-sheet reading. The useful question is narrower: why does DGX Spark matter for local agents?

Alloy NVIDIA is now explicitly framing DGX Spark and RTX PCs as machines for local agents. The company talks about agent computers, personal agents, local privacy, and no token costs. Its GTC material highlights Nemotron 3 Nano 4B, Nemotron 3 Super 120B, Qwen 3.5 optimizations, Mistral Small 4, and local agent stacks running through Ollama, LM Studio, and llama.cpp.

Nova The important DGX Spark number is 128GB of unified memory. Memory is the local-agent bottleneck that often matters more than raw peak compute. A model might be open. It might be downloadable. It might even be quantized. But if the machine cannot hold the model and the context comfortably, the local agent story falls apart.

Alloy NVIDIA positions DGX Spark as enough for models above 120B parameters. Nemotron 3 Super is described as a 120B open model with 12B active parameters. That active-parameter distinction matters because mixture-of-experts models can be very large in total size while activating only part of the network per token.

Nova That gives local builders a new middle tier. On the low end, you have laptops and desktops running smaller models. On the high end, you rent cloud GPUs or use hosted model APIs. DGX Spark sits in the middle: expensive and specialized, but local, private, and more capable than a normal consumer box.

Alloy The local-agent angle is not just bigger chat. Agents are different from chat because they run loops. They read context, call tools, inspect outputs, recover from failures, and often need to keep working for longer than a single prompt. That means inference cost, latency, privacy, and availability matter differently.

Nova A local agent machine can run all day without every step becoming a cloud API event. It can touch private context without shipping everything off-machine. It can be paired with local tools. And it can host models that are too large for a laptop but still fit inside a local workstation-class memory envelope.

Alloy NVIDIA also points to local model access through Ollama, LM Studio, and llama.cpp. That is the part builders should care about. Hardware only becomes useful when the software stack recognizes it. If the common local runtimes support the machine, then the hardware can slot into existing builder habits.

Nova The model names matter too. Nemotron 3 Nano 4B is a small local model direction. Nemotron 3 Super 120B is the larger local-agent direction. Qwen 3.5 optimizations and Mistral Small 4 show that this is not one model family. It is an ecosystem push around local open models and local agent execution.

Alloy The caveat is obvious: DGX Spark is not the default machine for every builder. But it changes the ceiling for local-first agents. It says local no longer only means small. Local can mean a dedicated inference box on the network, serving agents and tools without becoming a cloud bill.

Nova That is why DGX Spark belongs in this episode. It is not just an NVIDIA product announcement. It is a sign that local agent hardware is getting a serious tier, and the surrounding runtimes are starting to treat that tier as something builders might actually use.

Alloy The recommendation is to watch software support as much as hardware availability. The useful DGX Spark story is Ollama, LM Studio, llama.cpp, EXO, and agent frameworks treating it as a first-class node. Without that, it is just expensive hardware. With that, it becomes local agent infrastructure.

Nova The practical build question is where Spark sits in the stack. It should not be treated like a bigger laptop. It is better understood as a local inference appliance: one machine that can host larger models, serve multiple clients, and let smaller devices stay responsive.

Alloy That means the surrounding software has to make model serving boring. A builder should be able to load a model, expose a compatible endpoint, point a coding agent at it, and know what happens when the model runs out of memory or the server restarts. Hardware expands the ceiling, but the software decides whether that ceiling is usable.

Nova There is another implication for cost. Local hardware does not make inference free; it moves the cost from per-token billing to capital cost, power, maintenance, and setup time. That trade can make sense for persistent agents and private context, but only when the machine is used enough to justify it.

Alloy That is why DGX Spark should be evaluated as part of a system. What models run well on it? Which runtimes support it? Can EXO discover it? Can LM Studio serve from it? Can Ollama or llama.cpp use it cleanly? Can a coding agent call it without custom patches? Those answers matter more than the spec sheet alone.

[15:00-20:00] EXO Plus DGX Spark Shows Distributed Local Inference Is Real But Still Rough

Alloy The fourth story is EXO, and this is the most grounded one because it comes from an actual issue around DGX Spark joining a local EXO cluster.

Nova The setup is exactly the kind of setup local-agent builders care about: Macs plus DGX Spark on the same local network, trying to behave like one distributed inference pool. Basic connectivity worked. The dashboard was reachable. Ports were reachable. The network was not simply broken.

Alloy But the nodes still did not form a working cluster. The problem landed in the layer between reachability and reliable peer formation. That is a very common distributed-systems trap. Ping works, a dashboard loads, and still the thing you actually need, peer discovery and private-network agreement, does not work.

Nova The reported fix had two parts. First, the Rust exo_pyo3_bindings networking module needed to be compiled on Linux aarch64. That module contains libp2p networking, mDNS discovery, and private-network logic. On macOS, the app path had prebuilt pieces. On DGX Spark's Linux aarch64 environment, the missing build meant EXO could appear alive while the important peer connection layer was degraded.

Alloy Second, the nodes needed the same EXO_LIBP2P_NAMESPACE. EXO uses a libp2p private network key derived from its discovery namespace. If nodes derive different keys, they may see parts of the network environment without actually forming the same trusted peer network.

Nova After compiling the Rust networking module and aligning the namespace, the DGX Spark appeared in the EXO dashboard and participated in distributed inference. That final state is the important part: the Spark node did join the EXO cluster.

Alloy This is why EXO matters. Local inference is usually discussed machine by machine: this Mac can run this model, this GPU can run that quant, this desktop can serve this API. EXO is working on the harder question: can multiple local machines become one practical inference pool?

Nova That is the right problem for local-first agents because a real local environment often has uneven hardware. One machine has huge memory. One has a strong Apple Silicon setup. One is a smaller always-on node. One has an RTX card. If the agent stack can combine them, local inference becomes more flexible.

Alloy But this issue also shows the rough edge. Distributed inference depends on boring but critical systems pieces: mDNS discovery, libp2p behavior, architecture-specific packaging, namespace alignment, and clear failure messages. Raw model performance is only one part of the job.

Nova The best technical lesson is that distributed local inference fails in layers. Network reachability is layer one. Service discovery is layer two. Private-network identity is layer three. Runtime packaging is layer four. Model scheduling and inference performance come after that. If any earlier layer is wrong, the model never gets a chance to be fast.

Alloy For builders watching EXO, that means the most important updates may not look glamorous. Automated Rust module builds for Linux aarch64, clearer errors when networking bindings are missing, better namespace UX, and stronger discovery diagnostics are all product-quality features.

Nova Exactly. A local cluster product has to make failure legible. If a node is reachable but not joinable, the system should say why. If a private-network key differs, it should be visible. If a compiled module is missing, the app should not quietly limp forward.

Alloy The recommendation: keep EXO high on the watchlist, especially if your local agent setup spans more than one machine. The idea is important. The current lesson is equally important: distributed inference is not just model math. It is networking, packaging, and trust alignment.

Nova And that brings us to a very different kind of agent surface: Grok Build.

Alloy For builders, EXO is interesting because it suggests a different way to scale local inference. Instead of replacing every machine with one giant box, you try to combine the machines already available. That is attractive for homes, small labs, and studios where hardware accumulates unevenly over time.

Nova But the build pattern needs guardrails. A distributed inference layer should expose which nodes are present, which transport is active, which namespace is in use, which model shards are where, and whether a node is only visible at the dashboard layer or actually usable for inference. Without that visibility, debugging becomes guesswork.

Alloy The DGX Spark issue is a good reminder that successful local clusters need first-class diagnostics. The best user experience would not be a silent failure followed by hours of packet captures. It would be a clear message: the Linux aarch64 networking binding is missing, or the private-network namespace does not match, or this node can see the dashboard but cannot join the libp2p swarm.

Nova If EXO gets those edges right, the payoff is large. A local agent could route small tasks to a lightweight node, larger prompts to a memory-rich machine, and distributed jobs across several devices. That is a much more flexible local stack than one model tied to one computer.

[20:00-25:00] Grok Build Arrives, But Model Redirects And Pricing Need Attention

Nova The fifth story is xAI's Grok Build. The official docs describe a coding-agent CLI with an interactive terminal UI, headless scripting, JSON and streaming JSON output, resumable sessions, custom model configuration, skills, plugins, hooks, MCP servers, and ACP support through Grok agent stdio.

Alloy In plain terms, Grok Build is not only a web chat front end. It is positioned as a terminal-native coding agent that can run interactively or inside scripts and bots. That puts it in the same category as the broader coding-agent CLI wave.

Nova The feature surface is worth unpacking. The interactive TUI is for human-in-the-loop coding. Headless mode is for automation. Streaming JSON matters when another tool needs to observe the agent as it works. ACP support matters because IDEs and agent clients increasingly need a standard way to talk to coding agents over a structured protocol.

Alloy Custom model configuration is also important. The docs show a model block with a model ID, base URL, display name, and environment key. That means Grok Build is not only tied to one default backend in concept. It can be configured to point at custom model endpoints.

Nova For builders, that matters because coding-agent shells are becoming model routers. You may want one model for quick edits, another for deep reasoning, another local model for private code, and another hosted model for large context. The CLI becomes the control surface where those choices happen.

Alloy But there is a second xAI story this week: model redirects and pricing. xAI's May 15 migration page says retired reasoning model slugs redirect to Grok 4.3 with low reasoning effort. Retired non-reasoning slugs redirect to Grok 4.3 with no reasoning effort. grok-code-fast-1 redirects to Grok 4.3.

Nova The pricing number on that page is concrete: Grok 4.3 API pricing is listed at $1.25 per million input tokens and $2.50 per million output tokens. That is the number builders should use when evaluating the official API migration page.

Alloy The risk is not only the price. The risk is silent behavior change. If code keeps calling an old model slug and the provider redirects it, the request may still work, but the reasoning effort, latency, cost, and quality profile can change. That is dangerous for production agents and expensive coding loops.

Nova This is especially relevant for coding agents because they can consume a lot of tokens quickly. A headless coding task can read files, inspect diffs, propose patches, run tests, and iterate. If the model behind the slug changes, the economics of that loop change too.

Alloy There has also been chatter about lower promotional pricing, but the official docs checked for this episode do not show a clear $99 plan. The visible migration pricing is the API token pricing for Grok 4.3, and the broader subscription price people are reacting to is much higher than what many individual builders consider casual.

Nova The recommendation is straightforward: do not let deprecated slugs pick your economics. If you use xAI models in agents, choose the replacement model explicitly, set reasoning effort intentionally, and monitor cost after the redirect date.

Alloy And on Grok Build itself, the thing to watch is whether it becomes a serious cross-model coding shell or mostly a front door into xAI's own model stack. The docs support custom model config, and that is the part that makes it interesting for builders who care about routing.

Nova Grok Build is relevant. The pricing and redirect story is relevant. The right stance is not hype or dismissal. It is: test the CLI, pin models explicitly, and make sure the cost profile fits the builder budget before putting it into a frequent agent loop.

Alloy The build pattern here is a model-aware coding shell. A CLI like this should make it easy to run an interactive session, run a headless task, emit machine-readable progress, and integrate with editors or agent clients. Those pieces are what let a coding agent become part of a larger system instead of staying trapped in one terminal.

Nova But model-aware also means cost-aware. A builder should know which model is being called, which reasoning effort is active, and whether a deprecated name is being redirected. If a long-running coding job silently moves to a different model tier, the agent may still complete the task, but the bill and latency profile can surprise you.

Alloy That is especially important for teams building automation on top of coding agents. Headless mode is powerful because it can run in bots, CI-like checks, and maintenance scripts. But that same power means repeated calls. Repeated calls turn small pricing differences into real monthly cost.

Nova The clean recommendation is to treat Grok Build like any other serious coding agent surface: test it on real repositories, inspect its output format, verify custom model routing, and put cost monitoring around it before it becomes a default automation path.

[25:00-30:00] LiteLLM And Envoy Harden The Model Gateway Layer

Alloy The sixth story is the gateway layer. LiteLLM and Envoy AI Gateway both matter because every serious agent stack eventually needs a control plane between agents and models.

Nova LiteLLM v1.84.0 is a hardening release. The release changes version naming to PEP 440, authenticates pass-through endpoints by default, improves multi-pod budget enforcement, avoids Prisma reconnect freezes, reduces memory footprint through lazy-loaded feature routers, adds MCP OAuth and Azure Entra discovery support, and introduces durable agent run tracking through a workflow-runs API surface.

Alloy The pass-through endpoint change is a good example of the release's tone. Authenticated by default is less convenient for careless setups, but better for real ones. A model gateway should not accidentally expose forwarders just because a default was loose.

Nova Multi-pod budget enforcement is another practical point. Agents can fan out across workers. If spend counters are stale or inconsistent across pods, budgets become advisory instead of real. LiteLLM's refresh behavior and Redis counter fixes are about making spend accounting more accurate in distributed deployments.

Alloy The Prisma reconnect fix is also more important than it sounds. If a database reconnect path freezes the event loop, the gateway can fail liveness probes during database flaps. For an agent stack, that looks like random provider failure even though the underlying issue is control-plane reliability.

Nova Then there is memory footprint. Lazy-loading routers and the front page reportedly cuts memory by hundreds of megabytes in a two-worker Docker deployment. For local or small-server stacks, that is not trivial. The gateway should not become the heaviest thing in the room.

Alloy The MCP OAuth and Azure Entra discovery work points to a broader reality: model gateways are also tool gateways now. Agents are not only routing prompts to models. They are touching MCP servers, OpenAPI tools, auth flows, and user-scoped capabilities.

Nova Envoy AI Gateway v0.6.0 is moving from the Kubernetes side. It graduates core custom resources to v1beta1, adds AWS Bedrock InvokeModel support for Claude, supports Anthropic endpoints on OpenAI-compatible backends, adds Gemini embeddings and context caching, supports MCP per-backend header forwarding, adds request and response body redaction, and updates the Envoy and Gateway baseline.

Alloy The Anthropic-on-OpenAI-compatible backend piece is a provider-normalization story. A gateway can make different model providers look more consistent to clients. That is useful when agents need to swap models without rewriting every client integration.

Nova Gemini embeddings and context caching matter because not every model call is chat completion. Agents need retrieval, memory, context reuse, and cost control. Embeddings and caching are part of the economics of keeping an agent useful over time.

Alloy MCP per-backend header forwarding is a small phrase with real consequences. If an agent gateway talks to multiple MCP backends, each backend may need different headers, credentials, or routing metadata. Per-backend forwarding makes that cleaner and less brittle.

Nova Body redaction is another serious agent-stack feature. Agents often carry sensitive context. If the gateway logs everything raw, the control plane becomes a privacy problem. Request and response redaction are table stakes for production use.

Alloy The local-first connection is this: local does not mean simple. The moment a builder combines Ollama, LM Studio, cloud fallbacks, coding agents, MCP tools, and maybe a DGX Spark node, routing becomes a real system. Gateways decide auth, budgets, observability, provider compatibility, and failure behavior.

Nova The recommendation: do not treat gateways as optional glue once an agent stack has more than one model or more than one user. LiteLLM is relevant for multi-provider routing and budget control. Envoy AI Gateway is relevant when Kubernetes-native traffic management and provider normalization matter. In both cases, the useful updates are the ones that reduce surprise.

Alloy A practical builder pattern is to put the gateway in front of every non-trivial agent, even when some inference is local. That does not mean every small experiment needs Kubernetes. It means the agent should have one clear place where model names, auth, budgets, fallbacks, and logging policy are defined.

Nova This is where LiteLLM's routing groups are worth watching. Different model groups can have different routing strategies. A builder might want latency-based routing for high-quality hosted models, simple shuffle for cheaper fallback models, and a separate local path for private tasks. The value is not abstraction for its own sake. The value is making model choice explicit instead of scattering it across every agent script.

Alloy Envoy AI Gateway's direction is similar but more infrastructure-native. The v1beta1 API surface matters because teams are more willing to build on APIs that are stabilizing. The body redaction and header forwarding pieces matter because agents carry credentials, private prompts, and tool-specific metadata through the gateway. When those details are handled centrally, the rest of the stack gets easier to reason about.

Nova The trap is thinking a gateway magically fixes model quality or agent design. It does not. A gateway cannot make a weak model reason better, and it cannot make a confused agent plan better. What it can do is make the surrounding system less fragile: fewer accidental unauthenticated routes, better budget accounting, clearer provider compatibility, cleaner tool authorization, and safer logs.

Alloy For local-first builders, that is exactly the right level of ambition. Keep the models close when privacy and cost demand it. Use hosted models when they are clearly better for the task. Put a control layer between the agent and all of those choices so the system can evolve without rewriting everything.

Alloy And that is the theme across the whole episode, without needing to force one. Local agents are becoming more practical because the stack is filling in below the model: runtimes, hardware, distributed inference, coding-agent shells, and routing infrastructure.

Nova Ollama is becoming more agent-shaped. LM Studio is improving local vision inference and pointing toward shared local servers. DGX Spark is giving local agents a more serious hardware tier. EXO is proving that distributed local inference is real, while showing exactly where it still needs polish. Grok Build adds another serious coding-agent CLI, but the model redirect and pricing details need attention. And the gateway layer is hardening because agents need reliable control planes.

Alloy The main builder takeaway is simple: local-first AI is no longer one tool. It is a stack. Model runners matter. APIs matter. Hardware matters. Peer discovery matters. CLI surfaces matter. Gateways matter.

Nova The second takeaway is that the local stack is becoming more modular. Ollama can be the quick local runtime. LM Studio can be a desktop app and a private model server. DGX Spark can be a heavy inference node. EXO can try to make multiple machines act like a cluster. Grok Build can be a coding-agent shell. LiteLLM or Envoy can sit in front of model calls. Those pieces do not all have to be used at once, but they are starting to fit into recognizable roles.

Alloy The third takeaway is that builders should evaluate local AI by loops, not demos. A demo asks whether a model can answer one prompt. A builder loop asks whether the agent can inspect context, choose the right model, call a tool, recover from an error, keep costs visible, and run again tomorrow. That is why the small release details matter.

Nova Ollama's faster metadata calls matter inside loops. LM Studio's MLX vision work matters inside loops. EXO's namespace and networking details matter inside loops. Grok Build's headless JSON output matters inside loops. Gateway auth, budget counters, and redaction matter inside loops. The stack either supports repeated agent work, or it stays a collection of impressive one-off tests.

Alloy The final recommendation is to build the local stack in layers. First, pick the runtime that can actually run the models you need. Second, expose it through APIs your agents can use. Third, decide whether one machine is enough or whether a dedicated inference box or cluster layer makes sense. Fourth, put routing and cost control somewhere visible. Fifth, test the whole loop with real tasks, not benchmark prompts.

Nova That is where local-first agent building is headed: less magic, more systems thinking, and better tools for keeping inference close when close actually matters.

Alloy One more point before we close: the stack is also becoming more testable. A builder can now ask sharper questions. Does Ollama serve the model fast enough for this coding loop? Does LM Studio handle the vision model locally? Does Spark give enough memory headroom for the larger model? Does EXO actually see every node and form the private network? Does Grok Build expose output that another tool can consume? Does the gateway show cost and route behavior clearly?

Nova Those questions are better than asking whether local AI is ready in the abstract. Local AI is ready for some tasks, immature for others, and changing quickly. The useful work is matching each task to the right layer of the stack. A private coding task may belong on a local model. A very hard reasoning task may still need a hosted model. A repetitive agent loop may need local economics. A team deployment may need gateway policy more than another benchmark result.

Alloy So the builder stance is not local-only and not cloud-only. It is control. Put local runtimes where privacy, latency, and cost make sense. Use larger hosted models where quality clearly wins. Keep the interface stable enough that the agent can move between those choices without becoming a rewrite project.

Nova That is the practical line to watch this week.

Nova For episode notes and links, go to Toby On Fitness Tech dot com.

Alloy We'll be back soon.

Nova I'm NOVA. This was AgentStack Daily. [PAUSE]