Running AI on your own hardware

Running a model locally means its weights are stored on a computer or device you control and the computation happens there. A runtime loads the model and makes it usable. llama.cpp is a C/C++ inference project with a command-line tool and an OpenAI-compatible server. Ollama runs open models and serves a REST API on the local machine. llamafile, now developed by Mozilla.ai, packages a model and a llama.cpp-based runtime into a single executable that runs without installation.124

Local does not automatically mean offline. Ollama also offers cloud models that run on Ollama's servers after you sign in, and its documentation says to disable cloud features if you want to use only local models. Its integrations can also connect a local setup to outside services such as messaging apps. Check a tool's settings before assuming that nothing leaves your machine.32

A local model can also power an agent. Hermes Agent, from Nous Research, needs at least one model provider, and its documentation covers self-hosted options including Ollama, llama.cpp's llama-server, LM Studio, and vLLM alongside cloud APIs. Where the agent's commands run is a separate setting: its terminal backends include local, Docker, SSH, and serverless options such as Modal and Daytona. Check where both the model and the commands run before treating a setup as local.109

Hardware shapes what is practical. llama.cpp supports backends including CUDA for NVIDIA GPUs, HIP for AMD GPUs, Metal for Apple silicon, and Vulkan, and it can split a model between CPU and GPU when the model is larger than the GPU's memory. MLX, from Apple machine learning research, is an array framework for Apple silicon in which arrays live in memory shared by the CPU and GPU. ExecuTorch, PyTorch's on-device stack, captures a model with torch.export and runs it on phones, laptops, embedded systems, and microcontrollers.156

Quantization stores a model's weights at lower numerical precision to reduce memory use. llama.cpp supports integer quantization from 1.5 to 8 bits, and AWQ, from MIT's Han Lab, is a method for 3- and 4-bit weight-only quantization of large language models. Published results can depend on the precision used. OpenAI's gpt-oss-20b card says the model's mixture-of-experts weights were post-trained with MXFP4 quantization so that it runs within 16 GB of memory, and that all of its evaluations used that same quantization.178

Running a model on your own hardware does not change its license, so read the terms for the specific release before you rely on it. gpt-oss-20b, for example, is published under Apache 2.0. Its card describes it for lower-latency, local, or specialized uses and gives instructions for running it with Ollama or LM Studio on consumer hardware.8