Single-file local LLM + speech-to-text runner for offline repo tooling

What they did

They wanted local inference that a teammate could run without a Python environment, CUDA toolkit, or model-download dance. Starting from the llamafile README, they downloaded a small pre-built llamafile (Qwen3.5 0.8B Q8_0) from the mozilla-ai Hugging Face repo, chmod +x'd it, and ran it directly to confirm the bundled server came up and served an OpenAI-compatible endpoint on localhost. They then added a second single-file binary, whisperfile, for speech-to-text so meeting recordings and voice notes could be transcribed on the same machine with the same no-install story. In the repo they added a small scripts/ directory: a fetch script that curls the llamafile and whisperfile artifacts into a gitignored .local/ folder, a run script that launches the llamafile server on a fixed port with a modest context size, and a transcribe script that pipes audio through whisperfile and writes plain-text transcripts next to the source files. Application code points at http://localhost:PORT/v1 with a dummy API key, so the same client works whether the backend is the local llamafile or a hosted model later. They documented the Windows caveat (rename to .exe, and executables over 4GB will not run, so use the small llamafile binary with external GGUF weights instead) and pinned the 0.10.* build so the bundled llama.cpp version is known. Finally they wrote a short README section and a smoke test that checks the server responds before any script runs.

How it went

Worked end to end on macOS and Linux: the 0.8B llamafile started in seconds and answered prompts through the local endpoint, and whisperfile produced usable transcripts from short audio clips. The whole setup is a couple of shell scripts plus two downloaded binaries, so onboarding a new machine is a curl and a chmod. Quality is fine for drafting, summarizing, and transcription triage but noticeably weaker than hosted frontier models on long or nuanced tasks, and the 0.8B model is best treated as a smoke-test default rather than a daily driver.

Worth knowing

Windows needs the .exe rename and cannot run single-file executables above 4GB, so larger models require the standalone llamafile binary plus external GGUF weights. The 0.10.* build system is newer and may lack features from the classic 0.9.* releases, so pin versions deliberately. Local models are slower and less capable than hosted ones, and running them pegs CPU and RAM. Nothing here is a security boundary: the local server has no real auth, so bind it to localhost only. Model licenses and download sizes vary per artifact.

Wiring

Order: (1) fetch artifacts with curl into .local/, (2) chmod +x the llamafile and whisperfile binaries, (3) start the llamafile server in the background on a fixed port, (4) poll the health endpoint until ready, (5) run app code or transcription against it. Config lives in a single .env-style file: MODEL_PATH, WHISPERFILE_PATH, LOCAL_PORT, CTX_SIZE, and BASE_URL defaulting to http://localhost:PORT/v1 with a dummy API key. The app uses an OpenAI-compatible client so the base URL is the only switch between local and hosted. Transcription is a separate one-shot process per file, invoked by script rather than through the chat server. A Makefile target chains fetch, serve, and smoke-test so a fresh clone is one command.

Try it yourself

Set up offline local inference in this repo using llamafile and whisperfile. Create a gitignored .local/ directory and a scripts/fetch-local-models.sh that downloads a small pre-built llamafile (Qwen3.5 0.8B Q8_0 from the mozilla-ai Hugging Face repo) and a whisperfile binary, then chmod +x both. Add scripts/serve-local.sh that launches the llamafile server on a configurable port with a modest context window and logs to .local/server.log, plus scripts/transcribe.sh that runs whisperfile over an audio file and writes a .txt transcript beside it. Point the app's model client at http://localhost:PORT/v1 with a placeholder API key so it can swap to a hosted backend via env var. Add a smoke test that waits for the server to respond before running, and a README section covering the Windows .exe rename and the 4GB executable limit.