cactus

C++NOASSERTION★ 6.2k

Cactus is a hybrid edge-cloud AI engine with custom quantization, CPU/GPU kernels, and a zero-copy graph, exposing OpenAI-compatible APIs for text

Runtime

Runs: LLMs, speech, and vision models on mobile, wearable, smart home, and robotics hardware
Hardware: CPU and GPU on Apple, Samsung, Pixel, and other ARM devices
API: OpenAI-compatible APIs

From the project’s README

GitHub ↗

Written by the project's own authors, not by runchr. Snapshot taken 2026-11-02.

Cactus

A hybrid edge-cloud AI engine for mobile devices & wearables.

┌─────────────────┐
│  Cactus Engine  │ ←── OpenAI-compatible APIs for text, speech, and vision.
└─────────────────┘     
         │
┌─────────────────┐
│  Cactus Graph   │ ←── Zero-copy computation graph
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Kernels  │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.)
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Quants   │ ←── Custom rotation-based quantization technique
└─────────────────┘

Quick Demo (Mac)

Step 1: brew install cactus-compute/cactus/cactus
Step 2: cactus run

Cactus Engine

#include "cactus_engine.h"

cactus_model_t model = cactus_init(
    "path/to/weight/folder",
    "path to txt or dir of txts for auto-rag",
    false
);

const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Henry Ndubuaku"}
])";

const char* options = R"({
    "max_tokens": 50,
    "stop_sequences": ["<|im_end|>"]
})";

char response[4096];
int result = cactus_complete(
    model,            // model handle
    messages,         // JSON chat messages
    response,         // response buffer
    sizeof(response), // buffer size
    options,          // generation options
    nullptr,          // tools JSON
    nullptr,          // streaming callback
    nullptr,          // user data
    nullptr,          // pcm audio buffer
    0                 // pcm buffer size
);

Example response from Gemma4-E2B

{
    "success": true,        // generation succeeded
    "error": null,          // error details if failed
    "cloud_handoff": false, // true if cloud model used
    "response": "Hi there!",
    "function_calls": [],   // parsed tool calls
    "segments": [],         // transcription segments (empty for chat)
    "confidence": 0.8193,   // model confidence
    "confidence_threshold": 0.7, // resolved handoff threshold (model-dependent)
    "time_to_first_token_ms": 45.23,
    "total_time_ms": 163.67,
    "prefill_tps": 1621.89,
    "decode_tps": 168.42,
    "ram_usage_mb": 245.67,
    "prefill_tokens": 28,
    "decode_tokens": 50,
    "total_tokens": 78
}

Cactus Graph

#include "cactus_graph.h"

CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);

auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);

float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};

graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);

graph.execute();
void* output_data = graph.get_output(result);

graph.hard_reset();

Inference Speed

LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)
VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)
Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)
1k-Context RAM: peak MB during the LLM benchmark
No speculative decode or MTP, pure decode

Command: cactus benchmark [optional --ios or --android]

Device	LLM	VLM	Transcribe	RAM
Mac M5 Max	2964tps / 154tps	0.09s / 168tps	0.15s	1348MB
Mac M4 Pro	1963tps / 101tps	0.25s / 112tps	0.21s	1225MB
Mac M3 Pro	1294tps / 64tps	0.40s / 72tps	0.37s	735MB
iPad/Vision Pro M5	1336tps / 71tps	0.25s / 80tps	0.27s	703MB
iPhone 17 Pro	729tps / 37tps	0.5s / 39tps	0.51s	644MB
iPhone 15 Pro	517tps / 26tps	1.15s / 27tps	0.82s	633MB

N/B: With 1k-context prefill and decode for 100 runs on M5 Max

LFM2.5-VL-1.6B = 289toks/sec
Qwen3-1.7B = 155toks/sec
LFM2.5-VL-450m = 472toks/sec, image encodes in 43ms
LFM22.5-VL-230m = 555toks/sec

Output Quality

Gemma-4-E2B-it accuracy across bit widths, averaged over 3 seeds.
CQ3.26 and CQ2.54 are mixed-precision, CQ2/CQ3/CQ4 are uniformly quantized.
Full results in docs/cactus_quants.md:

Task	F16 (Original)	CQ4	CQ3.26	CQ2.54	CQ2
ARC-E	73.80	73.73	74.20	68.20	50.80
ARC-C	56.47	52.47	51.53	37.20	24.73
HellaSwag	46.93	47.07	45.20	40.73	35.87
WinoGrande	61.00	61.13	59.60	60.13	51.27
MMLU	62.33	59.45	57.63	47.19	33.18
GPQA	34.34	34.34	31.82	30.81	23.23
GSM8K	73.67	71.20	66.20	22.00	0.40
HumanEval	54.88	57.11	53.66	15.24	1.02
BFCL Simple	92.00	92.42	91.50	82.25	18.75
BFCL Multi	89.00	88.33	89.00	52.50	13.67
BFCL Parallel	84.00	83.67	82.50	30.00	3.33
BFCL Parallel-Multi	78.00	83.33	82.00	37.00	1.33

Supported Models

Any HuggingFace model can be converted using cactus convert [HF-Name], though experimental.
Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested.
Some models have been pre-uploaded here, just run cactus download [HF-Name].
cactus run [HF-Name] albeit first downloads or convert the model if not found.

Needle

Needle is a 26m parameter model for on-device tool calling:

cactus run Cactus-Compute/needle [--tools my_tools.json]  # OpenAI function-calling format; demo toolset by default

Learn More

Reference	Language	Description
Cactus Engine	C	Chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, vector index, cloud handoff
Cactus Graph	C++	Tensor operations, matrix multiplication, attention, normalization, activation functions
Cactus Kernels	C++	ARM NEON SIMD kernels for matmul, attention, convolution, quantization, DSP, image processing
Cactus Quants	C++	Rotation-and-codebook quantization from 4-bit to 1-bit for all weight tensors
Cactus Hybrid	C/Python	Route hard queries to the cloud automatically based on local model confidence
Python Package	Python	Python package and CLI

Bindings

Using this repo

┌────────────────────────────────────────────────────────────────────────────────┐
│                                                                                │
│ Step 0: if on Linux (Ubuntu/Debian)                                            │
│ sudo apt-get install python3.12 python3.12-venv python3-pip cmake              │
│   build-essential libcurl4-openssl-dev                                         │
│                                                                                │
│ Step 1: clone and setup                                                        │
│ git clone https://github.com/cactus-compute/cactus && cd cactus                │
│ source ./setup                                                                 │
│                                                                                │
│ Step 2: use the commands                                                       │
│────────────────────────────────────────────────────────────────────────────────│
│                                                                                │
│  cactus auth                         manage cloud API key                      │
│    --status                          show key status                           │
│    --clear                           remove saved key                          │
│                                                                                │
│  cactus run [model|path]             run a model (downloads if needed)         │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --image <path>                    image file for VLM inference              │
│    --audio <path>                    audio file for audio chat                 │
│    --system <prompt>                 system prompt                             │
│    --prompt <text>                   send prompt immediately                   │
│    --tools <json|file>               tool definitions for tool calling         │
│    --thinking                        enable thinking/reasoning mode            │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│                                                                                │
│  cactus transcribe [model]           live microphone transcription with a model│
│    --file <audio.wav>                audio file to transcribe (WAV)            │
│    --language <code>                 language code (default: en)               │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│                                                                                │
│  cactus download [model]             get a bundle (prebuilt, else build)       │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       refresh cached bundle                     │
│                                                                                │
│  cactus convert <model> [dir]        HuggingFace -> Cactus CQ weights          │
│    --bits 1|2|3|4                    CQ quantization (default: 4)              │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       force weight conversion from source       │
│    --lora <path>                     merge a LoRA adapter before converting    │
│    --weights-only                    stop after CQ weights (skip the graph)    │
│    --artifact-dir <path>             bundle output (default: weights/<model>)  │
│                                                                                │
│  cactus serve [model]                OpenAI-compatible local HTTP server       │
│    --host <addr>                     bind address (default: 127.0.0.1)         │
│    --port <port>                     port (default: 8080)                      │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│    --no-cloud-handoff                disable automatic cloud handoff           │
│    --confidence-threshold <0..1>     handoff to cloud below this confidence    │
│    --cloud-timeout-ms <n>            max wait for cloud handoff                │
│                                                                                │
│  cactus code                         run the AI coding agent (TUI / print)     │
│    --serve-model <id>                auto-start a server with this model       │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│    --host <addr>                     server address (default: 127.0.0.1)       │
│    --port <port>                     server port (default: 8080)               │
│    --no-serve                        require a running server (no auto-start)  │
│    --no-cloud-handoff                disable automatic cloud handoff           │
│    --confidence-threshold <0..1>     handoff to cloud below this confidence    │
│    --cloud-timeout-ms <n>            max wait for cloud handoff                │
│    -- <args...>                      pass remaining args to the agent          │
│                                                                                │
│  cactus list                         list downloaded models                    │
│                                                                                │
│  cactus build                        build cactus libraries                    │
│    --apple                           Apple (iOS/macOS)                         │
│    --android                         Android                                   │
│    --python                          shared lib for Python FFI                 │
│                                                                                │
│  cactus test                         run the test suite                        │
│    --component <name>                kernels | graph | engine | all            │
│                                      (default: all)                            │
│    --model <hf-id>                   default: google/gemma-4-E2B-it            │
│    --transcription-model <hf-id>     default: nvidia/parakeet-tdt-0.6b-v3      │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --token <token>                   HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild of test models        │
│    --suite <name>                    run a single test suite by name           │
│                                      (resolved across components,              │
│                                      e.g. llm → engine)                        │
│    --list                            list components and suites                │
│    --ios                             run on connected iPhone                   │
│    --android                         run on connected Android                  │
│    --enable-telemetry                send cloud telemetry (off by default)     │
│                                                                                │
│  cactus benchmark                    run the engine benchmark suite            │
│    --model <hf-id>                   default: google/gemma-4-E2B-it            │
│    --transcription-model <hf-id>     default: nvidia/parakeet-tdt-0.6b-v3      │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --ios                             run on connected iPhone                   │
│    --android                         run on connected Android                  │
│                                                                                │
│  cactus clean                        delete build artifacts, weights, venv     │
│  cactus --help                       show this help                            │
│                                                                                │
└────────────────────────────────────────────────────────────────────────────────┘

Citation

If you use Cactus in your research, please cite it as follows:

@software{cactus,
  title        = {Cactus: AI Inference Engine for Phones & Wearables},
  author       = {Ndubuaku, Henry and Cactus Team},
  url          = {https://github.com/cactus-compute/cactus},
  year         = {2025}
}

N/B: Scroll all the way up and click the shields link for resources!

Other tools that do the same work

activepieces GitHub ↗

Apps

Multi-agent Data DevOps Research Knowledge

TypeScript25.3ktoday

Open-source AI workflow automation with 280+ pieces exposed as MCP servers.

Agently GitHub ↗

Frameworks

Coding Multi-agent Knowledge DevOps

Python1.7k20 days ago

Python framework for building reliable GenAI agent backends with structured

ai-agent-book GitHub ↗

Frameworks

Coding Multi-agent Knowledge Memory Research

Python56.9k25 days ago

Open-source book on AI agent design with 109 hands-on experiment projects.