Glossary Definition · Physical AI & Local LLMs

What Is Edge AI & GGUF Quantization?

Edge AI is the execution of artificial intelligence models directly on local hardware—inside your computer or single-board computer's RAM, CPU, or GPU—rather than on remote servers. GGUF Quantization compresses 16-bit floating-point model weights (.safetensors) into compact 4-bit (Q4_K_M) or 8-bit (Q8_0) integer blocks so open-weight language models can run locally via llama.cpp [1, 2, 4].

01 // MATHEMATICAL COMPRESSION

Why Local Models Need Block Quantization

When AI research labs release open-weight language models on Hugging Face (such as Google Gemma, Qwen, Llama, Mistral, or SmolLM2), the parameters are typically stored as 16-bit floating-point numbers (FP16 or BF16) inside .safetensors files [1, 3]. At 2 bytes per parameter, a 7-billion-parameter model requires roughly 14 GB of memory just to hold the weights—plus additional RAM for the key-value (KV) context cache [1].

GGUF (commonly expanded as GPT-Generated Unified Format), introduced by Georgi Gerganov and the ggml / llama.cpp contributors in August 2023, solves this memory bottleneck by grouping weights into small blocks (commonly 32 weights per block) and scaling them into 4-bit or 8-bit integer representations [1]:

4-Bit K-Quant (Q4_K_M)

Uses ~4.8 bits per weight with super-block scaling for critical attention and feed-forward tensors [1]. Cuts RAM consumption by ~70% compared to FP16, allowing 1B–3B parameter models to run inside 2–4 GB of RAM on laptops and 64-bit ARM single-board computers [1, 4].

8-Bit Linear Quant (Q8_0)

Uses ~8.5 bits per weight (32 eight-bit integers per block with a 16-bit float scale factor) [1]. Cuts FP16 size nearly in half while retaining numerical precision closely aligned with the unquantized checkpoint [1, 4].

02 // ON-DEVICE COMPILATION

The 3-Tab Model Forge Workflow in VTM Studio

After you download official .safetensors model weights from Hugging Face in your browser, Model Forge Mode (#view-forge, included in the Physical AI Pro Pack) executes the entire conversion, quantization, and runtime registration pipeline locally on your own machine across three tabs [1, 3, 4]:

Tab 1 // 1. Convert HF

Convert Local Safetensors to GGUF

Select a local directory containing downloaded Hugging Face .safetensors weights, config.json, and tokenizer files. Model Forge converts the tensor dictionary into an initial .gguf file on your local disk [1, 3, 4].

Tab 2 // 2. Compact GGUF

4-Bit & 8-Bit Local Quantization

Run llama-quantize locally to compress your .gguf model into Q4_K_M, Q5_K_M, Q3_K_M, or Q8_0 precision matched to your target computer's RAM or GPU VRAM capacity [1, 4].

Tab 3 // 3. Run Model

Click-to-Browse & Live Hot-Swap

Use the CLICK TO BROWSE AND ADD .GGUF FILE importer to register any local .gguf file and click ENGAGE LOCAL MODEL (SWAP) to hot-swap the active model on llama-server (Port 8080) without restarting VTM Studio [4].

Offline Speech Stack

Click-to-Talk Whisper STT & Piper ONNX TTS

In VTM Studio, local Edge AI inference pairs directly with an on-device speech pipeline. Clicking the microphone button in Chat Mode, the VAS-X sidebar, Spec Mode, or Log Mode captures voice input, converts it to 16 kHz mono WAV via ffmpeg, and transcribes it locally using faster-whisper (base.en INT8 quantized model) [4, 5]. Responses are synthesized into spoken audio readback locally via piper ONNX voice models with sentence-boundary streaming [4, 6].

Hybrid Workstation Flexibility

When to Use Edge AI vs. Cloud AI

The top header CLOUD / EDGE switch (#ai-core-toggle) lets you choose the right engine for each task: use Local Edge AI for offline operation, zero API cost, and testing quantized 1B–12B GGUF models; flip to Cloud AI (Google Gemini for 1M-token context windows, or Groq for high-speed cloud inference) whenever you need large-context reasoning, multi-file C++ refactoring, VAS-X full-workbench navigation, or Cloud Vision analysis [4, 7].

04 // REALISTIC EXPECTATIONS

Measured Local Edge AI Benchmarks Across Hardware Tiers

Local LLM inference speed depends on parameter count, quantization level (Q4 vs. Q8), memory bandwidth, and GPU acceleration. The Arduino UNO Q and 2017 MacBook Pro cards below come from verified VTM Studio telemetry logs, while the Windows 10 and RTX 3060 cards are informal observed references [4]:

Arduino UNO Q (4 GB · Full UI + TTS)
1.34–2.00 tok/s
~27s Wait Before First Token
Gemma 3 1B Q4 + Live Inference Stream + Concurrent Piper ONNX TTS in 3.58 GB Usable RAM [4]
2017 MacBook Pro (8 GB · Full UI + TTS)
3.24–5.41 tok/s
3.94 tok/s Combined 210-Token Avg (~9–11s First-Token Wait)
Gemma 3 1B Q4 + 3 Parallel Piper ONNX WAV Chunks in 8.0 GB RAM (Darwin 22.6.0 x86_64) [4]
Win 10 (4 GB GPU / 24 GB RAM)
~2 tok/s
2.7B Phi-2 Q8_0 (Observed Reference, Not Logged) [4]
12 GB NVIDIA RTX 3060
~32 tok/s
12B Gemma (Observed GPU Reference, Not Logged) [4]
First-Party SBC Telemetry Case Study // Not a Headless Terminal Demo

Running Gemma 3 1B Q4 + Live Inference Streaming + Concurrent Piper ONNX TTS on Arduino UNO Q (4 GB)

Open Full Benchmarks & 4-Way Comparison Page →

Most single-board computer LLM benchmarks measure a bare llama-cli process inside a headless SSH terminal with no graphical user interface and no audio synthesis. In contrast, the telemetry table below was captured from VTM Studio 2.2.0 running its complete graphical Electron workstation interface, Python 3.12.8 backend, all three commercial packs (maker, maker_pro, physical_ai_pro), the live X-Panel Inference Monitor token stream, 3.00-second /system_stats hardware telemetry polling, and concurrent Piper ONNX text-to-speech synthesis directly on an Arduino UNO Q (4 GB, SKU ABX00173) with 3.58 GB of usable Linux system RAM [1, 4, 6]:

Interactive 43-Point Hardware Oscilloscope // Click or Hover Any Token Node (T01–T43)
Instantaneous Token Throughput (tok/s) vs. Concurrent Piper ONNX TTS Synthesis
3.5 2.6 1.8 0.9 0.0 THROUGHPUT (TOK/S) PHASE 1: PURE DECODE (2.00 TOK/S AVG) PHASE 2: PARALLEL PIPER ONNX TTS Sentence 1 Audio Synthesizing @ t=6.56s PHASE 3: POST-TTS RECOVERY (1.54 TOK/S AVG) Phase 1 Avg: 2.00 tok/s Phase 3 Avg: 1.54 tok/s Full Turn Avg: 1.34 tok/s T01 (0.0s) T14 (6.56s · "." TTS Trigger) T24 (19.34s · "Python") T34 (25.90s) T43 (31.98s) ACCUMULATED TOKEN INDEX (T01–T43) & WALL-CLOCK ELAPSED TIME (SECONDS)
[HOVER OR CLICK ANY POINT] Token #24 ("Python") · Elapsed: 19.337s · Δ = 318 ms · Phase 3 Avg: 1.54 tok/s (1.34 tok/s Turn Avg)
Phase 3: Post-TTS Decode
[VTM-HOST STREAM]: Micro Python is a Python implementation designed for microcontrollers and embedded systems. It’s a compact, optimized version of Python, focusing on speed and efficiency, often used for real-time control and sensor data processing.
Execution Phase (43-Token MicroPython Query) Token Span & Wall Clock Measured Throughput Concurrent Workstation Workload on QRB2210 CPU
Phase 1: Pure Token Decode (Sentence 1) Tokens 1–15 (7.035 s) 2.00 tok/s sustained average (~502 ms/token) Gemma 3 1B Q4 inference + live UI token stream + 3.0s /system_stats polling
Phase 2: Sentence-Boundary Piper TTS Overlap Tokens 15–23 (11.984 s) 0.67 tok/s (Zero dropped tokens / zero UI freeze) Token 14 (".") triggers parallel Piper ONNX voice synthesis of Sentence 1 while Sentence 2 keeps generating
Phase 3: Post-TTS Decode Recovery (Sentence 2) Tokens 23–43 (12.963 s) 1.54 tok/s sustained average (~648 ms/token) Sentence-1 audio playing back while llama-server finishes Sentence 2 and commits JSON payload
Full End-to-End Turn (Text + Spoken Audio + UI) All 43 Tokens (31.982 s decode · ~27.2 s first-token wait) 1.34 tok/s overall · TTS triggered 6.56 s into decode Cuts post-first-token decode wait before speech by ~79% (6.56 s vs 31.98 s decode; ~33.8 s vs ~59.2 s total wall-clock, ~43% sooner); 7 ms session archive export
1. Fits Inside 3.58 GB Usable RAM
Running an Electron GUI, Python 3.12 backend, ONNX voice synthesizer, and 1B LLM on a 4 GB SBC without triggering Linux Out-Of-Memory (OOM) resets is achieved via VTM Studio's Smart Auto-SBC Profile (512 MB V8 heap cap, /dev/shm RAM cache, and detached inactive DOM views) [4].
2. Sentence-Boundary Time-to-Voice
Because VTM Studio streams completed sentences to Piper ONNX TTS at the first period (Token 14, 6.56 s after Token 1 arrives, or ~33.8 s total wall-clock after prompt submission vs. ~59.2 s for full 43-token completion), spoken audio begins 25.42 s earlier while Sentence 2 continues streaming [4, 6].
3. Multi-Turn Workload Stability
Across four consecutive local Edge AI turns on the UNO Q—(1) arithmetic problem generation, (2) an 87-word creative story, (3) a 69-token C++ Arduino digitalWrite code snippet, and (4) the 43-token MicroPython definition—/system_stats maintained an unbroken 3.00-second HTTP 200 heartbeat [4].
Matched 4-Prompt Head-to-Head Telemetry // 2017 Intel MacBook Pro (8.0 GB RAM) vs. Arduino UNO Q (4 GB / 3.58 GB Usable RAM)

Both machines ran the exact same four prompts in VTM Studio 2.2.0 (Gemma 3 1B Q4 + live X-Panel Inference Monitor streaming + sentence-boundary Piper ONNX TTS /generate_audio_chunk WAV synthesis + 3.0s /system_stats polling):

Test Prompt (Identical Across Both Machines) 2017 MacBook Pro (8.0 GB RAM · x86_64) Arduino UNO Q (4 GB / 3.58 GB RAM · aarch64) Laptop vs. SBC Speedup
1. Basic Arithmetic ("7 + 5 = ?") 6 tokens in 0.39 s decode (12.82 tok/s) · ~9.0s first-token wait (9.7s total) 6 tokens · 29.0s total turn (first-token wait + decode + TTS) 2.99× faster total turn
2. Short Story (≤100 Words + Multi-Chunk TTS) 91 tokens in 25.18 s decode (3.61 tok/s overall · 6.22 tok/s Sent. 1) · 34.6s total ~110 tokens · 114.0s total turn (first-token wait + decode + TTS) 3.29× faster total turn
3. C++ Arduino Blink Snippet (No Comments) 69 tokens in 12.75 s decode (5.41 tok/s sustained) · ~9.1s first-token wait (22.0s total) ~45 tokens · 61.0s total turn (first-token wait + decode + TTS) 2.77× faster total turn
4. "What is MicroPython. Be concise" + Piper TTS 50 tokens in 15.43 s decode (3.24 tok/s overall · 4.91 tok/s Sent. 1) · 10.9s first-token wait · Sent. 1 WAV plays @ 6.75s 43 tokens in 31.98 s decode (1.34 tok/s overall · 2.00 tok/s Sent. 1 avg) · 27.2s first-token wait · Sent. 1 TTS triggers @ 6.56s 2.42× faster decode · 2.49× faster first-token wait
Practical Maker Guidance — When to Use 1B Edge Models vs. Cloud AI for Arduino C++: Compact 1-billion-parameter quantized models (Gemma 3 1B Q4) are well suited for offline concept explanations, prose summaries, and spoken voice Q&A. However, on both the UNO Q and the 2017 MacBook Pro, the 1B model omitted pinMode() and void loop() when asked for a raw C++ blink sketch. When generating compilable microcontroller firmware, use VTM Studio's 1-click header toggle to switch to Cloud Mode (Google Gemini or Groq) or load a hardware-verified sketch from Code Mode → Science Mode Examples.
05 // VERIFIED ACADEMIC TEXTBOOKS, DATASHEETS & SPECIFICATIONS

References & Technical Citations

  1. Gerganov, G. & GGML Contributors. llama.cpp & GGUF Model File Format Specification (Q4_K_M 4-Bit and Q8_0 8-Bit Block Quantization). https://github.com/ggml-org/llama.cpp
  2. IBM Technology / NVIDIA Technical Glossary. What Is Edge AI? On-Device Artificial Intelligence & Local Hardware Inference. https://www.ibm.com/think/topics/edge-ai
  3. Hugging Face Inc. Safetensors: Simple, Safe Way to Store and Distribute Tensors. https://github.com/huggingface/safetensors
  4. Padilla, T. VTM Studio Desktop Builds, Local Edge AI Benchmarks & System Requirements. https://videotronicmaker.com/download/
  5. Radford, A. et al. (OpenAI) & Klein, G. (SYSTRAN). Robust Speech Recognition via Large-Scale Weak Supervision (Whisper) & faster-whisper CTranslate2 INT8 Runtime. https://github.com/SYSTRAN/faster-whisper
  6. Hansen, M. & Open Home Foundation. Piper: Fast, Local ONNX Text-to-Speech Synthesis System. https://github.com/OHF-Voice/piper1-gpl
  7. Google DeepMind. Gemma 3 Technical Report (arXiv:2503.19786) & Gemini API Token Context Window Documentation. https://arxiv.org/abs/2503.19786 · https://ai.google.dev/gemini-api/docs/models