Skip to content

Project 02 · Private inference

Self-Hosted LLM

A 30-billion-parameter mixture-of-experts model generating tokens on four ARM cores. No GPU, no external API, no data ever leaving the machine.

llama.cppNemotron 3.5 30B-A3BAmpere ARMNGINX + TLS

Live demo app.ingfrerod.com

Open in new tab

The application runs live on my own infrastructure. It loads on demand to keep this page fast.

A private chat application powered by NVIDIA Nemotron 3.5 Lightning 30B-A3B, running entirely on a self-managed Oracle Cloud Ampere ARM instance. Every prompt and response stays on the server — no external APIs, no third-party telemetry.

Model & hardware

30B

NVIDIA Nemotron 3.5 Lightning 30B-A3B

Open-weight mixture-of-experts model: 30B parameters in total, only ~3B active per token — which keeps CPU inference fast.

ARM

Ampere Altra · 4 cores

Server-grade ARM64 CPU on Oracle Cloud, tuned for sustained inference.

24G

24 GB RAM

Enough headroom to keep the quantized GGUF weights fully in memory.

CPU

100% CPU inference

No GPU required — llama.cpp with ARM NEON optimizations.

Key features

  • Fully offline inference, no external API required
  • Served with NGINX + HTTPS on Oracle Cloud
  • Powered by llama.cpp using the GGUF format
  • Interactive web UI built with Streamlit
  • Complete data privacy by design

Technology

llama.cpp
Inference engine
GGUF
Quantized weights
Streamlit
Chat UI
NGINX
TLS reverse proxy