Most AI products route every prompt through remote infrastructure regardless of complexity, sensitivity, or cost. Gödel 1.6 proposes a different model: run locally by default through the Core, escalate to Doors only when required, and let a lightweight Router decide which path to take.
The architecture consists of four discrete layers operating in sequence. Each is independently replaceable. The system is designed so that the majority of user interactions, casual conversation, short-form Q&A, summarization ; never leave the device. Doors are reserved for tasks demanding external access or deep specialist reasoning.
Layer 1 ; The Tokenizer
The tokenizer converts raw input text into a sequence of numerical token IDs that the Core can process. It operates entirely on-device and is the first component in the pipeline.
Two practical options exist depending on the chosen Core model. If the Gemma 3 270M path is taken, SentencePiece is already bundled no separate tokenization layer is required. For the MoE pruning path, a standalone tokenizer such as TikToken or Hugging Face's BPE implementation can be used independently, adding roughly 100KB to the local footprint.
Important: The tokenizer serves the Core only. When the Router sends a request to Doors, raw decoded text is transmitted ; not token IDs. Each cloud model maintains its own internal vocabulary. Sending local token IDs to a remote model produces malformed output.
Layer 2 ; The Core
The Core is the local brain of Gödel 1.6. It is a compact language model focused on reasoning, planning, and understanding user intent. Its design keeps it fast, low-cost to run, and capable of operating entirely on local hardware without network dependency. Two construction paths exist.
Gemma 3 270M fine-tuned
A pre-trained dense model with native SentencePiece tokenizer. Fine-tuned on domain data using LoRA or full fine-tuning. Runs on CPU at roughly 540MB in 4-bit quantized form.
MoE expert pruning
Start with a 1B+ Mixture-of-Experts model. Profile expert activation across a diverse text corpus, remove dormant experts, then quantize. Produces a structurally leaner model not just compressed.
On expert pruning
MoE architectures like Mixtral and DeepSeek-MoE route each token through only a subset of available expert sub-networks. For text-only, tool-free inference ; exactly what the Core handles ; many experts never activate. The process: run 500–1,000 diverse prompts through the base model, log which expert networks fire, remove those that remain dormant, then apply INT4 or INT8 quantization to the remaining weights. The result is structurally leaner ; fewer compute paths per token, translating directly to faster inference rather than just reduced memory.
Caution: Expert activation patterns are domain-sensitive. Experts dormant on general prompts may activate on structured outputs, rare vocabulary, or mathematical notation. A minimum of 500 diverse prompts is the practical lower bound before any pruning decisions are finalized.
Layer 3 ; The Router
The Router is the decision node of the entire system. It receives signals from the Core and determines whether Core execution is sufficient or whether a Door is required. In the Gödel architecture, the Router selects the smallest set of specialists needed ; keeping the system efficient even as the number of Doors grows.
Having the Core self-route by evaluating its own confidence is self-defeating: that reasoning pass consumes the same local resources the architecture is trying to preserve. A dedicated, standalone Router sidesteps this entirely. Routing strategies can evolve from simple rule-based decisions, to classifiers trained from usage logs, to reinforcement learning when sufficient data is available.
What the Router evaluates
| Signal | Description | Example trigger |
|---|---|---|
| Intent type | Casual vs. research vs. tool-use | "search for", "buy", "find me" |
| Query complexity | Entity count, reasoning depth, ambiguity | Multi-hop questions, named entities |
| Explicit triggers | Keyword patterns for tool-required actions | "watch", "browse", "shop", "run" |
| Core confidence | Output logit entropy from Core pass | High entropy = uncertain = escalate |
The Router itself must be extremely lightweight ; a fine-tuned BERT-tiny or a rule-based keyword system is sufficient for an initial implementation. Target inference time is under 5ms.
Design note: A two-stage approach is more efficient. A pre-Router (keyword/rule-based, sub-millisecond) intercepts obvious Door-bound requests before the Core runs at all. The Core's output confidence score then serves as a secondary signal for borderline cases ; eliminating wasted local compute on requests that will always be escalated.
Layer 4 ; Doors
Doors are where domain intelligence lives. Each Door is a cluster of specialist models ; or tool interfaces ; trained on focused, real-world data. Rather than building one giant model capable of everything, Gödel 1.6 lets many smaller experts handle different jobs. Each Door has its own internal logic, tools, and validators, and returns concrete, validated outputs rather than raw text. This makes the system modular: a Door can be upgraded or replaced without touching the Core or Router.
Groq is a well-suited provider for Door workloads ; specifically because of its LPU (Language Processing Unit) hardware, which achieves 300–500 tokens per second on Llama 3.3 70B and Mixtral 8x7B. This is critical for maintaining a seamless experience after the local Core phase.
Full Architecture ; End to End
The four layers compose into a single coherent system. The local zone ; Tokenizer, Core, and Router ; handles the vast majority of interactions autonomously. Doors are a contingency, not a dependency. The system remains functional and private without a network connection for most use cases.
Why This Architecture Matters
For users
Unlimited local usage with no rate limits or subscription requirement for standard interactions. Sensitive conversations never leave the device. The system functions offline for the majority of use cases, with Doors intelligence available on demand and transparently invoked.
For providers
Cloud API costs are incurred only for the subset of requests ; estimated at 15–30% ; that genuinely require escalation to Doors. Local devices absorb the compute cost of all remaining interactions. This fundamentally changes the unit economics of serving intelligence at scale. The modular Doors design also means individual specialist clusters can be upgraded independently as requirements evolve.
Technical Notes
Tokenizer on the Doors path. Token IDs generated by the local tokenizer must not be sent to Doors. Cloud-hosted models maintain their own internal vocabularies ; Groq's Llama 3 tokenizer and a locally-run SentencePiece tokenizer share no vocabulary. Raw decoded text must be transmitted, and each cloud model handles its own tokenization internally.
MoE pruning test corpus size. Expert utilization is domain-sensitive. Experts dormant on general-purpose prompts may activate on structured outputs, code, or rare vocabulary. A minimum of 500 diverse prompts is the practical lower bound before pruning decisions are made.
Router placement. Positioning the Router after the Core means local compute has already been spent before the routing decision is made. A two-stage approach is more efficient: a pre-Router using keyword and intent rules intercepts obvious Doors-bound requests before the Core even runs, with Core output confidence as a secondary signal.
Groq model selection for Doors. For agentic and tool-calling workloads, Llama 3.3 70B or Mixtral 8x7B are the appropriate choices. Smaller Groq-hosted models offer lower latency but reduced capability for multi-step reasoning and structured tool use.
References
- [1]Frantar et al. ; SparseGPT: Massive Language Models Can Be Accurately Pruned in One Shot, 2023.
- [2]Shazeer et al. ; Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, 2017.
- [3]Jiang et al. ; Mixtral of Experts, 2024.
- [4]Google DeepMind ; Gemma 3 Model Card. Specifications, SentencePiece integration, quantization guidance.
- [5]Groq ; Groq API Documentation. Models, LPU hardware specifications, tool-calling reference.
- [6]OpenAI ; Function Calling in LLMs. Tool-calling interface patterns for the Doors layer.
- [7]Ggerganov et al. ; llama.cpp. Quantized local model inference runtime for the Core.
- [8]MLC Team ; MLC-LLM. On-device LLM serving with WebGPU and native backends.