Monarch is a thought-grounded multimodal architecture. It separates perception, thought formation, memory, specialist processing, reasoning, and output instead of asking one token stream to perform every job at once. This page explains the current design block by block, the data being prepared for it, the compute limits shaping the build, and the parts that still require real training signals before they can be trusted.
The complete system in one view
Monarch begins with text, images, audio, video, or document chunks. A frozen multimodal encoder converts those inputs into native semantic vectors. Monarch then converts those vectors into its own internal space, forms an active ThoughtVector, retrieves relevant memories, builds a workspace, selects useful specialists, and runs one fixed-depth reasoning transformer before generating an answer.
Perception, thought formation, retrieval, reasoning, and output are separate jobs. The reasoning transformer never operates directly on raw pixels, audio samples, or unprepared text.
The central architectural decision: perception asks “what is present?”, thought formation asks “what is this situation about?”, and reasoning asks “what follows from this prepared situation?”
The compute envelope
This is a single-GPU research build. Data preparation has been running on Google Colab instances with either an 80 GB A100 or a 95.6 GB Blackwell-class GPU. The open multimodal encoder itself occupies roughly 9.5–10.5 GB during tested inference.
| Constraint | Current decision | Reason |
|---|---|---|
| Frontend encoder | Frozen Omni-Embed-Nemotron-style encoder | Avoid retraining multimodal perception from scratch. |
| Raw dataset storage | Disposable Colab local disk | Local reads are faster than repeatedly reading media from Drive. |
| Permanent storage | FP16 embeddings, metadata, manifests and checkpoints on Drive | Raw media can disappear with the session; reusable vectors cannot. |
| Reasoning compute | One six-layer transformer pass | Lower latency and fewer instability risks than recurrent reasoning. |
| Memory search | Approximate nearest-neighbour indexes | Avoid scanning every semantic or episodic record. |
| Knowledge strategy | semantic memory & episodic memory | Weights provide competence; explicit memory supplies searchable concepts and cases. |
One normalized 2048-dimensional FP16 vector uses 4096 bytes, or about 4 KB. A corpus of 250,000 vectors therefore requires roughly one gigabyte for the dense arrays before metadata and indexes.
What the hardware changes: it encourages precomputation, short reproducible shards, conservative batch sizes, fixed-depth reasoning, and staged training rather than one enormous end-to-end run.
Inputs and thought formation
Raw multimodal input
Monarch accepts supported combinations of text, images, audio, video, documents, and extracted structured chunks. Raw data is not passed directly into Monarch’s reasoning core.
Omni-Embed encoder layer
The current frontend uses NVIDIA’s Omni-Embed-Nemotron model. It maps supported modalities into one 2048-dimensional semantic space. This replaces the older design with separate CLIP, Whisper, text, and hand-built structural encoders.
The encoder supplies perception and cross-modal alignment. It does not produce Monarch’s final answer or its internal ThoughtVector.
Native vector storage
Encoder vectors are stored in their native width before any Monarch-specific projection. Every shard is accompanied by metadata describing its source, modality, domain and record identity.
Keeping the native vectors means the adapter or Monarch width can change later without encoding the raw corpus again.
Input Adapter
The adapter translates the encoder’s 2048-dimensional representation into Monarch’s 1536-dimensional internal space. Encoder-space and Monarch-space vectors must never be compared directly.
Thought Transformer
The Thought Transformer organizes perceived meaning into an active thought. It uses learned latent thought tokens representing global, visual, relational, causal and task-focused views of the input.
It does not generate text and it does not perform final reasoning. Its output is a single 1536-dimensional ThoughtVector.
Persistent state and explicit memory
PersistentSelf
The ThoughtVector describes the current situation. PersistentSelf is a separate 1536-dimensional runtime vector carrying capability estimates, recent interaction context and session-sensitive information.
New users begin from a learned blank_self vector. Each runtime
session receives its own copy, preventing one user’s state from mutating the
shared default.
Retrieval Query Head
Memory is not searched directly with the raw ThoughtVector. A Query Head combines the current thought with PersistentSelf and produces a normalized retrieval key.
Semantic and episodic keys must be produced in this same learned key space. This alignment requires contrastive retrieval training with relevant and irrelevant examples.
Semantic memory
Semantic memory stores reusable concepts and procedures: load distribution, proof strategies, stoichiometric relationships, repair procedures and other abstractions that should be useful across many tasks.
Search uses a trained FAISS IVF-PQ index. IVF-PQ cannot accept useful searches immediately after construction: its centroids and product-quantisation codebooks must first be trained on representative normalized keys.
Episodic memory
Episodic memory stores specific past cases rather than general concepts. Examples include a previous solved problem, a failed attempt and correction, or user-specific context from an earlier interaction.
Episodes use HNSW because new records can be inserted dynamically. The stored
key is q.detach(), not the raw ThoughtVector, ensuring writes and
later searches use the same key space.
Memory Fusion
Semantic and episodic retrieval both return 1536-dimensional value summaries. They are concatenated and compressed into one clean memory context.
Assessment, workspace, and learned specialists
Thought Assessment
The older “emotional analog” has been replaced with a more operational assessment module. A shared trunk reads the current thought, self state and retrieved memory. Five separate heads estimate novelty, risk, confidence, importance and social context.
These scores are not meaningful merely because the network outputs numbers. They require auxiliary labels or defensible self-supervised training signals.
Workspace Builder
The workspace is Monarch’s current reasoning table. It combines the active thought, retrieved memory and projected assessment signals into one 1536-dimensional state.
Specialist Gate
The gate routes using both the workspace and PersistentSelf:
[w0 ; s]. It does not route directly from raw input or the
ThoughtVector alone.
It produces a probability for every learned expert, selects the top-k, and normalizes their selected weights. A load-balancing loss is required to prevent every task from collapsing into one popular expert.
Selected specialists
Each selected expert transforms the same workspace from a learned perspective. Names such as causal, spatial, planning, mathematical or language may be attached for monitoring, but the underlying behavior is learned rather than manually programmed.
Cross-specialist fusion
Selected specialist outputs attend to one another, then merge using the gate probabilities. This allows one high-signal specialist to contribute more than a weaker one.
Specialist names are not guarantees. Useful specialization must emerge through task pressure, balanced routing, and measurable expert behavior during training.
Reasoning: one transformer pass, not a loop
The current reasoning design intentionally rejects the older recursive system. There is no repeated shared backbone, adaptive pass count, ACT halt controller, or “think again” loop in v1.
Fixed-depth Reasoning Transformer
The merged workspace passes once through a normal transformer module containing six distinct layers. Each layer has its own parameters and executes once.
“One transformer” means one fixed stack invoked once, not literally one layer. This gives predictable compute, parallel training, simpler debugging and fewer stability problems than recurrent reasoning.
Quality Head
A Quality Head estimates whether the final reasoned state appears reliable. In v1 it is diagnostic only. It does not decide how many reasoning passes to run, because there is only one pass.
It requires verified correctness, task success, reward labels, preference data or another real target before its score can be trusted.
Output and output embedding
The reasoned state conditions the output or decoder path. The LM Head maps decoder hidden states to token logits, and decoding turns those logits into text.
The separate output_embedding is not a vocabulary-sized logit
vector. It is a pooled 1536-dimensional representation of decoder hidden states,
later used for episodic compression and self updating.
What happens after an answer
Optional episodic write
Monarch does not save every interaction. When an interaction is important enough, an Episode Compressor combines the reasoned state, output embedding, assessment and quality estimate into a 1536-dimensional episodic value.
The key is the normalized retrieval query from the interaction. HNSW top-k neighbours are checked for duplicates before adding a new record.
Bounded self update
An Experience Builder combines the thought, final reasoned state, output embedding, assessment and quality into one 1536-dimensional experience vector.
A GRU proposes a new self state, but a learned mixing coefficient controls how much of that proposal is accepted. Its output bias starts near −3, making the initial update rate roughly 0.05.
Memory and self are different: episodic memory can grow into a searchable collection of cases. PersistentSelf stays one bounded vector representing runtime state.
The data being prepared
The corpus is encoded once with the frozen multimodal frontend and stored as normalized FP16 vectors. Raw data is temporary. Permanent output uses sharded NumPy arrays, matching JSONL metadata, manifests and completion markers.
General foundation
| Source | Allocation | Role | Status |
|---|---|---|---|
| English Wikipedia | 100,000 | Factual and encyclopaedic grounding | Validated |
| OpenWebText | 60,000 | General language and web knowledge | Encoded |
| COCO 2017 | 40,000 | Image-caption alignment | Validated |
| CodeSearchNet Python | 20,000 | Code and documentation alignment | Encoded |
| ConceptNet | 15,000 | Common-sense concept relationships | Validated |
| AudioCaps | 8,000 | Short environmental audio-caption grounding | Validated |
| Clotho | 3,750 | Longer, diverse sound descriptions | Validated |
| OASST1 & bAbI | 3,250 | Dialogue format and compact logical reasoning | Final general allocation |
| Total | 250,000 | Balanced general multimodal foundation | Target |
STEM specialization plan
| Domain | Primary sources | What they add |
|---|---|---|
| Mathematics | Big-Math-RL-Verified, OpenMathInstruct, Hendrycks MATH, MathVista training | Worked solutions, verified reasoning, competition problems and visual mathematics. |
| Physics | SciInstruct Physics, selected arXiv Physics, OpenAlex concepts | Scientific reasoning, research language and concept relationships. |
| Chemistry | SMolInstruct, PubChem descriptions, ChEBI, selected ChemRxiv | Chemical reasoning, entities, ontology and research context. |
| Biology | BioInstruct, BioAlchemy, Swiss-Prot, selected PMC, PDB descriptions | Biomedical instruction, proteins, structures and scientific literature. |
Domain-level packaging: each STEM domain is prepared as one combined corpus and saved under one domain folder. Metadata preserves the original source of every row without requiring a separate training pipeline for every dataset.
Evaluation-only boundary
Evaluation sets remain separate from training data. Current protected examples include ARC-AGI, HumanEval, OlymMATH, Omni-MATH, GPQA domain subsets, UGPhysics, PhysReason, SUPERChem, ChemCoTBench-V2 and BioProBench. Validation or test splits from multimodal datasets must also remain isolated.
Why this matters: benchmark contamination can make a weak model look capable. A meaningful evaluation requires keeping benchmark questions and answers outside the training and semantic-memory build.
What still has to be learned
The blocks are now dimensionally and operationally defined, but architecture alone does not make their outputs meaningful. Several modules require explicit objectives.
| Component | Required pressure | Failure without it |
|---|---|---|
| Input Adapter | Preserve useful encoder geometry while adapting to Monarch space | Information loss or arbitrary projection. |
| Thought Transformer | Task losses, alignment objectives and anti-collapse regularisation | All ThoughtVectors become similar or remain encoder copies. |
| Query Head | Contrastive positive and negative retrieval pairs | Queries and stored keys occupy incompatible geometry. |
| Thought Assessment | Labels or defensible self-supervised proxies | Risk, novelty and confidence are meaningless numbers. |
| Specialist Gate | Task loss plus expert load balancing | One expert receives nearly every example. |
| Quality Head | Verified answers, preferences, rewards or task success | Quality scores cannot be trusted. |
| Bounded self update | Longer-horizon evidence that updates improve behaviour | Self drift or changes with no measurable benefit. |
Ground rule: if a module requires a learning signal that does not yet exist, keep it diagnostic or disabled. Do not mistake a returned scalar for learned intelligence.
What Monarch is and what it is not
Monarch is an attempt to build a concept-oriented multimodal reasoning system under realistic compute limits. It uses a capable frozen frontend for perception, but the internal adapter, thought space, memory queries, workspace, specialists, reasoning transformer and state updates remain trainable parts of the research system.
It is not yet evidence that a vector-based architecture will outperform a standard language model. It is a concrete, testable architecture with explicit interfaces, dimensions, indexes, failure modes and training dependencies. Its value will come from measured retrieval quality, specialist behaviour, reasoning accuracy, multimodal transfer and resistance to representation collapse, not from the names assigned to its internal blocks.
No benchmark results are claimed here. Dataset preparation and architectural specification are progress, not proof. Performance claims should appear only after clean held-out evaluation.
References
- [1] NVIDIA Omni-Embed-Nemotron-3B model card : current multimodal frontend and 2048-dimensional output.
- [2]Johnson, Douze and Jégou, Billion-scale similarity search with GPUs : FAISS and indexed semantic retrieval.
- [3]Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using HNSW : dynamic episodic-memory indexing.
- [4]Bardes, Ponce and LeCun, VICReg : variance, invariance and covariance regularisation against collapse.
- [5]Wikipedia, OpenWebText, COCO, CodeSearchNet Python, and ConceptNet : general text, vision, code and conceptual grounding.
- [6]AudioCaps and Clotho : audio-caption alignment and environmental sound understanding.
- [7]OASST1 and bAbI QA : dialogue structure and compact logical reasoning.