REAL DATA · FROZEN SMOLLM2-135M
Can a trainable memory module compete with LoRA?
One pretrained checkpoint. A shared classifier design. An Engram-inspired module with hashed 2/3-gram lookups, a context gate, and causal convolution, compared with rank-8 LoRA on query/value projections.
Small data: 20 examples per intent
Test: 3,080 previously held-out BANKING77 queries · 77 intents
| Method | Test accuracy | Trainable parameters | Peak allocated VRAM | Saved module |
|---|
| Frozen model + classifier | 52.86% | 44,429 | 628 MB | 0.18 MB |
| LoRA + classifier | 80.62% | 505,229 | 1667 MB | 2.06 MB |
| Engram + classifier | 63.73% | 504,781 | 1358 MB | 2.02 MB |
Learning curves and settings
Frozen model + classifier
Validation by epoch: 19.5% → 41.6% → 46.9% → 51.6%
Selected learning rate: 0.001; saved module reload verified.
LoRA + classifier
Validation by epoch: 65.5% → 77.1% → 79.2% → 78.6%
Selected learning rate: 0.001; saved module reload verified.
Engram + classifier
Validation by epoch: 21.2% → 46.1% → 54.8% → 59.9%
Selected learning rate: 0.001; saved module reload verified.
Does the trained memory matter?
| Engram condition | Test accuracy |
|---|
| Trained module | 63.73% |
| Same reader, randomly rearranged memory rows | 52.53% |
| Module disabled, same trained classifier | 48.60% |
Rearranging rows breaks the mapping between a token group and its learned vector. This is a diagnostic, not a separately retrained competing model.
How much do familiar token patterns help?
| Method | Low overlap | Medium overlap | High overlap |
|---|
| Frozen model + classifier | 43.6% (n=555) | 53.9% (n=1795) | 57.4% (n=730) |
| LoRA + classifier | 75.3% (n=555) | 80.3% (n=1795) | 85.3% (n=730) |
| Engram + classifier | 50.1% (n=555) | 64.0% (n=1795) | 73.4% (n=730) |
Overlap is the fraction of input token pairs/triples seen at least twice in training. Bins: below one third, one to two thirds, and at least two thirds. These are observational groups, not a controlled causal test.
Exact normalized train/test query overlap: 1. Duplicate-excluded accuracy is saved in analysis.json.All available training examples
Test: 3,080 previously held-out BANKING77 queries · 77 intents
| Method | Test accuracy | Trainable parameters | Peak allocated VRAM | Saved module |
|---|
| Frozen model + classifier | 77.14% | 44,429 | 628 MB | 0.18 MB |
| LoRA + classifier | 89.94% | 505,229 | 2666 MB | 2.06 MB |
| Engram + classifier | 85.42% | 504,781 | 2086 MB | 2.02 MB |
Learning curves and settings
Frozen model + classifier
Validation by epoch: 53.8% → 66.6% → 72.2% → 75.1%
Selected learning rate: 0.001; saved module reload verified.
LoRA + classifier
Validation by epoch: 84.9% → 87.3% → 89.0% → 89.5%
Selected learning rate: 0.001; saved module reload verified.
Engram + classifier
Validation by epoch: 66.0% → 81.8% → 81.7% → 83.5%
Selected learning rate: 0.001; saved module reload verified.
Does the trained memory matter?
| Engram condition | Test accuracy |
|---|
| Trained module | 85.42% |
| Same reader, randomly rearranged memory rows | 71.04% |
| Module disabled, same trained classifier | 55.49% |
Rearranging rows breaks the mapping between a token group and its learned vector. This is a diagnostic, not a separately retrained competing model.
How much do familiar token patterns help?
| Method | Low overlap | Medium overlap | High overlap |
|---|
| Frozen model + classifier | 51.8% (n=83) | 69.7% (n=925) | 81.5% (n=2072) |
| LoRA + classifier | 71.1% (n=83) | 86.1% (n=925) | 92.4% (n=2072) |
| Engram + classifier | 60.2% (n=83) | 76.1% (n=925) | 90.6% (n=2072) |
Overlap is the fraction of input token pairs/triples seen at least twice in training. Bins: below one third, one to two thirds, and at least two thirds. These are observational groups, not a controlled causal test.
Exact normalized train/test query overlap: 6. Duplicate-excluded accuracy is saved in analysis.json.Next-token prediction on real text
WikiText-2 pilot: 512 training blocks, 64 validation blocks, 128 test blocks; 256 tokens per block. Each adapter receives two epochs and two learning-rate choices. Validation chooses the checkpoint. Test scores cover 32,640 predicted tokens per model.
Lower perplexity means better probability assigned to the real next tokens. These are subset results with short context resets, not the standard full WikiText benchmark. Pretraining overlap with WikiText has not been audited.
| Run | Original model | LoRA | N-gram memory |
|---|
| text-pilot-r01 | 32.01 | 17.07 | 24.19 |
| text-seed43-r01 | 32.01 | 17.13 | 24.15 |
| text-seed44-r01 | 32.01 | 17.14 | 24.10 |
| Mean ± sample SD | 32.01 ± 0.00 | 17.11 ± 0.04 | 24.15 ± 0.04 |
Same data and evaluation blocks across seeds; initialization and training order vary. Classification above remains a single-seed experiment. Standard deviation describes these runs; it is not a confidence interval or evidence across datasets.
Read every saved continuation from seed 42
Prompts were fixed before training. Greedy generation, up to 40 new tokens. These are raw outputs, including errors and repetition; no instruction tuning was performed.
The history of artificial intelligence
frozen
is a long one, and it is not a new one. The first artificial intelligence was developed in the 1950s by Alan Turing, who was a British mathematician. Turing’s
lora
is a long one . The first AI research was done in the 1950s . The first AI program was developed in 1956 by John McCarthy . The first AI program
engram
is a long and complex one. The first AI was developed in the 1950s by John McCarthy, who was a professor at Dartmouth College. He was the first to develop a formal
The city of New York
frozen
is a city of 100 million people. It is the largest city in the United States. It is the capital of the state of New York. It is the largest city in the United
lora
was the first to be built on the site of the original site of the site of the site of the site of the site of the site of the site of the site of the site of the site
engram
is the largest city in the United States. It is the capital of the state of New York and the largest city in the United States. It is also the largest city in the United Kingdom. It
In a scientific experiment,
frozen
the researchers used a small, 10-centimeter-long, 10-millimeter-wide, 10-millimeter-deep, 10-millimeter-
lora
the researchers used a 1990s model of the human brain to simulate the brain of a 1950s model of the human brain. The model was then used to simulate
engram
the researchers used a small, 10-centimeter-long, 10-gram-weight, 10-millimeter-long, 10-millimeter-wide
The company announced that
frozen
it would be able to sell its products to the public in 2015.
The company is also working on a new product called the “Sport-Touch” that will
lora
it would be selling the product in 2010 .
engram
it would be able to sell its products to the public in 2015.
The company has also announced that it will be able to sell its products to the public in 2
All methods recompute the entire prefix during generation. The saved timings are affected by shared GPU load and are not optimized KV-cache speed comparisons.
What this experiment establishes
We train and evaluate real task adapters on BANKING77. The base model stays frozen; classifier-only adaptation is the baseline. Each method gets two learning-rate trials and four epochs; validation chooses the learning rate and checkpoint. Trainable adapter budgets differ by less than 0.1%.
Classification currently reports one seed and one dataset. These are preliminary classification results, not proof of general reasoning, factual-memory portability, or superiority across tasks. The GPU had another workload, so timings are not clean throughput benchmarks. SST-2 remains untrained. Text-generation results, when available, are shown separately above.
Saved artifacts contain the task head and trained adapter; no base-model copy. Model and dataset source revisions are recorded. Engram uses raw token IDs and one insertion layer; this is an adaptation study, not an exact DeepSeek reproduction.
BANKING77 source · Engram source