On this page
Keep experiments separate
A benchmark becomes misleading when its configuration disappears. The SinGatedLM record contains multiple experiments; they should not collapse into one best number.
One ~1M run reports 1,027,048 SinGated parameters against 1,027,047 baseline parameters. After 8,000 steps on Tiny Shakespeare, validation losses were 1.5902 and 1.7592, with reported perplexities approximately 4.90 and 5.81.
A separate ~64K report gives losses of 2.5603 and 2.6931. Its training-step count, context, and seed are not supplied in the summary. Missing settings should remain visible rather than borrowed from another run.
Observation and interpretation
The observation is lower reported validation loss for the gated model in these configurations. Parameter matching controls an obvious difference in the larger run.
The interpretation is narrower than general superiority. Tiny Shakespeare is small; early models were small and shallow, seeds and architecture breadth were limited, and current SGCA differs from the predecessor gate.
Strengthening evidence
Repeat across seeds, broaden data and depth, and compare sin with identity, tanh, and sigmoid. Report parameters and compute. Hold tokenizer, data order, optimizer, token budget, context, and evaluation fixed where possible.
The goal is to understand which change caused which effect, whether it repeats, and whether its cost is acceptable—not merely to collect favorable final losses.
Preserve the record
Reproduction needs a source revision, split, tokenizer, seed, precision, context, batch construction, optimizer, schedule, and evaluation settings. The final number should lead back to that record.
The SinGatedLM page keeps the supplied configurations separate. It omits training curves because no complete per-step series was supplied. A plausible invented curve would weaken the evidence.