SinGatedLM

Overview

SinGatedLM is an earlier architecture experiment using attention-derived output as a sinusoidal multiplicative gate. It motivated further investigation in Alethic.

Mechanism

$f(x,y)=(Wx+b)\odot(\alpha\sin(y))$

Here $y$ is derived from attention. This predecessor expression is not the current SGCA control formulation.

Experimental setups

Two reported parameter-matched configurations used Tiny Shakespeare. Their settings are kept separate. Context and seed identifiers were not supplied for these reports.

Final reported metrics; different setups remain separate. Missing settings are marked unreported. No statistical significance is implied.

Parameter matching

The ~1M comparison has 1,027,048 SinGated parameters versus 1,027,047 baseline parameters: one parameter apart. This controls one confounder; it does not prove that compute, optimization, and every initialization detail matched.

Reading the results

For the ~1M run, validation loss was 1.5902 versus 1.7592 after 8,000 steps, an observed difference of 0.1690. Reported perplexities were approximately 4.90 and 5.81.

The separate ~64K report recorded 2.5603 versus 2.6931, a difference of 0.1328. Other historical run values must not be silently merged into these configurations.

Training curves

A complete per-step series has not been supplied. No interpolated or reconstructed training curve is shown.

What came next

Priorities include multiple seeds, broader datasets, deeper networks, sin/tanh/sigmoid/identity ablations, matched parameters, matched compute, and downstream evaluation.

Repository and reproduction

Inspect the repository. Use its actual configurations and instructions. Resolve missing seed and context information before claiming an independently reproduced run.