Neel Rajani

Existing approaches to combat the opacity of large language models tend to centre around post-hoc explanations of model processes. We explore an alternative approach to fostering critical understanding through intuitive, experience-driven exploration of an LLM’s inner workings. Toward that end, we present “Stepping Into the Black Box” (SIBB): a critical artefact which enables participants to “see like an LLM” and take on an active role in its generative process. Through the box, pairs of participants enact a dynamic interaction with a chatbot presented as a local expert. Participants are invited to intervene in its algorithmic decisions, building responses in tandem with the model. Their interactions and reflections speak to the kinds of criticality developed by experiential learning, highlighting the particular ways that exploring an enclosed system from within affects trust. In this way, SIBB demonstrates the potential of collaborative, experiential research-through-design practices to foster community understanding of emerging technologies.

Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforcement learning (RL) and supervised fine-tuning (SFT), but their training dynamics are poorly understood. We present a comparative analysis of RL and SFT on the same maths problems with the same model and similar hyperparameters. We find that RL yields minor in-domain gains on maths and slight degradation on knowledge-intensive benchmarks like MMLU, while both trends are more pronounced in SFT. We also analyse model parameters across checkpoints, observing that both algorithms modify query and key weights the most. Meanwhile, SFT exhibits greater updates and also affects mid-layer MLPs more, leading us to hypothesise that this may have caused the out-of-domain degradation. We therefore investigate whether freezing parts of the model during training can mitigate the reduced performance on knowledge-intensive benchmarks. However, our results are inconclusive, with benefits on GPQA:Diamond and degradation on other benchmarks. Taken together, our observations provide a preliminary indication for why RL amplifies existing capabilities, while SFT replaces old skills with new ones.