Training transformers to predict "any-to-any" as opposed to just next token solves the reversal curse and can improve planning capabilites.
DiSK is a generative framework for structured (dictionary-like) data that can handle various data types, from numbers to complex hierarchical types. This model excels in tasks like populating missing data and is especially proficient at predicting numerical values. Its potential extends to augmenting language models for better information retrieval and knowledge manipulation.
What information can we extract from neural networks? Do they tend to learn useful representations that can be interpreted by human experts? In this work, we investigate the learned representations of a model trained on nuclear physics data and show that it captures some of the underlying physical principles.
NEEMo is a technique to fit arbitrary geometries to an arbitrary collection of points. Much like WGANs, it relies on the neural estimation of the Wasserstein metric through the KR dual formulation.
A small package to make neural networks monotonic in any subset of their inputs (this works for individual neurons, too!).
We develop a novel neural architecture with an exact bound on its Lipschitz constant. The model can be made monotonic in any subset of its features. This inductive bias is especially important for fairness and interpretability considerations.
This study investigates *grokking*, a generalization phenomenon first observed in transformer models trained on arithmetic data, using microscopic and macroscopic analyses, revealing four learning phases and a “Goldilocks zone” for optimal representation learning, while emphasizing the value of physics-inspired tools in understanding deep learning.
A regularization to make neural networks' output independent from certain features.
Moment Decorrelation (MoDe) is a tool designed to ensure that a model's output remains uncorrelated with certain parameters, commonly termed as protected attributes in fairness contexts. Beyond mere decorrelation, MoDe can even shape the output of a model to adopt linear or quadratic relationships with any input/protected attribute.