Dense Associative Memory with Epanechnikov Energy

We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Moreover, it introduces abundant additional \emph{emergent} local minima while preserving perfect pattern recovery – a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR’s emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method’s potential for both large-scale memory storage and generative tasks.

💡 Research Summary

The paper introduces a novel energy function for Dense Associative Memory (DenseAM) networks called log‑sum‑ReLU (LSR), which is derived from the Epanechnikov kernel—a kernel known to be optimal for kernel density estimation (KDE). Traditional DenseAMs rely on the log‑sum‑exponential (LSE) energy, where the separation function F(x)=exp(x) and scaling Q(x)=log produce a softmax‑like gradient. While LSE can achieve exponential memory capacity M★ ≈ exp(d) by using an exponential separation function, exact retrieval of stored patterns requires an infinite inverse temperature (β→∞) and LSE does not generate new local minima beyond the stored patterns.

The authors observe that the energy of an associative memory defines an unnormalized probability density p(x)∝exp