Efficient Action Counting with Dynamic Queries

Temporal repetition counting aims to quantify the repeated action cycles within a video. The majority of existing methods rely on the similarity correlation matrix to characterize the repetitiveness of actions, but their scalability is hindered due to the quadratic computational complexity. In this work, we introduce a novel approach that employs an action query representation to localize repeated action cycles with linear computational complexity. Based on this representation, we further develop two key components to tackle the essential challenges of temporal repetition counting. Firstly, to facilitate open-set action counting, we propose the dynamic update scheme on action queries. Unlike static action queries, this approach dynamically embeds video features into action queries, offering a more flexible and generalizable representation. Secondly, to distinguish between actions of interest and background noise actions, we incorporate inter-query contrastive learning to regularize the video representations corresponding to different action queries. As a result, our method significantly outperforms previous works, particularly in terms of long video sequences, unseen actions, and actions at various speeds. On the challenging RepCountA benchmark, we outperform the state-of-the-art method TransRAC by 26.5% in OBO accuracy, with a 22.7% mean error decrease and 94.1% computational burden reduction. Code is available at https://github.com/lizishi/DeTRC.

💡 Research Summary

The paper “Efficient Action Counting with Dynamic Queries” addresses a critical bottleneck in the field of temporal repetition counting: the computational inefficiency of existing methods. Traditionally, identifying repeated action cycles in a video has relied on calculating similarity correlation matrices between frames. While effective for short clips, this approach suffers from quadratic computational complexity ($O(N^2)$), making it practically unusable for long-duration videos or high-resolution sequences where the number of frames is substantial.

To overcome this, the authors propose a groundbreaking approach centered around an “action query representation.” By shifting the paradigm from frame-to-frame similarity to a query-based localization mechanism, the researchers have successfully reduced the computational complexity to linear ($O(N)$). This innovation allows the model to scale efficiently to much longer video sequences without the exponential growth in processing time.

The methodology is underpinned by two sophisticated technical components. The first is the “Dynamic Update Scheme.” A major challenge in action counting is the “open-set” problem, where the model encounters actions it was not explicitly trained on. Unlike traditional static queries, this scheme dynamically embeds video features into the action queries themselves. This allows the queries to adapt to the specific characteristics of the input video, providing a highly flexible and generalizable representation that can handle unseen actions with high precision.

The second component is “Inter-query Contrastive Learning.” In complex video environments, distinguishing between the target action and background noise (unrelated movements) is notoriously difficult. The authors implement a contrastive learning framework that regularizes video representations by maximizing the distinction between different action queries. This ensures that the model focuses on the relevant temporal cycles while effectively suppressing irrelevant background fluctuations.

The empirical results are highly significant. When evaluated on the challenging RepCountA benchmark, the proposed method demonstrated a massive 26.5% improvement in OBO accuracy over the previous state-of-the-art method, TransRAC. Furthermore, it achieved a 22.7% reduction in mean error and, most impressively, a 94.1% reduction in computational burden. These metrics highlight the method’s ability to provide unprecedented efficiency and accuracy, particularly in scenarios involving long videos, varying action speeds, and previously unseen action classes. This work represents a major leap forward in making automated temporal action analysis scalable and robust for real-world applications.