NeuralOS: Towards Simulating Operating Systems via Neural Generative Models

We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Beyond reproducing existing systems, NeuralOS shows that synthesized training data can teach the model to simulate applications that were never installed, as illustrated by a Doom application, and suggests a path toward learning user interfaces purely from synthetic demonstrations.

💡 Research Summary

NeuralOS introduces a novel neural framework that simulates an operating system’s graphical user interface by directly generating screen frames from user inputs such as mouse movements, clicks, and keyboard events. The authors formalize OS GUI simulation as an autoregressive generative modeling problem: at each discrete timestep the model predicts the next frame conditioned on all previous frames and the current input sequence. To meet the unique demands of OS interaction—instantaneous response to abrupt state changes and long‑term state tracking—the architecture combines two complementary components.

The first component is a hierarchical recurrent neural network (RNN) that maintains an internal representation of the system state. It consists of a lower‑level LSTM that encodes the current input event (cursor coordinates, click flags, binary key‑press vectors) and attends over the previous frame’s latent representation via multi‑head attention. The attended vector is added to the lower‑level output and fed into an upper‑level LSTM, which captures longer‑range dependencies and feeds back its hidden state to the lower LSTM at the next step. This design keeps per‑step computational complexity constant, unlike transformers whose cost grows with context length, making it suitable for real‑time, long‑horizon OS simulation.

The second component is a diffusion‑based renderer operating in a latent space. High‑resolution screen images are first compressed by a pretrained autoencoder into lower‑dimensional latent tensors. The RNN produces a “renderer context” tensor that concatenates transformed lower‑ and upper‑level hidden states with a Gaussian spatial map encoding the cursor position. This context, together with a noisy latent frame, is input to a UNet‑style diffusion model that denoises and generates the next latent frame, which is finally decoded back to RGB. The Gaussian cursor map is crucial for sub‑pixel accuracy, reducing typical cursor‑position errors from hundreds of pixels to a few pixels.

Training proceeds in four stages. (1) RNN pre‑training: the RNN is first trained with a mean‑squared‑error loss to predict latent frames directly, ensuring that its outputs carry meaningful spatial information. (2) Joint training: the pretrained RNN and diffusion renderer are optimized together with the standard diffusion loss, preventing the renderer from ignoring the RNN. (3) Scheduled sampling: to mitigate exposure bias, a small probability is used during training to replace the most recent ground‑truth frame with the model’s own prediction, making the system robust to its own errors at inference time. (4) Context‑length extension: after initial training on short sequences for efficiency, a curriculum expands the context window so the model can learn long‑term dependencies such as delayed application launches.

The dataset consists of recordings from Ubuntu XFCE environments, totaling over ten thousand hours of interaction data. It mixes randomly generated scripts with human‑like interactions produced by large‑language‑model agents. Synthetic demonstrations are also added, notably a fabricated Doom game application that never existed on the host system; NeuralOS learns to launch, play, and close this app purely from the synthetic demonstrations, showcasing the ability to acquire UI behavior without real installation.

Quantitative results show an average cursor‑position error below 3 px, application‑launch prediction accuracy of 92 %, and high perceptual similarity to real recordings. Human evaluators could not reliably distinguish generated sequences from genuine ones. Ablation studies confirm the importance of hierarchical RNN state, Gaussian cursor encoding, and the multi‑stage training pipeline.

Limitations include coarse handling of keyboard input due to latent‑space resolution, high computational cost of diffusion sampling that approaches but does not fully meet real‑time constraints, and limited generalization to UI elements unseen during training. Security and privacy considerations are also discussed: because NeuralOS never executes real system commands, it must be run in an isolated environment.

Future work aims at lightweight diffusion architectures, higher‑resolution keyboard/text modeling, integration of multimodal inputs (speech, gestures), and safe interfacing with actual operating systems. Overall, NeuralOS represents a first step toward neural operating systems that can adapt interfaces on the fly, providing a valuable platform for HCI research and opening a path toward fully generative, learned user interfaces.