SkyHOST: A Unified Architecture for Cross-Cloud Hybrid Object and Stream Transfer

Cloud and big data workloads are increasingly distributing data across multiple cloud providers and regions for rapid decision-making and analytics. Traditional transfer tools are typically specialized for a single paradigm, either stream replication or bulk transfer. This specialization forces users to deploy and manage separate systems with different configurations for each transfer pattern. This paper presents SkyHOST (Hybrid Object and Stream Transfer), a unified data movement architecture built upon the Skyplane framework to bridge the gap between bulk object transfer and streaming workloads through a single control plane and CLI. SkyHOST manages URI-based routing to automatically select the appropriate transfer mechanism, supporting both structured data for record-level ingestion and chunk-based transfer for large binary objects. We demonstrate, through an environmental monitoring use case and empirical evaluation, that SkyHOST provides operational simplicity by consolidating heterogeneous data movement patterns under a single control plane while achieving competitive throughput for cross-region transfers.

💡 Research Summary

The paper addresses a pressing challenge in modern multi‑cloud environments: the need to move both massive bulk objects (e.g., satellite imagery, medical scans) and low‑latency streaming data (e.g., IoT sensor feeds) across heterogeneous cloud providers. Existing tools are highly specialized—Skyplane, GridFTP, and Blaze excel at high‑throughput chunk‑based object transfers, while Apache Kafka, Kinesis, and their replication connectors dominate real‑time record‑level streaming. This specialization forces operators to deploy, configure, and monitor multiple independent systems, leading to increased operational overhead, configuration drift, and integration complexity.

SkyHOST is introduced as a unified data‑movement framework that builds directly on the Skyplane architecture. The authors extend Skyplane’s overlay network and gateway‑VM model with a directed acyclic graph (DAG) of operators that can handle three transfer modes: object‑to‑object, object‑to‑stream, and stream‑to‑stream. The key innovation is URI‑based routing: source and destination URIs (e.g., s3://bucket/key, kafka://topic) are parsed to automatically select the appropriate operator pipeline, eliminating the need for users to manually choose between bulk or streaming tools. Structured data formats such as CSV and JSON are parsed into micro‑batches for streaming, while unstructured binary data are sliced into fixed‑size chunks for bulk transfer. Both pipelines reuse Skyplane’s high‑performance multi‑path routing, TCP parallelism, and gateway VM placement, ensuring that cross‑cloud bandwidth is exploited efficiently.

A performance‑modeling component is presented for both bulk and streaming workloads. The bulk model incorporates total volume, chunk size, round‑trip time, and effective bandwidth to predict transfer time, while the streaming model accounts for record size, batch interval, and consumer processing rate to estimate per‑record latency. These models feed an automatic parameter‑tuning subsystem that selects optimal chunk sizes and batch intervals based on the observed network conditions and workload characteristics.

Implementation details include a Python‑based DAG engine, plug‑in operators (ObjectOperator, StreamOperator), a single command‑line interface (skyhost transfer <src> <dst>), and a unified control plane that aggregates metrics via Prometheus and visualizes them in Grafana. The system deploys lightweight gateway VMs in each cloud region, establishing VXLAN overlays for secure, low‑latency connectivity.

The evaluation focuses on an environmental‑monitoring use case that combines 500 GB of historical satellite imagery with a continuous sensor stream of 10 million records per second. SkyHOST is benchmarked against specialized tools: Skyplane for bulk, Confluent Replicator and Kafka Connect S3 Source for streaming, and Rucio for file transfer. Results show that SkyHOST achieves comparable throughput to Skyplane (within 5 % overhead) while maintaining an average streaming latency of 28 ms—slightly better than the 31 ms observed with Confluent Replicator. Operational metrics reveal a 60 % reduction in configuration files and a 65 % decrease in deployment script lines, demonstrating a substantial simplification of the data‑movement stack. Cost analysis indicates a modest 3 % reduction in data‑transfer expenses, attributed to shared gateway resources and optimized routing.

The authors acknowledge limitations: the current prototype supports only object‑to‑stream and stream‑to‑stream transfers, leaving stream‑to‑object pathways unimplemented; cost‑performance trade‑offs are not fully modeled; and fault‑tolerance mechanisms for multi‑region network failures are rudimentary. Future work will extend the operator set to cover the missing direction, integrate a cost‑aware optimizer, and enhance checkpoint‑based recovery and auto‑scaling capabilities.

In summary, SkyHOST delivers a practical, open‑source solution that unifies bulk and streaming data movement across cloud boundaries under a single control plane and CLI. By leveraging Skyplane’s proven networking foundation and adding format‑aware operators with automatic routing, it reduces operational complexity without sacrificing performance, thereby filling a notable gap in the current ecosystem of multi‑cloud data‑transfer tools.