2024 article
CAROL: Significantly Improving Fixed-Ratio Compression Framework for Resource-limited Applications
53RD INTERNATIONAL CONFERENCE ON PARALLEL PROCESSING, ICPP 2024, pp. 845–855.
open access via dl.acm.org (publisher PDF)Scientific simulations running on HPC facilities generate massive amount of data, putting significant pressure onto supercomputers' storage capacity and network bandwidth. To alleviate this problem, there has been a rich body of work on reducing data volumes via error-controlled lossy compression. However, fixed-ratio compression is not very well-supported, not allowing users to appropriately allocate memory/storage space or know the data transfer time over the network in advance. To address this problem, recent ratio-controlled frameworks, such as FXRZ, have incorporated methods to predict required error bound settings to reach a user-specified compression ratio. However, these approaches fail to achieve fixed-ratio compression in an accurate, efficient and scalable fashion on diverse datasets and compression algorithms.