Speaker
Description
The MWA archive is creaking; data is being deleted.
All of that data represents valuable observing time, yet some of that data has never been examined.
Both of these problems relate to the size of the file; when the processing takes too long the data never gets imaged, particularly if one starts to consider division by frequency or time.
However, if we could reduce the sizes of the files we could both avoid discarding the data and
fully exploit the MWA archive to extract the full potential of the instrument,
as the I/O limit is one of the bottlenecks to investigating all of the dimensions of the data held.
Compressing the data is an obvious solution; for visibilities only lossy compression can deliver a significant saving and we have been investigating the impacts on the final data products, and find these can be minimal.
We are using the lossy compression tool MGARD, which is a richly featured multi-grid compression application.
This outperforms existing solutions, i.e. DYSCO, by allowing us to apply physical error limits (i.e. a fraction of the system temperature) and to alter our compression conditions as a function of observing parameters.
That is, for example, we can implement compression that is time dependent, or baseline length dependent.
We will present our results, demonstrating that no significant losses are introduced.
We are building all our tools into a simple user interface, which will allow a user to pull their data from the archive, compress it by an order of magnitude and replace the original version, rather than delete it.
