Research question
I studied how storage layout and compression affect the retrieval of large-scale synchrotron and neutron scattering data from AWS S3.
Method
The benchmark compared HDF5 through HSDS, Zarr, TileDB and ROOT/TTree using a synthetic detector stack of approximately 20 GiB derived from real detector frames. It examined interactive single-frame reads and full sequential scans, alongside request counts and stored size.
Findings
In this tested setup, Zarr with zstd had the lowest reported geometric-mean interactive-read latency, approximately 155 ms. TileDB completed full scans faster than Zarr across the tested codecs.
These results describe this dataset, implementation and measurement environment. They are not a universal ranking of storage formats or measurements of Scatterin’s deployed service.
Reproducibility
The public repository contains the thesis, LaTeX source, experiment notebooks, figures and benchmark tables. The result tables and plotting notebook provide the source behind the reported comparisons.
This is a master’s thesis completed at KTH in 2026, not a peer-reviewed journal publication.