Block size

As we mentioned previously, the two most obvious optimizations that we can perform on the storage code are to add compression and adjust the storage block size. We have tried compression, now let’s try adjusting the block size.

To this end, please add a new runner option that controls the block size, wire it up to HDF5Writer::create(), fix callers, and then try to tune the setting up and see if it helps.

On modern local NVMe storage, you will find that this is an unfavorable tradeoff. But I wouldn’t be surprised if higher-latency storage, like the parallel filesystems used in HPC centers, would benefit from a larger block size. And in any case, it feels good to have one less hardcoded performance-sensitive number in the codebase.