Undeduplicatable/uncompressible data

Testing anything that uses deduplication and/or compression? Use data that is entirely unique.

You can create unique data with the /dev/random (or urandom) but it’s terribly slow if you need allot of data, 10-20MB/s(per thread). OpenSSL has a faster and better way to create random data, using OpenSSL will improve the speed up to 120MB/s(per thread).
Create a 1GB unique file with the following command(openssl required):
openssl rand -out 1GB-random-A -base64 $(( 2**30 * 3/4 ))

Want to fill 1TB of storage with random data, use the line below (openssl & bash required):
for i in {1..1000}; do openssl rand -out 1GB-random-%i -base64 $(( 2**30 * 3/4 )); done

Using this while testing an PureStorage storage array I was able to keep the dedup/compression ratio on a 1:1 with 10TB of pseudo data. (FYI same result on Nexenta)

4 unique 1GB files you can download and use directly:
[wpdm_file id=1] [wpdm_file id=2] [wpdm_file id=3] [wpdm_file id=4]

Leave a Reply

Your email address will not be published. Required fields are marked *