Komprise KEDM migrates file data ‘6 times faster than the status quo – and at half the cost’

Komprise today released a file migration utility incorporating parallelisation, which runs 27 times faster than Linux Rsync, according to its internal benchmarks.

Komprise Elastic Data Management (KEDM) takes file data from an NFS or SMB/CIFS source and transfers it across a LAN or WAN to a target NAS system or via S3 or NFS/SMB to object storage systems or the public cloud.

Kumar Goswami, Komprise CEO, provided this quote: “Komprise Elastic Data Migration minimises migration downtime by using analytics to identify the right files to move, maximising data migration efficiency.”

According to the file data management company, KEDM migrates data six times faster than the status quo and at half the cost. Komprise COO Krishna Subramanian explained that; “Six times faster than status quo is an estimate only as we did not have access to commercial tools to test against them.

“On the cost, we are typically less than a half to a third of commercial solutions like DataDobi or DataDynamics StorageX for the standalone Komprise Elastic Data Migration solution.”

KEDM is an acceleration of Komprise’s transparent move technology. It executes in virtual machines running the Komprise Observer software portion of its Intelligent Data Management suite. Observers are data movers and can be scaled out in a grid to increase the migration resource.

KEDM scans the source file system, extracts the metadata, with its permissions and file attributes, and migrates the filesystem. The software parallelises operations at the share, volume, directory and file level. It uses a specially-written NFS client to minimise the number of NFS protocol interactions. This is helpful when migrating large numbers of small files as the protocol chatter takes up network bandwidth otherwise used for data payload transfer. Obviously this does not work for SMB files.

Komprise Elastic Data Migration diagram.

KEDMe is multi-threaded and uses multi-processing in a multi-core host. Every transferred file or object has an MD5 checksum calculated before and after the transfer. A post-transfer comparison verifies migration accuracy and the software retries operations automatically when a fault is detected.

KEDM provides a dashboard and has API access so it can be programmatically controlled from existing data management software.

Benchmark

Komprise ran a benchmark test against the Rsync utility using a 74GB Android open source project data set with 990,000 files -some large (1.5GB – 10GB) and thousands of small files (500B – 100KB). Source files from a disk system were transferred to an all-flash filer across a LAN and also across a simulated WAN by adding a 30ms delay to each transfer. 

KEDM was 27 times faster than Rsync. Komprise didn’t provide raw number but noted KEDM completed the run across the simulated WAN it in minute, whereas Rsync did not complete in 48 hours.

Datadobi, a competitor, also uses parallelisation and checksumming to verify migration integrity. If we assume Datadobi is part of the migration status quo, Komprise is claiming it is six times faster than DobiMigrate.

KEDM can be included in Komprise’s Intelligent Data Management suite or purchased standalone. It is available through Komprise’s channel.

Here is a one page KEDM datasheet.