Handling and Processing large amount of data between s3 buckets

S3DistCp is faster than DistCp

S3DistCp is an extension of DistCp with optimizations to work with AWS, particularly Amazon S3. S3DistCp copies data using distributed map–reduce jobs, which is similar to DistCp. S3DistCp runs mappers to compile a list of files to copy to the destination. Upon completion of the mappers compiling a list of files, the reducers perform the actual data copy. The main optimization that S3DistCp provides over DistCp is by having a reducer run multiple HTTP upload threads to upload the files in parallel.

Problem Statement

In our use case, we need to copy a large number of small files ranging from 200-600 KB, the time taken to process these 9 lakhs files was more than two hours. Our EMR setup was a 6-node cluster setup with 1 master and 5 core nodes to parallelize the processing of the data. We tried an approach of splitting data into multiple splits of input paths in lists and parallelize. But still, the smaller file issue existed, and job took longer time.

Aggregation with file-based pattern

To overcome the problem of large number of smaller files, we found a solution to aggregate these small files to fewer larger files based on their timestamp using S3DistCp.

The same cluster is used for the process where we used EMR step feature to submit the S3DistCp job.

Below are the sample of many numbers of small files for a single day present in a S3 bucket.

So, we launched an EMR cluster and using add Step we can leverage S3Distcp, and the Jar location is in built location (command-runner.jar). The S3Distcp command takes source path, destination path and target size of the aggregated file in the destination path and the aggregation Regex in the group By. The recommended approach for the target file size is keeping files larger than the default block size, which is 128 MB on EMR.

Below is the sample code format snippet

s3-dist-cp –src <source_path> –dest <destination_path> –targetSize <target_file_size> –groupBy <REGEX>

Below is the snippet of an example run, where the EMR step completed successfully

Upon completion of the S3Distcp run, below snippet shows that for each day the destination path contained single large files of the input.

The same approach can also be achieved using the AWS CLI EMR add-steps command.

aws emr add-steps –cluster-id <id> –steps ‘Name=<name>, Jar=command-runner.jar, ActionOnFailure=<action>, Type=CUSTOM_JAR, Args= s3-dist-cp –src <source_path> –dest <destination_path> –targetSize <target_file_size> –groupBy <REGEX>

S3DistCp is a magic tool to optimize the raw files of different sizes and selectively copy different files between locations.