Example usage for org.apache.hadoop.mapreduce.lib.input FileInputFormat SPLIT

Introduction

In this page you can find the example usage for org.apache.hadoop.mapreduce.lib.input FileInputFormat SPLIT_MINSIZE.

Prototype

String SPLIT_MINSIZE

To view the source code for org.apache.hadoop.mapreduce.lib.input FileInputFormat SPLIT_MINSIZE.

Click Source Link

Usage

From source file:co.cask.hydrator.plugin.batch.source.FTPBatchSource.java

License:Apache License

private static String limitSplits(@Nullable String fsProperties) {
    Map<String, String> providedProperties;
    if (fsProperties == null) {
        providedProperties = new HashMap<>();
    } else {/*from   w  w  w. j  av a2 s.  c o  m*/
        providedProperties = GSON.fromJson(fsProperties, MAP_STRING_STRING_TYPE);
    }
    providedProperties.put(FileInputFormat.SPLIT_MINSIZE, Long.toString(Long.MAX_VALUE));
    return GSON.toJson(providedProperties);
}

From source file:org.apache.eagle.jpm.analyzer.mr.suggestion.MapReduceSplitSettingProcessor.java

License:Apache License

@Override
public Result.ProcessorResult process(MapReduceAnalyzerEntity jobAnalysisEntity) {
    StringBuilder sb = new StringBuilder();

    if (context.getJobconf().getLong(FileInputFormat.SPLIT_MINSIZE, 0) > 1) {
        sb.append("Best practice: don't set " + FileInputFormat.SPLIT_MINSIZE);
        sb.append(", because it may lower data locality, hence maps will run slower.\n");
        return new Result.ProcessorResult(Result.RuleType.SPLIT, Result.ResultLevel.INFO, sb.toString());
    }//from w w w  . ja  v a 2 s. c  om
    return null;
}

From source file:org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper.java

License:Apache License

private static void populateMRSettingsToRetain() {

    // FileInputFormat
    mrSettingsToRetain.add(FileInputFormat.INPUT_DIR);
    mrSettingsToRetain.add(FileInputFormat.SPLIT_MAXSIZE);
    mrSettingsToRetain.add(FileInputFormat.SPLIT_MINSIZE);
    mrSettingsToRetain.add(FileInputFormat.PATHFILTER_CLASS);
    mrSettingsToRetain.add(FileInputFormat.NUM_INPUT_FILES);
    mrSettingsToRetain.add(FileInputFormat.INPUT_DIR_RECURSIVE);

    // FileOutputFormat
    mrSettingsToRetain.add(MRConfiguration.OUTPUT_BASENAME);
    mrSettingsToRetain.add(FileOutputFormat.COMPRESS);
    mrSettingsToRetain.add(FileOutputFormat.COMPRESS_CODEC);
    mrSettingsToRetain.add(FileOutputFormat.COMPRESS_TYPE);
    mrSettingsToRetain.add(FileOutputFormat.OUTDIR);
    mrSettingsToRetain.add(FileOutputCommitter.SUCCESSFUL_JOB_OUTPUT_DIR_MARKER);
}

Example usage for org.apache.hadoop.mapreduce.lib.input FileInputFormat SPLIT_MINSIZE

Introduction

Prototype

Usage