Example usage for org.apache.hadoop.mapreduce TaskCounter COMBINE_INPUT

Introduction

In this page you can find the example usage for org.apache.hadoop.mapreduce TaskCounter COMBINE_INPUT_RECORDS.

Prototype

TaskCounter COMBINE_INPUT_RECORDS

To view the source code for org.apache.hadoop.mapreduce TaskCounter COMBINE_INPUT_RECORDS.

Click Source Link

Usage

From source file:org.apache.tez.mapreduce.combine.MRCombiner.java

License:Apache License

public MRCombiner(TaskContext taskContext) throws IOException {
    this.conf = TezUtils.createConfFromUserPayload(taskContext.getUserPayload());

    assert (taskContext instanceof InputContext || taskContext instanceof OutputContext);
    if (taskContext instanceof OutputContext) {
        this.keyClass = ConfigUtils.getIntermediateOutputKeyClass(conf);
        this.valClass = ConfigUtils.getIntermediateOutputValueClass(conf);
        this.comparator = ConfigUtils.getIntermediateOutputKeyComparator(conf);
        this.reporter = new MRTaskReporter((OutputContext) taskContext);
    } else {/*  w w w.  ja v a2s  . c  o m*/
        this.keyClass = ConfigUtils.getIntermediateInputKeyClass(conf);
        this.valClass = ConfigUtils.getIntermediateInputValueClass(conf);
        this.comparator = ConfigUtils.getIntermediateInputKeyComparator(conf);
        this.reporter = new MRTaskReporter((InputContext) taskContext);
    }

    this.useNewApi = ConfigUtils.useNewApi(conf);

    combineInputKeyCounter = taskContext.getCounters().findCounter(TaskCounter.COMBINE_INPUT_RECORDS);
    combineInputValueCounter = taskContext.getCounters().findCounter(TaskCounter.COMBINE_OUTPUT_RECORDS);

    boolean isMap = conf.getBoolean(MRConfig.IS_MAP_PROCESSOR, false);
    this.mrTaskAttemptID = new TaskAttemptID(
            new TaskID(String.valueOf(taskContext.getApplicationId().getClusterTimestamp()),
                    taskContext.getApplicationId().getId(), isMap ? TaskType.MAP : TaskType.REDUCE,
                    taskContext.getTaskIndex()),
            taskContext.getTaskAttemptNumber());

    LOG.info("Using combineKeyClass: " + keyClass + ", combineValueClass: " + valClass + ", combineComparator: "
            + comparator + ", useNewApi: " + useNewApi);
}

Example usage for org.apache.hadoop.mapreduce TaskCounter COMBINE_INPUT_RECORDS

Introduction

Prototype

Usage