Example usage for org.apache.hadoop.mapred SkipBadRecords setAutoIncrReducerProcCount

Introduction

In this page you can find the example usage for org.apache.hadoop.mapred SkipBadRecords setAutoIncrReducerProcCount.

Prototype

public static void setAutoIncrReducerProcCount(Configuration conf, boolean autoIncr) 

Source Link

Document

Set the flag which if set to true, SkipBadRecords#COUNTER_REDUCE_PROCESSED_GROUPS is incremented by framework after invoking the reduce function.

Usage

From source file:edu.stolaf.cs.wmrserver.streaming.PipeReducer.java

License:Apache License

public void configure(JobConf job) {
    super.configure(job);
    //disable the auto increment of the counter. For streaming, no of 
    //processed records could be different(equal or less) than the no of 
    //records input.
    SkipBadRecords.setAutoIncrReducerProcCount(job, false);
    skipping = job.getBoolean("mapred.skip.on", false);

    try {//  www. j a  v  a2  s  .  c om
        reduceOutFieldSeparator = job_.get("stream.reduce.output.field.separator", "\t").getBytes("UTF-8");
        reduceInputFieldSeparator = job_.get("stream.reduce.input.field.separator", "\t").getBytes("UTF-8");
        this.numOfReduceOutputKeyFields = job_.getInt("stream.num.reduce.output.key.fields", 1);
    } catch (UnsupportedEncodingException e) {
        throw new RuntimeException("The current system does not support UTF-8 encoding!", e);
    }
}

---

From source file:it.crs4.pydoop.pipes.PipesReducer.java

License:Apache License

public void configure(JobConf job) {
    this.job = job;
    //disable the auto increment of the counter. For pipes, no of processed 
    //records could be different(equal or less) than the no of records input.
    SkipBadRecords.setAutoIncrReducerProcCount(job, false);
    skipping = job.getBoolean(MRJobConfig.SKIP_RECORDS, false);
}

---