Example usage for org.apache.hadoop.mapred.lib NLineInputFormat getSplits

Introduction

In this page you can find the example usage for org.apache.hadoop.mapred.lib NLineInputFormat getSplits.

Prototype

public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException 

Source Link

Document

Logically splits the set of input files for the job, splits N lines of the input as one split.

Usage

From source file:crunch.MaxTemperature.java

License:Apache License

@Test
    public void nLine() throws Exception {
        String input = "On the top of the Crumpetty Tree\n" + "The Quangle Wangle sat,\n"
                + "But his face you could not see,\n" + "On account of his Beaver Hat.";

        writeInput(input);/*w w w.  j a va  2  s  .c o m*/

        conf.setInt("mapred.line.input.format.linespermap", 2);
        NLineInputFormat format = new NLineInputFormat();
        format.configure(conf);
        InputSplit[] splits = format.getSplits(conf, 2);
        RecordReader<LongWritable, Text> recordReader = format.getRecordReader(splits[0], conf, Reporter.NULL);
        checkNextLine(recordReader, 0, "On the top of the Crumpetty Tree");
        checkNextLine(recordReader, 33, "The Quangle Wangle sat,");
        recordReader = format.getRecordReader(splits[1], conf, Reporter.NULL);
        checkNextLine(recordReader, 57, "But his face you could not see,");
        checkNextLine(recordReader, 89, "On account of his Beaver Hat.");
    }

---