Example usage for org.apache.mahout.common.iterator.sequencefile SequenceFileDirIterable iterator

Introduction

In this page you can find the example usage for org.apache.mahout.common.iterator.sequencefile SequenceFileDirIterable iterator.

Prototype

@Override
    public Iterator<Pair<K, V>> iterator() 

Source Link

Usage

From source file:org.conan.mymahout.clustering.streaming.tools.ResplitSequenceFiles.java

License:Apache License

private void run(PrintWriter printWriter) throws IOException {
    conf = new Configuration();
    SequenceFileDirIterable<Writable, Writable> inputIterable = new SequenceFileDirIterable<Writable, Writable>(
            new Path(inputFile), PathType.LIST, conf);
    fs = FileSystem.get(conf);//  w w w .ja  v  a2s  . com

    int numEntries = Iterables.size(inputIterable);
    int numEntriesPerSplit = numEntries / numSplits;
    int numEntriesLastSplit = numEntriesPerSplit + numEntries - numEntriesPerSplit * numSplits;
    Iterator<Pair<Writable, Writable>> inputIterator = inputIterable.iterator();

    printWriter.printf("Writing %d splits\n", numSplits);
    for (int i = 0; i < numSplits - 1; ++i) {
        printWriter.printf("Writing split %d\n", i);
        writeSplit(inputIterator, i, numEntriesPerSplit);
    }
    printWriter.printf("Writing split %d\n", numSplits - 1);
    writeSplit(inputIterator, numSplits - 1, numEntriesLastSplit);
}

---