Example usage for org.apache.spark.api.java StorageLevels DISK

Introduction

In this page you can find the example usage for org.apache.spark.api.java StorageLevels DISK_ONLY.

Prototype

StorageLevel DISK_ONLY

To view the source code for org.apache.spark.api.java StorageLevels DISK_ONLY.

Click Source Link

Usage

From source file:my.first.streaming.JavaNetworkWordCount.java

License:Apache License

public static void main(String[] args) throws Exception {
    if (args.length < 2) {
        System.err.println("Usage: JavaNetworkWordCount <hostname> <port>");
        System.exit(1);/*from w  w  w . j  a  v a 2s. c  o m*/
    }

    //    StreamingExamples.setStreamingLogLevels();

    // Create the context with a 1 second batch size
    SparkConf sparkConf = new SparkConf().setMaster("spark://192.168.1.26:7077")
            .setAppName("JavaNetworkWordCount");//.setMaster("local[2]").setAppName("JavaNetworkWordCount");
    //    sparkConf.getsp.addJar("E:\\work\\workspace\\spark-study\\target\\spark-study-1.0-SNAPSHOT.jar");
    JavaSparkContext sc = new JavaSparkContext(sparkConf);
    sc.addJar("E:\\work\\workspace\\spark-study\\target\\spark-study-1.0-SNAPSHOT.jar");
    JavaStreamingContext ssc = new JavaStreamingContext(sc, Durations.seconds(5));

    // Create a JavaReceiverInputDStream on target ip:port and count the
    // words in input stream of \n delimited text (eg. generated by 'nc')
    // Note that no duplication in storage level only for running locally.
    // Replication necessary in distributed scenario for fault tolerance.
    JavaReceiverInputDStream<String> lines = ssc.socketTextStream(args[0], Integer.parseInt(args[1]),
            StorageLevels.DISK_ONLY);
    JavaDStream<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
        @Override
        public Iterator<String> call(String x) {
            return Arrays.asList(SPACE.split(x)).iterator();
        }
    });
    JavaPairDStream<String, Integer> wordCounts = words.mapToPair(new PairFunction<String, String, Integer>() {
        @Override
        public Tuple2<String, Integer> call(String s) {
            return new Tuple2<>(s, 1);
        }
    }).reduceByKey(new Function2<Integer, Integer, Integer>() {
        @Override
        public Integer call(Integer i1, Integer i2) {
            return i1 + i2;
        }
    });

    wordCounts.print();
    ssc.start();
    ssc.awaitTermination();
}

Example usage for org.apache.spark.api.java StorageLevels DISK_ONLY

Introduction

Prototype

Usage