Example usage for org.apache.spark.api.java StorageLevels MEMORY

Introduction

In this page you can find the example usage for org.apache.spark.api.java StorageLevels MEMORY_ONLY.

Prototype

StorageLevel MEMORY_ONLY

To view the source code for org.apache.spark.api.java StorageLevels MEMORY_ONLY.

Click Source Link

Usage

From source file:demo.jaxrs.server.socket.Server.java

License:Apache License

protected Server(String args[]) throws Exception {

    ServerSocket sparkServerSocket = new ServerSocket(9999);
    ServerSocket jaxrsResponseServerSocket = new ServerSocket(10000);
    Socket jaxrsResponseClientSocket = new Socket("localhost", 10000);

    SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("JAX-RS Spark Socket Connect");
    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(1));

    SparkStreamingOutput streamOut = new SparkStreamingOutput(jssc);
    SparkStreamingListener sparkListener = new SparkStreamingListener(streamOut);
    jssc.addStreamingListener(sparkListener);

    JavaDStream<String> receiverStream = jssc.socketTextStream("localhost", 9999, StorageLevels.MEMORY_ONLY);

    JavaPairDStream<String, Integer> wordCounts = SparkUtils.createOutputDStream(receiverStream, true);
    PrintStream sparkResponseOutputStream = new PrintStream(jaxrsResponseClientSocket.getOutputStream(), true);
    wordCounts.foreachRDD(new SocketOutputFunction(sparkResponseOutputStream));

    jssc.start();//from  w w  w  .ja v a2s .com

    Socket receiverClientSocket = sparkServerSocket.accept();
    PrintStream sparkOutputStream = new PrintStream(receiverClientSocket.getOutputStream(), true);
    BufferedReader sparkInputStream = new BufferedReader(
            new InputStreamReader(jaxrsResponseServerSocket.accept().getInputStream()));

    JAXRSServerFactoryBean sf = new JAXRSServerFactoryBean();

    sf.setResourceClasses(StreamingService.class);
    sf.setResourceProvider(StreamingService.class,
            new SingletonResourceProvider(new StreamingService(sparkInputStream, sparkOutputStream)));
    sf.setAddress("http://localhost:9000/spark");
    sf.create();

    jssc.awaitTermination();
    sparkServerSocket.close();
    jaxrsResponseServerSocket.close();
    jaxrsResponseClientSocket.close();

}

Example usage for org.apache.spark.api.java StorageLevels MEMORY_ONLY

Introduction

Prototype

Usage