Example usage for org.apache.spark.network.util JavaUtils byteStringAsMb

Introduction

In this page you can find the example usage for org.apache.spark.network.util JavaUtils byteStringAsMb.

Prototype

public static long byteStringAsMb(String str) 

Source Link

Document

Convert a passed byte string (e.g.

Usage

From source file:azkaban.jobtype.HadoopSecureSparkWrapper.java

License:Apache License

/**
 * Get the memory GB size of Spark executor containers. The logic is as follows:
 * 1) Transforms requested memory String into a number representing amount of MBs requested.
 * 2a) If memory overhead is not set by the user, use the default logic to calculate it,
 * which is to add max(requestedMemInMB * 10%, 384) to the requested memory size.
 * 2b) If memory overhead is set by the user, directly add it.
 * 3) Use the logic inside YARN to round up the container size according to defined min
 * allocation for memory size.// w w w .j  ava2  s .com
 * 4) Return the memory GB size.
 *
 * @param mem requested executor memory size, of the format 2G or 1024M
 * @param memOverhead user defined memory overhead
 * @param config Hadoop Configuration object
 * @return the rounded executor memory GB size
 */
private static double getRoundedMemoryGb(String mem, String memOverhead, Configuration config) {
    long memoryMb = JavaUtils.byteStringAsMb(mem);
    if (memOverhead == null || !NumberUtils.isDigits(memOverhead)) {
        memoryMb += Math.max(memoryMb / 10, 384);
    } else {
        memoryMb += Long.parseLong(memOverhead);
    }
    int increment = config.getInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
            YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
    return Math.ceil(memoryMb * 1.0 / increment) * increment / 1024;
}

---