Example usage for org.apache.hadoop.mapreduce Job getCacheArchives

Introduction

In this page you can find the example usage for org.apache.hadoop.mapreduce Job getCacheArchives.

Prototype

public URI[] getCacheArchives() throws IOException 

Source Link

Document

Get cache archives set in the Configuration

Usage

From source file:org.apache.solr.hadoop.SolrOutputFormat.java

License:Apache License

public static void addSolrConfToDistributedCache(Job job, File solrHomeZip) throws IOException {
    // Make a reasonably unique name for the zip file in the distributed cache
    // to avoid collisions if multiple jobs are running.
    String hdfsZipName = UUID.randomUUID().toString() + '.' + ZIP_FILE_BASE_NAME;
    Configuration jobConf = job.getConfiguration();
    jobConf.set(ZIP_NAME, hdfsZipName);//w w w . j av  a 2s. c o m

    Path zipPath = new Path("/tmp", getZipName(jobConf));
    FileSystem fs = FileSystem.get(jobConf);
    fs.copyFromLocalFile(new Path(solrHomeZip.toString()), zipPath);
    final URI baseZipUrl = fs.getUri().resolve(zipPath.toString() + '#' + getZipName(jobConf));

    DistributedCache.addCacheArchive(baseZipUrl, jobConf);
    LOG.debug("Set Solr distributed cache: {}", Arrays.asList(job.getCacheArchives()));
    LOG.debug("Set zipPath: {}", zipPath);
    // Actually send the path for the configuration zip file
    jobConf.set(SETUP_OK, zipPath.toString());
}

---