List of usage examples for com.google.common.collect Iterables limit
public static <T> Iterable<T> limit(final Iterable<T> iterable, final int limitSize)
From source file:org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.java
private List<ScheduledTask> getTasks(TaskQuery query) { requireNonNull(query);// w w w.j a v a2s.c o m Iterable<IScheduledTask> tasks = Storage.Util.fetchTasks(storage, Query.arbitrary(query)); if (query.getOffset() > 0) { tasks = Iterables.skip(tasks, query.getOffset()); } if (query.getLimit() > 0) { tasks = Iterables.limit(tasks, query.getLimit()); } return IScheduledTask.toBuildersList(tasks); }
From source file:com.torodb.torod.db.backends.postgresql.PostgreSQLDbConnection.java
@Override public void insertSubdocuments(String collection, SubDocType type, Iterable<? extends SubDocument> subDocuments) { Connection connection = getJooqConf().connectionProvider().acquire(); try {/*from w w w. ja va 2 s . c o m*/ int maxCappedSize = 10; int cappedSize = Iterables.size(Iterables.limit(subDocuments, maxCappedSize)); if (cappedSize < maxCappedSize) { //there are not enough elements on the insert => fallback LOGGER.debug("The insert window is not big enough to use copy (the " + "limit is {}, the real size is {}).", maxCappedSize, cappedSize); super.insertSubdocuments(collection, type, subDocuments); } else { if (!connection.isWrapperFor(PGConnection.class)) { LOGGER.warn("It was impossible to use the PostgreSQL way to " + "insert documents. Inserting using the standard " + "implementation"); super.insertSubdocuments(collection, type, subDocuments); } else { copyInsertSubdocuments(connection.unwrap(PGConnection.class), collection, type, subDocuments); } } } catch (SQLException ex) { //TODO: Change exception throw new ToroRuntimeException(ex); } finally { getJooqConf().connectionProvider().release(connection); } }
From source file:org.diqube.flatten.Flattener.java
/** * Flattens a single {@link TableShard}. * // w ww .ja va 2 s . c o m * <p> * This works as follows: * * <ol> * <li>Find all patterns the flatten-by-field pattern matches to. These are then the prefixes of the column names of * which a new row will be created. * <li>Also find the names of the length columns of these patterns. * <li>Produce a to-do list: What is the name of the output columns and what input columns is that output column * created from? * <ul> * <li>Is the new column a "multiplicating col"? These cols are cols that are outside of the path of the repeated * column that is flattened over. Nevertheless each input col contains a value for that row: A single row-value of the * input columns needs to be available for multiple cols on the output table. * <li>Remove previously found length-columns from to-be-created col list (when flattening over a[*] we do not want a * a[length] column to appear in the output!). * </ul> * <li>Iterate over all rows of the input col and identify for each row and identify (1) how many output rows that row * will create (taking into account the length columns of the flatten-by field in that row) and (2) if this row is * missing of any child-fields (i.e. there is an array a[*].c[*], when flattening over a[*], there are output cols * a.c[0], a.c[1], a.c[2], but it could be that a specific row does not contain a.c[2], because that row simply does * not have that many entries in the array. * <li>Build the new columns - each new column can be either "multiplicating" (see above), in which case the col pages * are repeated accordingly (and no-longer repeated rows are removed from the repeated colpages) or they can be * "flattned" - in which case the col is a sub-field of the flattened one and we only need to remove rows that do not * contain any value. * </ol> * * We need to ensure that we do not mess up with the row-ordering of the various output columns: Each output column * needs to have the same number of rows and the rowIds need to match correctly. Therefore, when creating a column * e.g. based on inputColumns where we do not have realized all, we need to insert "constant" column pages into the * output which will then resolve to default values. Example: * * Source table: * * <pre> * {a:[ { b:[1] }, * { b:[2, 3] }]}, * {a:[ { b:[4] }, * { b:[5, 6] }]} * </pre> * * In this example, there will be no column a[0].b[1] in the input (as all a[0]s only have at max a single entry in * .b). Would we now map new columns to col pages of old columns in the following way (flattened over a[*]; displayed * is the list of col pages that are consecutively accessed for a new column): * * <pre> * a.b[0] = [ all col pages of a[0].b[0] ] * a.b[1] = [ all col pages of a[0].b[1], all col pages of a[1].b[1] ] * a.b[length] = [ all col pages of a[0].b[length], all col pages of a[1].b[length] ] * </pre> * * .. in that way we would mess up as a.b[0] would have less rows than a.b[1] -> we need to add a "constant" colPage * to a.b[0] to resolve to a default value. Note that we nevertheless will probably never resolve those default values * (at least in this example) as the a.b[length] value will not allow us to iterate that far in the corresponding * rows. * * <p> * Note that the resulting TableShard will have the same first Row ID as the input TableShard. If multiple TableShards * of the same table are flattened (this is usually the case), then after flattening them, the row IDs might overlap * (since every TableShard has the original firstRow ID, but each table shard contains more rows). The rowIds need to * be adjusted afterwards!. */ private TableShard flattenTableShard(String resultTableName, TableShard inputTableShard, String flattenByField) throws PatternException, LengthColumnMissingException, IllegalStateException { String[] flattenFieldSplit = flattenByField .split(Pattern.quote(repeatedColNameGen.allEntriesIdentifyingSubstr() + ".")); List<String> repeatedFieldsAlongPath = new ArrayList<>(); String prev = ""; for (String splitPart : flattenFieldSplit) { if (!"".equals(prev)) prev += "."; prev += splitPart; if (!splitPart.endsWith(repeatedColNameGen.allEntriesIdentifyingSubstr())) prev += repeatedColNameGen.allEntriesIdentifyingSubstr(); repeatedFieldsAlongPath.add(prev); } // calculate the most specific patterns first - colPatternUtil will return its lists in the same ordering! repeatedFieldsAlongPath = Lists.reverse(repeatedFieldsAlongPath); Set<String> allInputLengthColsOfFlattenedFields = new HashSet<>(); ColumnPatternContainer patterns = colPatternUtil.findColNamesForColNamePattern(lengthColName -> { allInputLengthColsOfFlattenedFields.add(lengthColName); return new QueryableLongColumnShardFacade(inputTableShard.getLongColumns().get(lengthColName)); }, repeatedFieldsAlongPath); // transpose result of colPatternUtil: Collect all the most specific patterns in a set, then the second-most // specific patterns etc. // Later we want to first check if a colname matches one of the most specfic patterns as prefix and replace that, // before checking if it matches some less-specific patterns. List<Set<String>> prefixesToReplace = new ArrayList<>(); for (int i = 0; i < repeatedFieldsAlongPath.size(); i++) prefixesToReplace.add(new HashSet<>()); for (List<String> patternList : patterns.getMaximumColumnPatterns()) { for (int i = 0; i < patternList.size(); i++) prefixesToReplace.get(i).add(patternList.get(i)); } // Prefix replacements based on index in prefixesToReplace: If a prefix of prefixesToReplace.get(0) is found, that // prefix needs to be replaced by replacements.get(0). List<String> replacements = repeatedFieldsAlongPath.stream().map( pattern -> pattern.replaceAll(Pattern.quote(repeatedColNameGen.allEntriesIdentifyingSubstr()), "")) .collect(Collectors.toList()); // map from new column name to input column names that column is based upon. Note that input col names might not // exist in inputTableShard, see comments below when newColumn is filled. Map<String, SortedSet<String>> newColumns = new HashMap<>(); // output cols whose row-values are based on using input cols values and each row value of those inputs is the value // of multiple output cols Set<String> multiplicatingOutputCols = new HashSet<>(); Set<String> allInputColNames = inputTableShard.getColumns().keySet(); for (String inputColName : allInputColNames) { if (allInputLengthColsOfFlattenedFields.contains(inputColName)) // Remove certian length columns from the set of to-be-created columns. For example when flattenning over a[*], // we do not want to create a[length] column, as it simply does not make sense any more as each of the entries // in a[*] is now a separate row. continue; String newColName = null; String foundPrefix = null; int foundPatternIdx = -1; for (int patternIdx = 0; patternIdx < prefixesToReplace.size(); patternIdx++) { Set<String> prefixes = prefixesToReplace.get(patternIdx); for (String prefix : prefixes) { if (inputColName.startsWith(prefix)) { newColName = inputColName.replaceFirst(Pattern.quote(prefix), replacements.get(patternIdx)); foundPrefix = prefix; foundPatternIdx = patternIdx; if (patternIdx > 0) // not the first list of prefixes matched (= created from pattern equalling the "flatten-by"), but // less-specific patterns matched. That means that this column needs to act in a way, that the value of // one input row needs to be projected to multiple rows on the output side. // Example: matched: a[0], but flattened over a[*].b[*] multiplicatingOutputCols.add(newColName); break; } } if (newColName != null) break; } if (newColName == null) { // no replacement found, this column is on different path than the flattened one, do not flatten, do not // replace. newColName = inputColName; // At the same time, this column needs to be multiplied: One row of the input col needs to be available in // multiple rows in the output. multiplicatingOutputCols.add(newColName); } if (!newColumns.containsKey(newColName)) newColumns.put(newColName, new TreeSet<>()); // Add all "potentially available" input columns to the newColName. It could be that for a specific repetition, a // child-field is missing, e.g. a[0].c does not exist, but a[1].c does. Nevertheless, we need to reserve some // "space" for a[0].c in the new column a.c, because otherwise the rows of an existing a[0].d will mess up with // the rows of a[1].c, because a.c does contain the values of rows of a[1].c first, but a.d does contain a[0].d // first if (foundPatternIdx == -1) newColumns.get(newColName).add(inputColName); else { // add all eg. a[*].c as input columns, no matter if they exist or not. for (String inputPref : prefixesToReplace.get(foundPatternIdx)) newColumns.get(newColName) .add(inputColName.replaceFirst(Pattern.quote(foundPrefix), inputPref)); } } logger.trace("Will flatten following columns using following input cols (limit): {}", Iterables.limit(newColumns.entrySet(), 100)); logger.trace("Following columns will be multiplicating (limit): {}", Iterables.limit(multiplicatingOutputCols, 100)); // prepare information of single rows: Map<Long, Integer> multiplicationFactorByRowId = new HashMap<>(); // map from input col prefix to rowIds that are not available for all cols starting with that prefix. NavigableMap<String, NavigableSet<Long>> rowIdsNotAvailableForInputCols = new TreeMap<>(); // number of rows that are generated for one of the prefixes created based on the flatten-by value. Example: When // flattening over a[*], this will contain: a[0] -> generates X rows, a[1] -> generates Y rows. Map<String, Integer> numberOfRowsByFlattenedPrefix = new HashMap<>(); for (long inputRowId = inputTableShard.getLowestRowId(); inputRowId < inputTableShard.getLowestRowId() + inputTableShard.getNumberOfRowsInShard(); inputRowId++) { // find the cols of the "flatten-by" field that actually exist for this row. Set<List<String>> colPatterns = patterns.getColumnPatterns(inputRowId); Set<String> mostSpecificColPatterns = // most-specific = the flatten-by field! colPatterns.stream().flatMap(l -> Stream.of(l.get(0))).collect(Collectors.toSet()); // This row will produce this many rows in the output. int numberOfNewRows = mostSpecificColPatterns.size(); multiplicationFactorByRowId.put(inputRowId, numberOfNewRows); mostSpecificColPatterns .forEach(colPattern -> numberOfRowsByFlattenedPrefix.merge(colPattern, 1, Integer::sum)); // This row might not have valid values for all those repeated cols that are available in the Table for the // flatten-by field. Find those columns that are missing. for (String notAvailableColName : Sets.difference(prefixesToReplace.get(0), mostSpecificColPatterns)) { if (!rowIdsNotAvailableForInputCols.containsKey(notAvailableColName)) rowIdsNotAvailableForInputCols.put(notAvailableColName, new TreeSet<>()); rowIdsNotAvailableForInputCols.get(notAvailableColName).add(inputRowId); } } logger.trace("Multiplication factors are the following for all rows (limit): {}", Iterables.limit(multiplicationFactorByRowId.entrySet(), 100)); int maxMultiplicationFactor = multiplicationFactorByRowId.values().stream().mapToInt(Integer::intValue) .max().getAsInt(); // Build new col shards List<StandardColumnShard> flattenedColShards = new ArrayList<>(); for (String newColName : newColumns.keySet()) { long nextFirstRowId = inputTableShard.getLowestRowId(); // find colType by searching an input col that exists and taking the coltype of that one. ColumnType colType = newColumns.get(newColName).stream() .filter(inputColName -> inputTableShard.getColumns().containsKey(inputColName)) .map(inputColName -> inputTableShard.getColumns().get(inputColName).getColumnType()).findAny() .get(); // Collect all the col dictionaries of the input columns: // map from an artificial ID to the dictionary of an input column. The artificial ID is built the following way: // The first dict has artificial ID 0. // The second dict has artificial ID = number of entries in first dict // The third dict has artificial ID = number of entries in second dict // and so on // -> basically every entry in the dict has it's own artificial ID. These must not be overlapping! // The artificial ID is defined in a way so it can be fed to #mergeDicts(.) Map<Long, Dictionary<?>> origColDicts = new HashMap<>(); long nextColAndColDictId = 0L; for (String inputColName : newColumns.get(newColName)) { Dictionary<?> dict; if (inputTableShard.getColumns().containsKey(inputColName)) dict = inputTableShard.getColumns().get(inputColName).getColumnShardDictionary(); else { // assume we had an input col dict for this non-existing col. if (inputColName.endsWith(repeatedColNameGen.lengthIdentifyingSuffix())) // length cols get "0" as default. dict = new ConstantLongDictionary(0L); else dict = createDictionaryWithOnlyDefaultValue(colType); } origColDicts.put(nextColAndColDictId, dict); nextColAndColDictId += dict.getMaxId() + 1; } // merge the input column dicts into the new column dict. Pair<Dictionary<?>, Map<Long, Map<Long, Long>>> mergeDictInfo = mergeDicts(newColName, colType, origColDicts); Dictionary<?> colDict = mergeDictInfo.getLeft(); // new col pages. List<ColumnPage> flattenedColPages = new ArrayList<>(); // we'll use the same counting mechanism that we used fot origColDicts. nextColAndColDictId = 0L; long[] nextPageValues = new long[ColumnShardBuilder.PROPOSAL_ROWS]; int nextPageValueNextIdx = 0; // build col pages for (String inputColName : newColumns.get(newColName)) { long curColId = nextColAndColDictId; Map<Long, Long> columnValueIdChangeMap = mergeDictInfo.getRight().get(curColId); if (!inputTableShard.getColumns().containsKey(inputColName)) { // This col does not exist, therefore we add an "empty" colPage, which resolves statically to the colTypes' // default value. // The size of the page is identified by the number of rows that flattened prefix would have. int noOfRows = -1; for (String prefix : numberOfRowsByFlattenedPrefix.keySet()) { if (inputColName.startsWith(prefix)) { noOfRows = numberOfRowsByFlattenedPrefix.get(prefix); break; } } if (noOfRows == -1) throw new IllegalStateException("Could not find number of rows for empty values."); for (int i = 0; i < noOfRows; i++) { if (nextPageValueNextIdx == nextPageValues.length) { flattenedColPages.add( buildColPageFromValueArray(nextPageValues, -1, nextFirstRowId, newColName)); nextPageValueNextIdx = 0; nextFirstRowId += nextPageValues.length; } nextPageValues[nextPageValueNextIdx++] = columnValueIdChangeMap.get(0L); // constant dict -> always id 0L. } nextColAndColDictId++; // single entry dict! continue; } Dictionary<?> colShardDict = inputTableShard.getColumns().get(inputColName) .getColumnShardDictionary(); nextColAndColDictId += colShardDict.getMaxId() + 1; if (multiplicatingOutputCols.contains(newColName)) { // decompress whole column at once, so we can access it quickly later on. StandardColumnShard inputCol = inputTableShard.getColumns().get(inputColName); Map<Long, Long[]> colValueIds = new HashMap<>(); for (ColumnPage inputPage : inputCol.getPages().values()) { long[] pageValueIds = inputPage.getValues().decompressedArray(); Long[] colValueIdsByRow = inputPage.getColumnPageDict() .decompressValues(LongStream.of(pageValueIds).boxed().toArray(l -> new Long[l])); colValueIds.put(inputPage.getFirstRowId(), colValueIdsByRow); } for (int multiplication = 0; multiplication < maxMultiplicationFactor; multiplication++) for (ColumnPage inputPage : inputTableShard.getColumns().get(inputColName).getPages() .values()) { final int curMultiplicationNo = multiplication; for (int i = 0; i < inputPage.getValues().size(); i++) { Integer thisIndexMultiplicationFactor = multiplicationFactorByRowId .get(inputPage.getFirstRowId() + i); if (thisIndexMultiplicationFactor == null) thisIndexMultiplicationFactor = 1; if (thisIndexMultiplicationFactor > curMultiplicationNo) { // we need to multiplicate this row! if (nextPageValueNextIdx == nextPageValues.length) { flattenedColPages.add(buildColPageFromValueArray(nextPageValues, -1, nextFirstRowId, newColName)); nextPageValueNextIdx = 0; nextFirstRowId += nextPageValues.length; } long origColValueId = colValueIds.get(inputPage.getFirstRowId())[i]; nextPageValues[nextPageValueNextIdx++] = (columnValueIdChangeMap != null) ? columnValueIdChangeMap.get(origColValueId) : origColValueId; } } } } else { for (ColumnPage inputPage : inputTableShard.getColumns().get(inputColName).getPages() .values()) { // decompress whole column page at once, so we can access it quickly later on. long[] pageValueIds = inputPage.getValues().decompressedArray(); Long[] colValueIdsByRow = inputPage.getColumnPageDict() .decompressValues(LongStream.of(pageValueIds).boxed().toArray(l -> new Long[l])); Set<Long> sortedNotAvailableIndices; String interestingPrefix = rowIdsNotAvailableForInputCols.floorKey(inputColName); if (interestingPrefix != null && inputColName.startsWith(interestingPrefix)) { sortedNotAvailableIndices = rowIdsNotAvailableForInputCols.get(interestingPrefix) .subSet(inputPage.getFirstRowId(), inputPage.getFirstRowId() + inputPage.getValues().size()); } else sortedNotAvailableIndices = new HashSet<>(); // peek next unavailable index, works because indices are sorted. PeekingIterator<Long> notAvailableIndicesIt = Iterators .peekingIterator(sortedNotAvailableIndices.iterator()); for (int i = 0; i < inputPage.getValues().size(); i++) { if (notAvailableIndicesIt.hasNext() && notAvailableIndicesIt.peek() == inputPage.getFirstRowId() + i) { notAvailableIndicesIt.next(); continue; } if (nextPageValueNextIdx == nextPageValues.length) { flattenedColPages.add( buildColPageFromValueArray(nextPageValues, -1, nextFirstRowId, newColName)); nextPageValueNextIdx = 0; nextFirstRowId += nextPageValues.length; } long origColValueId = colValueIdsByRow[i]; nextPageValues[nextPageValueNextIdx++] = (columnValueIdChangeMap != null) ? columnValueIdChangeMap.get(origColValueId) : origColValueId; } } } } if (nextPageValueNextIdx > 0) { flattenedColPages.add(buildColPageFromValueArray(nextPageValues, nextPageValueNextIdx, nextFirstRowId, newColName)); nextFirstRowId += nextPageValueNextIdx; nextPageValueNextIdx = 0; } NavigableMap<Long, ColumnPage> navigableFlattenedColPages = new TreeMap<>(); for (ColumnPage flattendColPage : flattenedColPages) navigableFlattenedColPages.put(flattendColPage.getFirstRowId(), flattendColPage); StandardColumnShard flattenedColShard = null; switch (colType) { case STRING: flattenedColShard = columnShardFactory.createStandardStringColumnShard(newColName, navigableFlattenedColPages, (StringDictionary<?>) colDict); break; case LONG: flattenedColShard = columnShardFactory.createStandardLongColumnShard(newColName, navigableFlattenedColPages, (LongDictionary<?>) colDict); break; case DOUBLE: flattenedColShard = columnShardFactory.createStandardDoubleColumnShard(newColName, navigableFlattenedColPages, (DoubleDictionary<?>) colDict); break; } flattenedColShards.add(flattenedColShard); logger.trace("Created flattened column {}", newColName); } TableShard flattenedTableShard = tableFactory.createDefaultTableShard(resultTableName, flattenedColShards); logger.trace("Created flattened table shard " + resultTableName); return flattenedTableShard; }
From source file:brooklyn.entity.rebind.PeriodicDeltaChangeListener.java
private static String limitedCountString(Collection<?> items) { if (items == null) return null; int size = items.size(); if (size == 0) return "[]"; int MAX = 12; if (size <= MAX) return items.toString(); List<Object> itemsTruncated = Lists.newArrayList(Iterables.limit(items, MAX)); if (items.size() > itemsTruncated.size()) itemsTruncated.add("... (" + (size - MAX) + " more)"); return itemsTruncated.toString(); }
From source file:omero.cmd.graphs.DiskUsageI.java
/** * Calculate the disk usage of the model objects specified in the request. * @return the total usage, in bytes/* w w w . j a v a 2 s . c o m*/ */ private DiskUsageResponse getDiskUsage() { final IQuery queryService = helper.getServiceFactory().getQueryService(); final int batchSize = 256; final SetMultimap<String, Long> objectsToProcess = HashMultimap.create(); final SetMultimap<String, Long> objectsProcessed = HashMultimap.create(); final Usage usage = new Usage(); /* original file ID to types that refer to them */ final SetMultimap<Long, String> typesWithFiles = HashMultimap.create(); /* original file ID to file ownership and size */ final Map<Long, OwnershipAndSize> fileSizes = new HashMap<Long, OwnershipAndSize>(); /* note the objects to process */ for (final String className : classes) { final String hql = "SELECT " + getIdPropertyFor(className) + " FROM " + className; for (final Object[] resultRow : queryService.projection(hql, null)) { if (resultRow != null) { final Long objectId = (Long) resultRow[0]; objectsToProcess.put(className, objectId); } } } for (final Map.Entry<String, List<Long>> objectList : objects.entrySet()) { objectsToProcess.putAll(objectList.getKey(), objectList.getValue()); if (LOGGER.isDebugEnabled()) { final List<Long> ids = Lists.newArrayList(objectsToProcess.get(objectList.getKey())); Collections.sort(ids); LOGGER.debug("size calculator to process " + objectList.getKey() + " " + Joiner.on(", ").join(ids)); } } /* check that the objects' class names are valid */ for (final String className : objectsToProcess.keySet()) { getIdPropertyFor(className); } /* iteratively process objects, descending the model graph */ while (!objectsToProcess.isEmpty()) { /* obtain canonical class name and ID list */ final Map.Entry<String, Collection<Long>> nextClass = objectsToProcess.asMap().entrySet().iterator() .next(); String className = nextClass.getKey(); final int lastDot = className.lastIndexOf('.'); if (lastDot >= 0) { className = className.substring(lastDot + 1); } else if (className.charAt(0) == '/') { className = className.substring(1); } /* get IDs still to process, and split off a batch of them for this query */ final Collection<Long> ids = nextClass.getValue(); ids.removeAll(objectsProcessed.get(className)); if (ids.isEmpty()) { continue; } final List<Long> idsToQuery = Lists.newArrayList(Iterables.limit(ids, batchSize)); ids.removeAll(idsToQuery); objectsProcessed.putAll(className, idsToQuery); final Parameters parameters = new Parameters().addIds(idsToQuery); if ("Pixels".equals(className)) { /* Pixels may have /OMERO/Pixels/<id> files */ final String hql = "SELECT id, details.owner.id, details.group.id FROM Pixels WHERE id IN (:ids)"; for (final Object[] resultRow : queryService.projection(hql, parameters)) { if (resultRow != null) { final Long pixelsId = (Long) resultRow[0]; final Long ownerId = (Long) resultRow[1]; final Long groupId = (Long) resultRow[2]; final String pixelsPath = pixelsService.getPixelsPath(pixelsId); usage.bumpTotals().add(ownerId, groupId, className, getFileSize(pixelsPath)); usage.bumpTotals().add(ownerId, groupId, className, getFileSize(pixelsPath + PixelsService.PYRAMID_SUFFIX)); usage.bumpTotals().add(ownerId, groupId, className, getFileSize( pixelsPath + PixelsService.PYRAMID_SUFFIX + BfPyramidPixelBuffer.PYR_LOCK_EXT)); } } } else if ("Thumbnail".equals(className)) { /* Thumbnails may have /OMERO/Thumbnails/<id> files */ final String hql = "SELECT id, details.owner.id, details.group.id FROM Thumbnail WHERE id IN (:ids)"; for (final Object[] resultRow : queryService.projection(hql, parameters)) { if (resultRow != null) { final Long thumbnailId = (Long) resultRow[0]; final Long ownerId = (Long) resultRow[1]; final Long groupId = (Long) resultRow[2]; final String thumbnailPath = thumbnailService.getThumbnailPath(thumbnailId); usage.bumpTotals().add(ownerId, groupId, className, getFileSize(thumbnailPath)); } } } else if ("OriginalFile".equals(className)) { /* OriginalFiles have their size noted */ final String hql = "SELECT id, details.owner.id, details.group.id, size FROM OriginalFile WHERE id IN (:ids)"; for (final Object[] resultRow : queryService.projection(hql, parameters)) { if (resultRow != null && resultRow[3] instanceof Long) { final Long fileId = (Long) resultRow[0]; final Long ownerId = (Long) resultRow[1]; final Long groupId = (Long) resultRow[2]; final Long fileSize = (Long) resultRow[3]; fileSizes.put(fileId, new OwnershipAndSize(ownerId, groupId, fileSize)); } } } else if ("Experimenter".equals(className)) { /* for an experimenter, use the list of owned objects */ for (final String resultClassName : OWNED_OBJECTS) { final String hql = "SELECT " + getIdPropertyFor(resultClassName) + " FROM " + resultClassName + " WHERE details.owner.id IN (:ids)"; for (final Object[] resultRow : queryService.projection(hql, parameters)) { objectsToProcess.put(resultClassName, (Long) resultRow[0]); } } } else if ("ExperimenterGroup".equals(className)) { /* for an experimenter group, use the list of owned objects */ for (final String resultClassName : OWNED_OBJECTS) { final String hql = "SELECT " + getIdPropertyFor(resultClassName) + " FROM " + resultClassName + " WHERE details.group.id IN (:ids)"; for (final Object[] resultRow : queryService.projection(hql, parameters)) { objectsToProcess.put(resultClassName, (Long) resultRow[0]); } } } /* follow the next step from here on the model object graph */ for (final Map.Entry<String, String> query : TRAVERSAL_QUERIES.get(className)) { final String resultClassName = query.getKey(); final String hql = query.getValue(); for (final Object[] resultRow : queryService.projection(hql, parameters)) { if (resultRow != null && resultRow[0] instanceof Long) { final Long resultId = (Long) resultRow[0]; objectsToProcess.put(resultClassName, resultId); if ("OriginalFile".equals(resultClassName)) { typesWithFiles.put(resultId, className); } } } } if (ANNOTATABLE_OBJECTS.contains(className)) { /* also watch for annotations on the current objects */ final String hql = "SELECT child.id FROM " + className + "AnnotationLink WHERE parent.id IN (:ids)"; for (final Object[] resultRow : queryService.projection(hql, parameters)) { objectsToProcess.put("Annotation", (Long) resultRow[0]); } } if (LOGGER.isDebugEnabled()) { Collections.sort(idsToQuery); LOGGER.debug("usage is " + usage + " after processing " + className + " " + Joiner.on(", ").join(idsToQuery)); } } /* collate file counts and sizes by referer type */ for (final Map.Entry<Long, OwnershipAndSize> fileIdSize : fileSizes.entrySet()) { final Long fileId = fileIdSize.getKey(); final OwnershipAndSize fileSize = fileIdSize.getValue(); Set<String> types = typesWithFiles.get(fileId); if (types.isEmpty()) { types = ImmutableSet.of("OriginalFile"); } usage.bumpTotals(); for (final String type : types) { usage.add(fileSize.owner, fileSize.group, type, fileSize.size); } } if (LOGGER.isDebugEnabled()) { LOGGER.debug("usage is " + usage + " after including " + OriginalFile.class.getSimpleName() + " sizes"); } return usage.getDiskUsageResponse(); }
From source file:org.apache.accumulo.server.master.balancer.HostRegexTableLoadBalancer.java
@Override public long balance(SortedMap<TServerInstance, TabletServerStatus> current, Set<KeyExtent> migrations, List<TabletMigration> migrationsOut) { long minBalanceTime = 20 * 1000; // Iterate over the tables and balance each of them TableOperations t = getTableOperations(); if (t == null) return minBalanceTime; Map<String, String> tableIdMap = t.tableIdMap(); long now = System.currentTimeMillis(); Map<String, SortedMap<TServerInstance, TabletServerStatus>> currentGrouped = splitCurrentByRegex(current); if ((now - this.lastOOBCheck) > this.oobCheckMillis) { try {//from w w w . jav a 2 s .com // Check to see if a tablet is assigned outside the bounds of the pool. If so, migrate it. for (String table : t.list()) { LOG.debug("Checking for out of bounds tablets for table {}", table); String tablePoolName = getPoolNameForTable(table); for (Entry<TServerInstance, TabletServerStatus> e : current.entrySet()) { // pool names are the same as table names, except in the DEFAULT case. // If this table is assigned to a pool for this host, then move on. List<String> hostPools = getPoolNamesForHost(e.getKey().host()); if (hostPools.contains(tablePoolName)) { continue; } String tid = tableIdMap.get(table); if (tid == null) { LOG.warn("Unable to check for out of bounds tablets for table {}," + " it may have been deleted or renamed.", table); continue; } try { List<TabletStats> outOfBoundsTablets = getOnlineTabletsForTable(e.getKey(), TableId.of(tid)); if (outOfBoundsTablets == null) { continue; } Random random = new SecureRandom(); for (TabletStats ts : outOfBoundsTablets) { KeyExtent ke = new KeyExtent(ts.getExtent()); if (migrations.contains(ke)) { LOG.debug("Migration for out of bounds tablet {} has already been requested", ke); continue; } String poolName = getPoolNameForTable(table); SortedMap<TServerInstance, TabletServerStatus> currentView = currentGrouped .get(poolName); if (currentView != null) { int skip = random.nextInt(currentView.size()); Iterator<TServerInstance> iter = currentView.keySet().iterator(); for (int i = 0; i < skip; i++) { iter.next(); } TServerInstance nextTS = iter.next(); LOG.info("Tablet {} is currently outside the bounds of the" + " regex, migrating from {} to {}", ke, e.getKey(), nextTS); migrationsOut.add(new TabletMigration(ke, e.getKey(), nextTS)); if (migrationsOut.size() >= this.maxTServerMigrations) { break; } } else { LOG.warn("No tablet servers online for pool {}, unable to" + " migrate out of bounds tablets", poolName); } } } catch (TException e1) { LOG.error("Error in OOB check getting tablets for table {} from server {} {}", tid, e.getKey().host(), e); } } } } finally { // this could have taken a while...get a new time this.lastOOBCheck = System.currentTimeMillis(); } } if (migrationsOut.size() > 0) { LOG.warn("Not balancing tables due to moving {} out of bounds tablets", migrationsOut.size()); LOG.info("Migrating out of bounds tablets: {}", migrationsOut); return minBalanceTime; } if (migrations != null && migrations.size() > 0) { if (migrations.size() >= maxOutstandingMigrations) { LOG.warn("Not balancing tables due to {} outstanding migrations", migrations.size()); if (LOG.isTraceEnabled()) { LOG.trace("Sample up to 10 outstanding migrations: {}", Iterables.limit(migrations, 10)); } return minBalanceTime; } LOG.debug("Current outstanding migrations of {} being applied", migrations.size()); if (LOG.isTraceEnabled()) { LOG.trace("Sample up to 10 outstanding migrations: {}", Iterables.limit(migrations, 10)); } migrationsFromLastPass.keySet().retainAll(migrations); SortedMap<TServerInstance, TabletServerStatus> currentCopy = new TreeMap<>(current); Multimap<TServerInstance, String> serverTableIdCopied = HashMultimap.create(); for (TabletMigration migration : migrationsFromLastPass.values()) { TableInfo fromInfo = getTableInfo(currentCopy, serverTableIdCopied, migration.tablet.getTableId().toString(), migration.oldServer); if (fromInfo != null) { fromInfo.setOnlineTablets(fromInfo.getOnlineTablets() - 1); } TableInfo toInfo = getTableInfo(currentCopy, serverTableIdCopied, migration.tablet.getTableId().toString(), migration.newServer); if (toInfo != null) { toInfo.setOnlineTablets(toInfo.getOnlineTablets() + 1); } } migrations = EMPTY_MIGRATIONS; } else { migrationsFromLastPass.clear(); } for (String s : tableIdMap.values()) { TableId tableId = TableId.of(s); String tableName = tableIdToTableName.get(tableId); String regexTableName = getPoolNameForTable(tableName); SortedMap<TServerInstance, TabletServerStatus> currentView = currentGrouped.get(regexTableName); if (currentView == null) { LOG.warn("Skipping balance for table {} as no tablet servers are online.", tableName); continue; } ArrayList<TabletMigration> newMigrations = new ArrayList<>(); getBalancerForTable(tableId).balance(currentView, migrations, newMigrations); if (newMigrations.isEmpty()) { tableToTimeSinceNoMigrations.remove(s); } else if (tableToTimeSinceNoMigrations.containsKey(s)) { if ((now - tableToTimeSinceNoMigrations.get(s)) > ONE_HOUR) { LOG.warn("We have been consistently producing migrations for {}: {}", tableName, Iterables.limit(newMigrations, 10)); } } else { tableToTimeSinceNoMigrations.put(s, now); } migrationsOut.addAll(newMigrations); if (migrationsOut.size() >= this.maxTServerMigrations) { break; } } for (TabletMigration migration : migrationsOut) { migrationsFromLastPass.put(migration.tablet, migration); } LOG.info("Migrating tablets for balance: {}", migrationsOut); return minBalanceTime; }
From source file:com.google.devtools.build.lib.skyframe.SequencedSkyframeExecutor.java
private static void logDiffInfo(Iterable<Path> pathEntries, Collection<SkyKey> changedWithoutNewValue, Map<SkyKey, ? extends SkyValue> changedWithNewValue) { int numModified = changedWithNewValue.size() + changedWithoutNewValue.size(); StringBuilder result = new StringBuilder("DiffAwareness found ").append(numModified) .append(" modified source files and directory listings for ") .append(Joiner.on(", ").join(pathEntries)); if (numModified > 0) { Iterable<SkyKey> allModifiedKeys = Iterables.concat(changedWithoutNewValue, changedWithNewValue.keySet()); Iterable<SkyKey> trimmed = Iterables.limit(allModifiedKeys, 5); result.append(": ").append(Joiner.on(", ").join(trimmed)); if (numModified > 5) { result.append(", ..."); }//from w w w . j a v a2s. c o m } LOG.info(result.toString()); }
From source file:com.google.idea.blaze.base.sync.BlazeSyncTask.java
private void printWorkingSet(BlazeContext context, WorkingSet workingSet) { List<String> messages = Lists.newArrayList(); messages.addAll(workingSet.addedFiles.stream().map(file -> file.relativePath() + " (added)") .collect(Collectors.toList())); messages.addAll(workingSet.modifiedFiles.stream().map(file -> file.relativePath() + " (modified)") .collect(Collectors.toList())); Collections.sort(messages);/*from www .j av a2 s .co m*/ if (messages.isEmpty()) { context.output(PrintOutput.log("Your working set is empty")); return; } int maxFiles = 20; for (String message : Iterables.limit(messages, maxFiles)) { context.output(PrintOutput.log(" " + message)); } if (messages.size() > maxFiles) { context.output(PrintOutput.log(String.format(" (and %d more)", messages.size() - maxFiles))); } context.output(PrintOutput.output("")); }
From source file:org.apache.solr.analytics.accumulator.FacetingAccumulator.java
@Override @SuppressWarnings("unchecked") public NamedList<?> export() { final NamedList<Object> base = (NamedList<Object>) super.export(); NamedList<NamedList<?>> facetList = new NamedList<>(); // Add the field facet buckets to the output base.add("fieldFacets", facetList); for (FieldFacetRequest freq : request.getFieldFacets()) { final String name = freq.getName(); if (hiddenFieldFacets.contains(name)) { continue; }//w ww . ja va 2s . com final Map<String, Expression[]> buckets = fieldFacetExpressions.get(name); final NamedList<Object> bucketBase = new NamedList<>(); Iterable<Entry<String, Expression[]>> iter = buckets.entrySet(); final FieldFacetRequest fr = (FieldFacetRequest) freq; final FacetSortSpecification sort = fr.getSort(); final int limit = fr.getLimit(); final int offset = fr.getOffset(); final boolean showMissing = fr.showsMissing(); if (!showMissing) { buckets.remove(MISSING_VALUE); } // Sorting the buckets if a sort specification is provided if (sort != null && buckets.values().iterator().hasNext()) { int sortPlace = Arrays.binarySearch(expressionNames, sort.getStatistic()); final Expression first = buckets.values().iterator().next()[sortPlace]; final Comparator<Expression> comp = (Comparator<Expression>) first.comparator(sort.getDirection()); final List<Entry<String, Expression[]>> sorted = new ArrayList<>(buckets.size()); Iterables.addAll(sorted, iter); Collections.sort(sorted, new EntryComparator(comp, sortPlace)); iter = sorted; } // apply the limit if (limit > AnalyticsContentHandler.DEFAULT_FACET_LIMIT) { if (offset > 0) { iter = Iterables.skip(iter, offset); } iter = Iterables.limit(iter, limit); } // Export each expression in the bucket. for (Entry<String, Expression[]> bucket : iter) { bucketBase.add(bucket.getKey(), export(bucket.getValue())); } facetList.add(name, bucketBase); } // Add the range facet buckets to the output facetList = new NamedList<>(); base.add("rangeFacets", facetList); for (RangeFacetRequest freq : request.getRangeFacets()) { final String name = freq.getName(); final Map<String, Expression[]> buckets = rangeFacetExpressions.get(name); final NamedList<Object> bucketBase = new NamedList<>(); Iterable<Entry<String, Expression[]>> iter = buckets.entrySet(); for (Entry<String, Expression[]> bucket : iter) { bucketBase.add(bucket.getKey(), export(bucket.getValue())); } facetList.add(name, bucketBase); } // Add the query facet buckets to the output facetList = new NamedList<>(); base.add("queryFacets", facetList); for (QueryFacetRequest freq : request.getQueryFacets()) { final String name = freq.getName(); final Map<String, Expression[]> buckets = queryFacetExpressions.get(name); final NamedList<Object> bucketBase = new NamedList<>(); Iterable<Entry<String, Expression[]>> iter = buckets.entrySet(); for (Entry<String, Expression[]> bucket : iter) { bucketBase.add(bucket.getKey(), export(bucket.getValue())); } facetList.add(name, bucketBase); } return base; }
From source file:io.prestosql.verifier.Validator.java
public String getResultsComparison(int precision) { List<List<Object>> controlResults = controlResult.getResults(); List<List<Object>> testResults = testResult.getResults(); if (valid() || (controlResults == null) || (testResults == null)) { return ""; }//from www .j av a 2s . c om Multiset<List<Object>> control = ImmutableSortedMultiset.copyOf(rowComparator(precision), controlResults); Multiset<List<Object>> test = ImmutableSortedMultiset.copyOf(rowComparator(precision), testResults); try { Iterable<ChangedRow> diff = ImmutableSortedMultiset.<ChangedRow>naturalOrder() .addAll(Iterables.transform(Multisets.difference(control, test), row -> new ChangedRow(Changed.REMOVED, row, precision))) .addAll(Iterables.transform(Multisets.difference(test, control), row -> new ChangedRow(Changed.ADDED, row, precision))) .build(); diff = Iterables.limit(diff, 100); StringBuilder sb = new StringBuilder(); sb.append(format("Control %s rows, Test %s rows%n", control.size(), test.size())); if (verboseResultsComparison) { Joiner.on("\n").appendTo(sb, diff); } else { sb.append("RESULTS DO NOT MATCH\n"); } return sb.toString(); } catch (TypesDoNotMatchException e) { return e.getMessage(); } }