Warning: We are still working on the document infrastructure. It should be pretty stable, but changes should not be unexpected. Suggestions are welcome. Note also that most of the classes in this package should be considered examples and suggestions: while a casual user will find them invaluable in indexing data, a custom, large-scale application will usually require writing your own {@link it.unimi.di.mg4j.document.DocumentCollection}.
MG4J
aims at indexing the content of entities called documents. The main
classes that describe documents and sets of documents are included in the it.unimi.di.mg4j.document
package. In particular, documents are instances of the {@link it.unimi.di.mg4j.document.Document}
interface. A document is characterized abstractly by the following data:
Object
returned by the
{@link it.unimi.di.mg4j.document.Document#content(int)} method; the type of object that
this method returns must be known by the calling class in advance; in particular, for
textual fields (see below), the content will be a {@link java.io.Reader};
Users should always close a document after usage by calling the {@link it.unimi.di.mg4j.document.Document#close()} method: the method is responsible for relinquishing all resources that a document instantiated for its very existence.
Documents usually do not come alone, but they are grouped into collections: documents within a collection are of the same type, and this fact explains why the document structure (number and type of fields) are not contained in the document itself. Indeed, documents are produced by document factories.
A document factory is an instance of the {@link it.unimi.di.mg4j.document.DocumentFactory} interface, that in particular is able to produce a document. All documents produced by the same factory are of the same kind, and exhibit the same number and type of fields. A factory gives information about the documents it produces through the following methods:
The abovementioned methods provide information about documents produced by the factory. The actual documents are produced by the {@link it.unimi.di.mg4j.document.DocumentFactory#getDocument(java.io.InputStream,it.unimi.dsi.fastutil.objects.Reference2ObjectMap) getDocument(rawContent,metadata)} method.
This method returns a new document from the factory. The rawContent
parameter is
the most important one: it is a stream
of bytes that the factory uses to produce the document.
The factory knows how the sequence of bytes should be interpreted
to produce a document of the desired kind.
Note that even though the interpretation of the sequence of bytes representing the raw
document content is entirely left to implementors, often you
might prefer to think of the input byte sequence as of a list of consecutive self-delimiting byte
subsequences, one for each field: in this case, the
{@link java.io.InputStream#reset()} method of the {@link java.io.InputStream} class is used to divide the
subsequences from one another.
The metadata
parameter is a map providing some basic
data about the document as derived by the collection. The map is a reference map with
suitable {@link java.lang.Enum} keys, and as such must be
queried using the keys in {@link it.unimi.di.mg4j.document.PropertyBasedDocumentFactory.MetadataKeys}, or other
similar factory-specific keys. For instance, the key {@link it.unimi.di.mg4j.document.PropertyBasedDocumentFactory.MetadataKeys#TITLE}
gives a suggested title to the document (but the factory may ignore it, if it has a better way to determine
a title for the document), whereas {@link it.unimi.di.mg4j.document.PropertyBasedDocumentFactory.MetadataKeys#URI} specifies a suggested URI for
the document (which, once more, may be ignored by the factory).
Usually, a factory is built using a list of properties that define default values for metadata such as charset encoding or MIME types. There properties can be passed in several ways, and usually the main method of a collection provides an option (typically, -p) that let the user specify default metadata for the factory. The property resolution algorithm is explained in the documentation for {@link it.unimi.di.mg4j.document.PropertyBasedDocumentFactory}.
Up to this point, we have interpreted documents and document factories in a very abstract manner, but we gave no importance on the way the byte sequences representing the raw data are produced.
Basically, a source of documents is a {@link it.unimi.di.mg4j.document.DocumentSequence}. More precisely, instances of this class represent sources that are able to generate documents. Typically, a document sequence is able to produce a stream of documents after one another, through a special kind of iterator, called {@link it.unimi.di.mg4j.document.DocumentIterator}, returned by the {@link it.unimi.di.mg4j.document.DocumentSequence#iterator()} method.
A document iterator is not really a Java {@link java.util.Iterator}: it is simply a class that
exposes a method {@link it.unimi.di.mg4j.document.DocumentIterator#nextDocument()} that returns the next document,
if any, or null
when there are no more documents. Thus, document iterators can be lazy, which
is preferrable in several circumstance (e.g., documents coming from an input stream).
In some cases, the documents only appear as an uninterrupted stream and applications do not have direct accesses to single documents; in particular, it might be the case that documents just disappear after being enumerated (as it happens when the document source is standard input). In all such situations, a {@link it.unimi.di.mg4j.document.DocumentSequence} provides the only way to get the documents, because it only guarantees sequential access.
Nonetheless, there are other cases where documents can be easily accessed in a direct fashion, and can be read many times (for example, when documents are files in the file system). In such cases, {@link it.unimi.di.mg4j.document.DocumentCollection} (an extension of {@link it.unimi.di.mg4j.document.DocumentSequence}) can be used.
Apart for the methods of a document sequence, a document collection provides the following additional access methods to the documents:
size()
, exclusive);
After a document collection has been created, for example, starting from a set of files in the file system, it can usually be saved (serialized) on a file: the extension used for the filename is, by default, {@link it.unimi.di.mg4j.document.DocumentCollection#DEFAULT_EXTENSION}. Implementors of this interface should always specify explicitly which assumptions on the existence of external data are made for the consistency of a collection to be preserved. For example, a collection produced from a set of files will be consistent until no file has been changed or deleted; if the latter situation happens, the collection usually becomes inconsistent, and in any case you might expect that the indices thereof produced will no longer match the content of the collection.
Note that a document collection has very weak requirements (and thus very weak obligations) on the concurrent creation of several objects (documents, iterators, etc.). Please read carefully the {@link it.unimi.di.mg4j.document.DocumentCollection class description}.
As we explained above, document sequences/collections extract raw data (byte sequences) from some source and use a specified document factory to turn such data into documents. Hence, there exists a tight connection between each document sequence and the document factory it uses.
Typically, the document factory is provided to the document sequence at construction time, and this fact provides a form of flexibility, because different sources (e.g., the file system and the standard input) may be coupled with the same document factory (e.g., a document factory parsing HTML documents into text), or, conversely, the same document source may be used to produce documents with different formats.
Users should always be careful, however. Often, document sequences make assumptions about the factory they use, which reduces the number of possible combinations the user may adopt. Implementors of the {@link it.unimi.di.mg4j.document.DocumentSequence} interface should always clarify all the assumptions they make about the factories that can be used for the sequence.
Warning: it is a common mistake forgetting to specify some property (typically, the character encoding) for the factory when creating a document sequence (or collection). This problem cannot be detected at construction time because, in principle, the document sequence could guess the property and pass it to the factory. The problem arises, usually, in the indexing phase, when the first document is retrieved. We realise that this is confusing and counterintuitive—the collection has already been created and serialized, so what's the problem?—but there is no way to check what the sequence will do with a particular document.
Recall that a document factory is an object that is capable of producing homogeneous documents (documents with the same number/type of fields). Every document is produced starting from a raw bytestream.
The simplest possible document factory is the {@link it.unimi.di.mg4j.document.IdentityDocumentFactory}: this factory produces documents with a single textual field, called text, that is actually obtained by transforming the byte sequence into a sequence of characters, using some default encoding. Actually, a document factory must also provide a way to break the text into words. With this aim, the identity factory may be provided with a {@link java.util.Locale} that is used to determine how words are best broken in the given locale's language.
Other implementations of document factories that are provided with MG4J
are:
metadata
field of {@link it.unimi.di.mg4j.document.DocumentFactory#getDocument(java.io.InputStream,it.unimi.dsi.fastutil.objects.Reference2ObjectMap)}
should pass this information.
As we said, many document factories interpret the raw content data (an {@link java.io.InputStream}, i.e., a sequence of bytes) as if it is really made by a concatenation of many {@link java.io.InputStream}s, where each stream is typically parsed to a field; to pass from one stream to the next, the {@link java.io.InputStream#reset()} method is called.
Suppose you have n document factories D1, …, Dn, with f1, …, fn fields, respectively. One may want to build a new factory with f1+…+fn fields, where each document is produced by composing the document factories D1, …, Dn sequentially: in other words, the raw data are first passed to the first factory (that extracts f1 fields, typically resetting the stream as many times), then it is passed to the second factory (that extracts f2 fields) etc.
The {@link it.unimi.di.mg4j.document.CompositeDocumentFactory} does the job, and also allows one to change the field names (that are otherwise named as they were in the subfactories).
The class {@link it.unimi.dsi.io.MultipleInputStream} is a useful tool to produce raw data for composite factories: it allows one to convert an array of input streams into a single input stream: each time the resulting stream is reset, the multiple input stream will offer you the next stream in the array.
A special form of composite document factory is obtained using {@link it.unimi.di.mg4j.document.ReplicatedDocumentFactory}, that allows one to compose sequentially the same document factory with itself a certain number of times.
This is the simplest kind of document sequence: it just breaks a single {@link java.io.InputStream} on the basis of a given separator character; each piece of the stream is interpreted as the raw data corresponding to a document, and it is passed to a factory (specified at construction time) for converting it into a {@link it.unimi.di.mg4j.document.Document}.
This kind of collection is built starting from a set of files in the file system. Each file is interpreted as a document, and passed to a factory (specified at construction time). The suggested title for a document is the corresponding filename, and the suggested URI is the URI of the file.
There are cases in which one would like to turn a document sequence into a document collection. This may happen for one of the following reasons:
In all such cases, it may be advisable to produce a compact copy of the sequence that is easily and efficiently accessible at random. To do this, one may use the {@link it.unimi.di.mg4j.document.ZipDocumentCollectionBuilder}, that takes a document sequence and produces a "zipped clone" of the documents in the sequence: there are some mild limitations to the sequences that can be used in this context, and the resulting collection is only a partial copy of the original one, but in most cases this is sufficient for all indexing purposes. The builder will save two files: one contains the essential data concerning the zipped collection, and the other contains the zipped version of the documents.
After this, the produced {@link it.unimi.di.mg4j.document.ZipDocumentCollection} may be used as any other collection.