The gCube Document Library (gDL) is a client library for adding, updating, deleting and retrieving document descriptions to, in, and from remote collections in a gCube infrastructure.

The gDL is a high-level component of the subsystem of gCube Information Services and it interacts with lower-level components of the subsystem to support document management processes within the infrastructure:

the gCube Document Model (gDM) defines the basic notion of document and the gCube Model Library (gML) implements that notion into objects;
the objects of the gML can be exchanged in the infrastructure as edge-labelled trees, and the Content Manager Library (CML) can dispatch them to the read and write operations of the Content Manager (CM) service;
the CM implements its operations by translating trees to and from the content models of diverse repository back-ends.

The gDL builds on the gML and the CML to implement a local interface of CRUD operations that lift those of the CM to the domain of documents, efficiently and effectively.

Preliminaries

The core functionality of the gDL lies in its operations to read and write document descriptions. The operations trigger interactions with the Content Manager service and the movement of potentially large volumes of data across the infrastructure. This may have a non-trivial and combined impact on the responsiveness of clients and the overall load of the infrastructure. The operations have been designed to minimise this impact. In particular:

when reading, clients can qualify the documents that are relevant to their queries, and indeed what properties of those documents should be actually retrieved. These retrieval directives are captured in the gDL by the notion of document projections.

when reading and writing, clients can move large numbers of documents across the infrastructure. The gDL streams this I/O movements so as to make efficient use of local and remote resources. It then defines a facilities with which clients can conveniently consume input streams, produce output streams, and more generally convert one stream into an other regardless of its origin. These facilities are collected into the stream DSL, an Embedded Domain-Specific Language (EDSL) for stream conversion and processing.

Understanding document projections and the stream DSL is key to reading and writing documents effectively with the gDL. We discuss these preliminary concepts first, and then consider their use as input and outputs in read and write the operations of the library.

Projections

A projection is a set of constraints over the properties of document descriptions. It can be be used in the read operations of the gDL to:

characterise relevant descriptions as those that match the constraints (projections as types);
specify what properties of relevant descriptions should be retrieved (projections as retrieval directives).

Constraints take accordingly two forms:

include constraints apply to properties that must be matched and retrieved;
filter constraints apply to properties that must be matched but not retrieved.

note: in both cases, the constraints take the form of predicates of the Content Manager Library (CML). The projection itself converts into a complex predicate which is amenable for processing by the Content Manager service in the execution of its retrieval operations. In this sense, projections are a key part of the document-oriented layer that the gDL defines over lower-level components of the gCube subsystem dedicated to content management.

As a first example, a projection may specify an include constraint over the name of metadata elements and a filter constraint over the time of last update. It may then be used to:

characterise document descriptions with at least one metadata element that matches both constraints;
retrieve of those descriptions only the name of matching metadata elements, excluding the time of last update, any other metadata property, and any other document property, include other inner elements and their properties.

Projections have the Projection interface, which can be used to access their constraints in element-generic computations. To build projections, however, clients deal with one of the following implementation of the interface:

DocumentProjection
MetadataProjection
AnnotationProjection
PartProjection
AlternativeProjection

A further implementation of the interface:

PropertyProjection

allows clients to express constraints on the generic properties of documents and their inner elements.

Streams

In some of its operations, the gDL relies on streams to model, process, and transfer large-scale data collections. Streams may consist of document descriptions, document identifiers, and document updates. More generally, they may consist of the outcomes of operations that take in turn large-scale collections in input. Streamed processing makes efficient use of both local and remote resources, from local memory to network bandwidth, promoting the overall responsiveness of clients and services through reduced latencies.

Clients that use these operations will need to route streams towards and across the operations of the gDL, converting between different stream interfaces, often injecting application logic in the process. As a common example, a client may need to:

route a remote result set of document identifiers towards the read operations of the gDL;
process the document descriptions returned by the read operations, e.g. in order to update some of their properties;
feed the modified document descriptions to the write operations of the gDL, so as to commit the changes;
inspect commit outcomes, so as to report or otherwise handle the failures that may have occurred in the process.

Throughout the workflow, it is important that the client remains within the paradigm of streamed processing, avoiding the accumulation of data in memory in all cases but where strictly required. Document identifiers will be streaming from the remote location of the original result set as documents descriptions will be flowing back from yet another remote location, as updated document descriptions will be leaving towards the same remote location, and as failures will be steadily coming back for handling.

Streaming raises significant opportunities for clients, as well as non-trivial challenges. In recognition of the difficulties, the gDL includes a set of general-purpose facilities for stream conversion that simplify the tasks of filtering, transforming, or otherwise processing streams. These facilities are cast as the sentences of the Stream DSL, an Embedded Domain-Specific Language (EDSL).

Standard and Remote Iterators

As all the sentences of the Stream DSL take and return streams, we begin by looking look at how streams are represented in the gDL.

Streams have the interface of iterators, i.e. yield elements on demand and are typically consumed within loops. There are two such interfaces:

Iterator<T>, the standard Java interface for iterations.
RemoteIterator<T>, a variation over Iterator<T> which marks explicitly the remote origin of the stream.

In particular, a RemoteIterator differs from a standard Iterator in two respects:

the method next() may throw a checked Exception. This witnesses to the fact that iterating over the stream involves fallible I/O operations;
there is a method locator() that returns a reference to the remote stream as a plain String in some implementation-specific syntax.

Locators aside, the key difference between the two interfaces is in their assumptions about the possibility of iteration failures. A standard Iterator does not present failures to its clients other than for requests made past end of the stream (an unchecked NoSuchElementException). This may be because failures do not occur at all, e.g. the iteration is over an in-memory collection; it may also be because the iterator knows how to handle failures when these occur. In this sense, Iterator<T> may well be defined over external, even remote collections, but it assumes that all failure handling policies are responsibilities of its implementations.

In contrast, RemoteIterator<T> makes it clear that:

failures are likely to occur;
clients are expected to handle them.

The operations of the gDL make use of both interfaces:

when they take streams, they expect them as standard Iterators;
when they return streams, the provide them as RemoteIterators.

This choice emphasises two points:

streams that are provided by clients are of unknown origin, those provided by the library originate in remote services of the gCube Content Management infrastructure.
all fault handling policies are in the hands of clients, where they should be. When clients provide an Iterator to the library, they will have embedded a fault handling policy in its implementation. When they receive a RemoteIterator from the library, they will apply a fault handling policy when consuming the stream.

Simple Conversions

The sentences of the DSL begin with verbs, which can be statically imported from the Streams class:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...

The verb convert introduces the simplest of sentences, those that convert between Iterators and RemoteIterators. The following example shows the conversion of an Iterator into a RemoteIterator:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
Iterator<SomeType> it = ...
RemoteIterator<SomeType> rit = convert(it);

The result is a RemoteIterator that promises to return failures but never does. The implementation is just a wrapper around the standard Iterator which returns it.toString() as the locator of the underlying collection.

Converting a RemoteIterator to an Iterator is more interesting because it requires the encapsulation of a fault handling policy. The following example shows the possibilities:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<SomeType> rit = ...
 
//iterator will return any fault raised by the remote iterator
Iterator<SomeType> it1 = convert(rit).with(IGNORE_POLICY); 
//iterator will stop at the first fault raised by the remote iterator
Iterator<SomeType> it2 = convert(rit).with(FAILFAST_POLICY); 
//iterator will handle fault as specified by given policy
FaultPolicy policy = new FaultPolicy() {...}; 
Iterator<SomeType> it3 = convert(rit).with(policy);

In this example, the clause with() introduces the fault handling policy to encapsulate in the resulting Iterator. Two common policies are predefined and can be named directly, as shown for it1 and it2 above:

IGNORE_POLICY: any faults raised by the RemoteIterator are discarded by the resulting Iterator<code>, which will ensure that <code>hasNext()>/code> and <code>next() behave as if they had not occurred;
FAILFAST_POLICY: the first fault raised by the RemoteIterator halts the resulting Iterator, which will ensure that hasNext()>/code> and <code>next() behave as if they stream had reached its natural end;

Custom policies can be defined by implementing the interface FaultPolicy:

public interface FaultPolicy ... {
 
	boolean onFault(Exception e, int count); 
}

In onFault(), clients are passed the fault raised by the RemoteIterator, as well as the count of faults raised so far during the iteration (this will be greater than 1 only if the policy will have tolerated some previous faults during the iteration). Clients apply the policy and return true if the fault should be tolerated and the iteration continue, false if they instead wish the iteration to stop. Here's an example of a fault handling policy that tolerates only the first error and uses two aliases for the boolean values to improve the legibility of the policy (CONTINUE and STOP, also defined in the Streams class and statically imported):

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
FaultPolicy policy = new FaultPolicy() {
 
       public boolean onFault(Exception e, int count) {
             if (count=1) {
                   ....dealing with fault ...
		   return CONTINUE;
	      }
             else 
                  return STOP;	
        }
};

Note also that the IGNORE_POLICY is the default policy from conversion to standard iterators. Clients can use the clause withDefaults() to avoid naming it.

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<SomeType> rit = ...
 
//iterator will handle faults with the default policy: IGNORE_POLICY
Iterator<SomeType> it = convert(rit).withDefaults();

Finally, note that stream conversions may also be applied between RemoteIterators, so as to change their FaultPolicy:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<SomeType> rit1 = ...
 
//iterator will handle faults with the default policy: IGNORE_POLICY
RemoteIterator<SomeType> rit2 = convert(rit1).withRemote(IGNORE_POLICY);

Here, the clause withRemote() introduces a fault policy for the RemoteIterator in output. Fault policies for RemoteIterator are a superset of those that can be configured on standard Iterators. In particular, they implement the interface RemoteFaultPolicy:

public interface RemoteFaultPolicy ... { 
	boolean onFault(Exception e, int count) throws Exception; 
}

Note that the only difference between a FaultPolicy and a RemoteFaultPolicy is that the latter has the additional option to raise a fault of its own in onFault(). Thus, when a fault occurs during iteration, the RemoteIterator can continue iterating, stop the iteration, but also re-throw the same or another fault to the iterating client, which is indeed what makes a RemoteIterator different from a standard Iterator.

In particular, the Stream DSL predefines a third policy which is available only for RemoteIterators:

RETHROW_POLICY: any faults raised during iteration will be immediately propagated to clients;

This is the in fact the default policy for RemoteIterators and clients can use the clause withRemoteDefaults() to avoid naming it:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<SomeType> rit1 = ...
 
RemoteIterator<SomeType> rit2 = convert(rit1).withRemoteDefaults();

In summary, the Stream DSL allows clients to formulate the following sentences for simple stream conversion:

convert(Iterator): converts a standard Iterator into a RemoteIterator;
convert(RemoteIterator).with(FaultPolicy): converts a RemoteIterator into a standard Iterator that encapsulates a given FaultPolicy;
convert(RemoteIterator).withDefaults(): converts a RemoteIterator into a standard Iterator that encapsulates the IGNORE_POLICY for faults;
convert(RemoteIterator).withRemote(RemoteFaultPolicy): converts a RemoteIterator into another RemoteIterator that encapsulates a given RemoteFaultPolicy;
convert(RemoteIterator).withRemoteDefaults(): converts a RemoteIterator into another RemoteIterator that encapsulates the RETHROW_POLICY for faults;

ResultSet Conversions

A different but very common form of conversion is between gCube result sets and RemoteIterators. While result sets are the preferred way of modelling remote streams within the system, their iterators do not natively implement the RemoteIterator<T> interface, which has been independently defined in the CML as an abstraction over an underlying result set mechanism. The CML defines an initial set of facilities to perform the conversion from result sets of untyped string payloads to RemoteIterators of typed objects. The Stream DSL builds on these facilities to cater for a few common conversions:

payloadsIn(RSLocator): converts an arbitrary result set into a RemoteIterator<String> defined over the record payloads in the result set;
urisIn(RSLocator): converts a result set of URI serialisations into a RemoteIterator<URI>;
documentIn(RSLocator): converts a result set of GCubeDocument serialisations into a RemoteIterator<GCubeDocument>;
metadataIn(RSLocator): converts a result set of GCubeDocument serialisations into a RemoteIterator<GCubeMetadata> defined over the metadata elements of the GCubeDocuments in the result set;
annotationsIn(RSLocator): converts a result set of GCubeDocument serialisations into a RemoteIterator<GCubeAnnotations> defined over the annotations of the GCubeDocuments in the result set;
partsIn(RSLocator): converts a result set of GCubeDocument serialisations into a RemoteIterator<GCubePart> defined over the parts of the GCubeDocuments in the result set;
alternativesIn(RSLocator): converts a result set of GCubeDocument serialisations into a RemoteIterator<GCubeAlternative> defined over the alternatives of the GCubeDocuments in the result set;

Essentially, documentsIn() adapts the result set to a RemoteIterator<T> that parses documents as it iterates over their serialisations. The following methods do the same, but extract the corresponding GCubeElements from the GCubeDocuments obtained from parsing. All the methods are based on the last one, payloadsIn, which is also immediately useful to feed result set of GCubeDocument identifiers to the read operations the gDL that perform stream-based document lookups.

note: all the conversions above produce RemoteIterators that return the locator of the original result set from invocations of locator(). Clients can use the locator to process the stream with standard set-based APIs, as usual.

The usage pattern is straightforward and combines with the previous conversions. The following example illustrates:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RSLocator rs = ...
Iterator<GCubeDocument> it = convert(documentsIn(rs)).with(FAILFAST_POLICY);

Piped Conversions

The conversions introduced above do not alter the original streams, i.e. the output iterators produce the same elements of the input iterators. The exception is with result set-based conversions: documentsIn() parses the untyped payloads of the input result sets into typed objects, while methods such as metadataIn() extract GCubeMetadata elements from GCubeDocuments. Parsing and extraction are only examples of the kind of post-processing that clients may wish to apply to the elements of existing stream to produce a new stream of post-processed elements. All the remaining sentences of the Stream DSL are dedicated precisely to this kind of conversions.

Sentences introduced by the verb pipe take a stream and return a second stream that applies an arbitrary filter to the elements of the first stream, encapsulating a fault handing policy in the process. The following example illustrates basic usage:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
Iterator<GCubeDocument> it1 = ...
 
Filter<GCubeDocument,String> filter = new Filter<GCubeDocument,String>() { 
                  public String apply(GCubeDocument doc) throws Exception {                           return doc.name();
                  }
};
 
Iterator<GCubeDocument> it2 = pipe(it1).though(filter).withDefaults();

Here, a standard Iterator of GCubeDocuments is piped through a filter that extracts the names of GCubeDocuments. The result is another standard Iterator that produces document names from the original stream. The clause through() introduces the filter on the output stream and, as already discussed for conversion methods, the clause withDefaults() configures the default IGNORE_POLICY for it.

As shown in the example, filters are implementations of the Filter<FROM,TO> interface. The method apply() is self-explanatory: clients are given the elements of the unfiltered stream as the filtered stream is being iterated over, and they have the onus to produce and return an element of the filtered stream. The only point worth stressing is that apply()s can throw a fault if it cannot produce an element of the filtered stream. The filtered stream passes these faults to the FaultPolicy configured for it. In this example, faults clearly cannot occur. If they did, however, the configured policy would simply ignore them, i.e. the problematic elements of the input stream would not contribute to the contents of the filtered stream.

In the example the input stream and the filtered one are both standard Iterators. The construct, however, is generic and can be used to filter any form of stream into any other. In this sense, the construct embeds stream conversions within its clauses. As an example, consider the common case in which a RemoteIterator is filtered into a standard Iterator:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<GCubeDocument> rit = ...
 
Filter<GCubeDocument,SometType> filter = ....;
 
Iterator<SomeType> it = pipe(rit).though(filter).with(FAILFAST_POLICY);

Here, filter is applied to the elements of a RemoteIterator to produce a standard Iterator that stops as soon as the input stream raises a fault. Conversely, in the following example:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<GCubeDocument> rit1 = ...
 
Filter<GCubeDocument,SometType> filter = ....;
 
RemoteIterator<SomeType> rit2 = pipe(rit1).though(filter).withRemote(IGNORE_POLICY);

Here, filter is applied to the elements of a RemoteIterator to produce yet another RemoteIterator that ignores any fault raised by the input iterator.

To conclude with pipe-based sentences, note that the Stream DSL includes Processor<T>, a base implementation of Filter&ltFROM,TO> that simplifies the declaration of filters that simply mutate the input and return it. The following example illustrates usage:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<GCubeDocument> rit1 = ...
 
Processor<GCubeDocument> processor = new Processor<GCubeDocument>() { 
            public void process(GCubeDocument doc) throws Exception {                       doc.setName(doc.name()+"-modified");
            }
} ;
 
RemoteIterator<GCUBEDocument> rit2 = pipe(rit1).though(processor).withRemoteDefaults();

Here, the processor simply updates the GCubeDocuments in the input stream by changing their name. The output stream thus returns the same elements of the input stream, albeit updated. During iteration, its policy is simply to re-throw any fault that may be raised by the input iterator.

In summary, the Stream DSL allows clients to formulate the following sentences for piped stream conversion:

pipe(Iterator|RemoteIterator).through(Filter|Processor).with(FaultPolicy): uses a given Filter or Processor to convert a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates a given FaultPolicy;
pipe(Iterator|RemoteIterator).through(Filter|Processor).withDefaults(): uses a given Filter or Processor to convert a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates a IGNORE_POLICY for faults;
pipe(Iterator|RemoteIterator).through(Filter|Processor).withRemote(RemoteFaultPolicy): uses a given Filter or Processor to convert a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates a given RemoteFaultPolicy;
pipe(Iterator|RemoteIterator).through(Filter|Processor).withRemoteDefaults(): uses a given Filter or Processor to convert a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates the RETHROW_POLICY for faults;

Folding Conversions

With pipe-based sentences, clients can filter the elements of a stream into the elements of another streams. While the elements of the two stream can vary arbitrarily in type, the correspondence between elements of the two streams is fairly strict: for each element of the input stream there may be at most one element of the output stream (elements that raise iteration failures in the input stream may have no counterpart in the output stream, i.e. may be discarded). In this sense, the streams are always consumed in phase.

In some cases, however, clients may wish to:

fold a stream, i.e. produce another stream that has one List element for each N elements of the original stream;
unfold a stream, i.e. produce another stream that has N elements for each element in the original stream.

Conceptually, these requirements are still within the scope of filtering, but the fact that the consumption of the filtered stream is out of phase with respect to the unfiltered stream requires a rather different treatment. For this reason, the Stream DSL offers two dedicated classes of sentences:

group-based sentences for stream folding;
unfold-based sentences for stream unfolding.

To fold a stream, clients indicate how many elements of the stream should be grouped into elements of the folded stream, what filter should be applied to each of the elements of the stream and, as usual, what fault handling policy should be used for the folded stream. The following example illustrates usage in the common case in which a RemoteIterator is folded into a standard Iterator:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<GCubeDocument> rit = ...
 
Filter<GCubeDocument,SometType> filter = ....;
 
Iterator<List<SomeType>> it = group(rit).in(10).pipingThrough(filter).withDefaults();

The RemoteIterator is here folded in Lists of 10 elements, (or smaller, if the end of the input stream is reached before a List of The clause in() indicates the maximum size of the output Lists. Each of the GCubeDocuments in the original stream is then passed through filter, which produces one of the List elements for it. The clause pipingThrough allows the configuration of the filer. Finally, the default IGNORE_POLICY is set on the folded stream with the clause withDefaults(), meaning that any fault raised by the RemoteIterator or filter will be tolerated and the element that caused the failure will simply not contribute to the accumulation of the next 10 elements of the folded stream.

note: the example shows the folding of a RemoteIterator into a standard Iterator but, as for all the sentences of the DSL, all combinations of input and output streams are possible, with the usual implications on the fault handing policies that can be set on the folded stream and with the optional choice of Processors over Filters in cases where folding simply groups updated elements of the stream.

It is a common requirement to fold a stream without transforming or altering otherwise its elements. In this case, the clause pipingThrough can be omitted altogether from the sentence:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<GCubeDocument> rit = ...
 
Iterator<List<GCubeDocument>> it = group(rit).in(10).withDefaults();

Effectively, the stream is here being filtered with a pass-through filter that simply returns the elements of the unfolded streams. As we shall see, t his kind of folding is particularly useful to 'slice' a stream in small in-memory collections that can be used with the write operations of the gDL that work in bulk and by-value.

In summary, the Stream DSL allows clients to formulate the following sentences for folding stream conversion:

group(Iterator|RemoteIterator).in(N).pipingThrough(Filter|Processor).with(FaultPolicy): uses a given Filter or Processor to N-fold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates a given FaultPolicy;
group(Iterator|RemoteIterator).in(N).pipingThrough(Filter|Processor).withDefaults(): uses a given Filter or Processor to N-fold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates the IGNORE_POLICY for faults;
group(Iterator|RemoteIterator).in(N).pipingThrough(Filter|Processor).withRemote(RemoteFaultPolicy): uses a given Filter or Processor to N-fold a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates a given RemoteFaultPolicy;
group(Iterator|RemoteIterator).in(N).pipingThrough(Filter|Processor).withRemoteDefaults(): uses a given Filter or Processor to N-fold a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates the RETHROW_POLICY for faults;
group(Iterator|RemoteIterator).in(N).with(FaultPolicy): uses a pass-through filter to N-fold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates a given FaultPolicy;
group(Iterator|RemoteIterator).in(N).withDefaults(): uses a pass-through filter to N-fold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates the IGNORE_POLICY for faults;
group(Iterator|RemoteIterator).in(N).withRemote(RemoteFaultPolicy): uses a pass-through filter to N-fold a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates a given RemoteFaultPolicy;
group(Iterator).in(N).withRemoteDefaults(): uses a pass-through filter to N-fold a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates the RETHROW_POLICY for faults

Unfolding Conversions

Unfolding a stream follows a similar pattern, as shown in the following example:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*;
...
RemoteIterator<GCubeDocument> rit = ...
 
Filter<GCubeDocument,List<SometType>> filter = ....;
 
Iterator<SomeType> it = unfold(rit).pipingThrough(filter).withDefaults();

This time we cannot dispense with using a Filter, which is necessary to map a single element of the stream into a List of elements that the unfolded stream, a standard Iterator in this example, will then yield one at the time at the client's demand. As usual, all combinations of standard Iterators, RemoteIterators, and fault handling policies are allowed. Using Processors is instead disallowed here, as it's in the nature of unfolding to convert a element into a number of different elements. Unfolding and updates, in other words, do not interact well.

The most common application of unfolding is for the extraction of inner elements from documents, e.g. unfold a stream of GCubeDocuments into a stream of GCubeMetadata elements, where each element in the unfolded stream belongs to some GCubeDocument in the document stream. Accordingly, the Stream DSL predefines a comprehensive number of these unfoldings. We have seen some of them already, where the document input stream was in the form of a result set (e.g. metadataIn(RSLocator)). Similar unfoldings are directly available on RemoteIterator<GCubeDocument>s.

In summary, the Stream DSL allows clients to formulate the following sentences for unfolding stream conversion:

unfold(Iterator|RemoteIterator).pipingThrough(Filter).with(FaultPolicy): uses a given Filter to unfold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates a given FaultPolicy;
- unfold(Iterator|RemoteIterator).pipingThrough(Filter).withDefaults(): uses a given Filter to unfold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates the IGNORE_POLICY for faults;
unfold(Iterator|RemoteIterator).pipingThrough(Filter).withRemote(RemoteFaultPolicy): uses a given Filter to unfold a standard Iterator or a RemoteIterator into a standard Iterator that encapsulates a given RemoteFaultPolicy for faults
unfold(Iterator|RemoteIterator).pipingThrough(Filter).withRemoteDefaults(): uses a given Filter to unfold a standard Iterator or a RemoteIterator into a RemoteIterator that encapsulates the RETHROW_POLICY for faults;
metadataIn(Iterator<GCubeDocument>|RemoteIterator<GCubeDocument>): unfolds a standard Iterator<GCubeDocument> or a RemoteIterator<GCubeDocument> into, respectively, a Iterator<GCubeMetadata> or a RemoteIterator<GCubeMetadata> defined over the metadata elements of the GCubeDocuments in the original stream;
annotationsIn(Iterator<GCubeDocument>|RemoteIterator<GCubeDocument>): unfolds a standard Iterator<GCubeDocument> or a RemoteIterator<GCubeDocument> into, respectively, a Iterator<GCubeAnnotation> or a RemoteIterator<GCubeAnnotation> defined over the annotations of the GCubeDocuments in the original stream;
partsIn(Iterator<GCubeDocument>|RemoteIterator<GCubeDocument>): unfolds a standard Iterator<GCubeDocument> or a RemoteIterator<GCubeDocument> into, respectively, a Iterator<GCubePart> or a RemoteIterator<GCubePart> defined over the parts of the GCubeDocuments in the original stream;
alternativesIn(Iterator<GCubeDocument>|RemoteIterator<GCubeDocument>): unfolds a standard Iterator<GCubeDocument> or a RemoteIterator<GCubeDocument> into, respectively, a Iterator<GCubeAlternative> or a RemoteIterator<GCubeAlternative> defined over the alternatives of the GCubeDocuments in the original stream;

Operations

The operations of the gDL allows clients to add, update, delete, and retrieve document descriptions to, in, and from remote collections within the infrastructure. These CRUD operations target (instances of) a specific back-end within the infrastructure, the Content Manager (CM) service. It is a direct implication of the CM that the document descriptions may be stored in different forms within repositories which are inside or outside the strict boundaries of the infrastructure. While the gDL operations clearly expose the remote nature of document descriptions, the actual location of document descriptions, hosting repositories, and Content Manager instances is hidden to their clients.

In what follows, we discuss first read operations, i.e. operations that localise document descriptions from remote collections. We then discuss write operations, i.e. operations that persist in remote collections document descriptions which have been created or else modified locally. In all cases, operations are overloaded to work with different forms of inputs and outputs. In particular, we distinguish between:

singleton operations: these are operations that read, add, or change individual document descriptions. Singleton operations are used for punctual interactions with the infrastructure, most noticeably those required by front-end clients to implement user interfaces. All singleton operations that target existing document descriptions require the specifications of their identifiers;

bulk operations: these are operations that read, add, or change multiple document descriptions in a single interaction with the infrastructure. Bulk operations can be used for batch interactions with the infrastructure, most noticeably those required by back-end clients to implement workflows. They can also be used for real-time interactions with the infrastructure, such as those required by front-end clients that process user queries. Bulk operations may be further classified in:
- by-value operations are defined over in-memory collections of document descriptions. Accordingly, these operations are indicated for small-scale data transfer scenarios. As we shall see, they may also be used to move segments of larger data collections, when the creation of such fragments is a functional requirement.
- by-reference operations are defined over streams of document descriptions. These operations are indicated for medium-scale to large-scale data transfer scenarios, where the streamed processing promotes the responsiveness of clients and the effective use of network resources.

Read and write operations work with document descriptions that align with the gCube document model (gDM) and its implementation in the gCube Model Library (gML). In the terminology of the gML, in particular, operations that create document descriptions expect new elements, all the others take or produce element proxies.

Finally, read and write operations build on the facilities of the Content Manager Library (CML) to interact with the Content Manager service, including the adoption of best-effort strategies to discover and interact with instances of the service. These facilities are thus indirectly available to gDL clients as well.

Reading Documents

Clients retrieve document descriptions from remote collections with the operations of a DocumentReader. Readers are created in the scope of the target collection, as follows:

GCubeScope scope = ...
String collectionID =...
 
DocumentReader reader = new DocumentReader(collectionID,scope);

In a secure infrastructure, a security manager is also required:

GCUBEScope scope = ...
GCUBESecurityManager manager = ...
String collectionID =...
 
DocumentReader reader = new DocumentReader(collectionID,scope,manager);

Readers expose three get() operations to retrieve document descriptions from target collections, two lookup operations and one query operation:

get(String,Projection): retrieves the description of a document with a given identifier, where the description matches a given projection and reflects the retrieval directives therein;
get(Iterator<String>,Projection): retrieves a stream of document descriptions from a stream with their identifiers, where the descriptions match a given projection and reflect the retrieval directives therein;
get(Projection): returns a stream of document descriptions, where the descriptions match a given projection and reflect the retrieval directives therein.

The operations and their return values can be illustrated as follows:

DocumentReader reader = ...
 
DocumentProjection p = ....
 
String id = ...
GCubeDocument doc = reader.get(id,p); 
Iterator<String> ids = ...
RemoteIterator<GCubeDocument> docs = reader.get(ids,p); 
 
RemoteIterator<GCubeDocument> docs = reader.get(p);

A few points are worth emphasising:

get(Iterator<String>,Projection) takes a stream of identifiers under the standard Iterator interface. As discussed at length above, this indicates that the operation makes no assumption as to the origin of the stream and that it has no policy of its own to deal with possible iteration failures; clients need to provide one in the implementation of the Iterator. Conversely, get(Projection) returns a RemoteIterator because it can guarantee the remote origin of the stream, though it still has no policy of its own to handler possible iteration failures. Again, clients are responsible for providing one. Clients can use the pipe sentences of the Stream DSL, to derive the Iterators in input from other form of streams and to post-process the RemoteIterators in output.

as a convenience, all the retrieval operations can take projections other than DocumentProjections. Projections over the inner elements of documents are equally accepted, e.g.:

DocumentReader reader = ...
MetadataProjection mp = ....
 
RemoteIterator<GCubeDocument> docs = reader.get(mp);

Here, matched documents are characterised directly with a MetadataProjection. The operation will derive a corresponding DocumentProjection with a single include constraint that requires matching documents to have that at least one metadata element that satisfy the projection. As usual, the output stream will retrieve of such documents no more than what the original MetadataProjection specifies in its include constraints. Again, clients are recommended to use the Stream DSL to extract the metadata elements from the output stream and possibly to process it further, e.g.:

import static org.gcube.contentmanagement.gcubedocumentlibrary.projections.Projections.*;
 
DocumentReader reader = ...
MetadataProjection mp = .... 
RemoteIterator<GCubeMetadata> metadata = metadataIn(reader.get(mp));

Similarly, the Stream DSL can be relied upon in the common case in which input streams originate in remote result sets, or when the output streams must be computed over using the result set API. The following example illustrates some of the possibilities:

import static org.gcube.contentmanagement.gcubedocumentlibrary.projections.Projections.*;
 
DocumentReader reader = ...
MetadataProjection mp = ....
 
//a result set of document identifiers
RSLocator idRS = ....
 
//extracts identifiers from result set into remote iterator and converts it into a local iterator
Iterator<String> ids = convert(payloadsIn(idRS)).withDefaults(); 
RemoteIterator<GCubeMetadata> metadata = metadataIn(reader.get(ids,mp)); 
//extract result set locator from remote iterator
RSLocator docRS = new RSLocator(metadata.locator()); 
//use locator with result set API
...

Finally, note that the example above does not handle possible failures. Clients may consult the code documentation for a list of the faults that the individual operations may raise.

Resolving Elements

A DocumentReader may also be used to resolve content URIs into individual elements of document descriptions. It offers two operations for this purpose:

resolve(URI,Projection): takes a content URI and returns the description of the document element identified by it, where the description matches a given projection and reflects its retrieval directives;
resolve(Iterator<URI>,Projection): takes a stream of content URIs and returns a stream of description of the document elements identified by it, where the descriptions match a given projection and reflect its retrieval directives.

The operations and their return values can be illustrated as follows:

import static org.gcube.contentmanagement.gcubedocumentlibrary.projections.Projections.*;
 
//a reader over a given collection
DocumentReader = ...
 
//a sample content URI to a metadata element of a document
URI metadataURI = new URI("cms://somecollection/somedocument/somemetadata"); 
//a sample projection over metadata elements
MetadataProjection mp = metadata().with(BYTESTREAM);
 
GCubeMetadata element = reader.resolve(metadataURI,mp);

Here the client resolves a metadata element and uses a projection to limit retrieval to its bytestream alone.

Do note that the following points:

the URI must point to an element within the target collection of the DocumentReader. In this example, the reader must be bound to somecollection or the operation will fail;

the resolution is typed, i.e. the client must know the type of element identified by the URI. Providing a projection gives a hint to the reader as to what type of element is expected. Resolution will fail if the URI points to an element of a different type as much as it fails if it points to an unknown element;

as usual, empty projections can be used for conventional resolution, i.e. to retrieve the whole element unconditionally;

clients can resolve content URIs that identify to whole documents, in combination with document projections. In this case, resolve() behaves exactly like get() when the latter is invoked with the document identifier inside the URI;

remember that the CML defines a set of facilities to compose and decompose content URIs;

remember also that the gML defines a method uri() on documents and their elements. Clients that work with element proxies can use it to obtain their content URI and then store it, move it over the network, etc. until it becomes available to the same or different clients for resolution.

As an example of stream-based resolution consider the following:

import static org.gcube.contentmanagement.gcubedocumentlibrary.projections.Projections.*;
 
//a reader over a given collection
DocumentReader = ...
 
//an iterator over content URIs of annotations
Iterator<URI> annotationURIs = ...; 
//a sample projection over annotations
AnnotationProjection ap =...
 
RemoteIterator<GCubeAnnotation> annotations = reader.resolve(annotationURIs,mp);

Like all the stream-based operations of the DocumentReader, resolve() takes stream as standard Iterators and returns streams as RemoteIterators. As usual, clients can use the facilities of the Stream DSL to convert to and from these iterators and other models of streams. In particular:

remember the method urisIn() of the Streams class can transparently convert a result set of content URI serialisations into an equivalent RemoteIterator<URI>.

Adding Documents

Clients add document descriptions to remote collections with the operations of a DocumentWriter. Writers are created in the scope of the target collection, as follows:

GCubeScope scope = ...
String collectionID =...
 
DocumentWriter writer = new DocumentWriter(collectionID,scope);

In a secure infrastructure, a security manager is also required:

GCUBEScope scope = ...
GCUBESecurityManager manager = ...
String collectionID =...
 
DocumentWriter writer = new DocumentWriter(collectionID,scope,manager);

Writers expose four operations to add document descriptions to the target collections, two singleton operations and two bulk operations. All the operations take new document descriptions built with the APIs of the gML. In addition, each description must satisfy certain basic criteria, including:

the consistency between the collection bound to it and the collection bound to the writer;
other constraints specific to its inner elements.

We say that the description must be valid for addition. The operations are the following:

add(GCubeDocument): adds a valid document description to the target collection and returns an identifier for it;

addAndSychronize(GCubeDocument): adds a valid document description to the target collection and returns a proxy for it. The proxy is synchronised with the changes applied to the description at the point of addition, including the assignment of identifiers to the whole description and its inner elements;

add(Iterator<GCubeDocument>): adds a stream of valid document descriptions to the target collection and returns a Future<?> of the completion of the operation.

add(List<GCubeDocument>): adds a list of valid document descriptions to the target collection and returns of a list of corresponding outcomes, where each outcome is either an identifier for the description or else a processing failure.

The operations and their return values can be illustrated as follows:

DocumentWriter writer = ...
 
//singleton add
GCubeDocument doc = ...
String id = writer.add(doc); 
//singleton with synchronization
GCubeDocument synchronized = writer.addAndSynchronize(doc); 
//bulk by-value
List<GCubeDocument>  docs = ...
List<AddOutcome> outcomes = writer.add(docs); 
//bulk by-ref
Iterator<GCubeDocument> docStream = ...
Future<?> future = writer.add(docStream);....
//poll for completion (see also other polling methods of Futures)
if (future.get()==null)    ...submission is completed...

A few points are worth emphasising:

addAndSynchronize(GCubeDocument) requires two remote interactions, one to add the document description and a one to retrieve its synchronised proxy. Clients are responsible for replacing the input description with the proxy in any local structure that may already contain references to the former;

add(Iterator<GCubeDocument>) follows the same pattern for stream-based operations already discussed for read operations. Invalid document descriptions found in the input stream are silently discarded. Due to the possibility of these pre-processing failures and its non-blocking nature, the operation cannot guarantee the fidelity of outcome reports. For this reason, the operation returns only a Future<?> that clients can poll to know when all the proxies in input have been submitted for addition. Clients that work with large or remote streams, and are interested in processing outcomes, are responsible for grouping the elements of the stream in 'chunks' of acceptable size and use add(List<GCubeDocument>);

add(List<GCubeDocument>) is indicated for small input collections and/or when reports on outcomes are important to clients. Invalid document descriptions found in the input list fail the operation before any attempt is made to add any document description to the target collection. Clients can use the group sentences of the Stream DSL to conveniently convert an arbitrarily large stream of GCubeDocuments into a stream of List<GCubeDocument>, which can then be fed to the operation element by element. The following example illustrates this processing pattern:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*; 
DocumentWriter writer = ...
Iterator<GCubeDocument> docs = ...
 
//fold stream in chunks of 30 descriptions
Iterator<List<GCubeDocument>> chunks = group(docs).in(30).withDefaults(); 
while (chunks.hasNext()) {
 List<AddOutcome> outcomes = writer.add(chunks.next()); for (AddOutcome outcome : outcomes) {
   if (outcome.getSuccess()!=null) {       ...outcome.getSuccess().getId()...    }
    else {
       ...outcome.getFailure().getFault()...    }
}

add(List<GCubeDocument>) and add(Iterator<GCubeDocument>) uses result set mechanisms to interact with remote services and thus can be invoked only inside a container;

the processing model for AddOutcomes is already defined in the CML.

Finally, note that the code in the examples above does not handle possible failures. Clients may consult the code documentation for an enumeration of the faults that the individual operations may raise.

Updating Documents

A DocumentWriter may also be used to update document descriptions already in the target collection.

It offers four operations for this purpose, two singleton operations and two bulk operations. The operations mirror those that add document descriptions to the target collection, as discussed above. Like the add operations, the update operations take document descriptions built with the APIs of the gML. However, the descriptions must be proxies of remote descriptions and each proxy must satisfy certain basic criteria, including:

the consistency between the collection bound to it and the collection bound to the writer;
the existence of tracked changes on it;
other constraints that are specific to its inner elements.

We say that the proxy must be valid for update. The operations are the following:

update(GCubeDocument): updates a document description in the target collection with the properties of a valid proxy;
updateAndSychronize(GCubeDocument): updates a document description in the target collection with the properties of a valid proxy, returning another proxy that is fully synchronised with the description, i.e. reflects all its properties after the update, including updates times of last update for the description and its inner elements;
update(Iterator<GCubeDocument>): updates multiple document descriptions in the target collection with the properties of a stream of valid proxies, returning a Future<?> for the future completion of the operation;
update(List<GCubeDocument>): updates multiple document descriptions in the target collection with the properties of a list of valid proxies, returning a map of processing failures indexed by the identifier of the corresponding description.

The operations and their return values can be illustrated as follows:

DocumentWriter writer = ...
 
//singleton add
GCubeDocument proxy = ...
writer.update(proxy); 
//singleton with synchronization
GCubeDocument synchronized = writer.updateAndSynchronize(proxy); 
//bulk by-value
List<GCubeDocument>  proxies = ...
Map<String,Throwable> failures = writer.update(proxies); 
//bulk by-ref
Iterator<GCubeDocument> proxyStream = ...
Future<?> future = writer.update(proxyStream);....
//poll for completion (see also other polling methods of Futures)
if (future.get()==null)    ...submission is completed...

A few points are worth emphasising:

updateAndSynchronize(GCubeDocument) requires two remote interactions, one to update the document description and one to retrieve its synchronised proxy;

update(Iterator<GCubeDocument>) follows the same pattern for stream-based operations already discussed for add operations. Invalid proxies found in the stream are silently discarded. Due to the possibility of these pre-processing failures and its non-blocking nature, the operation cannot guarantee the fidelity of outcome reports. For this reason, the operation returns only a Future<?> that clients can poll to know when all the proxies in input have been submitted for update. Clients that work with large or remote streams, and are interested in processing outcomes, are responsible for grouping the elements of the stream in 'chunks' of acceptable size and use add(List<GCubeDocument>);

update(List<GCubeDocument>) is indicated for small input collections and/or when outcome reports are important to clients. Invalid proxies found in the input list fail the operation before any attempt is made to update any document description in the target collection. Clients can use the group sentences of the Stream DSL to conveniently convert an arbitrarily large stream of GCubeDocuments into a stream of List<GCubeDocument>, which can then be fed to the operation element by element. The following example illustrates this processing pattern:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*; 
DocumentWriter writer = ...
Iterator<GCubeDocument> proxies = ...
 
//fold stream in chunks of 30 proxies
Iterator<List<GCubeDocument>> chunks = group(proxies).in(30).withDefaults(); 
while (chunks.hasNext()) { Map<String,Throwable> failures = writer.update(chunks.next()); ...process failures...
}

update(List<GCubeDocument>) and update(Iterator<GCubeDocument>) uses result set mechanisms to interact with remote services and thus can be invoked only inside a container.

Finally, note that the code in the examples above does not handle possible failures. Clients may consult the code documentation for a list of the faults that the individual operations may raise.

Deleting Documents

A DocumentWriter may also be used to delete document descriptions already in the target collection.

It offers three operations for this purpose, one singleton operation and two bulk operations. The operations mirror those that update document descriptions in the target collection, as discussed above. Like the update operations, the delete operations take proxies of remote descriptions built with the APIs of the gML. The operations are the following:

delete(GCubeDocument): deletes a document description from the target collection using a proxy for it;
delete(Iterator<GCubeDocument>): deletes multiple document descriptions from the target collection using a stream of proxies for them, returning a Future<?> for the future completion of the operation;
update(List<GCubeDocument>): deletes multiple document descriptions from the target collection using a list of proxies for them and returning a map of processing failures indexed by the identifier of the corresponding description.

The operations and their return values can be illustrated as follows:

DocumentWriter writer = ...
 
//singleton add
GCubeDocument proxy = ...
writer.delete(proxy); 
//bulk by-value
List<GCubeDocument>  proxies = ...Map<String,Throwable> failures = writer.delete(proxies);
 
//bulk by-ref
Iterator<GCubeDocument> proxyStream = ...Future<?> future = writer.delete(proxyStream);
....
//poll for completion (see also other polling methods of Futures)
if (future.get()==null)    ...submission is completed...

A few points are worth emphasising:

delete(Iterator<GCubeDocument>) follows the same pattern for stream-based operations already discussed for add operations and update operations. Document descriptions found in the stream which are not proxies are silently discarded. Due to the possibility of these pre-processing failures and its non-blocking nature, the operation cannot guarantee the fidelity of outcome reports. For this reason, the operation returns only a Future<?> that clients can poll to know when all the proxies in input have been submitted for deletion. Clients that work with large or remote streams, and are interested in processing outcomes, are responsible for grouping the elements of the stream in 'chunks' of acceptable size and use delete(List<GCubeDocument>);

delete(List<GCubeDocument>) is indicated for small input collections and/or when outcome reports are important to clients. Document descriptions found in the input list which are not proxies fail the operation before any attempt is made to delete any document description in the target collection. Clients can use the group sentences of the Stream DSL to conveniently convert an arbitrarily large stream of GCubeDocuments into a stream of List<GCubeDocument>, which can then be fed to the operation element by element. The following example illustrates this processing pattern:

import static org.gcube.contentmanagement.gcubedocumentlibrary.streams.dsl.Streams.*; 
DocumentWriter writer = ...
Iterator<GCubeDocument> proxies = ...
 
//fold stream in chunks of 30 proxies
Iterator<List<GCubeDocument>> chunks = group(proxies).in(30).withDefaults(); 
while (chunks.hasNext()) { Map<String,Throwable> failures = writer.delete(chunks.next()); ...process failures...
}

delete(List<GCubeDocument>) and delete(Iterator<GCubeDocument>) uses result set mechanisms to interact with remote services and thus can be invoked only inside a container.

Finally, note that the code in the examples above does not handle possible failures. Clients may consult the code documentation for a list of the faults that the individual operations may raise.

Views

Some clients interact with remote collections to work exclusively with subsets of document descriptions that share certain properties, e.g. are in a given language, have changed in the last month, have metadata in a given schema, have parts of a given type, and so on. Their queries and updates are always resolved within these subsets, rather than the whole collection. Essentially, such clients have their own view of the collection.

The gDL offers support for working with two types of view:

local views: these are views defined by individual clients as the context for a number of subsequent queries and updates. Local views may have arbitrary long lifetimes, and may even outlive the client that created them, they are never used by multiple clients. Thus local views are commonly transient and if their definitions are somehow persisted, they are persisted locally to the 'owning' client and remain under its direct responsibility.

remote views: these are views defined by some clients and used by many others within the system. Remote views outlive all such clients and persist in the infrastructure, typically for as long as the collection does. They are defined through the View Manager service (VM), which materialises them as WS-Resources. Each VM resource encapsulates the definition of the view as well as its descriptive properties, and it is responsible for managing its lifetime, e.g. keep track of its cardinality and notify interested clients of changes to its contents. However, VM resources are 'passive, i.e. do not mediate access to those content resources.

Naturally, the gDL uses projections as view definitions. It then offers specialised Readers that encapsulate such projections to implicitly resolve all their operations in the scope of the view. This yields view-based access to collections and allows clients to work with local views. In addition, the gDL provides local proxies of VM resources with which clients can create, discover, and inspect remote views. As these proxies map remote view definitions onto projections, remote views can be accessed with the same reading mechanisms available for local views.

Local Views

To work with a local view of a remote collection, a gDL client creates first the projection that defines the view. The client then injects the projection into a ViewReader, along with a DocumentReader already configured to access the target collection. Like the DocumentReader, the ViewReader implements the Reader interface, offering all the read operations discussed above. When any of its operations is called, however, the ViewReader merges the view definition and the input projection, combining their constraints. It then passes the merged projection to the inner DocumentReader, which executes the operation. Effectively, this resolves the operation in the scope of the view.

The following example illustrates the approach:

import static java.util.Locale.*;
import static org.gcube.contentmanagement.gcubedocumentlibrary.projections.Projections.*;
 
//a reader configured to access a target collection.
DocumentReader reader = ...
 
//define a local viewDocumentProjection view = document().withValue(LANGUAGE,FRENCH);
 //inject view and reader it into a view reader
ViewReader viewReader = new ViewReader(view,reader);
 
GCubeDocument doc = viewReader.get("...some id...", document().with(NAME)); assert(doc.language()!=null);
assert(doc.language().equals(FRENCH));
assert(doc.name()!=null);

Here, the view includes only the document descriptions (with a bytestream) in a given language. The lookup operation retrieves the target description only if it has a name and is in the view. At runtime, the DocumentProjection that defines the view is merged with the projection passed to the lookup operation. This produces the same effect that the following projection would produce if it was executed by a plain DocumentReader:

document().withValue(LANGUAGE,FRENCH).with(NAME);

The example above defines the view as a DocumentProjection but any projection can be used for the purpose (e.g. a MetadataProjection, an AnnotationProjection, etc). In general, clients have the same flexibility in defining views as they do in invoking the operations of Readers: any projection that can be used in one context can also be used in the other. Clients will choose DocumentProjections when the view needs to characterise properties of entire documents and/or inner elements of different types. They will instead prefer more specific projections when the view is predicated on properties of inner elements of the same type. For example, a view that characterises document descriptions based only on the schema of their metadata elements is more conveniently defined with a MetadataProjection:

...
//define a local view
DocumentProjection view = metadata().withValue(SCHEMA_URI,"..some schema..");...
GCubeDocument doc = viewReader.get("...some id...", document().with(NAME));
...

The operation above would lookup the target document description only if it has a name and at least one metadata element in the given schema.

Similarly, clients are free to pass any projection with the operations of the ViewReader, including those that "diverge" arbitrarily from the view:

...
//define a local view
DocumentProjection view = document().with(PART);...
GCubeDocument doc = viewReader.get("...some id...", metadata().with(BYTESTREAM));...

The operation above would lookup the target document descriptions only if it has at least one part and one metadata element with an inlined bytestream.

The freedom in merging view definitions with other projections is limited only by the obvious requirement: the merged projection must not retrieve documents that are outside the view. The ViewReader will detect projections that break the view abstraction and, in case, refuse them as parameters of its operations. For example:

...
//define a local view
DocumentProjection view = document().withValue(LANGUAGE,FRENCH);...
try {
  GCubeDocument doc = viewReader.get("...some id...", document().withValue(LANGUAGE,ENGLISH));  assert(false);
}
catch(InvalidProjectionException e) {	assert(true);
}
...

This attempts generates an InvalidProjectionException as document descriptions in English are not part of the view.

The possibility of an invalid projection emerges as soon as there is an overlap between the properties specified in view definitions and those specified in the input projections (as for LANGUAGE above). A crude policy to enforce views would be to prevent overlaps altogether. This policy is too inflexible, for two main reasons:

like all projections, view definitions specify retrieval directives (i.e. with(), where, opt()), i.e. what is to be retrieved and what it should not move over the network. While clients can exploit this directives to define default directives within view definitions, it's important that clients may freely override them on a per-operation basis, e.g. replace a with() in the view definition with a where() in the input projection (e.g. to avoid retrieval of unnecessary data). For this reason, it is important to handle overlaps that signal overriding of defaults, provided that the operation can still be guaranteed to remain within the scope of the view. An example illustrates the point:

...
//define a local view
DocumentProjection view = document().withValue(LANGUAGE,FRENCH);...
GCubeDocument doc = viewReader.get("...some id...", document().whereValue(LANGUAGE,FRENCH).with(NAME));...

Here the input projection is allowed to overlap with the view definition on LANGUAGE for overriding purposes. Notice that overriding is allowed here because the constraints imposed by the view are preserved. In general, it is good practice to use where() directives in view definitions, so as to limit the need for overriding and to allow clients to specify explicitly only what they wish to retrieve. Now that we illustrated this point we will align with this practice in all the examples below.

the other reason to deal with overlaps between view definitions and input projections is simply flexibility. In principle, input projections ought to be allowed to refine the view in their projections. While it is not possible to recognise all refinements statically (and in some case it is genuinely difficult even when it is possible), ViewReader recognise and allow a number of common refinements including:

refinements on existence constraints. If the view requires the mere existence of a property, then an input projection can specify a further constraint on it. For example, the view document().where(NAME) can be refined (and overridden) by the input projection document().withValue(NAME,"myname");

refinements on deep projections. If the view constraints properties of inner elements, then an input projection can constrain other properties of those elements, or even the same properties if the refinement is allowed in turn. For example, the view document().where(METADATA,metadata().with(NAME)) can be refined by the input projection document().with(METADATA, metadata().withValue(NAME,"myname")).

Flexibility is not only introduced by refinements:

a view constrain may also be overridden by a widening input projection. For example, the view document().where(NAME,"myname") can be overridden by the widerdocument().with(NAME). The two projections merge into the projection document().with(NAME,"myname"). Thus clients are not required to know the details of the view if they wish to retrieve the names of document descriptions.

To conclude on the possibilities for view-based access, notice that empty projections continue to retain their simplicity of use under views. For example:

...
//define a local view
PartProjection view = part().withValue(LANGUAGE,FRENCH);...
GCubeDocument doc = viewReader.get("...some id...", part());...

continues to retrieve all parts of documents in the view. Here, the view definition and the input projection merge into a projection that has the single constrain of the view and optional include constraints for all the other remaining documents properties.

Similarly , the catch-all constraints such as etc() and allexcept() retain their semantics under views, and can be used in input projections with the usual expectations (using them in view definitins as defaults for retrieval is also possible, though discouraged in the general case for the reasons discussed previously).

Remote Views

When working with remote views, accessing documents in the view is not the only client requirements. Publishing a remote view, i.e. share it with other remote clients,and discovering existing views with given properties are also common tasks for clients. In the gDL, support for these tasks is mostly found in CollectionViews.

A CollectionView is a local proxy a remote view, not dissimilarly from how GCubeDocuments are local proxies of remote document descriptions. More specifically, collection views are document-oriented abstractions over the tree-oriented View proxies of WS-Resources of the View Manager service, as offered by the client-library.

Most properties of View proxies carry over directly to CollectionViews, including:

the collection identifier (cf. collectionId());
the view identifier (cf. id());
the descriptive name of the view, e.g "myview" (cf. name());
the free-form description of the view, e.g. "all document descriptions such that..." (cf. description());
the broad type of the view, e.g. a QName like {http://...}mytype (cf. type());
the time of last update of the view (cf. lastUpdate());
the cardinality of the view (cf. cardinality());
the generic properties of the view (cf. properties()).

Most importantly, the CM Predicates used in View proxies to characterise the document descriptions in the view become Projections over GCubeElements, whether entire document descriptions (i.e. GCubeDocuments) or specific inner elements of such descriptions (e.g. GCubeMetadata).

CollectionView defines a read-only interface to the properties of the view, which supports generic programming tasks over views. Most clients however will work with concrete implementations of the interface. The current version of the gDL includes the following ones:

GenericView: a generic implementation for arbitrary remote views;
MetadataView: an implementation for remote views defined over a simple set of metadata element properties;
AnnotationView: an implementation for remote views defined over a simple set of annotation properties;

All CollectionView implementations inherit from the abstract BaseCollectionView, while MetadataView and AnnotationView inherit from the more specific SimpleView.

note: Custom implementations can be created as well, by specialising GenericView or, more commonly, BaseCollectionView which has been explicitly designed for open-ended extensibility.

Across all implementations, CollectionViews may be in either one of two states:

unbound: this is the state of instances that are not associated with remote views. In this state, CollectionViews can be used for view publication and view discovery, but not for view-based access to collections.
bound: this is the state of instances that are bound to remote views. In this state, CollectionViews can be used for view-based access but not for view publication or view discovery.

Typically, CollectionViews are created unbound and used for publication or discovery, depending on the general goals of the client. Both operations introduce bindings: a CollectionViews becomes bound after publication and all the CollectionViews returned from discovery operations are already bound; these can then be used for view-based access. In less common cases, clients may start with a VM proxy and inject it directly into a new CollectionViews, which is thus instantiated in a bound state. All the implementations of CollectionView are responsible for enforcing state-based constraints.

Publishing Views

We use GenericViews to illustrate how client can publish, discover and use remote views. We then show how working with MetadataViews and AnnotationViews changes the basic usage patterns.

Publishing a remote view involves creating a proxy for it in a given scope, setting its properties, and invoking the method publish() on it:

import static org.gcube.contentmanagement.gcubedocumentlibrary.projections.Projections.*;
import static java.util.Locale.*;
 
GCubeScope scope = ...
 
//creates unbound view in given scope
GenericView view = new GenericView(scope); 
assert(view.isBound()==false);
 
//sets view properties
view.setId("...");
view.setCollectionId("...");
view.setName("...");
view.setDescription("...");
 
//sets view definition
view.setProjection(document().where(LANGUAGE,FRENCH)); 
/publish view
view.publish(); 
assert(view.isBound());

We create a GenericView in a given scope and sets its properties, including the projection that defines it. We then publish the view in the scope in which we created it. The test shows that the GenericView is unbound until its publication, after which it is bound.

A few points to notice about view publication:

in the example, we have used a DocumentProjection for documents (with a bytestream) in French. As usual, clients can use the type of projection which is more convenient to express their constraints (e.g. a MetadataProjection). Regardless of the type of the injected projection, a GenericView returns always a DocumentProjection from its projection() method, as in the general case the definition of the view may be acquired during discovery, when its gDL type is statically unknown.

views can be published with the method publishAndBroadcast(). As the name suggests, this overload induces the View Manager service to broadcast a record of its creation. This is then used for autonomic state replication across its instances, in line with the scalability mechanisms of the service.

in the example, we have provided an explicit identifier for the view. We could have omitted it, in which case the View Manager service would have generated one for it.

like identifiers, other properties of the view (e.g. time of last update) are set on the view by the View Manager at the point of creation. During publication these properties are automatically synchronised on the local proxy.

we did not set the type of the view before publication. The type of a GenericView is in fact constant (the name of the class itself, GenericView) and clients cannot alter it. This constancy is found in all other implementations of CollectionView, as it ensures that upon discovering views we can recognise the classes to which we should bind them.

GCube Document Library (2.0)

Contents

Preliminaries

Projections

Streams

Standard and Remote Iterators

Simple Conversions

ResultSet Conversions

Piped Conversions

Folding Conversions

Unfolding Conversions

Operations

Reading Documents

Resolving Elements

Adding Documents

Updating Documents

Deleting Documents

Views

Local Views

Remote Views

Publishing Views

Discovering Views

Using Views

Advanced Topics

Caches

Buffers

Navigation menu

Views

Personal tools

gCube Wiki

gCube features

gCube documentation

Integration and Distribution

Search

Tools