OpenSearch Framework

From Gcube Wiki
Jump to: navigation, search

Description

The role of the gCube OpenSearch Framework is to enable the gCube Framework to access external providers which publish their results through search engines conforming to the OpenSearch Specification. The framework consists of two components

  • The OpenSearch Library, which includes a core library providing general-purpose OpenSearch functionality, and the OpenSearch Operator which utilizes functionality provided by the former.
  • The OpenSearch Service (also called OpenSearchDataSource Service), which binds collections with provider-specific information encapsulated in generic resources and invokes the OpenSearch Operator

To resolve ambiguity, the name "OpenSearch Library" will be used when referring to the whole OpenSearch Library component, whereas the name "OpenSearch Core Library" will be used when referring to the library constituent of the component.

A client library library for OpenSearch Service, called OpenSearchDataSource Client Library, also exists in order to assist the programmatically use of the service.

OpenSearch Library, OpenSearch Service and OpenSearchDataSource Client Library are available in our Maven repositories with the following coordinates: 

<!-- OpenSearch Library -->
<groupId>org.gcube.opensearch</groupId>
<artifactId>opensearchlibrary</artifactId>
<version>...</version>
 
<!-- OpenSearch Service -->
<groupId>org.gcube.opensearch</groupId>
<artifactId>opensearchdatasource-service</artifactId>
<version>...</version>
 
<groupId>org.gcube.opensearch</groupId>
<artifactId>opensearchdatasource-stubs</artifactId>
<version>...</version>
 
<!-- OpenSearchDataSource Client Library -->
<groupId>org.gcube.opensearch</groupId>
<artifactId>opensearchdatasource-client-library</artifactId>
<version>...</version>

The OpenSearch Library

The OpenSearch Core Library

Description

The OpenSearch Core Library conforms to the latest OpenSearch specification and provides general OpenSearch-related functionality to any component which needs to query OpenSearch providers. It can be optionally extended, as described in the Extensibility section, in order for OpenSearch Extensions whose parameters or other elements need special handling to be supported. The OpenSearch Operator, described in a following section functions atop this library.

Functionality

The central class which can be used in order to exploit the functionality provided by the library, is the DescriptionDocument class. For reasons explained in the following section, the DescriptionDocument class needs to be provided with a pair of URLElementFactory and QueryElementFactory factory classes. Provided that the query parameter namespaces present in the query string are extracted in some way and a namespace-to-factory mapping is available, this pair can be obtained by the FactoryResolver class, as follows: 

FactoryPair factories = FactoryResolver.getFactories(queryNamespaces, factoryMapping);

The DescriptionDocument is then instantiated as follows: 

DescriptionDocument dd = new DescriptionDocument(descriptionDocumentXML, factories.urlElFactory, factories.queryElFactory);

where the descriptionDocumentXML parameter corresponds to a DOM Document object containing the parsed Description Document. Properly instantiated, the DescriptionDocument class can provide any information relevant to the processed Description Document, as well as a mechanism to formulate search queries to send to the OpenSearch provider described by the Description Document. The latter is achieved by a QueryBuilder object, which can be obtained as follows: 

List<QueryBuilder> qbs = dd.getQueryBuilders(rel, MimeType);

where rel is a rel value as described in the OpenSearch Specification, e.g. results and MimeType is a MIME type, such as application/rss+xml. The returned list contains one QueryBuilder instance for each template contained in a URL Element with the specified rel and type attributes. Once the desired QueryBuilder is selected, it can be used to formulate a query by first assigning values to the parameters and then obtaining the constructed query. For example, searchTerms parameter can be set to some value as follows: 

qb.setParameter(OpenSearchConstants.searchTermsQName, searchTerms);

Once all the required parameters are set, the constructed query can be obtained as follows: 

URL query;
try {
    query = qb.getQuery();
}catch(IncompleteQueryException iqe) {
    //Incomplete query exception handling
}catch(MalformedQueryException mqe {
    //Malformed query exception handling
}

Once the query is properly constructed and is available, it can be sent to the search engine of the provider in order to retrieve results. The returned results should be passed to either HTMLResponse or XMLResponse, depending on the MIME type of the OpenSearch response, in order for the OpenSearch Response Elements and any other available information contained in the response to be processed. 

InputStream responseStream = query.openConnection().getInputStream();
OpenSearchResponse response = new XMLResponse(responseStream, factories.queryElFactory, qb, outputEncoding,  dd.getURIToPrefixMappings());

The raw XML data can then be obtained by the OpenSearchResponse object as follows: 

response.getResponse();

and any information available, mainly relevant to paging, can be obtained by one of the methods of the OpenSearchResponse class. For example the total number of results, as reported by the totalResults Response Element, if present, can be obtained as follows: 

response.getTotalResults();

Library Extensibility

Motivation

The core functionality provided by the OpenSearch Core Library is not limited to the processing of only standard OpenSearch parameters. More specifically, the basic components of the library treat all extended query parameters in a uniform way, making only the assumptions holding for any OpenSearch parameter, be it standard or extended. Furthermore, any unrecognized markup or element value is simply ignored. An example of an assumption made by the OpenSearch Core Library, in the form of a requirement, is that all parameter values passed to its QueryBuilder components should be URL-encoded. This requirement is in accordance with the OpenSearch Specification and causes no problems for most OpenSearch parameters. In fact, if a client failed to URL-encode free-text values, query formulation would fail in the query URL construction phase.

There are, however, cases in which the previous requirement proves problematic. For example, the OpenSearch Geo Extension presents examples of parameter values in which the comma character is not URL encoded, regardless of what the RFC specifications state. Such parameter values could therefore call for an extra URL decoding preprocessing step, or otherwise the caller should be required to not URL encode the values. Furthermore, it would be quite useful if the library could be aware of the specific format and other peculiarities and rules governing the syntax of extended parameters, for the purpose of query validation and for supporting any extra functionality provided by the extension. The support of value-adding functionality provided by extended OpenSearch elements by the library could also prove useful.

Extensibility Mechanism

The extensibility mechanism chosen for the library focuses on extensible elements, as described in the OpenSearch Specification, namely URL Elements and Query Elements. Furthermore, QueryBuilder components are included in the extensibility mechanism as they depend on the aforementioned elements.

Given that the number of available OpenSearch extensions is quite large and because of the fact that not all of these extensions are utilized by some OpenSearch provider at the same time, the extensibility mechanism should allow the easy inclusion of library extensions for specific OpenSearch extensions in a dynamic, pluggable fashion. Furthermore, it should allow extension-related functionality to be dynamically added depending on the complexity of the query of the caller.

The mechanism found to best satisfy the above requirements and implemented as the extensibility pattern for the library, is the construction of a Chain of Responsibility for each extensible component. A more detailed explanation follows:

  • The URLElement, QueryElement and QueryBuilder components are interfaces whose implementations support core or extension-related functionality.
  • Core functionality processing takes place in the last link of the chain of responsibility. For example, BasicQueryBuilder implements core QueryBuilder functionality.
  • Each component implementing extension-related functionality contains a reference to the next link in the chain of responsibility. For example if GeoQueryBuilder implements functionality related to the Geo OpenSearch Extension, it contains a reference to a QueryBuilder implementing either core functionality, or functionality related to some other extension.
  • Each link in the chain of responsibility should process whatever information it can handle, otherwise forward the request to the next link in the chain.


In order for a chain of responsibility to be dynamically created by the DescriptionDocument class, a similar chain of abstract factories should be implemented. The resulting factories, one for URL Elements and another for Query Elements can then be passed to the constructor of the DescriptionDocument in order for it to be able to construct the correct elements. The FactoryResolver utility is responsible for the construction of factories capable of constructing instances supporting no more than the functionality necessary to process a given query. Since the chain structure is already known when constructing QueryBuilder instances, the latter are constructed without explicitly supplying a factory to the DescriptionDocument, by the getQueryBuilder method of the already constructed URLElement.

The FactoryResolver requires that two things be known in order to be able to construct the factories:

  • A set of mappings from namespace URIs to factory class names, one for each component implementing either core functionality (in this case the namespace URI being equal to the OpenSearch namespace) or extension-related functionality. An example of such a mapping could be: <http://a9.com/-/spec/opensearch/1.1/, (org.gcube.opensearch.opensearchlibrary.urlelements.BasicURLElement, org.gcube.opensearch.opensearchlibrary.queryelements.BasicQueryElement)>, which declares that the implementations responsible for providing core functionality are the BasicURLElement and BasicQueryElement classes.
  • A list of all parameter namespaces present in the query string.

Having the above information, and provided that the implementations of all factories and classes are available, the FactoryResolver will be able to construct, via reflection, the factories which can be used in order for the query to be properly processed. For example, if there are implementations available for core functionality, as well as for Geo and Time extensions and the query contains parameters for both two of these methods, the resulting chain of responsibility of the constructed instances will contain all three implementations, the last one being the implementation which supports core functionality, and the other two appearing in the chain in any order. If the query contains only standard OpenSearch parameters, there is no need for the chain to be burdened with links that will never be used, therefore a chain consisting of only the core implementation is constructed. The same holds if, for example, there are not Geo parameters present in the query; the corresponding implementation will not be included in the chain.

It should be stressed again that it is not necessary for all extensions that are expected to be met to be implemented in order for the library to work. The extension of the library remains a purely optional task. There are, therefore, two choices when using the library

  • Do not extend the library when in need of using extended parameters, relying in the core functionality provided by the library. In that case, the caller should be careful to supply the library with the correct values and format of parameters, so that a query can be constructed, albeit without the option of query validation or the ability to exploit additional functionality related to the extension.
  • Extend the library whenever this proves useful or makes things easier.
Implementing a new Extension

In order to implement a new extension that will be correctly incorporated into the already existing library functionality, one should do the following:

  • Implement URLElement, QueryElement and QueryBuilder interface implementations, the constructor of which accepts at least a reference to a corresponding upcasted object which will be next in the chain of responsibility. All requests that cannot be handled, or require additional processing by subsequent links in the chain, should be forwarded to the next link in the chain.
  • Implement a URLElementFactory and a QueryElementFactory interface implementations, the constructors of which accept a single argument, which is a reference to an upcasted factory of the same type corresponding to the factory used to create instances next in the chain of responsibility. An example of a URLElementFactory used for the construction of GeoURLElements which implement Geo extension functionality is the following:
public class GeoURLElementFactory implements URLElementFactory {
 
	URLElementFactory f;
 
	public GeoURLElementFactory(URLElementFactory f) {
		this.f = f;
	}
 
	public GeoURLElement newInstance(Element url, Map<String, String> nsPrefixes) throws Exception {
		URLElement el = f.newInstance(url, nsPrefixes);
		return new GeoURLElement(url, el);
	}
}
  • See that the getQueryBuilder method of the URLElement implementation of the new extension correctly constructs a QueryBuilder instance. For example, the getQueryBuilder method of the GeoQueryElement presented above could look like this:
public QueryBuilder getQueryBuilder() throws Exception {
	return new GeoQueryBuilder(el.getQueryBuilder());
}

where el is the next URLElement in the chain.

  • Add a mapping for the two new factories to the set of mappings passed to the FactoryResolver utility upon initialization of the library. For example, given that GeoURLElementFactory and GeoQueryElementFactory are implemented for Geo Extensions, one could add the mapping as follows:
factoryMappings.add("http://a9.com/-/opensearch/extensions/geo/1.0/",
        new FactoryClassNamePair("org.gcube.opensearch.opensearchlibrary.urlelements.extensions.geo.GeoURLElement", "org.gcube.opensearch.opensearchlibrary.queryelements.extensions.geo.GeoQueryElement"));

The OpenSearch Operator

Description

The role of the OpenSearch Operator is to provide support for querying and retrieval of search results via OpenSearch from providers which expose an OpenSearch description document. The operator accepts a query string consisting of a set query parameters which may include a number of search terms and an OpenSearch Resource reference which contains the URL of an OpenSearch description document and various specifications relevant to the OpenSearch provider to be queried. After performing the number of OpenSearch queries required to obtain the desired results, it returns these results wrapping them in a ResultSet.

Extensibility Points

The operator introduces and makes use of a set of functionalities beyond those of the standard OpenSearch specification. These extensions are supported by the introduction of a special OpenSearch Resource structure and by the internal logic of the operator, the latter using standard OpenSearch functionality provided by the OpenSearch Core Library. The extra functionalities are summarized as follows:

  • Both direct and brokered result processing is supported. Some OpenSearch-enabled providers diverge from the common case of returning a set of direct results and instead provide their results indirectly, by returning a set of links to other OpenSearch-enabled providers. Provided that both a transformation specification used to extract these links from the returned results as well as the OpenSearch resources for each one of the brokered OpenSearch services are available, the operator will return the full set of results provided by the brokered OpenSearch services.
  • The support of a set of fixed parameters, which override the user-provided parameters only at the level of the top provider, i.e either the broker or the single direct provider in the direct provider case. The purpose of these parameters is, first, to facilitate the creation of dynamic collections from results obtained by brokers by superseeding the caller's query parameters while querying the broker and using the full set of the caller parameters only on lower levels and, second, to customize the behaviour of some provider to the needs of the gCube Framework (for example set a value for a required query parameter that the framework cannot handle). These options can be used both in tandem, if desired.
  • Support for one or more security schemes is planned for a subsequent version of the OpenSearch Library.

OpenSearch Resource

The purpose of an OpenSearch Resource object is to describe the specifications of an OpenSearch provider. It encapsulates the extensions described in the Extensibility Points section. The attributes included are the following:

  • The name of the resource
  • The URL of the OpenSearch Description Document of the provider to be queried
  • Information about whether the provider returns direct or brokered results, used by the operator to adapt its operation to both kinds of providers.
  • Data transformation specifications for a subset of the MIME types of the results which the result provider returns. The data transformation consists of two or, optionally, three parts:
    • The RecordSplitXPath expression is used to split a page of search results into individual records. For example for the rss format, the <item> elements under rss/channel could be of interest
    • The presentationInfo expression which is actually a map between fieldnames and XPath expressions that are used to extract the desired value from the response.
    • The optional RecordIdXPath expression can be used to tag each record with a unique identifier, extracted from the record itself. If the element is found, its payload is added as a DocID record attribute. Otherwise, if the element is not found or is empty, no DocID attribute is added to the record.
  • Security specifications (planned for a future version, when the supported security specifications are decided on). This element is optional, its absence implying the absence of a security scheme,

The serialization of an OpenSearch Resource can be easily incorporated into a Generic Resource. The default mode of operation for the OpenSearch Operator in fact obtains the necessary OpenSearch resources by retrieving the corresponding Generic Resource from the IS. The Generic Resources utilized by the OpenSearch Operator is:

  • The OpenSearchResource which contains the body of the OpenSearch Resource as described below


Note that, solely for testing purposes, the OpenSearch Operator also supports a local mode of operation, whereby all OpenSearch Resources are loaded from the local file system.

The XML Schema that all OpenSearch Resource serializations should conform to is the following: 

<?xml version="1.0"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
   <xs:element name="OpenSearchResource">
      <xs:complexType>
         <xs:sequence>
            <xs:element name="name" type="xs:string"/>
            <xs:element name="descriptionDocumentURI" type="xs:string"/>
            <xs:element name="brokeredResults" type="xs:boolean"/>
            <xs:element name="transformation" maxOccurs="unbounded">
               <xs:complexType>
                  <xs:sequence>
                     <xs:element name="MIMEType" type="xs:string"/>
                     <xs:element name="recordSplitXPath" type="xs:string"/>
                     <xs:element name="recordIdXPath" type="xs:string" minOccurs="0" maxOccurs="1"/>
 		     <xs:element name="presentationInfo" maxOccurs="unbounded">
		         <xs:element name="presentable" maxOccurs="unbounded">
		             <xs:element name="fieldName" type="xs:string"/>
                             <xs:element name="expression" type="xs:string"/>
                         </xs:element>
                     </xs:element>
                  </xs:sequence>
               </xs:complexType>
            </xs:element>
            <xs:element name="security" minOccurs="0">
               <xs:complexType>
                  <xs:sequence>
                  </xs:sequence>
               </xs:complexType>
            </xs:element>		
         </xs:sequence>
      </xs:complexType>
   </xs:element>
</xs:schema>

The transformation element can appear multiple times within an OpenSearch Resource. The usual case is for a single transformation element per provider to be specified, but if transformation elements are present for more than one MIME type, the operator has the alternative of resorting to the next available transformation in sequence, if the result retrieval procedure fails for some reason. This strategy can only be meaningful if the same amount of information can be obtained from different result MIME types.

In the case of querying providers which return brokered results, the transformation element is used to specify a data transformation that extracts the URLs of the Description Documents of the brokered OpenSearch providers from the initial results provided by the OpenSearch provider acting as a broker.

An example of an OpenSearch Resource serialization describing the Bing external repository as a direct OpenSearch provider, currently in use by the gCube Framework is the following: 

<OpenSearchResource>
            <name>Bing</name>
            <descriptionDocumentURI>http://imarine.web.cern.ch/imarine/OpenSearch/bing.xml</descriptionDocumentURI>
            <brokeredResults>false</brokeredResults>
            <parameters>
                   <parameter>
                          <fieldName>allIndexes</fieldName>
                          <qName>http%3A%2F%2Fa9.com%2F-%2Fspec%2Fopensearch%2F1.1%2F:searchTerms</qName>
                   </parameter>
            </parameters>
            <transformation>
                   <MIMEType>application/rss+xml</MIMEType>
                   <recordSplitXPath>*[local-name()='rss']/*[local-name()='channel']/*[local-name()='item']</recordSplitXPath>
                   <recordIdXPath>//*[local-name()='item']/*[local-name()='link']</recordIdXPath>
                   <presentationInfo>
                          <presentable>
                                 <fieldName>title</fieldName>
                                 <expression>//*[local-name()='item']/*[local-name()='title']</expression>
                          </presentable>
                          <presentable>
                                 <fieldName>link</fieldName>
                                 <expression>//*[local-name()='item']/*[local-name()='link']</expression>
                          </presentable>
                          <presentable>
                                 <fieldName>description</fieldName>
                                 <expression>//*[local-name()='item']/*[local-name()='description']</expression>
                          </presentable>
                          <presentable>
                                 <fieldName>S</fieldName>
                                 <expression>//*[local-name()='item']/*[local-name()='description']</expression>
                          </presentable>
                          <presentable>
                                 <fieldName>pubDate</fieldName>
                                 <expression>//*[local-name()='item']/*[local-name()='pubDate']</expression>
                          </presentable>
                   </presentationInfo>
            </transformation>
</OpenSearchResource>

In case our datasrouce is broker the brokeredResults should be changed to true as: 

   <brokeredResults>true</brokeredResults>

OpenSearch Operator Logic

Figure 1: A simplified flowchart of the operations performed by the OpenSearch operator

The OpenSearch Operator employs the functionality provided by the OpenSearch Core Library in order to extract the required information from the Description Document of the external provider and the QueryBuilders needed in order to perform queries, in a fashion similar to that described in the OpenSearch Library Functionality section.

It should be noted that the OpenSearch Operator abstracts away the MIME type of the results that are to be obtained, treating it as low-level information which can be exploited by the way OpenSearch Resources are structured. Given that the OpenSearch Specification makes no assumptions about differences in the amount of information returned by results of different MIME types, there are two options

  • If the amount of information returned from results of MIME Type A and MIME Type B are different, the desired MIME Type should be selected and an OpenSearch Resource constructed with only this MIME Type present in the transformation specifications. If needed, additional OpenSearch Resources can be constructed to exploit information returned from different MIME types. In this way, the MIME Type is abstracted away by the conceptual level of information detail obtained by the provider.
  • If there more than one result MIME types exposing the same amount of information, or containing the same subset of information of interest, there exists the option of specifying more than one transformation specifications, in a way which will result in the uniform presentation of the data to the caller. In this way, the MIME Types are abstracted away by unifying result formats to a provider-specific schema. The option of having a single transformation specification is of course available in this case as well.

In order for the proper way of constructing queries to be selected, a preprocessing step is performed by the operator, before issuing any actual queries. Given that an OpenSearch Resource can contain more that one transformation specifications and that the number of the templates present in a URL Element is not necessarily limited to one, there can be more than one potential template matches for the caller's query. The purpose of the preprocessing step is to select the QueryBuilder whose parameters best match the query of the latter. First, all MIME types supported by the external provider, but lacking an associated transformation specification in the OpenSearch Resource describing the provider are discarded. The QueryBuilders of the first available MIME type for which a transformation specification exists are processed and reordered according to the following rules:

  • QueryBuilders whose required parameters are not covered by the parameters of the caller's query are discarded
  • QueryBuilders are reordered so that the first one best matches the caller's query, i.e. all of its required parameters and as many of its optional parameters as possible are covered
  • A QueryBuilder which lacks a parameter present in the caller's query is considered a match. In that case the extra parameter is discarded. This rule assumes that query parameters narrow the search down and is enforced in order to account for brokered providers exposing slightly different sets of parameters than the broker or their siblings.

The most usual case is for the provider's OpenSearch Resource to be set up with a single transformation specification for the MIME type of interest. Furthermore, most URL Elements provide a single template for each MIME type. This case results to only one QueryBuilder being available to construct queries, thereby resulting to a degenerate reordering step.

The functions performed by the operator in order for a set of results to be retrieved, given that the proper QueryBuilder is selected, are summarized in the simplified diagram of Figure 1.

As shown, the operator accepts a set of query terms and a set of query parameters.

The operator's main course of action is to formulate and send queries requesting pages of search results as long as there still are results to be returned and the caller requirement of the number of results, if present, is not met. A pager component sees that page switching is performed correctly, managing the relevant standard OpenSearch query parameters, namely startPage or startIndex and count. These parameters are therefore abstracted away by the OpenSearch Operator.

In the case of resources which return brokered results, the operator first retrieves the endpoints of the underlying brokered OpenSearch providers and reads their corresponding OpenSearch Resources so as to be able to retrieve the actual search results from them, either sequentially or concurrently. The extraction of brokered provider endpoints is not explicitly shown in the diagram. Furthermore, if an OpenSearch Resource structure is missing for one or more of the brokered services, the operator continues with the retrieval of results from the next available brokered service, ignoring it if it cannot obtain information for it. The same holds if all query formulation attempts for a provider fail.

Configurable Parameters

The OpenSearch Operator can be programmaticaly configured by passing to it a special configuration construct upon creation. The configuration parameters are the following:

  • The resultsPerPage parameter instructs the operator as to how many results per page should be requested when no other paging restrictions are in effect. The default value for this parameter currently is 100.
  • The sequentialResults parameter disables or enables multi-threaded result retrieval from brokered providers. When enabled, the results are retrieved from each provider in a sequential manner i.e the results retrieved from different providers are not intermingled. There is, however, a negative impact on performance. The default value for this configuration parameter currently is false.
  • The useLocalResource parameter, when enabled, permits the operator to operate in the absence of an IS. The OpenSearch Resources are instead retrieved from the local file system. It is used solely for testing reasons, which is why its default value is, and will remain equal to false.

An additional configurable element are the mappings from query namespaces to the corresponding factories, as described in Library Extensibility. The sequentialResults parameter can also be configured in a per-query manner, including it in the query string as a query parameter.

Query Format

The OpenSearch Operator expects to receive all query parameters, including the search terms, in a single query string. All query parameters should be of the form

<URL-Encoded_Namespace_URI>:<Parameter_Name>="<Parameter_Value>"

and should be space-delimited. Note that the presence of a namespace is mandatory for standard OpenSearch parameters as well. Any free-text parameter value should be URL-encoded.

The reserved keyword config when used as a parameter namespace denotes a configuration parameter. The query configuration parameters under the special configuration namespace include the sequentialResults parameter described in Configurable Parameters, plus the numOfResults parameter, which can be used to impose a limit on the number of retrieved results. These two query configuration parameters are optional. The following hold for the query configuration parameter values

  • The sequentialResults parameter should be assigned a value equal to true or false. Its absence implies the default value of the corresponding configurable parameter of the operator.
  • The numOfResults parameter should be assigned an integral valie, its absence implying that all available results should be retrieved.

Taking everything into account, an example of a legitimate query for the OpenSearch Operator could be the following:

http%3A%2F%2Fa9.com%2F-%2Fspec%2Fopensearch%2F1.1%2F:searchTerms="Hello+World" config:numOfResults="300"

which instructs the operator to use the string Hello World as the value for the SearchTerms standard OpenSearch parameter and to retrieve up to 300 results from the provider.

The OpenSearch Service

Description

The OpenSearch Service is a stateful web service responsible for the invocation of the OpenSearch Operator in the context of the provider to be queried. It also maintains a cache per provider WS-Resource, which contains the Generic Resources relevant to the top provider, the Generic Resources of all previously queried brokered providers and the corresponding Description Documents.


Deployment Instructions

In order to deploy and run Index Service on a node we will need the following:

  • opensearchdatasource-{version}.war
  • smartgears-distribution-{version}.tar.gz (to publish the running instance of the service on the IS and be discoverable)
    • see here for installation
  • an application server (such as Tomcat, JBoss, Jetty)

There are a few things that need to configured in order for the service to be functional. All the service configuration is done in the file deploy.properties that comes within the service war. Typically, this file should be loaded in the classpath so that it can be read. The default location of this file (in the exploded war) is webapps/service/WEB-INF/classes.

The hostname of the node as well as the port and the scope that the node is running on have to set in the in the variables hostname and scope in the deploy.properties.

Example : 

hostname=dl08.madgik.di.uoa.gr
port=8080
scope=/gcube/devNext

Finally, Resource Registry should be configured to not run in client mode. This is done in the deploy.properties by setting: 

clientMode=false

NOTE: it is important to note that resourcesFoldername property has relative path in its default value (./resources/opensearch). In some cases this value maybe evaluated by taking into account the folder that the container was started, so in order to avoid problems related to this behavior it is better for this property to take an absolute path as value.

WS and Generic Resource Interrelation

Provided that a Collection for the provider to be queried is available, the OpenSearch Service uses a WS-Resource for each OpenSearch provider in order to bind the Service to the Collection corresponding to the provider.

An OpenSearch Service WS-Resource contains the following properties:

  • The AdaptorID which is unique for every WSResource and is used for referencing the right WSResource on querying (it is optional).
  • The CollectionID of the collection to be used.
  • The ID of the Generic Resource of the top-provider, where the top-provider is the broker in the brokered case or the one and only direct provider in the direct case.
  • The URI of the Description Document (DescriptionDocumentURI) of the top-provider.
  • A set of fields (presentables and searchables) as extracted from the OpenSearchGenericResource.
  • A set of FixedParameters, which are used in every invocation of the Operator. See also Extensibility Points.

As mentioned above, the WS-Resource contains a reference only to the OpenSearchResource of the top-provider. The Generic Resources of any providers reached through a broker are retrieved through an Information System implementation and are therefore not directly referenced by the WS-Resource.

Some properties are dependent on information residing in the Generic Resources describing the providers and should, therefore, be updated accordingly when these Generic Resources are modified. Examples of such properties include the Description Document URI and the templates.

On WS-Resource creation, only the Metadata Collection ID and the ID of the OpenSearchResource of the top-provider need to be supplied to the create operation of the service's factory. All other properties are created internally by the service itself.

Resource Caching

For performance and reliability reasons, the OpenSearch Service maintains one cache per WS-Resource which initially contains the Generic Resource (OpenSearchResource) and the Description Document of the top-provider. In the brokered case, the cache is updated at run-time by the relevant OpenSearch Operator module with the Generic Resources and Description Documents of all providers reached through the broker.

To account for potential updates of Description Documents and/or the Generic Resources, the cache can be refreshed either on demand or periodically, based on a configurable time interval. The periodic refresh operation can be disabled if the Description Documents and the Generic Resource configuration is considered to be stable enough.

The cache refresh cycle policy used is described as follows:

  • The Description Document and Generic Resources of the top-provider are discarded and their updated versions are retrieved from the external site and the Information System respectively. All dependent WS-Resource properties are also updated accordingly.
  • The Description Documents and Generic Resources of all brokered providers, if any, are discarded. Because of the potentially large number of brokered providers, the cache is not repopulated with their updated versions to avoid locking the cache for large periods of time or using a partially updated cache. These pieces of information are re-cached at run-time instead.
  • In the event of failure, the previously cached version is kept.

Operations

The operations exposed by the OpenSearch Service are the following:

  • The query operation, with a single input message containing the query string to be sent to the operator, whose format is described in Query Format.
  • The refreshCache operation, which sends a request in order to force the cache of the service to be refreshed. No refresh cycle will be initiated if a periodic refresh cycle is currently in progress.

Configurable Parameters

The Service currently supports three configurable parameters, which are exposed to its deployment descriptor

  • The clearCacheOnStartup parameter, of boolean type, when enabled instructs the service to discard the stored cache on startup.
  • The cacheRefreshIntervalMillis parameter, of integral type, defines a time interval for the periodic cache refresh operation. The value 0 can be used to disable periodic cache refresh cycles.
  • The openSearchLibraryFactories parameter, of string type is used to supply the OpenSearch Core Library with the factory mappings for all namespaces for which there exists an implementation of a library extension. For more information on the mappings, see also the section referring to the extensibility mechanism of the library.

The openSearchLibraryFactories parameter is encoded as a sequence of mappings from strings to pairs, where each mapping is enclosed in braces, association is denoted by the = sign and each pair is enclosed in parentheses. For example, given that there are implementations for core functionality, Geo and Time extensions, the value of this configuration parameter could be the following:

[http://a9.com/-/spec/opensearch/1.1/=(org.gcube.opensearch.opensearchlibrary.urlelements.BasicURLElementFactory,org.gcube.opensearch.opensearchlibrary.queryelements.BasicQueryElementFactory)] [http://a9.com/-/opensearch/extensions/geo/1.0//=(org.gcube.opensearch.opensearchlibrary.urlelements.extensions.geo.GeoURLElementFactory,org.gcube.opensearch.opensearchlibrary.queryelements.extensions.geo.GeoQueryElementFactory)] [http://a9.com/-/opensearch/extensions/time/1.0/=(org.gcube.opensearch.opensearchlibrary.urlelements.extensions.time.TimeURLElementFactory,org.gcube.opensearch.opensearchlibrary.queryelements.extensions.time.TimeQueryElementFactory)]

The OpenSearchDataSource Client Library

In this section some examples of usage of the OpenSearchDataSource Client Library are provided.

Query example: 

final String scope = "/gcube/devNext";
 
final OpenSearchClient client = new OpenSearchClient.Builder()
    // .endpoint(endpoint) // you can also give a specific endpoint
    .scope(scope)
    .build();
 
final String queryString = "((((gDocCollectionID == \"ea5b9c70-01ce-4b45-96e7-6db037ebf2bc\") and (gDocCollectionLang == \"en\"))) and (5575bbdb-6d47-4297-ad12-2259b3405ce7 = greece)) project d91f3c47-e46e-4737-9496-a0f72361a397 3e3584f0-eed3-4089-99cd-86a7def1471e";
 
String grs2Locator = client.query(queryString);
 
// or
 
List<Map<String, String>> records = client2.queryAndRead(queryString);

Create Resource method example: 

static void createResource(List<String> fieldParameters, List<String> fixedParameters, String collectionID, String openSearchResourceID, String scope) throws OpenSearchClientException {
 
    OpenSearchFactoryClient factory = injector.getInstance(OpenSearchFactoryClient.Builder.class)
        .scope(scope)
        .build();
 
 
     List<String> fieldParams = Lists.newArrayList();
 
     for (int i=0; i<fieldParameters.size(); i++) {
         fieldParams.add(collectionID + ":" + fieldParameters.get(i));
         System.out.println("Field parameter: " + (i+1) + " " + fieldParams.get(i));
     }
 
     Provider p = new Provider();
     p.setCollectionID(collectionID);
     p.setOpenSearchResourceID(openSearchResourceID);
     p.setFixedParameters(fixedParameters);
 
     List<Provider> providers = Lists.newArrayList();
     providers.add(p);
 
     factory.createResource(fieldParams, providers, scope);
}

External Links

Some useful external links for further reading are provided here:

Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=OpenSearch_Framework&oldid=27342"