Creating Indices at the VO Level

From Gcube Wiki
Jump to: navigation, search

Indexing Procedure

The Indexing procedure refers to the creation of indices for the collections imported in a Virtual Organization. It consists of three steps:

  • Creation of the Rowset XSLT generic resources, that transform collection data into data that can be fed to an Index.
  • Creation of the Index type generic resources, that define the Index configuration.
  • Definition of an IRBootstrapper job that will perform the steps required to create the Indices.

In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the Resource Management portlet . You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following section:

You can find detailed descriptions for the Index Type definition here:

For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the IR Bootstrapper section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps.

Creating an Index for an OAI-DC collection

DataTransformation Programs

FtsRowset_Transformer

The following transformation program is called for fulltext rowset creation. Transformation unit with id="6" takes multiple XSLTs and applies final XSLT at the end.

File:FtsRowset Transformer.xml

Index Types

In this section we present the required IndexType for (FullText) Index.

FullTextIndexType

In order to extract the fields from the OAI-DC payload and build the FullText Index the following FullTextIndexType is required: 

<Name>IndexType_ft_oai_dc_1.0</Name>
<SecondaryType>FullTextIndexType</SecondaryType>
<Description>Definition of the fulltext index type for the 'oai dc' schema</Description>
<Body>
    <index-type name="default">
        <field-list sort-xnear-stop-word-threshold="2E8">
              <field name="contributor">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="coverage">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="creator">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="date">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="description">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="format">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="identifier">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="language">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="publisher">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="relation">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="rights">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="source">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="subject">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="title">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="type">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="ObjectID">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="gDocCollectionID">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="gDocCollectionLang">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="S">
                     <index>no</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>no</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
        </field-list>
    </index-type>
</Body>


Bootstrapper Configuration

The IRBootstrapper portlet requires a Generic Resource to be available on the IS with name: IRBootstrapperConfiguration and secondary type: IRBootstrapperConfig. For more information please refer to this section IRBoostrapperConfiguration Generic Resource

An example of the configuration is the following:

File:Bootstrapper Configuration.xml

Metadata Broker XSLT

  • BrokerXSLT_oai_dc_anylanguage_to_ftRowset_anylanguage

The following XSLT transforms data elements with oai-dc schema to fulltext rowsets:

File:BrokerXSLT oai dc anylanguage to ftRowset anylanguage.xml

Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Creating_Indices_at_the_VO_Level&oldid=21391"