Difference between revisions of "Creating Indices at the VO Level"

From Gcube Wiki
Jump to: navigation, search
(→‎Metadata Broker XSLT)
(→‎FtsRowset_Transformer)
 
(2 intermediate revisions by the same user not shown)
Line 8: Line 8:
 
* Definition of an [[ IR Bootstrapper | IRBootstrapper]] job that will perform the steps required to create the Indices.
 
* Definition of an [[ IR Bootstrapper | IRBootstrapper]] job that will perform the steps required to create the Indices.
  
−
In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the [[ Resource Management | Resource Management portlet ]]. You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following sections:
+
In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the [[ Resource Management | Resource Management portlet ]]. You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following section:
  
 
* [[ Index_Management_Framework#RowSet| Full Text Index Rowset ]]
 
* [[ Index_Management_Framework#RowSet| Full Text Index Rowset ]]
−
* [[ Index_Management_Framework#RowSet_2 | Forward Index Rowset ]]
 
  
 
You can find detailed descriptions for the Index Type definition here:
 
You can find detailed descriptions for the Index Type definition here:
  
 
* [[ Index_Management_Framework#IndexType | Full Text Index Type ]]
 
* [[ Index_Management_Framework#IndexType | Full Text Index Type ]]
−
* [[ Index_Management_Framework#Forward_Index | Forward Index key-value pairs ]]
 
  
 
For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the [[ IR Bootstrapper ]] section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps.
 
For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the [[ IR Bootstrapper ]] section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps.
  
−
==Creating a Full Text and a Forward Index for a OAI-DC collection==
+
==Creating an Index for an OAI-DC collection==
  
 
=== DataTransformation Programs ===
 
=== DataTransformation Programs ===
Line 28: Line 26:
  
 
[[File:FtsRowset_Transformer.xml]]
 
[[File:FtsRowset_Transformer.xml]]
−
 
−
====FwRowset_Transformer====
 
−
The following transformation program is called for forward rowset creation. Transformation unit with id="1" takes multiple XSLTs and applies final XSLT at the end.
 
−
 
−
[[File:FwRowset_Transformer.xml]]
 
  
 
=== Index Types ===
 
=== Index Types ===
−
In this section we present the required IndexTypes for both FullText and Forward Indices.
+
In this section we present the required IndexType for (FullText) Index.
  
 
====FullTextIndexType====
 
====FullTextIndexType====
Line 204: Line 197:
 
</source>
 
</source>
  
−
====ForwardIndexType====
 
−
In OAI-DC many fields, such as "title" and "creator",  have string values, so we just need to create a ForwardIndexType for string-string key-value pairs in order to
 
−
be able to create the Forward Index:
 
−
 
−
<source lang="xml">
 
−
<SecondaryType>ForwardIndexType</SecondaryType>
 
−
<Name>IndexType_fwd_string_string</Name>
 
−
<Description>Definition of the index type 'string_string' for the forward index</Description>
 
−
<Body>
 
−
<field-list>       
 
−
    <field name="key">   
 
−
      <type>string</type>
 
−
      <sort>ascending</sort>
 
−
    </field>
 
−
    <field name="value">
 
−
      <type>string</type>
 
−
    </field>
 
−
</field-list>
 
−
</Body>
 
−
</source>
 
−
 
−
Note that, in contrast to the FullTextIndexType in ForwardIndexType there is no field-datatype mapping but just declaration of the datatypes supported in the index.
 
  
 
=== Bootstrapper Configuration ===
 
=== Bootstrapper Configuration ===

Latest revision as of 17:04, 9 May 2014

Indexing Procedure

The Indexing procedure refers to the creation of indices for the collections imported in a Virtual Organization. It consists of three steps:

  • Creation of the Rowset XSLT generic resources, that transform collection data into data that can be fed to an Index.
  • Creation of the Index type generic resources, that define the Index configuration.
  • Definition of an IRBootstrapper job that will perform the steps required to create the Indices.

In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the Resource Management portlet . You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following section:

You can find detailed descriptions for the Index Type definition here:

For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the IR Bootstrapper section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps.

Creating an Index for an OAI-DC collection

DataTransformation Programs

FtsRowset_Transformer

The following transformation program is called for fulltext rowset creation. Transformation unit with id="6" takes multiple XSLTs and applies final XSLT at the end.

File:FtsRowset Transformer.xml

Index Types

In this section we present the required IndexType for (FullText) Index.

FullTextIndexType

In order to extract the fields from the OAI-DC payload and build the FullText Index the following FullTextIndexType is required: 

<Name>IndexType_ft_oai_dc_1.0</Name>
<SecondaryType>FullTextIndexType</SecondaryType>
<Description>Definition of the fulltext index type for the 'oai dc' schema</Description>
<Body>
    <index-type name="default">
        <field-list sort-xnear-stop-word-threshold="2E8">
              <field name="contributor">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="coverage">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="creator">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="date">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="description">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="format">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="identifier">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="language">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="publisher">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="relation">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="rights">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="source">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="subject">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="title">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="type">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="ObjectID">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="gDocCollectionID">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="gDocCollectionLang">
                     <index>yes</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>yes</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
              <field name="S">
                     <index>no</index>
                     <store>yes</store>
                     <return>yes</return>
                     <tokenize>no</tokenize>
                     <sort>no</sort>
                     <boost>1.0</boost>
              </field>
        </field-list>
    </index-type>
</Body>


Bootstrapper Configuration

The IRBootstrapper portlet requires a Generic Resource to be available on the IS with name: IRBootstrapperConfiguration and secondary type: IRBootstrapperConfig. For more information please refer to this section IRBoostrapperConfiguration Generic Resource

An example of the configuration is the following:

File:Bootstrapper Configuration.xml

Metadata Broker XSLT

  • BrokerXSLT_oai_dc_anylanguage_to_ftRowset_anylanguage

The following XSLT transforms data elements with oai-dc schema to fulltext rowsets:

File:BrokerXSLT oai dc anylanguage to ftRowset anylanguage.xml

Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Creating_Indices_at_the_VO_Level&oldid=21391"