|
|
(11 intermediate revisions by 2 users not shown) |
Line 8: |
Line 8: |
| * Definition of an [[ IR Bootstrapper | IRBootstrapper]] job that will perform the steps required to create the Indices. | | * Definition of an [[ IR Bootstrapper | IRBootstrapper]] job that will perform the steps required to create the Indices. |
| | | |
− | In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the [[ Resource Management | Resource Management portlet ]]. You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following sections: | + | In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the [[ Resource Management | Resource Management portlet ]]. You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following section: |
| | | |
| * [[ Index_Management_Framework#RowSet| Full Text Index Rowset ]] | | * [[ Index_Management_Framework#RowSet| Full Text Index Rowset ]] |
− | * [[ Index_Management_Framework#RowSet_2 | Forward Index Rowset ]]
| |
| | | |
| You can find detailed descriptions for the Index Type definition here: | | You can find detailed descriptions for the Index Type definition here: |
| | | |
| * [[ Index_Management_Framework#IndexType | Full Text Index Type ]] | | * [[ Index_Management_Framework#IndexType | Full Text Index Type ]] |
− | * [[ Index_Management_Framework#Forward_Index | Forward Index key-value pairs ]]
| |
| | | |
| For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the [[ IR Bootstrapper ]] section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps. | | For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the [[ IR Bootstrapper ]] section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps. |
| | | |
− | ==Creating a Full Text and a Forward Index for a OAI-DC collection== | + | ==Creating an Index for an OAI-DC collection== |
| | | |
| === DataTransformation Programs === | | === DataTransformation Programs === |
| | | |
− | *FtsRowset_Transformer
| + | ====FtsRowset_Transformer==== |
− | <gDTSTransformationProgram>
| + | The following transformation program is called for fulltext rowset creation. Transformation unit with id="6" takes multiple XSLTs and applies final XSLT at the end. |
− | <Transformer>
| + | |
− | <Class>org.gcube.datatransformation.datatransformationlibrary.programs.metadata.indexfeed.FtsRowset_Transformer</Class>
| + | |
− | <GlobalProgramParameters />
| + | |
− | </Transformer>
| + | |
− | <TransformationUnits>
| + | |
− | <TransformationUnit id="6" isComposite="false">
| + | |
− | <Sources>
| + | |
− | <<nowiki>Source</nowiki>>
| + | |
− | <Input id="TRInput0" />
| + | |
− | <ContentType>
| + | |
− | <Mimetype>text/xml</Mimetype>
| + | |
− | <Parameters>
| + | |
− | <Parameter isOptional="false" name="schemaURI" value="-" />
| + | |
− | </Parameters>
| + | |
− | </ContentType>
| + | |
− | <<nowiki>/Source</nowiki>>
| + | |
− | </Sources>
| + | |
− | <ProgramParameters>
| + | |
− | <Parameter isOptional="false" name="finalftsxslt" value="-" />
| + | |
− | <Parameter isOptional="false" name="indexType" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:1" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:2" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:3" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:4" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:5" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:6" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:7" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:8" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:9" value="-" />
| + | |
− | </ProgramParameters>
| + | |
− | <Target>
| + | |
− | <Output id="TROutput" />
| + | |
− | <ContentType>
| + | |
− | <Mimetype>text/xml</Mimetype>
| + | |
− | <Parameters>
| + | |
− | <Parameter isOptional="false" name="schemaURI" value="http://ftrowset.xsd" />
| + | |
− | </Parameters>
| + | |
− | </ContentType>
| + | |
− | </Target>
| + | |
− | </TransformationUnit>
| + | |
− | </TransformationUnits>
| + | |
− | </gDTSTransformationProgram>
| + | |
| | | |
− | *FwRowset_Transformer
| + | [[File:FtsRowset_Transformer.xml]] |
− | <gDTSTransformationProgram>
| + | |
− | <Transformer>
| + | |
− | <Class>org.gcube.datatransformation.datatransformationlibrary.programs.metadata.indexfeed.FwRowset_Transformer</Class>
| + | |
− | <GlobalProgramParameters />
| + | |
− | </Transformer>
| + | |
− | <TransformationUnits>
| + | |
− | <TransformationUnit id="1" isComposite="false">
| + | |
− | <Sources>
| + | |
− | <<nowiki>Source</nowiki>>
| + | |
− | <Input id="TRInput0" />
| + | |
− | <ContentType>
| + | |
− | <Mimetype>text/xml</Mimetype>
| + | |
− | <Parameters />
| + | |
− | </ContentType>
| + | |
− | <<nowiki>/Source</nowiki>>
| + | |
− | </Sources>
| + | |
− | <ProgramParameters>
| + | |
− | <Parameter isOptional="false" name="finalfwdxslt" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:1" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:2" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:3" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:4" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:5" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:6" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:7" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:8" value="-" />
| + | |
− | <Parameter isOptional="true" name="xslt:9" value="-" />
| + | |
− | </ProgramParameters>
| + | |
− | <Target>
| + | |
− | <Output id="TROutput" />
| + | |
− | <ContentType>
| + | |
− | <Mimetype>text/xml</Mimetype>
| + | |
− | <Parameters>
| + | |
− | <Parameter isOptional="false" name="schemaURI" value="http://fwrowset.xsd" />
| + | |
− | </Parameters>
| + | |
− | </ContentType>
| + | |
− | </Target>
| + | |
− | </TransformationUnit>
| + | |
− | </TransformationUnits>
| + | |
− | </gDTSTransformationProgram>
| + | |
| | | |
| === Index Types === | | === Index Types === |
− | In this section we present the required IndexTypes for both FullText and Forward Indices. | + | In this section we present the required IndexType for (FullText) Index. |
| | | |
| ====FullTextIndexType==== | | ====FullTextIndexType==== |
Line 283: |
Line 197: |
| </source> | | </source> |
| | | |
− | ====ForwardIndexType====
| |
− | In OAI-DC many fields, such as "title" and "creator", have string values, so we just need to create a ForwardIndexType for string-string key-value pairs in order to
| |
− | be able to create the Forward Index:
| |
− |
| |
− | <source lang="xml">
| |
− | <SecondaryType>ForwardIndexType</SecondaryType>
| |
− | <Name>IndexType_fwd_string_string</Name>
| |
− | <Description>Definition of the index type 'string_string' for the forward index</Description>
| |
− | <Body>
| |
− | <field-list>
| |
− | <field name="key">
| |
− | <type>string</type>
| |
− | <sort>ascending</sort>
| |
− | </field>
| |
− | <field name="value">
| |
− | <type>string</type>
| |
− | </field>
| |
− | </field-list>
| |
− | </Body>
| |
− | </source>
| |
− |
| |
− | Note that, in contrast to the FullTextIndexType in ForwardIndexType there is no field-datatype mapping but just declaration of the datatypes supported in the index.
| |
| | | |
| === Bootstrapper Configuration === | | === Bootstrapper Configuration === |
Line 311: |
Line 203: |
| | | |
| An example of the configuration is the following: | | An example of the configuration is the following: |
− | <source lang='xml'>
| + | |
− | <BootstrapInfo>
| + | [[File:Bootstrapper_Configuration.xml]] |
− | <types>
| + | |
− | <type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.TreeManagerCollectionDataType" name="TreeManagerCollection" />
| + | |
− | <type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.OpenSearchDataType" name="OpenSearch" />
| + | |
− | <type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.FullTextIndexNodeDataType" name="FullTextIndexNode" />
| + | |
− | <type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.ForwardIndexNodeDataType" name="ForwardIndexNode" />
| + | |
− | <type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.GCUBECollectionDataType" name="GCUBECollection" />
| + | |
− | <tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.OpenSearchGenerationTaskType" name="OpenSearchGenerationTaskType">
| + | |
− | <input type="GCUBECollection" />
| + | |
− | <output type="OpenSearch" />
| + | |
− | <run>true</run>
| + | |
− | </tasktype>
| + | |
− | <tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.ForwardIndexNodeGenerationTaskType" name="ForwardIndexNodeGenerationTask">
| + | |
− | <input type="TreeManagerCollection" />
| + | |
− | <output type="ForwardIndexNode" />
| + | |
− | <run>true</run>
| + | |
− | </tasktype>
| + | |
− | <tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.FullTextIndexNodeGenerationTaskType" name="FullTextIndexNodeGenerationTask">
| + | |
− | <input type="TreeManagerCollection" />
| + | |
− | <output type="FullTextIndexNode" />
| + | |
− | <run>true</run>
| + | |
− | </tasktype>
| + | |
− | <jobtype description="Creates the required fulltext indices for a collection." name="FTIndexNodeCollection">
| + | |
− | <input type="TreeManagerCollection" />
| + | |
− | <jobDefinition>
| + | |
− | <parallel>
| + | |
− | <sequential>
| + | |
− | <assign to="%Create_ft_node_index.input" value="%FTIndexNodeCollection.input" />
| + | |
− | <assign to="%Create_ft_node_index.output.IndexedCollectionID" value="%Create_ft_node_index.input.ColID" />
| + | |
− | <task name="Create_ft_node_index" tasktype="FullTextIndexNodeGenerationTask" />
| + | |
− | </sequential>
| + | |
− | </parallel>
| + | |
− | </jobDefinition>
| + | |
− | </jobtype>
| + | |
− | <jobtype description="Creates the required forward indices for a collection." name="FWDIndexNodeCollection">
| + | |
− | <input type="TreeManagerCollection" />
| + | |
− | <jobDefinition>
| + | |
− | <parallel>
| + | |
− | <sequential>
| + | |
− | <assign to="%Create_fwd_node_index.input" value="%FWDIndexNodeCollection.input" />
| + | |
− | <assign to="%Create_fwd_node_index.output.IndexedCollectionID" value="%Create_fwd_node_index.input.ColID" />
| + | |
− | <task name="Create_fwd_node_index" tasktype="ForwardIndexNodeGenerationTask" />
| + | |
− | </sequential>
| + | |
− | </parallel>
| + | |
− | </jobDefinition>
| + | |
− | </jobtype>
| + | |
− | <jobtype description="Creates the open search resource for an open search collection." name="CreateOpenSearchCollectionResource">
| + | |
− | <input type="GCUBECollection" />
| + | |
− | <jobDefinition>
| + | |
− | <parallel>
| + | |
− | <sequential>
| + | |
− | <assign to="%Create_OSR.input" value="%CreateOpenSearchCollectionResource.input" />
| + | |
− | <assign to="%Create_OSR.output.OpenSearchCollectionID" value="%Create_OSR.input.ColID" />
| + | |
− | <task name="Create_OSR" tasktype="OpenSearchGenerationTaskType" />
| + | |
− | </sequential>
| + | |
− | </parallel>
| + | |
− | </jobDefinition>
| + | |
− | </jobtype>
| + | |
− |
| + | |
− | </types>
| + | |
− |
| + | |
− | <jobs>
| + | |
− | <job jobtype="FTIndexNodeCollection" name="FullText Index SPD Tree Collections">
| + | |
− | <initialization>
| + | |
− | <assign to="%FTIndexNodeCollection.input.Type" value="ns5:SPD" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.IndexTypeID" value="ft_SPD_1.0" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFT" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.XsltsIDs" value="[ $BrokerXSLT_PROVENANCE_anylanguage_to_ftRowset_anylanguage, $BrokerXSLT_DwC_anylanguage_to_ftRowset_anylanguage, $BrokerXSLT_Properties_anylanguage_to_ftRowset_anylanguage ]" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index node to append" value="%userInput" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="FTIndexNodeCollection" name="FullText Index OAI Tree Collections">
| + | |
− | <initialization>
| + | |
− | <assign to="%FTIndexNodeCollection.input.Type" value="ns5:OAI" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.IndexTypeID" value="ft_oai_dc_1.0" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFT" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.XsltsIDs" value="[ $BrokerXSLT_FARM_dc_anylanguage_to_ftRowset_anylanguage ]" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index node to append" value="%userInput" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="FTIndexNodeCollection" name="FullText Index FIGIS Tree Collections">
| + | |
− | <initialization>
| + | |
− | <assign to="%FTIndexNodeCollection.input.Type" value="ns5:FIGIS" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.IndexTypeID" value="ft_FIGIS_1.0" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFT" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.XsltsIDs" value="[ $BrokerXSLT_FIGIS_anylanguage_to_ftRowset_anylanguage ]" />
| + | |
− | <assign to="%Create_ft_node_index.FullTextIndexNodeGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index node to append" value="%userInput" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="FWDIndexNodeCollection" name="Forward Index SPD Tree Collections">
| + | |
− | <initialization>
| + | |
− | <assign to="%FWDIndexNodeCollection.input.Type" value="ns5:SPD" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFWD" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IndexedKeyNames" value="[ ObjectID, gDocCollectionID, gDocCollectionLang, scientificName, scientificNameAuthorship, genus, phylum, kingdom, order, family, specificEpithet ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IndexedKeyTypes" value="[ fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.XsltsIDs" value="[ $BrokerXSLT_DwC_anylanguage_to_fwRowset_anylanguage ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index node to append" value="%userInput" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="FWDIndexNodeCollection" name="Forward Index FIGIS Tree Collections">
| + | |
− | <initialization>
| + | |
− | <assign to="%FWDIndexNodeCollection.input.Type" value="ns5:FIGIS" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFWD" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IndexedKeyNames" value="[ ObjectID, gDocCollectionID, gDocCollectionLang, scientific_name, family, personal_author ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IndexedKeyTypes" value="[ fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.XsltsIDs" value="[ $BrokerXSLT_FIGIS_anylanguage_to_fwRowset_anylanguage ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index node to append" value="%userInput" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="FWDIndexNodeCollection" name="Forward Index OAI Tree Collections">
| + | |
− | <initialization>
| + | |
− | <assign to="%FWDIndexNodeCollection.input.Type" value="ns5:OAI" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFWD" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IndexedKeyNames" value="[ ObjectID, gDocCollectionID, gDocCollectionLang, title, creator, subject, coverage ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IndexedKeyTypes" value="[ fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.XsltsIDs" value="[ $BrokerXSLT_dc_anylanguage_to_fwRowset_anylanguage_title_creator_subject_coverage ]" />
| + | |
− | <assign to="%Create_fwd_node_index.ForwardIndexNodeGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index node to append" value="%userInput" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="CreateOpenSearchCollectionResource" name="CreateOSResourceForDRIVERCollection">
| + | |
− | <initialization>
| + | |
− | <assign to="%CreateOpenSearchCollectionResource.input.ColName" value="DRIVER" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.FieldParameters" value="[ en:s:allIndexes, en:p:title, en:p:creator, en:p:pubDate ]" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.OpenSearchResourceID" value="30602b90-603f-11e0-90e5-a7c4e0a7bbf8" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="CreateOpenSearchCollectionResource" name="CreateOSResourceForEcoscopeCollection">
| + | |
− | <initialization>
| + | |
− | <assign to="%CreateOpenSearchCollectionResource.input.ColName" value="Ecoscope" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.FieldParameters" value="[ en:s:allIndexes, en:p:title, en:p:link, en:p:description, en:p:S ]" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.OpenSearchResourceID" value="ae0210b0-dcd2-11e2-be24-9415b2540510" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="CreateOpenSearchCollectionResource" name="CreateOSResourceForBING">
| + | |
− | <initialization>
| + | |
− | <assign to="%CreateOpenSearchCollectionResource.input.ColName" value="Bing" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.FieldParameters" value="[ en:s:allIndexes, en:p:title, en:p:link, en:p:description, en:p:S, en:p:pubDate ]" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.FixedParameters" value="[ http%3A%2F%2Fa9.com%2F-%2Fspec%2Fopensearch%2F1.1%2F:count="25", config:numOfResults="200" ]" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.OpenSearchResourceID" value="bcd216d0-dce6-11e2-89e1-9415b2540510" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job jobtype="CreateOpenSearchCollectionResource" name="CreateOSResourceForINSPIRECollection">
| + | |
− | <initialization>
| + | |
− | <assign to="%CreateOpenSearchCollectionResource.input.ColName" value="INSPIRE" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.FieldParameters" value="[ en:s:allIndexes, en:p:title, en:p:creator, en:p:pubDate ]" />
| + | |
− | <assign to="%Create_OSR.OpenSearchGenerationTask.OpenSearchResourceID" value="30602b90-603f-11e0-90e5-a7c4e0a7bbf8" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | <job extends="FullText Index OAI Tree Collections" jobtype="FTIndexNodeCollection" name="Indexing">
| + | |
− | <initialization>
| + | |
− | <assign to="%FTIndexNodeCollection.input.ColID" value="FIGIS" />
| + | |
− | </initialization>
| + | |
− | </job>
| + | |
− | </jobs>
| + | |
− | </BootstrapInfo>
| + | |
− | </source>
| + | |
| | | |
| === Metadata Broker XSLT === | | === Metadata Broker XSLT === |
| | | |
− | *BrokerXSLT_wrapperFWD
| |
− | *BrokerXSLT_wrapperFT
| |
− | *BrokerXSLT_oai_dc_anylanguage_to_fwRowset_anylanguage_title_creator
| |
| *BrokerXSLT_oai_dc_anylanguage_to_ftRowset_anylanguage | | *BrokerXSLT_oai_dc_anylanguage_to_ftRowset_anylanguage |
| + | The following XSLT transforms data elements with oai-dc schema to fulltext rowsets: |
| | | |
− | ==Creating a Full Text and a Forward Index for a collection==
| + | [[File:BrokerXSLT_oai_dc_anylanguage_to_ftRowset_anylanguage.xml]] |
− | | + | |
− | In this example we will create a Full Text and a Forward Index for a collection that has DC schema. First of all let's create a generic resource for the Full Text Rowset XSLT for the collection data in ES schema:
| + | |
− | | + | |
− | Name := BrokerXSLT_dc_anylanguage_to_ftRowset_anylanguage
| + | |
− | | + | |
− | SecondaryType := MetadataBrokerXSLT
| + | |
− | | + | |
− | Description := XSLT to transform from DC schema to FTS rowset schema (suitable for feeding the full text index).
| + | |
− | | + | |
− | Body :=
| + | |
− | <source lang="xml">
| + | |
− | <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
| + | |
− | <xsl:output indent="yes" method="xml" omit-xml-declaration="yes" />
| + | |
− | <xsl:template match="/">
| + | |
− | <ROWSET>
| + | |
− | <ROW>
| + | |
− | <FIELD name="description.abstract">
| + | |
− | <xsl:value-of select="//*[local-name()='description.abstract']" />
| + | |
− | </FIELD>
| + | |
− | <xsl:for-each select="//*[local-name()='subject']">
| + | |
− | <xsl:if test="normalize-space(.)">
| + | |
− | <FIELD name="subject">
| + | |
− | <xsl:value-of select="normalize-space(.)" />
| + | |
− | </FIELD>
| + | |
− | </xsl:if>
| + | |
− | </xsl:for-each>
| + | |
− | <FIELD name="type">
| + | |
− | <xsl:value-of select="//*[local-name()='type']" />
| + | |
− | </FIELD>
| + | |
− | <FIELD name="identifier">
| + | |
− | <xsl:value-of select="//*[local-name()='identifier']" />
| + | |
− | </FIELD>
| + | |
− | <FIELD name="language">
| + | |
− | <xsl:value-of select="//*[local-name()='language']" />
| + | |
− | </FIELD>
| + | |
− | <FIELD name="publisher">
| + | |
− | <xsl:value-of select="//*[local-name()='publisher']" />
| + | |
− | </FIELD>
| + | |
− | <FIELD name="contributor">
| + | |
− | <xsl:value-of select="//*[local-name()='contributor']" />
| + | |
− | </FIELD>
| + | |
− | <FIELD name="source">
| + | |
− | <xsl:value-of select="//*[local-name()='source']" />
| + | |
− | </FIELD>
| + | |
− | </ROW>
| + | |
− | </ROWSET>
| + | |
− | </xsl:template>
| + | |
− | </xsl:stylesheet>
| + | |
− | </source>
| + | |
− | | + | |
− | Note that this XSLT extracts a number of fields from the collection payload.
| + | |
− | | + | |
− | Additionally we must create the corresponding Full Text Index Type for the fields extracted from the ES and DC payload:
| + | |
− | | + | |
− | | + | |
− | Name := IndexType_ft_es_dc_2.0
| + | |
− | | + | |
− | SecondaryType := FullTextIndexType
| + | |
− | | + | |
− | Description := Definition of the fulltext index type for the 'es' + 'dc' schema
| + | |
− | | + | |
− | Body :=
| + | |
− | | + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <index-type name="default">
| + | |
− |
| + | |
− | <field-list sort-xnear-stop-word-threshold="2E8">
| + | |
− |
| + | |
− | <field name="title">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="description">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="purpose">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="creator">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="keyword">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="date">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="suppInfo">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="description.abstract">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="subject">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="type">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="identifier">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="language">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="publisher">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="contributor">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="source">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="gDocCollectionID">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="gDocCollectionLang">
| + | |
− |
| + | |
− | <index>yes</index>
| + | |
− |
| + | |
− | <store>yes</store>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | <tokenize>yes</tokenize>
| + | |
− |
| + | |
− | <sort>no</sort>
| + | |
− |
| + | |
− | <boost>1.0</boost>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | </field-list>
| + | |
− |
| + | |
− | </index-type>
| + | |
− |
| + | |
− | </source>
| + | |
− | | + | |
− | Note, in the Index Type above, that we also need to define the default "gDocCollectionID" and "gDocCollectionLang" fields, although there are not referenced in the XSLTs(the are automatically added by the [[ Data_Transformation | DTS ]] ).
| + | |
− | Then we will create the generic resources for the Forward Index. We will present a simple case here and we will use only the ES payload to create the Forward Rowsets. The following Rowset XSLT extracts the "title" and "creator" fields from the ES payload:
| + | |
− | | + | |
− | | + | |
− | Name := BrokerXSLT_es_anylanguage_to_fwRowset_anylanguage_title_creator
| + | |
− | | + | |
− | SecondaryType := MetadataBrokerXSLT
| + | |
− | | + | |
− | Description := XSLT to transform from es schema to forward rowset schema including the 'title' and 'creator' fields (suitable for feeding the forward index).
| + | |
− | | + | |
− | Body :=
| + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
| + | |
− |
| + | |
− | <xsl:output xmlns:xsl="http://www.w3.org/1999/XSL/Transform" indent="yes" method="xml" omit-xml-declaration="yes" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="keys">
| + | |
− |
| + | |
− | <key xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <keyName xmlns:xsl="http://www.w3.org/1999/XSL/Transform">title</keyName>
| + | |
− |
| + | |
− | <keyXPath xmlns:xsl="http://www.w3.org/1999/XSL/Transform">//title</keyXPath>
| + | |
− |
| + | |
− | </key>
| + | |
− |
| + | |
− | <key xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <keyName xmlns:xsl="http://www.w3.org/1999/XSL/Transform">creator</keyName>
| + | |
− |
| + | |
− | <keyXPath xmlns:xsl="http://www.w3.org/1999/XSL/Transform">//creator</keyXPath>
| + | |
− |
| + | |
− | </key>
| + | |
− |
| + | |
− | </xsl:variable>
| + | |
− |
| + | |
− | <xsl:template xmlns:xsl="http://www.w3.org/1999/XSL/Transform" match="/">
| + | |
− |
| + | |
− | <xsl:element xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="ROWSET">
| + | |
− |
| + | |
− | <xsl:element xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="INSERT">
| + | |
− |
| + | |
− | <xsl:element xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="TUPLE">
| + | |
− |
| + | |
− | <xsl:element xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="VALUE">
| + | |
− |
| + | |
− | <xsl:for-each xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="//title">
| + | |
− |
| + | |
− | <xsl:if xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="normalize-space(.)">
| + | |
− |
| + | |
− | <FIELD xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="title">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="normalize-space(.)" />
| + | |
− |
| + | |
− | </FIELD>
| + | |
− |
| + | |
− | </xsl:if>
| + | |
− |
| + | |
− | </xsl:for-each>
| + | |
− |
| + | |
− | <xsl:for-each xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="//creator">
| + | |
− |
| + | |
− | <xsl:if xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="normalize-space(.)">
| + | |
− |
| + | |
− | <FIELD xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="creator">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="normalize-space(.)" />
| + | |
− |
| + | |
− | </FIELD>
| + | |
− |
| + | |
− | </xsl:if>
| + | |
− |
| + | |
− | </xsl:for-each>
| + | |
− |
| + | |
− | </xsl:element>
| + | |
− |
| + | |
− | </xsl:element>
| + | |
− |
| + | |
− | </xsl:element>
| + | |
− |
| + | |
− | </xsl:element>
| + | |
− |
| + | |
− | </xsl:template>
| + | |
− |
| + | |
− | </xsl:stylesheet>
| + | |
− |
| + | |
− | </source>
| + | |
− | | + | |
− | In this example "title" and "creator" will have string values, so we just need to create a generic resource for string-string key-value pairs:
| + | |
− | | + | |
− | Name := IndexType_fwd_string_string
| + | |
− | | + | |
− | SecondaryType := ForwardIndexType
| + | |
− | | + | |
− | Description := Definition of the index type 'string_string' for the forward index
| + | |
− | | + | |
− | Body :=
| + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <field-list>
| + | |
− |
| + | |
− | <field name="key">
| + | |
− |
| + | |
− | <type>string</type>
| + | |
− |
| + | |
− | <sort>ascending</sort>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="value">
| + | |
− |
| + | |
− | <type>string</type>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | </field-list>
| + | |
− |
| + | |
− | </source>
| + | |
− | | + | |
− | In case we had a field that would have a single date value in the format 'yyyy-MM-dd' for each document, we would also have to create a generic resource for a string-date key-value pair like the following one:
| + | |
− | | + | |
− | Name := IndexType_fwd_esDate_string
| + | |
− | | + | |
− | SecondaryType := ForwardIndexType
| + | |
− | | + | |
− | Description := Definition of the index type 'date_string' for the forward index (suitable for es schema indexing)
| + | |
− | | + | |
− | Body :=
| + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <field-list>
| + | |
− |
| + | |
− | <field name="key">
| + | |
− |
| + | |
− | <type>date</type>
| + | |
− |
| + | |
− | <format>yyyy-MM-dd</format>
| + | |
− |
| + | |
− | <sort>ascending</sort>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="value">
| + | |
− |
| + | |
− | <type>string</type>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | </field-list>
| + | |
− |
| + | |
− | </source>
| + | |
− | | + | |
− | Finally we would have to append to the IRBootStrapper configuration generic resource the following job that would create the Full Text and Forward Index:
| + | |
− | | + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <job jobtype="IndexCollection" name="IndexEsDcCollection">
| + | |
− |
| + | |
− | <initialization>
| + | |
− |
| + | |
− | <assign to="%IndexCollection.input.ColName" value="European Environment Agency: Topic Reports" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.IndexTypeID" value="ft_es_dc_2.0" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFT" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.Views" value="[ 5ec68209-41f1-4c1c-9359-ff055a38d981, 55404922-12ee-4808-9cc7-ac1ab38c4a2e ]" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.ViewsXSLTSIDs" value="[ $BrokerXSLT_es_anylanguage_to_ftRowset_anylanguage, $BrokerXSLT_dc_anylanguage_to_ftRowset_anylanguage ]" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of FT index management resource to append" value="%userInput" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFWD" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.IndexedKeyNames" value="[ ObjectID, gDocCollectionID, gDocCollectionLang, title, creator ]" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.IndexedKeyTypes" value="[ fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string, fwd_string_string ]" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.Views" value="[ 5ec68209-41f1-4c1c-9359-ff055a38d981 ]" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.ViewsXSLTSIDs" value="[ $BrokerXSLT_es_anylanguage_to_fwRowset_anylanguage_title_creator ]" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of FWD index management resource to append" value="%userInput" />
| + | |
− |
| + | |
− | </initialization>
| + | |
− |
| + | |
− | </job>
| + | |
− | | + | |
− | | + | |
− | </source>
| + | |
− | | + | |
− | where the jobtype is defined in the corresponding section of the IRBootstrapper configuration:
| + | |
− | | + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <jobtype description="Creates the required FT-FWD indices for a collection." name="IndexCollection">
| + | |
− |
| + | |
− | <input type="GCUBECollection" />
| + | |
− |
| + | |
− | <jobDefinition>
| + | |
− |
| + | |
− | <sequential>
| + | |
− |
| + | |
− | <sequential>
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.input" value="%IndexCollection.input" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.output.IndexedCollectionID" value="%Create_MC_ft_index.input.ColID" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.NumberOfLookups" value="2" />
| + | |
− |
| + | |
− | <task name="Create_MC_ft_index" tasktype="FullTextIndexGenerationTask" />
| + | |
− |
| + | |
− | </sequential>
| + | |
− |
| + | |
− | <sequential>
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.input" value="%IndexCollection.input" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.output.IndexedCollectionID" value="%Create_MC_fwd_index.input.ColID" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_fwd_index.ForwardIndexGenerationTask.NumberOfLookups" value="2" />
| + | |
− |
| + | |
− | <task name="Create_MC_fwd_index" tasktype="ForwardIndexGenerationTask" />
| + | |
− |
| + | |
− | </sequential>
| + | |
− |
| + | |
− | </sequential>
| + | |
− |
| + | |
− | </jobDefinition>
| + | |
− |
| + | |
− | </jobtype>
| + | |
− | | + | |
− | </source>
| + | |
− | | + | |
− | Note in the IRBootstrapper job, that for the Forward Index we additionally define the default fields ObjectID, gDocCollectionID and gDocCollectionLang which are all of string-string type.
| + | |
− | | + | |
− | ==Creating a Geo-Spatial Index for a collection with ES payload==
| + | |
− | | + | |
− | Just as before we need to create the generic resource for the Geo Rowset XSLT:
| + | |
− | | + | |
− | Name := BrokerXSLT_es_anylanguage_to_geoRowset_anylanguage
| + | |
− | | + | |
− | SecondaryType := MetadataBrokerXSLT
| + | |
− | | + | |
− | Description := XSLT to transform from ES schema to geo index rowset schema
| + | |
− | | + | |
− | Body :=
| + | |
− | <source lang="xml">
| + | |
− |
| + | |
− | <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
| + | |
− |
| + | |
− | <xsl:output xmlns:xsl="http://www.w3.org/1999/XSL/Transform" indent="yes" method="xml" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="zeroDate" select="'00000000T000000Z'" />
| + | |
− |
| + | |
− | <xsl:template xmlns:xsl="http://www.w3.org/1999/XSL/Transform" match="/">
| + | |
− |
| + | |
− | <ROWSET xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:apply-templates xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="//esObject" />
| + | |
− |
| + | |
− | </ROWSET>
| + | |
− |
| + | |
− | </xsl:template>
| + | |
− |
| + | |
− | <xsl:template xmlns:xsl="http://www.w3.org/1999/XSL/Transform" match="//esObject">
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="flag">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="location/westBL[.!='']">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="'true'" />
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="'false'" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:variable>
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="id" select="generate-id()" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="dateTimeStringBegin" select="timeFrame/begin" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="dateTimeStringEnd" select="timeFrame/end" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="beginDateEmpty" select="($dateTimeStringBegin=$zeroDate) or ($dateTimeStringBegin='unavailable') or (not(normalize-space($dateTimeStringBegin)))" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="endDateEmpty" select="($dateTimeStringEnd=$zeroDate) or ($dateTimeStringEnd='unavailable') or (not(normalize-space($dateTimeStringEnd)))" />
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="$flag='true'">
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="startField">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="$beginDateEmpty='true'">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="$endDateEmpty='true'">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="$zeroDate" />
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="concat(substring($dateTimeStringEnd,1,4), '-', substring($dateTimeStringEnd,5,2), '-', substring($dateTimeStringEnd,7,2))" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="concat(substring($dateTimeStringBegin,1,4), '-', substring($dateTimeStringBegin,5,2), '-', substring($dateTimeStringBegin,7,2))" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:variable>
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="endField">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="$endDateEmpty='true'">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="$beginDateEmpty='true'">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="$zeroDate" />
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="concat(substring($dateTimeStringBegin,1,4), '-', substring($dateTimeStringBegin,5,2), '-', substring($dateTimeStringBegin,7,2))" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="concat(substring($dateTimeStringEnd,1,4), '-', substring($dateTimeStringEnd,5,2), '-', substring($dateTimeStringEnd,7,2))" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:variable>
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="location/southBL[.!='']">
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="x1" select="location/westBL" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="y1" select="location/southBL" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="x2" select="location/eastBL" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="y2" select="location/northBL" />
| + | |
− |
| + | |
− | <ROW xmlns:xsl="http://www.w3.org/1999/XSL/Transform" id="{$id}" x1="{$x1}" x2="{$x2}" y1="{$y1}" y2="{$y2}">
| + | |
− |
| + | |
− | <FIELD xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="StartTime">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="$startField" />
| + | |
− |
| + | |
− | </FIELD>
| + | |
− |
| + | |
− | <FIELD xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="EndTime">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="$endField" />
| + | |
− |
| + | |
− | </FIELD>
| + | |
− |
| + | |
− | </ROW>
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <ROW xmlns:xsl="http://www.w3.org/1999/XSL/Transform" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="location/centerLat[.!=''] and location/centerLon[.!='']">
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="cLat" select="location/centerLat" />
| + | |
− |
| + | |
− | <xsl:variable xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="cLon" select="location/centerLon" />
| + | |
− |
| + | |
− | <ROW xmlns:xsl="http://www.w3.org/1999/XSL/Transform" id="{$id}" x1="{$cLon}" y1="{$cLat}">
| + | |
− |
| + | |
− | <FIELD xmlns:xsl="http://www.w3.org/1999/XSL/Transform" name="TimeStamp">
| + | |
− |
| + | |
− | <xsl:choose xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:when xmlns:xsl="http://www.w3.org/1999/XSL/Transform" test="$endDateEmpty='true'">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="$zeroDate" />
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <xsl:value-of xmlns:xsl="http://www.w3.org/1999/XSL/Transform" select="$dateTimeStringEnd" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </FIELD>
| + | |
− |
| + | |
− | </ROW>
| + | |
− |
| + | |
− | </xsl:when>
| + | |
− |
| + | |
− | <xsl:otherwise xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
| + | |
− |
| + | |
− | <ROW xmlns:xsl="http://www.w3.org/1999/XSL/Transform" />
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:otherwise>
| + | |
− |
| + | |
− | </xsl:choose>
| + | |
− |
| + | |
− | </xsl:template>
| + | |
− |
| + | |
− | </xsl:stylesheet>
| + | |
− |
| + | |
− | </Body>
| + | |
− |
| + | |
− | </Profile>
| + | |
− |
| + | |
− | </Resource>
| + | |
− | | + | |
− | </source>
| + | |
− | | + | |
− | Then we create the generic resource for the Geo Index Type:
| + | |
− | | + | |
− | Name := IndexType_geo_span_1.0
| + | |
− | | + | |
− | SecondaryType := GeoIndexType
| + | |
− | | + | |
− | Description := Definition of the geo span index type
| + | |
− | | + | |
− | Body :=
| + | |
− | <source lang="xml">
| + | |
− |
| + | |
− | <index-profile name="default2-geo">
| + | |
− |
| + | |
− | <field-list>
| + | |
− |
| + | |
− | <field name="StartTime">
| + | |
− |
| + | |
− | <type>date</type>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="EndTime">
| + | |
− |
| + | |
− | <type>date</type>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | <field name="TimeStamp">
| + | |
− |
| + | |
− | <type>date</type>
| + | |
− |
| + | |
− | <return>yes</return>
| + | |
− |
| + | |
− | </field>
| + | |
− |
| + | |
− | </field-list>
| + | |
− |
| + | |
− | </index-profile>
| + | |
− |
| + | |
− | </source>
| + | |
− | | + | |
− | And finally we append the following job(and the jobtype in the corresponding section if it is not already there) in the IRBootstrapper configuration generic resource:
| + | |
− | | + | |
− | <source lang="xml">
| + | |
− | | + | |
− | <job jobtype="GeoIndexMetadataCollection" name="IndexGeoCollection">
| + | |
− |
| + | |
− | <initialization>
| + | |
− |
| + | |
− | <assign to="%GeoIndexMetadataCollection.input.SchemaName" value="es" />
| + | |
− |
| + | |
− | <assign to="%GeoIndexMetadataCollection.input.SchemaURI" value="http://es.xsd" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_geo_index.GeoIndexGenerationTask.IndexTypeID" value="geo_span_1.0" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_geo_index.GeoIndexGenerationTask.EsToGeoXSLTID" value="$BrokerXSLT_es_anylanguage_to_geoRowset_anylanguage" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_geo_index.GeoIndexGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index management resource to append" value="%userInput" />
| + | |
− |
| + | |
− | </initialization>
| + | |
− |
| + | |
− | </job>
| + | |
− | | + | |
− | <jobtype description="Creates a geo index for a metadata view." name="GeoIndexMetadataCollection">
| + | |
− |
| + | |
− | <input type="MetadataCollection" />
| + | |
− |
| + | |
− | <jobDefinition>
| + | |
− |
| + | |
− | <sequential>
| + | |
− |
| + | |
− | <sequential>
| + | |
− |
| + | |
− | <assign to="%Create_MC_geo_index.input" value="%GeoIndexMetadataCollection.input" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_geo_index.output.IndexedCollectionID" value="%Create_MC_geo_index.input.ColID" />
| + | |
− |
| + | |
− | <assign to="%Create_MC_geo_index.GeoIndexGenerationTask.NumberOfLookups" value="2" />
| + | |
− |
| + | |
− | <task name="Create_MC_geo_index" tasktype="GeoIndexGenerationTask" />
| + | |
− |
| + | |
− | </sequential>
| + | |
− |
| + | |
− | </sequential>
| + | |
− |
| + | |
− | </jobDefinition>
| + | |
− |
| + | |
− | </jobtype>
| + | |
− | | + | |
− | | + | |
− | | + | |
− | </source>
| + | |
In the first two steps we create generic resources for the Rowset XSLTs and Index Types through the Resource Management portlet . You can find detailed descriptions for the Rowset data (the output of the Rowset XSLT transformation) in the following section:
For the third step, a definition of an IRBootstrapper job is required. You can find the details for defining such a job in the IR Bootstrapper section. To complete the Index creation, the administrator must go to the IRBootstrapper and run the job. The two examples that follow will clarify the three steps.
The following transformation program is called for fulltext rowset creation. Transformation unit with id="6" takes multiple XSLTs and applies final XSLT at the end.
In this section we present the required IndexType for (FullText) Index.
In order to extract the fields from the OAI-DC payload and build the FullText Index the following FullTextIndexType is required:
The IRBootstrapper portlet requires a Generic Resource to be available on the IS with name: IRBootstrapperConfiguration and secondary type: IRBootstrapperConfig.
For more information please refer to this section IRBoostrapperConfiguration Generic Resource
The following XSLT transforms data elements with oai-dc schema to fulltext rowsets: