Difference between revisions of "Document Store Framework"

From Gcube Wiki
Jump to: navigation, search
(→‎Buffering & Aggregating Record)
m (Luca.frosini moved page Document Store Library to Document Store Framework)
 
(52 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
Document Store Lib Provides facilities to store document in a Document Store NoSQL DB. It provides '''Aggregation''', '''Bufferization''' and '''Fallback''' capabilities.
+
[[Category: Developer's Guide]]
 +
 
 +
Document Store Library Provides facilities to store document in a Document Store NoSQL DB. It provides '''Aggregation''', '''Bufferization''', '''Storing''' and '''Fallback & Retry''' capabilities.
  
 
== Library ==
 
== Library ==
 
The library has been tough to provide a common way to store '''Record'''s in Document Store NoSQL DB. In particular is responsible for:
 
The library has been tough to provide a common way to store '''Record'''s in Document Store NoSQL DB. In particular is responsible for:
  
−
* [[Document_Store_Lib#Buffering_.26_Aggregating_Record | Buffering]]
+
* [[Document_Store_Lib#Buffering & Aggregating Record | Buffering]]
−
* [[Document_Store_Lib#Buffering_.26_Aggregating_Record | Aggregating]]
+
* [[Document_Store_Lib#Buffering & Aggregating Record | Aggregating]]
−
* [[Document_Store_Lib#Storing | Storing]]
+
* [[Document_Store_Lib#Storing Record | Storing Record ]]
 +
* Providing [[Document_Store_Lib#Fallback & Retry | Fallback & Retry]] Capability
 +
 
 +
The library is designed to separate the above mentioned capabilities from the logic needed to really store the Records on DB.
  
 
== Buffering & Aggregating Record ==
 
== Buffering & Aggregating Record ==
  
−
The library provide buffering capabilities.
+
The library provides buffering capabilities.
−
For each new request to store a record the library try to aggregate the received record with the ones has already buffered. The aggregation made in this phase is lossless. In other words, the ''Record'' aggregated are only the ones having for all properties the same values except the ones where the aggregation has been made (e.g. operationCount).
+
For each new requests to store a '''Record''' the library tries to aggregate the received '''Record''' with the ones has already buffered. The aggregation made in this phase is lossless.
  
 
== Storing Record ==
 
== Storing Record ==
  
−
The library discover dynamically the '''Document Store Lib Persistence'''. This library is responsible to connect to DS-NoSQL DB and to really store the Record. The configuration parameters are also discovered dynamically.
+
The library discovers dynamically the [[Document Store Lib#Available Document Store Lib Persistence | Document Store Lib Persistence]] (i.e. using Java Service Loader) to use. This library is responsible to connect to DocumentStoreS-NoSQL DB and to really store the Record. The configuration parameters are also discovered dynamically.
  
−
If no library is discovered or if the storing fails (e.g. for network issue) the ''Record''s are persisted using a '''Fallback Persistence''' (on filesystem).
+
If no library is discovered the ''Record''s are persisted using '''Fallback Persistence''' (which save '''Record'''s on filesystem).
−
Please note that '''Fallback Persistence''' is also used when a failure occurs (e.g. a network failure).
+
  
−
A recurrent task periodically (actually 10 min) check if there are Records stored from '''Fallback Persistence''' (A '''Document Store Lib Persistence''' must be discovered in advance to activate the recurrent task because otherwise the task has no reason to exist).
+
== Fallback & Retry ==
 +
When a '''Record''' cannot be persisted on a Document Store DB (e.g. a network failure, no persistence found or misconfigured), the '''Record''' is saved on filesystem. The Java class responsible to do it is FallbackPersistence.
 +
 
 +
A recurrent task periodically (i.e. 10 min) check if there are '''Record'''s stored from '''Fallback Persistence'''. Please note that the recurrent task is activated only when a '''Document Store Lib Persistence''' is discovered and configured correctly.
  
 
== Base Model ==
 
== Base Model ==
  
−
To set the various properties of a records two different methods are available by setting
+
'''Record''''s properties can be set:
  
−
* Each Single Property through a generic Key-Value method
+
* Individually by using a Key-Value method
 
** '''key''' :  <syntaxhighlight lang="java" inline enclose="none">String</syntaxhighlight>
 
** '''key''' :  <syntaxhighlight lang="java" inline enclose="none">String</syntaxhighlight>
 
** '''value''' : <syntaxhighlight lang="java" inline enclose="none">? extends Serializable</syntaxhighlight>
 
** '''value''' : <syntaxhighlight lang="java" inline enclose="none">? extends Serializable</syntaxhighlight>
−
* Multiple Properties at once passing a <syntaxhighlight lang="java" inline enclose="none">Map<String,? extends Serializable></syntaxhighlight>.
+
* All togheter by passing  a multiple Properties at once passing a <syntaxhighlight lang="java" inline enclose="none">Map<String,? extends Serializable></syntaxhighlight>.
  
 
The build blocks of this library are the interfaces:
 
The build blocks of this library are the interfaces:
Line 36: Line 43:
  
 
The first interface ('''Record''') must be implemented to create a class compliant with the library model, so that the library can provide it own facilities.
 
The first interface ('''Record''') must be implemented to create a class compliant with the library model, so that the library can provide it own facilities.
−
The second interface must be implemented if the model needs the aggregation facilities.
+
The second interface must be implemented if the model want take advance of aggregation facilities.
  
−
The library provides also a basic implementation '''AbstractRecord''' that the developer can use as building block fro its own model. By extending this abstract class the library also provide properties:
+
 
 +
The library provides also a basic implementation '''AbstractRecord''' that the developer can use as building block for your own model. By extending this abstract class the library can also provide support for properties:
  
 
* Validations
 
* Validations
 
* Harmonizations
 
* Harmonizations
 +
 +
=== Internals ===
 +
 +
[[File:DocumentStoreLib-ER.png]]
 +
 +
As already mentioned '''Record''' and '''AggregatedRecord''' represent the basic Java Interface to  be implemented in your own model.  It is at your own cost to be compliant with interface specification so that it can behave as expected.
 +
 +
The library already provide an abstract class '''AbstractRecord''' to be extended which made the model definition easier.
 +
 +
The building block to use this facilities are Java Annotations.
 +
The library provides two main categories of annotations:
 +
* ''Core Annotations''
 +
* ''Validation Annotations''
 +
 +
 +
''Core Annotations'' are used to determine the the 'importance' of the field and auto-calculate the result of some methods defined in '''Record''' and '''AggregatedRecord''' interface:
 +
* '''RequiredField''' : Assert that the filed is Required.
 +
* '''AggregatedField''' : Assert that the field will be the target of lossless aggregation.
 +
* '''ComputedField''' : Assert that the field is computed through and calculation on others required field.
 +
 +
''Validation Annotations'' are used for validation and conversion purpose. Each annotation is associated to a validation and conversion function (if any).
 +
* '''NotEmpty''' : The Value cannot be empty but can be null. Empty for a collections mean having no elements, for a string having an empty string, and no entry for a map.
 +
* '''NotEmptyIfNotNull''' : The Value cannot be empty if not null.
 +
* '''NotNull''' : The Value cannot be null.
 +
* '''ValidBoolean''' : The value must be a boolean. The associated validation function automatically convert from valid string representing the boolean value using [http://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#valueOf%28java.lang.String Boolean.valueof(String) function](e.g. 'false' or 'FALSE' string) .
 +
* '''Validinteger''' : The value must be an integer. The associated validation function automatically convert from valid string representing the int value using [http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html#valueOf%28java.lang.String Integer.valueof(String) function](e.g. '1234' string).
 +
* '''ValidLong''' : he value must be an long. The associated validation function automatically convert from valid string representing the long value using [http://docs.oracle.com/javase/7/docs/api/java/lang/Long.html#valueOf%28java.lang.String Long.valueof(String) function] (e.g. '1276347834635' string).
 +
 +
The library provides a mechanism to easily extends ''Validation Annotations''. This has been widely used in [[Accounting Lib]] model implementations (see also [[Accounting Model]]).
 +
 +
Developers will find detailed information on [[Build on top of Document Store Lib]] guide.
  
 
== Development ==
 
== Development ==
Line 47: Line 86:
 
The library is identified by the following maven coordinates:
 
The library is identified by the following maven coordinates:
  
−
''<groupId>'''''org.gcube.data-publishing'''''</groupId>''
+
''<groupId>'''''org.gcube.data.publishing'''''</groupId>''
  
 
''<artifactId>'''''document-store-lib'''''</artifactId>''
 
''<artifactId>'''''document-store-lib'''''</artifactId>''
  
−
Please set the scope of the library to PROVIDED, if your component runs on a container gHN or vHN.
+
Please set the scope of the library to '''PROVIDED''', if your component runs on a container gHN or vHN.
  
−
==  Document Store Lib Persistence ==
+
If you want to create your own Model and providing a custom connection configuration take a look to [[Accounting Lib]] and [[Accounting Model]] guides.
 +
 
 +
==  Available Document Store Lib Persistence ==
  
 
The current implementations of persistence for this library are:
 
The current implementations of persistence for this library are:
  
−
* couchdb
+
* CouchBase
  
−
''<groupId>'''''org.gcube.data-publishing'''''</groupId>''
+
''<groupId>'''''org.gcube.data.publishing'''''</groupId>''
  
−
''<artifactId>'''''document-store-lib-couchdb'''''</artifactId>''
+
''<artifactId>'''''document-store-lib-couchbase'''''</artifactId>''
  
−
* couchbase
 
  
−
''<groupId>'''''org.gcube.data-publishing'''''</groupId>''
+
* Accounting Service
  
−
''<artifactId>'''''document-store-lib-couchbase'''''</artifactId>''
+
''<groupId>'''''org.gcube.data.publishing'''''</groupId>''
 +
 
 +
''<artifactId>'''''document-store-lib-accounting-service'''''</artifactId>''
 +
 
 +
 
 +
* CouchDB
 +
 
 +
''<groupId>'''''org.gcube.data.publishing'''''</groupId>''
 +
 
 +
''<artifactId>'''''document-store-lib-couchdb'''''</artifactId>''
  
−
* mongodb
+
* MongoDB
  
−
''<groupId>'''''org.gcube.data-publishing'''''</groupId>''
+
''<groupId>'''''org.gcube.data.publishing'''''</groupId>''
  
 
''<artifactId>'''''document-store-lib-mongodb'''''</artifactId>''
 
''<artifactId>'''''document-store-lib-mongodb'''''</artifactId>''
  
 +
* InfluxDB (under investigation)
  
 
The developer MUST not depend from this libraries.
 
The developer MUST not depend from this libraries.

Latest revision as of 09:33, 15 September 2020


Document Store Library Provides facilities to store document in a Document Store NoSQL DB. It provides Aggregation, Bufferization, Storing and Fallback & Retry capabilities.

Library

The library has been tough to provide a common way to store Records in Document Store NoSQL DB. In particular is responsible for:

The library is designed to separate the above mentioned capabilities from the logic needed to really store the Records on DB.

Buffering & Aggregating Record

The library provides buffering capabilities. For each new requests to store a Record the library tries to aggregate the received Record with the ones has already buffered. The aggregation made in this phase is lossless.

Storing Record

The library discovers dynamically the Document Store Lib Persistence (i.e. using Java Service Loader) to use. This library is responsible to connect to DocumentStoreS-NoSQL DB and to really store the Record. The configuration parameters are also discovered dynamically.

If no library is discovered the Records are persisted using Fallback Persistence (which save Records on filesystem).

Fallback & Retry

When a Record cannot be persisted on a Document Store DB (e.g. a network failure, no persistence found or misconfigured), the Record is saved on filesystem. The Java class responsible to do it is FallbackPersistence.

A recurrent task periodically (i.e. 10 min) check if there are Records stored from Fallback Persistence. Please note that the recurrent task is activated only when a Document Store Lib Persistence is discovered and configured correctly.

Base Model

Record's properties can be set:

  • Individually by using a Key-Value method
    • key : String
    • value : ? extends Serializable
  • All togheter by passing a multiple Properties at once passing a Map<String,? extends Serializable>.

The build blocks of this library are the interfaces:

  • Record
  • AggregatedRecord

The first interface (Record) must be implemented to create a class compliant with the library model, so that the library can provide it own facilities. The second interface must be implemented if the model want take advance of aggregation facilities.


The library provides also a basic implementation AbstractRecord that the developer can use as building block for your own model. By extending this abstract class the library can also provide support for properties:

  • Validations
  • Harmonizations

Internals

DocumentStoreLib-ER.png

As already mentioned Record and AggregatedRecord represent the basic Java Interface to be implemented in your own model. It is at your own cost to be compliant with interface specification so that it can behave as expected.

The library already provide an abstract class AbstractRecord to be extended which made the model definition easier.

The building block to use this facilities are Java Annotations. The library provides two main categories of annotations:

  • Core Annotations
  • Validation Annotations


Core Annotations are used to determine the the 'importance' of the field and auto-calculate the result of some methods defined in Record and AggregatedRecord interface:

  • RequiredField : Assert that the filed is Required.
  • AggregatedField : Assert that the field will be the target of lossless aggregation.
  • ComputedField : Assert that the field is computed through and calculation on others required field.

Validation Annotations are used for validation and conversion purpose. Each annotation is associated to a validation and conversion function (if any).

  • NotEmpty : The Value cannot be empty but can be null. Empty for a collections mean having no elements, for a string having an empty string, and no entry for a map.
  • NotEmptyIfNotNull : The Value cannot be empty if not null.
  • NotNull : The Value cannot be null.
  • ValidBoolean : The value must be a boolean. The associated validation function automatically convert from valid string representing the boolean value using Boolean.valueof(String) function(e.g. 'false' or 'FALSE' string) .
  • Validinteger : The value must be an integer. The associated validation function automatically convert from valid string representing the int value using Integer.valueof(String) function(e.g. '1234' string).
  • ValidLong : he value must be an long. The associated validation function automatically convert from valid string representing the long value using Long.valueof(String) function (e.g. '1276347834635' string).

The library provides a mechanism to easily extends Validation Annotations. This has been widely used in Accounting Lib model implementations (see also Accounting Model).

Developers will find detailed information on Build on top of Document Store Lib guide.

Development

The library is identified by the following maven coordinates:

<groupId>org.gcube.data.publishing</groupId>

<artifactId>document-store-lib</artifactId>

Please set the scope of the library to PROVIDED, if your component runs on a container gHN or vHN.

If you want to create your own Model and providing a custom connection configuration take a look to Accounting Lib and Accounting Model guides.

Available Document Store Lib Persistence

The current implementations of persistence for this library are:

  • CouchBase

<groupId>org.gcube.data.publishing</groupId>

<artifactId>document-store-lib-couchbase</artifactId>


  • Accounting Service

<groupId>org.gcube.data.publishing</groupId>

<artifactId>document-store-lib-accounting-service</artifactId>


  • CouchDB

<groupId>org.gcube.data.publishing</groupId>

<artifactId>document-store-lib-couchdb</artifactId>

  • MongoDB

<groupId>org.gcube.data.publishing</groupId>

<artifactId>document-store-lib-mongodb</artifactId>

  • InfluxDB (under investigation)

The developer MUST not depend from this libraries.

Deployment

Document Store Lib and the Document Store Lib CouchDB are already available on gCore and smartgears bundles (gHN and wHN).


Document Store Lib in the Accounting Architecture

The following image evidences the use of Document Store Lib in Accounting Architecture:

Accounting-architecture-document-store-lib.png

Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Document_Store_Framework&oldid=33523"