Difference between revisions of "Accounting Aggregator"

From Gcube Wiki
Jump to: navigation, search
(→‎Accounting Aggregator Execution)
(→‎How to manage calledMethod lead to ServiceUsageRecord cannot be aggregated)
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
{| align="right"
 +
||__TOC__
 +
|}
 +
 
This component is responsible of aggregating the collected '''Accounting''' data.
 
This component is responsible of aggregating the collected '''Accounting''' data.
−
'''PLEASE NOTE THAT THIS COMPONENT IS NOT READY YET. IT IS STILL UNDER DEVELOPEMENT.'''
 
  
 
== Accounting Aggregator in the Accounting Architecture ==
 
== Accounting Aggregator in the Accounting Architecture ==
Line 10: Line 13:
 
== Accounting Aggregator Execution ==
 
== Accounting Aggregator Execution ==
  
−
This component is a [[Smart Executor | SmartExecutor]] plugin and is launched as '''Global Scheduled Task''' (see [[Smart Executor | SmartExecutor]] for further information)
+
This component is a [[SmartExecutor | Smart Executor]] plugin and is launched as '''Global Scheduled Task''' (see [[SmartExecutor | Smart Executor]] for further information)
 +
 
 +
It aggregates (lossless) accounted Usage Record. The scheduling plan is to run on different smart-executor equipped with this plugin with the following parameters:
 +
 +
* Hourly => accounting records per hour are available for the past week;
 +
* Daily  => accounting records per day are available for the past 3 months;
 +
* Monthly => accounting records per month are available for the past 3 years;
 +
* Yearly => accounting records per year are available for the past 10 years;
 +
 
 +
 
 +
== How to manage calledMethod lead to ServiceUsageRecord cannot be aggregated  ==
 +
 
 +
RESTFull (and REST-like) services contain the ID of the managed resource inside the path of the URL. Smartgears auto calculate '''calledMethod''' by using the relative path of the URL.
 +
This lead to have ServiceUsageRecord cannot be aggregated.
 +
 
 +
To solve such an issue some actions are need from developers:
 +
 
 +
* Rewrite '''calledMethod''' in the service code
 +
* Provide regular expression to match old accounted records to allow to rewrite them from accoutnign-aggrator
 +
 
 +
=== Rewrite calledMethod ===
 +
 
 +
The developer have to insert as first line of the entry method the following line of code to instruct Smartgears to set the desired '''calledMethod'''
 +
 
 +
<syntaxhighlight lang="java">
 +
CalledMethodProvider.instance.set("rewrittenCalledMethod");
 +
</syntaxhighlight>
 +
 
 +
Resource Registry is an example of how to properly use such a feature for RESTFull services.
 +
 
 +
For example Resource Registry use the following URLs to read/update/delete a Facet having UUID 4023d5b2-8601-47a5-83ef-49ffcbfc7d86
 +
 
 +
<pre>
 +
GET /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86
 +
</pre>
 +
 
 +
<pre>
 +
PUT /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86
 +
</pre>
 +
 
 +
<pre>
 +
DELETE /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86
 +
</pre>
 +
 
 +
If the service does not rewrite the '''calledMethod''' we have two effect:
 +
 
 +
* the three operation cannot be distinguished from accounting and lead to aggregate the records operating to the same facet.
 +
* the uuid does not allow to aggregate the operation across facets.
 +
 +
 
 +
Resource Registry instruct smartgears to rewrite the '''calledMethod''' as following:
 +
 
 +
<syntaxhighlight lang="java">
 +
CalledMethodProvider.instance.set("GET /resource-registry/er/facet/ID");
 +
</syntaxhighlight>
 +
 
 +
<syntaxhighlight lang="java">
 +
CalledMethodProvider.instance.set("PUT /resource-registry/er/facet/ID");
 +
</syntaxhighlight>
 +
 
 +
<syntaxhighlight lang="java">
 +
CalledMethodProvider.instance.set("DELETE /resource-registry/er/facet/ID");
 +
</syntaxhighlight>
 +
 
 +
 
 +
I  such a way the different methods are distinguishable and the ServiceUsageRecords can be aggregated across facets.
  
−
It aggregates (loseless) accounted Usage Record. The scheduling plan is to run on different smart-executor equipped with this plugin with the following parameters:
+
=== Provide regular expression ===
  
−
* Daily (every day at 02:00 AM UTC the Records accounted the day before the previous day)
+
Accounting provide a way to rewrite the the old '''calledMethod''' to be compliant with the one declared in the service by using ''CalledMethodProvider''.
−
* Weekly (every Wednesday at 2.00 AM UTC the Records accounted the previous week (monday->sunday) )
+
−
* Monthly (every 1st of the month at 2.00 AM UTC the Records accounted the month before the previous)
+
−
* Yearly (every 1st of the month at 2.00 AM UTC the Records accounted 2 months before)
+
  
−
== Accounting Aggregator Maven Coordinates ==
+
The developer has to provide the following a json declaring the service class and name to match to check if the provided regular expression match the old '''calledMethod''' so that is rewritten with the provided replacement.
  
−
* Accounting Aggregator (Smart Executor Plugin)
+
For example to properly manage the following '''ServiceUsageRecord'''
  
−
''<groupId>'''''org.gcube.accounting'''''</groupId>''
+
<pre>
 +
{
 +
"operationCount": 1,
 +
"creationTime": 1507334430724,
 +
"serviceClass": "Index",
 +
"callerHost": "88.197.53.47",
 +
"callerQualifier": "UNKNOWN",
 +
"recordType": "ServiceUsageRecord",
 +
"consumerId": "UNKNOWN",
 +
"aggregated": true,
 +
"serviceName": "FullTextIndexNode",
 +
"duration": 3,
 +
"maxInvocationTime": 58,
 +
"scope": "/d4science.research-infrastructures.eu/gCubeApps/iSearch",
 +
"host": "dewn10.madgik.di.uoa.gr:8080",
 +
"startTime": 1507334430481,
 +
"id": "12a631da-1541-4771-8961-8d7e4a694bc0",
 +
"calledMethod": "/delete/3121eccdcbf9c6e982dcc5a2164b8b4029f51d8b1c448ddaf46316178d755c6f/oai:oai.datacite.org:352019",
 +
"endTime": 1507334430481,
 +
"minInvocationTime": 1,
 +
"operationResult": "SUCCESS"
 +
}
 +
</pre>
  
−
''<artifactId>'''''accounting-aggregator-se-plugin'''''</artifactId>''
+
an example of json to provide is
  
−
* Persistence Connectors:
+
<pre>
 +
{
 +
serviceClass : "Index"
 +
serviceName : "FullTextIndexNode"
 +
regex : "/delete/[^\\t\\n\\r\\f\\v]+/[^\\t\\n\\r\\f\\v]+"
 +
replace : "/delete/COLLECTION_ID/ITEM_ID"
 +
}
 +
</pre>
  
−
** CouchDB
 
  
−
''<groupId>'''''org.gcube.accounting'''''</groupId>''
 
  
−
''<artifactId>'''''accounting-aggregator-persistence-couchdb'''''</artifactId>''
+
Please note that RESTFull services like the Resource Registry cannot properly managed in this way so it is important that the developer fixes the service as soon as possible. This is due to missing operation type in '''calledMethod''' which is not part of the URL (as in FullTextIndexNode) instead was provided from HTTP method.

Latest revision as of 12:01, 2 January 2018

This component is responsible of aggregating the collected Accounting data.

Accounting Aggregator in the Accounting Architecture

The following image evidences the accounting-aggregator components as part of Accounting Architecture:

Accounting-architecture-accounting-aggregator.png

Accounting Aggregator Execution

This component is a Smart Executor plugin and is launched as Global Scheduled Task (see Smart Executor for further information)

It aggregates (lossless) accounted Usage Record. The scheduling plan is to run on different smart-executor equipped with this plugin with the following parameters:

  • Hourly => accounting records per hour are available for the past week;
  • Daily => accounting records per day are available for the past 3 months;
  • Monthly => accounting records per month are available for the past 3 years;
  • Yearly => accounting records per year are available for the past 10 years;


How to manage calledMethod lead to ServiceUsageRecord cannot be aggregated

RESTFull (and REST-like) services contain the ID of the managed resource inside the path of the URL. Smartgears auto calculate calledMethod by using the relative path of the URL. This lead to have ServiceUsageRecord cannot be aggregated.

To solve such an issue some actions are need from developers:

  • Rewrite calledMethod in the service code
  • Provide regular expression to match old accounted records to allow to rewrite them from accoutnign-aggrator

Rewrite calledMethod

The developer have to insert as first line of the entry method the following line of code to instruct Smartgears to set the desired calledMethod 

CalledMethodProvider.instance.set("rewrittenCalledMethod");

Resource Registry is an example of how to properly use such a feature for RESTFull services.

For example Resource Registry use the following URLs to read/update/delete a Facet having UUID 4023d5b2-8601-47a5-83ef-49ffcbfc7d86 

GET /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86

PUT /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86

DELETE /resource-registry/er/facet/4023d5b2-8601-47a5-83ef-49ffcbfc7d86

If the service does not rewrite the calledMethod we have two effect:

  • the three operation cannot be distinguished from accounting and lead to aggregate the records operating to the same facet.
  • the uuid does not allow to aggregate the operation across facets.


Resource Registry instruct smartgears to rewrite the calledMethod as following: 

CalledMethodProvider.instance.set("GET /resource-registry/er/facet/ID");
CalledMethodProvider.instance.set("PUT /resource-registry/er/facet/ID");
CalledMethodProvider.instance.set("DELETE /resource-registry/er/facet/ID");


I such a way the different methods are distinguishable and the ServiceUsageRecords can be aggregated across facets.

Provide regular expression

Accounting provide a way to rewrite the the old calledMethod to be compliant with the one declared in the service by using CalledMethodProvider.

The developer has to provide the following a json declaring the service class and name to match to check if the provided regular expression match the old calledMethod so that is rewritten with the provided replacement.

For example to properly manage the following ServiceUsageRecord 

{
	"operationCount": 1,
	"creationTime": 1507334430724,
	"serviceClass": "Index",
	"callerHost": "88.197.53.47",
	"callerQualifier": "UNKNOWN",
	"recordType": "ServiceUsageRecord",
	"consumerId": "UNKNOWN",
	"aggregated": true,
	"serviceName": "FullTextIndexNode",
	"duration": 3,
	"maxInvocationTime": 58,
	"scope": "/d4science.research-infrastructures.eu/gCubeApps/iSearch",
	"host": "dewn10.madgik.di.uoa.gr:8080",
	"startTime": 1507334430481,
	"id": "12a631da-1541-4771-8961-8d7e4a694bc0",
	"calledMethod": "/delete/3121eccdcbf9c6e982dcc5a2164b8b4029f51d8b1c448ddaf46316178d755c6f/oai:oai.datacite.org:352019",
	"endTime": 1507334430481,
	"minInvocationTime": 1,
	"operationResult": "SUCCESS"
}

an example of json to provide is 

{
	serviceClass : "Index"
	serviceName : "FullTextIndexNode"
	regex : "/delete/[^\\t\\n\\r\\f\\v]+/[^\\t\\n\\r\\f\\v]+"
	replace : "/delete/COLLECTION_ID/ITEM_ID"
}


Please note that RESTFull services like the Resource Registry cannot properly managed in this way so it is important that the developer fixes the service as soon as possible. This is due to missing operation type in calledMethod which is not part of the URL (as in FullTextIndexNode) instead was provided from HTTP method.

Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Accounting_Aggregator&oldid=30661"