[xwiki-devs] Recording View activity: stats or activity stream module ? - xwiki-devs@xwiki.org

Sergiu Dumitriu

22 Feb 22 Feb

2:38 a.m.

New subject: [xwiki-devs] Recording View activity: stats or activity stream module ?

On 02/21/2010 11:18 PM, Ludovic Dubost wrote:

Hi XWiki devs, We have a project where it is needed to show what users has seen which page, in addition to aggregate statistics. Now there are multiple ways to implement this, namely either using the Activity Stream module (which records page level edit activity) or to use the Statistics module (which records aggregate level view activity). Which of the two systems would be best to use ? In any case, I would like to implement this as a patch to the standard module with a setting to activate it at a Wiki level. My first choice would be to use the Activity Stream module but I see that we have some code to clean-up the activity stream (is it active ?). I don't think I would want this data to be deleted. WDYT ?

Personally I'd also go for the activity stream. Another option would be to write a custom component that uses the notification manager to record view requests, since this is not a clear case for inclusion in the default activity stream module. -- Sergiu Dumitriu http://purl.org/net/sergiu/

Reply

Vincent Massol

9:17 a.m.

On Feb 22, 2010, at 8:11 AM, Vincent Massol wrote:

...

On Feb 21, 2010, at 11:18 PM, Ludovic Dubost wrote:

Hi XWiki devs, We have a project where it is needed to show what users has seen which page, in addition to aggregate statistics. Now there are multiple ways to implement this, namely either using the Activity Stream module (which records page level edit activity) or to use the Statistics module (which records aggregate level view activity). Which of the two systems would be best to use ?

IMO, in some near future the Stats module will be rewritten to use the activity stream module and perform computations to store aggregated data.

Some more details. I believe we have only 2 genera choices: 1) Decide to save raw data and then perform any aggregated computations (allowing OLAP-based queries, business reporting, data mining) 2) Decide to perform aggregation in memory and only save a few metrics Obviously option 1) is much more powerful and from an architectural POV is the best. For option 1) the main issue is write performance. The solution I have put in place in the past for this kind of architecture is the following: * Have the application write to a local queue (JMS queue to be standard) * Have several physical servers poll the queue and get data from it as fast as they can and then save the data in a database different from the main application database. Note that the scalability is perfect since servers get data from the queue as fast as they can process it but not more and you just need to add more servers to get better throughput (in addition the servers don't need to be of the same type). * Have some separate application querying the raw data database and perform computations on it and store the result in different tables/database. The key point for option 1) is to ensure that the performance or your app is not impacted at all by the storage of the raw events. So we'll need to decide if we want option 1 or 2. Thanks -Vincent

...

-Vincent > > In any case, I would like to implement this as a patch to the standard module with a setting to activate it at a Wiki level. > > My first choice would be to use the Activity Stream module but I see that we have some code to clean-up the activity stream (is it active ?). I don't think I would want this data to be deleted. > > WDYT ? > > Ludovic

Reply

Ludovic Dubost

7:36 p.m.

New subject: [xwiki-devs] Recording View activity: stats or activity stream module ?

Here is the issue + patch proposal for the view activity stream http://jira.xwiki.org/jira/browse/XPAS-18 Le 22/02/10 17:30, Ludovic Dubost a écrit :

...

I agree with both Sergiu and Vincent: - it should use the activity stream - in the future the activity stream should use a queue system to write it's data (the stats system already does that) - the stats should be rewritten to aggregate the activity stream So I have written an Activity Stream patch to record view events. It works pretty well, except that I have the same issue than with the stats with AJAX calls: http://jira.xwiki.org/jira/browse/XWIKI-4590 It would be really great to have this fixed for 2.2.1. For the activity stream patch I would like to propose it for review for 2.2.1 inclusion too. It's not dangerous as it's behind a setting that is off by default. I'll open a jira task soon. Ludovic Le 22/02/10 09:17, Vincent Massol a écrit :

On Feb 22, 2010, at 8:11 AM, Vincent Massol wrote:

On Feb 21, 2010, at 11:18 PM, Ludovic Dubost wrote:

Hi XWiki devs, We have a project where it is needed to show what users has seen which page, in addition to aggregate statistics. Now there are multiple ways to implement this, namely either using the Activity Stream module (which records page level edit activity) or to use the Statistics module (which records aggregate level view activity). Which of the two systems would be best to use ?

IMO, in some near future the Stats module will be rewritten to use the activity stream module and perform computations to store aggregated data.

Some more details. I believe we have only 2 genera choices: 1) Decide to save raw data and then perform any aggregated computations (allowing OLAP-based queries, business reporting, data mining) 2) Decide to perform aggregation in memory and only save a few metrics Obviously option 1) is much more powerful and from an architectural POV is the best. For option 1) the main issue is write performance. The solution I have put in place in the past for this kind of architecture is the following: * Have the application write to a local queue (JMS queue to be standard) * Have several physical servers poll the queue and get data from it as fast as they can and then save the data in a database different from the main application database. Note that the scalability is perfect since servers get data from the queue as fast as they can process it but not more and you just need to add more servers to get better throughput (in addition the servers don't need to be of the same type). * Have some separate application querying the raw data database and perform computations on it and store the result in different tables/database. The key point for option 1) is to ensure that the performance or your app is not impacted at all by the storage of the raw events. So we'll need to decide if we want option 1 or 2. Thanks -Vincent

-Vincent > In any case, I would like to implement this as a patch to the standard module with a setting to activate it at a Wiki level. > > My first choice would be to use the Activity Stream module but I see that we have some code to clean-up the activity stream (is it active ?). I don't think I would want this data to be deleted. > > WDYT ? > > Ludovic > >

_______________________________________________ devs mailing list devs(a)xwiki.org http://lists.xwiki.org/mailman/listinfo/devs

-- Ludovic Dubost Blog: http://blog.ludovic.org/ XWiki: http://www.xwiki.com Skype: ldubost GTalk: ldubost

Reply