18 Mar
2011
18 Mar
'11
9:39 a.m.
Hi,
Right now logs go to a file on the filesystem. However this is not right since most logs
are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
Hence I believe we need a Log Console available somewhere (we could make it avail in the
Admin UI FTM).
I'd like to discuss an implementation idea I've had this morning:
* Send application logs as Observation Events and make the available in the Activity
Stream (AS)
Pros:
* Infrastructure already in place
* Fits the AS goal: temporary information and is purged regularly
* (Of course the Activity gadget would not display them)
* They can be sent remotely as remote events in the future; this allows implementing a
remote console to monitor an XE or XEM from a distance
Cons:
* We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
* 2 ideas for scaling up the Observation/AS:
1) Have the Observation Manager save events to be notified into a Queue and have one or
several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
2) Have a way to tell the AS what storage to use for specific Event Types. For example the
AS could use an in-memory storage for Log Events while using a DB storage for other
events. This would be useful since I don't think we really need to store logs in the
DB. Note that the cache could be indexed on the message so only one instance of each log
message is preserved (no need for dups), possibly with a counter to mention how many of
them there were (that's an optimization).
I believe 2) might be enough for performances in a first implementation of the log
console.
WDYT?
Thanks
-Vincent
18 Mar
18 Mar
9:46 a.m.
I'm stupid, there's no need to relate it to the AS. The AS can simply ignore those
events and we can have a separate listener to receive them....
-Vincent
On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
Hi,
Right now logs go to a file on the filesystem. However this is not right since most logs
are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
Hence I believe we need a Log Console available somewhere (we could make it avail in the
Admin UI FTM).
I'd like to discuss an implementation idea I've had this morning:
* Send application logs as Observation Events and make the available in the Activity
Stream (AS)
Pros:
* Infrastructure already in place
* Fits the AS goal: temporary information and is purged regularly
* (Of course the Activity gadget would not display them)
* They can be sent remotely as remote events in the future; this allows implementing a
remote console to monitor an XE or XEM from a distance
Cons:
* We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
* 2 ideas for scaling up the Observation/AS:
1) Have the Observation Manager save events to be notified into a Queue and have one or
several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
2) Have a way to tell the AS what storage to use for specific Event Types. For example
the AS could use an in-memory storage for Log Events while using a DB storage for other
events. This would be useful since I don't think we really need to store logs in the
DB. Note that the cache could be indexed on the message so only one instance of each log
message is preserved (no need for dups), possibly with a counter to mention how many of
them there were (that's an optimization).
I believe 2) might be enough for performances in a first implementation of the log
console.
WDYT?
Thanks
-Vincent
9:50 a.m.
And this listener can also log the events using log4j (or slf4j or any other tool) if we
really want it...
-Vincent
On Mar 18, 2011, at 9:46 AM, Vincent Massol wrote:
I'm stupid, there's no need to relate it to
the AS. The AS can simply ignore those events and we can have a separate listener to
receive them....
-Vincent
On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
Hi,
Right now logs go to a file on the filesystem. However this is not right since most logs
are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
Hence I believe we need a Log Console available somewhere (we could make it avail in the
Admin UI FTM).
I'd like to discuss an implementation idea I've had this morning:
* Send application logs as Observation Events and make the available in the Activity
Stream (AS)
Pros:
* Infrastructure already in place
* Fits the AS goal: temporary information and is purged regularly
* (Of course the Activity gadget would not display them)
* They can be sent remotely as remote events in the future; this allows implementing a
remote console to monitor an XE or XEM from a distance
Cons:
* We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
* 2 ideas for scaling up the Observation/AS:
1) Have the Observation Manager save events to be notified into a Queue and have one or
several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
2) Have a way to tell the AS what storage to use for specific Event Types. For example
the AS could use an in-memory storage for Log Events while using a DB storage for other
events. This would be useful since I don't think we really need to store logs in the
DB. Note that the cache could be indexed on the message so only one instance of each log
message is preserved (no need for dups), possibly with a counter to mention how many of
them there were (that's an optimization).
I believe 2) might be enough for performances in a first implementation of the log
console.
WDYT?
Thanks
-Vincent
10:01 a.m.
On Fri, Mar 18, 2011 at 09:46, Vincent Massol <vincent(a)massol.net> wrote:
--
Thomas Mortagne
I'm stupid, there's no need to relate it to
the AS. The AS can simply ignore those events and we can have a separate listener to
receive them....
Yes can be a special tool for that.
-Vincent
On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
> Hi,
>
> Right now logs go to a file on the filesystem. However this is not right since most
logs are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
>
> Hence I believe we need a Log Console available somewhere (we could make it avail in
the Admin UI FTM).
>
> I'd like to discuss an implementation idea I've had this morning:
>
> * Send application logs as Observation Events and make the available in the Activity
Stream (AS)
>
> Pros:
> * Infrastructure already in place
> * Fits the AS goal: temporary information and is purged regularly
> * (Of course the Activity gadget would not display them)
> * They can be sent remotely as remote events in the future; this allows implementing
a remote console to monitor an XE or XEM from a distance
>
> Cons:
> * We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
> * 2 ideas for scaling up the Observation/AS:
> 1) Have the Observation Manager save events to be notified into a Queue and have one
or several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
IMO if a listener takes time in its onEvent() it can always do that
message queue itself, that's what Lucene plugin do for example. We
dono't absolutely need to implement that in Observation Manager. Also
you can't do that for all events since some of them are
question/answer events or event responsible for setting and unsetting
the contexte before and after a task for example.
> 2) Have a way to tell the AS what storage to use
for specific Event Types. For example the AS could use an in-memory storage for Log Events
while using a DB storage for other events. This would be useful since I don't think we
really need to store logs in the DB. Note that the cache could be indexed on the message
so only one instance of each log message is preserved (no need for dups), possibly with a
counter to mention how many of them there were (that's an optimization).
>
> I believe 2) might be enough for performances in a first implementation of the log
console.
>
> WDYT?
Make the default org.xwiki.component.logging.Logger take care of
producing theses events so that it support already existing log and
makes it easier for a component than having to do that using
Observation Manager.
+1 for the general idea
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
10:10 a.m.
On Mar 18, 2011, at 10:01 AM, Thomas Mortagne wrote:
On Fri, Mar 18, 2011 at 09:46, Vincent Massol
<vincent(a)massol.net> wrote:
Yes i know but it's nice from an architecture POV to do it at this level rather than
*hope* that all listeners (including those not coded by us) will be good citizens.
I'm stupid, there's no need to relate it
to the AS. The AS can simply ignore those events and we can have a separate listener to
receive them....
Yes can be a special tool for that.
-Vincent
On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
> Hi,
>
> Right now logs go to a file on the filesystem. However this is not right since most
logs are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
>
> Hence I believe we need a Log Console available somewhere (we could make it avail in
the Admin UI FTM).
>
> I'd like to discuss an implementation idea I've had this morning:
>
> * Send application logs as Observation Events and make the available in the Activity
Stream (AS)
>
> Pros:
> * Infrastructure already in place
> * Fits the AS goal: temporary information and is purged regularly
> * (Of course the Activity gadget would not display them)
> * They can be sent remotely as remote events in the future; this allows implementing
a remote console to monitor an XE or XEM from a distance
>
> Cons:
> * We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
> * 2 ideas for scaling up the Observation/AS:
> 1) Have the Observation Manager save events to be notified into a Queue and have one
or several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
IMO if a listener takes time in its onEvent() it can always do that
message queue itself, that's what Lucene plugin do for example. We
dono't absolutely need to implement that in Observation Manager. Also
you can't do that for all events since some of them are
question/answer events or event responsible for setting and unsetting
the contexte before and after a task for example.
Yes although I think I'd prefer to slowly move them to send events instead of logging
and have the logging be done in the listener (ie turn the logic upside down).
I'm not 100% sure though, needs some more thoughts.
> 2) Have a way to tell the AS what storage to
use for specific Event Types. For example the AS could use an in-memory storage for Log
Events while using a DB storage for other events. This would be useful since I don't
think we really need to store logs in the DB. Note that the cache could be indexed on the
message so only one instance of each log message is preserved (no need for dups), possibly
with a counter to mention how many of them there were (that's an optimization).
>
> I believe 2) might be enough for performances in a first implementation of the log
console.
>
> WDYT?
Make the default org.xwiki.component.logging.Logger take care of
producing theses events so that it support already existing log and
makes it easier for a component than having to do that using
Observation Manager. +1 for the general idea
Thanks
-Vincent
10:17 a.m.
On Fri, Mar 18, 2011 at 10:10, Vincent Massol <vincent(a)massol.net> wrote:
On Mar 18, 2011, at 10:01 AM, Thomas Mortagne wrote:
If we want all the log to produce events that's the way to go IMO. 1)
it's a lot easier to use a logging API to log 2) you can be sure you
will have very few log events if you don't do it that way, people will
always forget to send log events since that's really not natural. I
don't think it's really turn the logic upside down, it make the Logger
API an helper that's all. If you have to create event called
"LogEvent" and not something unrelated directly to logging that is
transformed as log by a listener it's exactly the same logic.
On Fri, Mar 18, 2011 at 09:46, Vincent Massol
<vincent(a)massol.net> wrote:
Yes i know but it's nice from an architecture POV to do it at this level rather than
*hope* that all listeners (including those not coded by us) will be good citizens.
I'm stupid, there's no need to relate it
to the AS. The AS can simply ignore those events and we can have a separate listener to
receive them....
Yes can be a special tool for that.
-Vincent
On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
> Hi,
>
> Right now logs go to a file on the filesystem. However this is not right since most
logs are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
>
> Hence I believe we need a Log Console available somewhere (we could make it avail in
the Admin UI FTM).
>
> I'd like to discuss an implementation idea I've had this morning:
>
> * Send application logs as Observation Events and make the available in the Activity
Stream (AS)
>
> Pros:
> * Infrastructure already in place
> * Fits the AS goal: temporary information and is purged regularly
> * (Of course the Activity gadget would not display them)
> * They can be sent remotely as remote events in the future; this allows implementing
a remote console to monitor an XE or XEM from a distance
>
> Cons:
> * We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
> * 2 ideas for scaling up the Observation/AS:
> 1) Have the Observation Manager save events to be notified into a Queue and have one
or several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
IMO if a listener takes time in its onEvent() it can always do that
message queue itself, that's what Lucene plugin do for example. We
dono't absolutely need to implement that in Observation Manager. Also
you can't do that for all events since some of them are
question/answer events or event responsible for setting and unsetting
the contexte before and after a task for example.
Yes although I think I'd prefer to slowly move them to send events instead of logging
and have the logging be done in the listener (ie turn the logic upside down).
> 2) Have a way to tell the AS what storage to
use for specific Event Types. For example the AS could use an in-memory storage for Log
Events while using a DB storage for other events. This would be useful since I don't
think we really need to store logs in the DB. Note that the cache could be indexed on the
message so only one instance of each log message is preserved (no need for dups), possibly
with a counter to mention how many of them there were (that's an optimization).
>
> I believe 2) might be enough for performances in a first implementation of the log
console.
>
> WDYT?
Make the default org.xwiki.component.logging.Logger take care of
producing theses events so that it support already existing log and
makes it easier for a component than having to do that using
Observation Manager. I'm not 100% sure though, needs some more
thoughts.
--
Thomas Mortagne
+1 for the general idea
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
10:22 a.m.
On Fri, Mar 18, 2011 at 10:17, Thomas Mortagne
<thomas.mortagne(a)xwiki.com> wrote:
If we want all the log to produce events that's the way to go IMO. 1)
it's a lot easier to use a logging API to log 2) you can be sure you
will have very few log events if you don't do it that way, people will
always forget to send log events since that's really not natural. I
don't think it's really turn the logic upside down, it make the Logger
API an helper that's all. If you have to create event called
"LogEvent" and not something unrelated directly to logging that is
transformed as log by a listener it's exactly the same logic.
--
Thomas Mortagne
On Fri, Mar 18, 2011 at 10:10, Vincent Massol
<vincent(a)massol.net> wrote:
>
> On Mar 18, 2011, at 10:01 AM, Thomas Mortagne wrote:
>
>> On Fri, Mar 18, 2011 at 09:46, Vincent Massol <vincent(a)massol.net> wrote:
>>> I'm stupid, there's no need to relate it to the AS. The AS can simply
ignore those events and we can have a separate listener to receive them....
>>
>> Yes can be a special tool for that.
>>
>>>
>>> -Vincent
>>>
>>> On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
>>>
>>>> Hi,
>>>>
>>>> Right now logs go to a file on the filesystem. However this is not right
since most logs are application logs and should be visible to wiki developers. For ex, if
I use a deprecated API, I need to see it. It shouldn't go to admins only and
shouldn't "pollute" the system logs.
>>>>
>>>> Hence I believe we need a Log Console available somewhere (we could make
it avail in the Admin UI FTM).
>>>>
>>>> I'd like to discuss an implementation idea I've had this
morning:
>>>>
>>>> * Send application logs as Observation Events and make the available in
the Activity Stream (AS)
>>>>
>>>> Pros:
>>>> * Infrastructure already in place
>>>> * Fits the AS goal: temporary information and is purged regularly
>>>> * (Of course the Activity gadget would not display them)
>>>> * They can be sent remotely as remote events in the future; this allows
implementing a remote console to monitor an XE or XEM from a distance
>>>>
>>>> Cons:
>>>> * We need to assess the performance risk and more generally we need to
make the AS scalable (I don't think it is now).
>>>> * 2 ideas for scaling up the Observation/AS:
>>>> 1) Have the Observation Manager save events to be notified into a Queue
and have one or several separate threads take those events and send them to listeners.
Right now if one listener takes time in its onEvent() method it slows down the whole chain
since they are called serially. Note that if we want even better scalability, the Queue
could be stored externally to XWiki (a JMS queue for ex) and scalability can be achieved
by app server instances listening to this queue to process it.
>>
>> IMO if a listener takes time in its onEvent() it can always do that
>> message queue itself, that's what Lucene plugin do for example.
>
> Yes i know but it's nice from an architecture POV to do it at this level rather
than *hope* that all listeners (including those not coded by us) will be good citizens.
All I'm saying is that this has nothing to do with this subject
specifically and can be discussed latter as Observation Manager
improvement (even if that's not that easy, don't forget the second
part of what I said), your idea can be implemented without that.
We
dono't absolutely need to implement that in Observation Manager. Also
you can't do that for all events since some of them are
question/answer events or event responsible for setting and unsetting
the contexte before and after a task for example.
Yes although I think I'd prefer to slowly move them to send events instead of logging
and have the logging be done in the listener (ie turn the logic upside down).
> 2) Have a way to tell the AS what storage to
use for specific Event Types. For example the AS could use an in-memory storage for Log
Events while using a DB storage for other events. This would be useful since I don't
think we really need to store logs in the DB. Note that the cache could be indexed on the
message so only one instance of each log message is preserved (no need for dups), possibly
with a counter to mention how many of them there were (that's an optimization).
>
> I believe 2) might be enough for performances in a first implementation of the log
console.
>
> WDYT?
Make the default org.xwiki.component.logging.Logger take care of
producing theses events so that it support already existing log and
makes it easier for a component than having to do that using
Observation Manager. I'm not 100% sure though, needs some more
thoughts.
--
Thomas Mortagne
+1 for the general idea
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
19 Mar
19 Mar
4:58 a.m.
On 03/18/2011 10:10 AM, Vincent Massol wrote:
Yes i know but it's nice from an architecture POV to do it at this level rather than
*hope* that all listeners (including those not coded by us) will be good citizens.
Once you have the need for synchronous messages, it's hard to break that
need for purely architectural reasons. How would you implement the
ScriptExecution filter that prevents nested scripts without synchronous
events?
We could split events into two categories, sync and async, but then we'd
have two architectures instead of one.
Yes although I think I'd prefer to slowly move them to send events instead of logging
and have the logging be done in the listener (ie turn the logic upside down).
I'm not 100% sure though, needs some more thoughts.
Or we could switch to the slf4j API and write an implementation that
creates events.
--
Sergiu Dumitriu
http://purl.org/net/sergiu/
On Mar 18, 2011, at 10:01 AM, Thomas Mortagne wrote:
> On Fri, Mar 18, 2011 at 09:46, Vincent Massol<vincent(a)massol.net> wrote:
>> I'm stupid, there's no need to relate it to the AS. The AS can simply
ignore those events and we can have a separate listener to receive them....
>
> Yes can be a special tool for that.
>
>>
>> -Vincent
>>
>> On Mar 18, 2011, at 9:39 AM, Vincent Massol wrote:
>>
>>> Hi,
>>>
>>> Right now logs go to a file on the filesystem. However this is not right
since most logs are application logs and should be visible to wiki developers. For ex, if
I use a deprecated API, I need to see it. It shouldn't go to admins only and
shouldn't "pollute" the system logs.
I agree and don't agree at the same time.
* Certain logs are indeed useful only if visible somewhere. If I, as a
user, am doing something wrong, I should be notified about it, but I
sure can't read catalina.out...
* Most logs generated on myxwiki.org don't concern user at all, so I
don't agree that we should move all the log data in the wiki. By the
way, myxwiki.org generates gigabytes of data, are you sure you want to
move that much information in the database?
* Stacktraces can't be stored in the activity stream, since it has a
limit of 2000 characters for the event body.
So, what I agree with is that *short* messages should be loggable both
from applications and from Java code. I'm not sure how to display them,
though. I can see three different usecases.
A. As a user, if I do something wrong in a request, I want to be
notified of it immediately. If I have to visit some page, then it's not
useful at all, I probably won't do it. I shouldn't see that the visited
page uses a deprecated API; this would mean that I get to be warned
about the programmer's errors, and in general error notifications tend
to scare users.
B. As an application writer, I want to be able to see all the unexpected
errors raised when people use my application. I shouldn't see expected
errors, like "X tried to change his password but provided the wrong
verification password".
C. As a site admin, I should be able to see all the logs generated by
the wiki.
These three usecases are completely different and require different
approaches.
A. should be fixed as a generic error reporting tool in the UI.
Currently there are some messy approaches to this, for example expecting
an $errors variable to be set in the context, or returning an error
string from APIs.
B. means that we should indeed make it easy to do commons-logging-like
error reporting visible in the wiki. Is this the main use case you were
thinking of when writing this proposal? There's also
http://extensions.xwiki.org/xwiki/bin/Extension/Log+Application which
tries to do something similar.
C. should instead make the current logs visible to administrators from
the wiki. I don't think that a rethinking of the whole logging mechanism
is needed for this; on the contrary, the current logs written to
catalina.out are very good. They can be accessed using the Show Log
snippet from http://extensions.xwiki.org/xwiki/bin/Extension/Show+Log
>>> Hence I believe we need a Log Console
available somewhere (we could make it avail in the Admin UI FTM).
>>>
>>> I'd like to discuss an implementation idea I've had this morning:
>>>
>>> * Send application logs as Observation Events and make the available in the
Activity Stream (AS)
Or send them as Events in the new EventStream (we still have to debate
about this some time). Personally I find it weird that logs are sent out
as observable events that anyone can listen to, and someone specific
*should* listen to in order to persist them. For me logs have a precise
configurable path, X tells Y to log something. Your approach suggests
that X yells something, and various Y that might be present note what
they find interesting. Sure, it's a lot more flexible, but is all that
flexibility really needed? Aren't you over-engineering something just
for the sake of engineering and reusability?
> Pros:
> * Infrastructure already in place
> * Fits the AS goal: temporary information and is purged regularly
> * (Of course the Activity gadget would not display them)
> * They can be sent remotely as remote events in the future; this allows implementing
a remote console to monitor an XE or XEM from a distance
>
> Cons:
> * We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
> * 2 ideas for scaling up the Observation/AS:
> 1) Have the Observation Manager save events to be notified into a Queue and have one
or several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
IMO if a listener takes time in its onEvent() it can always do that
message queue itself, that's what Lucene plugin do for example. > We
> dono't absolutely need to implement that in Observation Manager. Also
> you can't do that for all events since some of them are
> question/answer events or event responsible for setting and unsetting
> the contexte before and after a task for example.
>
>>> 2) Have a way to tell the AS what storage to use for specific Event Types.
For example the AS could use an in-memory storage for Log Events while using a DB storage
for other events. This would be useful since I don't think we really need to store
logs in the DB. Note that the cache could be indexed on the message so only one instance
of each log message is preserved (no need for dups), possibly with a counter to mention
how many of them there were (that's an optimization).
Well, how many logs do you expect to have? MyXWiki generates hundreds of
megabytes of log data in a day, do you expect all that to fit in memory?
The memory has the advantage that it's faster so it can quickly return
control to the log caller, but it has the disadvantage that it's very
limited in size.
> I believe 2) might be enough for performances in a
first implementation of the log console.
>
> WDYT?
Make the default org.xwiki.component.logging.Logger take care of
producing theses events so that it support already existing log and
makes it easier for a component than having to do that using
Observation Manager. +1 for the
general idea
Thanks
-Vincent 
18 Mar
18 Mar
10:22 a.m.
On Fri, Mar 18, 2011 at 9:39 AM, Vincent Massol <vincent(a)massol.net> wrote:
Hi,
Right now logs go to a file on the filesystem. However this is not right since most logs
are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
Hence I believe we need a Log Console available somewhere (we could make it avail in the
Admin UI FTM).
I'd like to discuss an implementation idea I've had this morning:
* Send application logs as Observation Events and make the available in the Activity
Stream (AS)
Pros:
* Infrastructure already in place
* Fits the AS goal: temporary information and is purged regularly
* (Of course the Activity gadget would not display them)
* They can be sent remotely as remote events in the future; this allows implementing a
remote console to monitor an XE or XEM from a distance
Cons:
* We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
* 2 ideas for scaling up the Observation/AS:
1) Have the Observation Manager save events to be notified into a Queue and have one or
several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
2) Have a way to tell the AS what storage to use for specific Event Types. For example
the AS could use an in-memory storage for Log Events while using a DB storage for other
events. This would be useful since I don't think we really need to store logs in the
DB. Note that the cache could be indexed on the message so only one instance of each log
message is preserved (no need for dups), possibly with a counter to mention how many of
them there were (that's an optimization).
I believe 2) might be enough for performances in a first implementation of the log
console.
WDYT?
That could be the occasion to introducte a mecanism that enables comet
communication between the server and the client. [1]
This way logs could be displayed live, without having to refresh the
UI every now and then.
Note that the comet communication is likely something that the
extension manager UI could leverage too (to display installation
progress for example)
Jerome
[1] http://en.wikipedia.org/wiki/Comet_%28programming%29
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
10:33 a.m.
On Fri, Mar 18, 2011 at 10:22, Jerome Velociter <jerome(a)xwiki.com> wrote:
On Fri, Mar 18, 2011 at 9:39 AM, Vincent Massol
<vincent(a)massol.net> wrote:
Yep that's planned ;) I was thinking of it mostly for question/answer
from the install/uninstall handlers and later install/uninstall
scripts.
Hi,
Right now logs go to a file on the filesystem. However this is not right since most logs
are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
Hence I believe we need a Log Console available somewhere (we could make it avail in the
Admin UI FTM).
I'd like to discuss an implementation idea I've had this morning:
* Send application logs as Observation Events and make the available in the Activity
Stream (AS)
Pros:
* Infrastructure already in place
* Fits the AS goal: temporary information and is purged regularly
* (Of course the Activity gadget would not display them)
* They can be sent remotely as remote events in the future; this allows implementing a
remote console to monitor an XE or XEM from a distance
Cons:
* We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
* 2 ideas for scaling up the Observation/AS:
1) Have the Observation Manager save events to be notified into a Queue and have one or
several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
2) Have a way to tell the AS what storage to use for specific Event Types. For example
the AS could use an in-memory storage for Log Events while using a DB storage for other
events. This would be useful since I don't think we really need to store logs in the
DB. Note that the cache could be indexed on the message so only one instance of each log
message is preserved (no need for dups), possibly with a counter to mention how many of
them there were (that's an optimization).
I believe 2) might be enough for performances in a first implementation of the log
console.
WDYT?
That could be the occasion to introducte a mecanism that enables comet
communication between the server and the client. [1]
This way logs could be displayed live, without having to refresh the
UI every now and then.
Note that the comet communication is likely something that the
extension manager UI could leverage too (to display installation
progress for example)
Jerome
[1] http://en.wikipedia.org/wiki/Comet_%28programming%29
--
Thomas Mortagne
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
10:31 a.m.
Hello all,
although I cannot judge all of the arguments, having the possibility of a web-display of a
log of chosen logging-categories (in the existing logging infrastructure please!) would be
something immensely useful.
There are very commonly too-long-running requests and offering a developer or even
advanced user display of the request process would really really be useful and safer, you
never know if it was a communication failure or a true failure.
I would recommend to make such an "application" be enabled to "start
anytime to hook on a category" (without thinking of storing) and to "stop on
keyword".
paul
Le 18 mars 2011 à 09:39, Vincent Massol a écrit :
Hi,
Right now logs go to a file on the filesystem. However this is not right since most logs
are application logs and should be visible to wiki developers. For ex, if I use a
deprecated API, I need to see it. It shouldn't go to admins only and shouldn't
"pollute" the system logs.
Hence I believe we need a Log Console available somewhere (we could make it avail in the
Admin UI FTM).
I'd like to discuss an implementation idea I've had this morning:
* Send application logs as Observation Events and make the available in the Activity
Stream (AS)
Pros:
* Infrastructure already in place
* Fits the AS goal: temporary information and is purged regularly
* (Of course the Activity gadget would not display them)
* They can be sent remotely as remote events in the future; this allows implementing a
remote console to monitor an XE or XEM from a distance
Cons:
* We need to assess the performance risk and more generally we need to make the AS
scalable (I don't think it is now).
* 2 ideas for scaling up the Observation/AS:
1) Have the Observation Manager save events to be notified into a Queue and have one or
several separate threads take those events and send them to listeners. Right now if one
listener takes time in its onEvent() method it slows down the whole chain since they are
called serially. Note that if we want even better scalability, the Queue could be stored
externally to XWiki (a JMS queue for ex) and scalability can be achieved by app server
instances listening to this queue to process it.
2) Have a way to tell the AS what storage to use for specific Event Types. For example
the AS could use an in-memory storage for Log Events while using a DB storage for other
events. This would be useful since I don't think we really need to store logs in the
DB. Note that the cache could be indexed on the message so only one instance of each log
message is preserved (no need for dups), possibly with a counter to mention how many of
them there were (that's an optimization).
I believe 2) might be enough for performances in a first implementation of the log
console.
WDYT?
Thanks
-Vincent
_______________________________________________
devs mailing list
devs(a)xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
5027
days inactive
5028
days old
10 comments
5 participants
participants (5)
-
Jerome Velociter -
Paul Libbrecht -
Sergiu Dumitriu -
Thomas Mortagne -
Vincent Massol