[xwiki-devs] [proposal] Promote xwiki-reporting-tool out of the sandbox and start sending periodic reports.
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time. I'm +1 to getting reports going. A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script. Caleb
Sounds good (I like experiments ;)) but before going too far I'd like to see: 1) the needs we have regarding hudson mails. For example, IMO we need: - immediate notification after a commit that makes the build fail - no false positives - know who broke the build very visibly (with the commit that broke it) - console logs in the mail not to have to click on the link to get details - short and clear mail subjects (right now they're long and all look the same which makes it hard to see which module is broken) - no duplicate mails. We receive one for the top level project and one for the each module right now - no mails when build is successful; only when it fails Do we agree about those needs? Any more? 2) how you report is addressing these needs in 1) and whether it's supposed to replace the existing mails sent or come in addition to them. Also I'd like to know how frequently it is sent? On each commit? On a time basis? Thanks -Vincent On Nov 15, 2010, at 8:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
On 11/16/2010 05:57 AM, Vincent Massol wrote:
Sounds good (I like experiments ;)) but before going too far I'd like to see:
1) the needs we have regarding hudson mails. I think we need to start seeing the reports before we can know exactly what we want.
For example, IMO we need: - immediate notification after a commit that makes the build fail
RegressionAlertReport should cater to this need.
- no false positives This is rather difficult. Reporter looks back in history to see if a test failed before and if yes then it disregards it is a flicker. This is limited by the number of builds which hudson keeps in history.
- know who broke the build very visibly (with the commit that broke it) This is implemented, it is used in the RegressionAlertReport, a report which should run very often and only send mail if there are new regressions. Adding it to other reports is trivial.
- console logs in the mail not to have to click on the link to get details This is handled, /dev/stderr is included in the reports, stack traces are in the TestCase class so adding them to a report is trivial.
- short and clear mail subjects (right now they're long and all look the same which makes it hard to see which module is broken) Option 1 (which is currently implemented) is one message goes out which includes information for the entire array of jobs and reports on them all. Option 2 is a message goes out for each job which contains a regression. Changing to this can be done in a few hours of coding. I think it's best to start seeing mail before we decide which is best.
- no duplicate mails. We receive one for the top level project and one for the each module right now - no mails when build is successful; only when it fails Both done.
Do we agree about those needs? Any more?
I would add that I think a daily report including all of the failures (not just regressions) is necessary so we can get a perspective of the overall health of the code.
2) how you report is addressing these needs in 1)
See above.
and whether it's supposed to replace the existing mails sent or come in addition to them. Replace all of the test report mail. Hudson does a very good job sending compilation failure mail and the reporting tool is designed to work with test reports which don't exist if the code does not compile.
Also I'd like to know how frequently it is sent? On each commit? On a time basis? I imagined a time based system, another option is to run some reports after every Hudson build, per commit makes no sense because the test reports are not available until after the build. If this runs as a hudson job (as I would like) then it can be run by any hudson trigger we choose. IMO something to fine tune later.
Caleb
Thanks -Vincent
On Nov 15, 2010, at 8:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
Hi Vincent, On 11/16/2010 12:57 PM, Vincent Massol wrote:
Sounds good (I like experiments ;)) but before going too far I'd like to see:
1) the needs we have regarding hudson mails.
For example, IMO we need:
- immediate notification after a commit that makes the build fail
Often a Hudson build includes multiple commits from different committers so it's not always possible to know what commit broke the build. My use case is this: multiple commits occur while all Hudson agents are busy (build pending queue size > 0). By the time Hudson gets the chance to (re)build a module, that module has been affected (directly or indirectly) by multiple commits.
- no false positives
- know who broke the build very visibly (with the commit that broke it)
Same, I don't think this is always possible.
- console logs in the mail not to have to click on the link to get details
+1
- short and clear mail subjects (right now they're long and all look the same which makes it hard to see which module is broken)
+1
- no duplicate mails. We receive one for the top level project and one for the each module right now
Grouping the test results for each module is good IMO. Maybe not in separate mails, a single mail would do just fine. I don't like the fact that in Caleb's reports test failures from different modules are listed together; in order to see the WYSIWYG selenium test failures I have check the URLs.
- no mails when build is successful; only when it fails
One mail to notify that the build is back to normal is good IMO. Thanks, Marius
Do we agree about those needs? Any more?
2) how you report is addressing these needs in 1) and whether it's supposed to replace the existing mails sent or come in addition to them. Also I'd like to know how frequently it is sent? On each commit? On a time basis?
Thanks -Vincent
On Nov 15, 2010, at 8:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Nov 16, 2010, at 12:31 PM, Marius Dumitru Florea wrote:
Hi Vincent,
On 11/16/2010 12:57 PM, Vincent Massol wrote:
Sounds good (I like experiments ;)) but before going too far I'd like to see:
1) the needs we have regarding hudson mails.
For example, IMO we need:
- immediate notification after a commit that makes the build fail
Often a Hudson build includes multiple commits from different committers so it's not always possible to know what commit broke the build. My use case is this: multiple commits occur while all Hudson agents are busy (build pending queue size > 0). By the time Hudson gets the chance to (re)build a module, that module has been affected (directly or indirectly) by multiple commits.
- no false positives
- know who broke the build very visibly (with the commit that broke it)
Same, I don't think this is always possible.
Yes but you can display the list of commits and committers during that time frame.
- console logs in the mail not to have to click on the link to get details
+1
- short and clear mail subjects (right now they're long and all look the same which makes it hard to see which module is broken)
+1
- no duplicate mails. We receive one for the top level project and one for the each module right now
Grouping the test results for each module is good IMO. Maybe not in separate mails, a single mail would do just fine. I don't like the fact that in Caleb's reports test failures from different modules are listed together; in order to see the WYSIWYG selenium test failures I have check the URLs.
- no mails when build is successful; only when it fails
One mail to notify that the build is back to normal is good IMO.
Well that would only prove that the CI is working which I don't think we need. It's not enough to justify that many emails IMO. Thanks -Vincent
Thanks, Marius
Do we agree about those needs? Any more?
2) how you report is addressing these needs in 1) and whether it's supposed to replace the existing mails sent or come in addition to them. Also I'd like to know how frequently it is sent? On each commit? On a time basis?
Thanks -Vincent
On Nov 15, 2010, at 8:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
On 11/16/2010 06:31 AM, Marius Dumitru Florea wrote:
Hi Vincent,
On 11/16/2010 12:57 PM, Vincent Massol wrote:
Sounds good (I like experiments ;)) but before going too far I'd like to see:
1) the needs we have regarding hudson mails.
For example, IMO we need:
- immediate notification after a commit that makes the build fail
Often a Hudson build includes multiple commits from different committers so it's not always possible to know what commit broke the build. My use case is this: multiple commits occur while all Hudson agents are busy (build pending queue size > 0). By the time Hudson gets the chance to (re)build a module, that module has been affected (directly or indirectly) by multiple commits. Reporter lists all changes to the job and it's dependencies in their last build cycles so there is a list of revisions which might have caused the failure.
- no false positives
- know who broke the build very visibly (with the commit that broke it)
Same, I don't think this is always possible.
- console logs in the mail not to have to click on the link to get details
+1
- short and clear mail subjects (right now they're long and all look the same which makes it hard to see which module is broken)
+1
- no duplicate mails. We receive one for the top level project and one for the each module right now
Grouping the test results for each module is good IMO. Maybe not in separate mails, a single mail would do just fine. I don't like the fact that in Caleb's reports test failures from different modules are listed together; in order to see the WYSIWYG selenium test failures I have check the URLs.
I will change the reports to include the job and module names with each test case but I'm concerned about sorting the test cases by module because I'd like to promote the idea that everyone should care about every test.
- no mails when build is successful; only when it fails
One mail to notify that the build is back to normal is good IMO.
I have a filter which directs this mail to the garbage. Adding a report which sends mail about any module which contains at least one test whose status is FIXED would be trivial but IMO unnecessary. Caleb
Thanks, Marius
Do we agree about those needs? Any more?
2) how you report is addressing these needs in 1) and whether it's supposed to replace the existing mails sent or come in addition to them. Also I'd like to know how frequently it is sent? On each commit? On a time basis?
Thanks -Vincent
On Nov 15, 2010, at 8:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
I'm amending this proposal to include what I want to do further in the future. My goal is to decrease the quantity of hudson mail while making it easier to use. I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions. The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports. The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory. WDYT? Caleb On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On 11/16/2010 12:05 PM, Caleb James DeLisle wrote:
I'm amending this proposal to include what I want to do further in the future.
My goal is to decrease the quantity of hudson mail while making it easier to use.
I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions.
The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports.
The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory.
WDYT?
Caleb
+1, it's been months since I haven't read hudson mails. I get hundreds of mails from it each month, that's not normal.
On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Sergiu Dumitriu http://purl.org/net/sergiu/
On Nov 16, 2010, at 1:34 PM, Sergiu Dumitriu wrote:
On 11/16/2010 12:05 PM, Caleb James DeLisle wrote:
I'm amending this proposal to include what I want to do further in the future.
My goal is to decrease the quantity of hudson mail while making it easier to use.
I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions.
The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports.
The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory.
WDYT?
Caleb
+1, it's been months since I haven't read hudson mails. I get hundreds of mails from it each month, that's not normal.
Well, unless we break things frequently and this happens for real... we do break stuff frequently... so it's kind of normal we receive lots of emails :) We can reduce them by: - not sending the same email for false positives (we know for example that when we have an X server issue it's not a real issue, no space left on device, etc We should be able to list these messages and filter them out to send them in a different kind of mail, grouped together for ex). - not sending emails when the build is successful - not sending several emails for the same problem -Vincent
On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
On 11/16/2010 08:30 AM, Vincent Massol wrote:
On Nov 16, 2010, at 1:34 PM, Sergiu Dumitriu wrote:
On 11/16/2010 12:05 PM, Caleb James DeLisle wrote:
I'm amending this proposal to include what I want to do further in the future.
My goal is to decrease the quantity of hudson mail while making it easier to use.
I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions.
The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports.
The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory.
WDYT?
Caleb
+1, it's been months since I haven't read hudson mails. I get hundreds of mails from it each month, that's not normal.
Well, unless we break things frequently and this happens for real... we do break stuff frequently... so it's kind of normal we receive lots of emails :)
We can reduce them by: - not sending the same email for false positives (we know for example that when we have an X server issue it's not a real issue, no space left on device, etc We should be able to list these messages and filter them out to send them in a different kind of mail, grouped together for ex). - not sending emails when the build is successful - not sending several emails for the same problem
- if 10 tests break, send one message about it instead of 10. Caleb
-Vincent
On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
On Tue, Nov 16, 2010 at 14:30, Vincent Massol <[email protected]> wrote:
On Nov 16, 2010, at 1:34 PM, Sergiu Dumitriu wrote:
On 11/16/2010 12:05 PM, Caleb James DeLisle wrote:
I'm amending this proposal to include what I want to do further in the future.
My goal is to decrease the quantity of hudson mail while making it easier to use.
I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions.
The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports.
The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory.
WDYT?
Caleb
+1, it's been months since I haven't read hudson mails. I get hundreds of mails from it each month, that's not normal.
Well, unless we break things frequently and this happens for real... we do break stuff frequently... so it's kind of normal we receive lots of emails :)
We can reduce them by: - not sending the same email for false positives (we know for example that when we have an X server issue it's not a real issue, no space left on device, etc We should be able to list these messages and filter them out to send them in a different kind of mail, grouped together for ex).
Yes like an adminsys related mailing list.
- not sending emails when the build is successful
Yep that's pretty useless.
- not sending several emails for the same problem
Depends, maybe resending when it's still not fixed in a week or another duration we thing is good as "timeout".
-Vincent
On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On Nov 16, 2010, at 2:37 PM, Thomas Mortagne wrote:
On Tue, Nov 16, 2010 at 14:30, Vincent Massol <[email protected]> wrote:
On Nov 16, 2010, at 1:34 PM, Sergiu Dumitriu wrote:
On 11/16/2010 12:05 PM, Caleb James DeLisle wrote:
I'm amending this proposal to include what I want to do further in the future.
My goal is to decrease the quantity of hudson mail while making it easier to use.
I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions.
The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports.
The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory.
WDYT?
Caleb
+1, it's been months since I haven't read hudson mails. I get hundreds of mails from it each month, that's not normal.
Well, unless we break things frequently and this happens for real... we do break stuff frequently... so it's kind of normal we receive lots of emails :)
We can reduce them by: - not sending the same email for false positives (we know for example that when we have an X server issue it's not a real issue, no space left on device, etc We should be able to list these messages and filter them out to send them in a different kind of mail, grouped together for ex).
Yes like an adminsys related mailing list.
- not sending emails when the build is successful
Yep that's pretty useless.
- not sending several emails for the same problem
Depends, maybe resending when it's still not fixed in a week or another duration we thing is good as "timeout".
I was talking about submodules. When hudson builds a module with submodules, it sends mail for the failed submodule + for the top level module it seems. -Vincent
-Vincent
On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
On Nov 16, 2010, at 12:05 PM, Caleb James DeLisle wrote:
I'm amending this proposal to include what I want to do further in the future.
My goal is to decrease the quantity of hudson mail while making it easier to use.
I would like to start by sending additional mail, one message per day with an aggregated build report and run a report which checks for new regressions and send mail every 10 minutes or so when there are regressions.
The second step is to solicit input about the usability of the new reports. I think seeing reports in action will make it easier to see what makes a report most useful. This will be finished when we are happy with the reports.
The third step is to remove the existing mail system. I will propose this step when I think the new reporting system is satisfactory.
WDYT?
Sounds good to me. Thanks -Vincent
Caleb
On 11/15/2010 02:50 PM, Caleb James DeLisle wrote:
I would like to get reports going out so that we can get a feel for the periodic reports, and work out bugs and make improvements in real time.
I'm +1 to getting reports going.
A technical challenge is that unless one of the build agents has access to a mail relay, it will not be able to send mail. If this is a problem, it can be fixed by signing up with an email service and placing the password in a file on the filesystem of the agent which the job is bound to. The file can be loaded by the configuration script.
Caleb
participants (5)
-
Caleb James DeLisle -
Marius Dumitru Florea -
Sergiu Dumitriu -
Thomas Mortagne -
Vincent Massol