[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bacula-devel] CTest testers needed

Kern Sibbald wrote:
>>> which does all the regression tests including the tape tests, and it
>>> turns on REGRESS_DEBUG so that the full output should be available.
>> Heh.  I was about to suggest that, but I see you beat me to the punch =)
> :-)
> I am interested in seeing the volume of data, because it might be necessary to 
> reduce it a bit.

Looking at a couple of tests, it seems to be somewhere around 100-200k each.
By default, it runs using a simple standalone database engine called derby
from the apache project.  If necessary, though, I can move it over to
something heavier duty such as mysql.  I believe that ctest can also be
configured with different size limits for successful tests vs. failed tests.

>> I guess the biggest question now, then, is what would you like the final
>> hostname for the dashboard to be? 
> Well, I think the official name should be dart.bacula.org or 
> regress.bacula.org.

I would vote for regress, but either one works for me.  The
DartConfiguration.tcl.in and README.ctest file would just have to be updated
to point to wherever the final decision is.

>> I can host it here, so aliasing whatever 
>> you'd like the final name to be to baculadart.wpi.edu will work. 
> Well, it would be really nice to have someone host it.  I suggest you discuss 
> it with Dan, because he offered to head up this project and host it on his 
> machine.  I hope the two of you can work it out to your satisfaction.  

I've been chatting with Dan on IRC, and I don't think he has any problem with
me hosting it (it's a big java app, and luckily I was able to easily get it
running on Fedora 8.  I don't know if freebsd would go so smoothly...), but
again either way works for me.

> We might think if there is a way to restrict the input to Dart.  I have no 
> problem if any Bacula user has access to it and can submit data, but it would 
> be a bit of a pity if one day when we are doing final testing for a release 
> if some spammer came along and filled everything up with a bunch of junk.  It 
> is probably pretty unlikely but there do seem to be a lot of weirdos out 
> there ...

The overall design of the application seems pretty resistant to it.  It's
listening on a non-standard port, the only URL that takes unauthenticated
input (test result submissions) isn't linked to on any pages, and it only
takes well-formed XML.  That said, I'll go through the docs and see if it has
provisions for securing itself.

>> Once 
>> that's decided, I have one more patch with a quick README.ctest and a few
>> more tweaks to send in, and then I'll purge out the testing data in the
>> current dashboard.
> OK.   

Here's the patch.

 - It moves the svn update responsibility from the update-ctest script back to
ctest itself.  This has the advantage that ctest will log the updates it
applies before each run.

 - Adds a CTestCustom.cmake file.  This file adds options to ctest that ignore
false errors on pgsql builds (warnings such as creating implicit indexes and
database already existing), and increases the maximum log message size to 1M
to hold the entire output of REGRESS_DEBUG tests.

 - Remove the absolute path from /bin/true in the ConfigureCommand placeholder
for systems that keep true somewhere else.

 - Adds in a README.ctest file.

One other warning - I'm going to take down and reprovision the virtual machine
hosting the dashboard, either tonight or tomorrow.  In addition to clearing
out all of the various testing runs, I'm also going to bump up the memory and
disk space allocated to it to accommodate the dashboard and my own set of
nightly regression runs.

> Many thanks to you and Dan (and other participants) for working on this.  It 
> is a really nice addition to the project, and it will make it much easier for 
> us to get test results from various machines ...

Glad to help out.  Given how much we depend on Bacula here, I simply think of
it as enlightened self interest =)

Frank Sweetser fs at wpi.edu  |  For every problem, there is a solution that
WPI Senior Network Engineer   |  is simple, elegant, and wrong. - HL Mencken
    GPG fingerprint = 6174 1257 129E 0D21 D8D4  E8A3 8E39 29E3 E2E8 8CEC
Index: scripts/update-ctest.in
--- scripts/update-ctest.in	(revision 6519)
+++ scripts/update-ctest.in	(working copy)
@@ -4,8 +4,6 @@
 if [ ! -d build ] ; then
     echo "Build directory not present, will run make setup"
-    cd @srcdir@
-    svn update > /dev/null
     cd @regressdir@
     make setup
@@ -17,8 +15,6 @@
 cd @srcdir@
-svn update > /dev/null
 new=`svn info | grep Revision: | awk '{print $2'}`
 cd @regressdir@
Index: CTestCustom.cmake
--- CTestCustom.cmake	(revision 0)
+++ CTestCustom.cmake	(revision 0)
@@ -0,0 +1,11 @@
+  "ERROR: *database \".*\" already exists"
+  "ERROR: *table \".*\" does not exist"
+  "NOTICE: .*will create implicit sequence"
+  "NOTICE: .*will create implicit index"
+  "ERROR: *role \".*\" already exists"
+  )
Index: DartConfiguration.tcl.in
--- DartConfiguration.tcl.in	(revision 6519)
+++ DartConfiguration.tcl.in	(working copy)
@@ -7,5 +7,5 @@
 DropLocation: Bacula
 NightlyStartTime: 21:00:00 EST
 MakeCommand: @regressdir@/scripts/update-ctest
-ConfigureCommand: /bin/true
-UpdateCommand: /bin/true
+ConfigureCommand: true
+UpdateCommand: svn
Index: README.ctest
--- README.ctest	(revision 0)
+++ README.ctest	(revision 0)
@@ -0,0 +1,86 @@
+Bacula Regression Suite and CTest
+The Bacula regression scripts have now been modified to use the ctest component
+of cmake.  The major gain from this, since Bacula already had a working test
+framework in place, is the ability to have the results of each test submitted
+to a centralized dashboard system.  All of the test results are aggregated and
+summarized, where all of the developers can quickly see how the regression
+tests are running.
+How to Use CTest
+For more complete documentation on ctest, go to:
+The first step is to install the cmake package, which includes ctest.  If your
+distribution does already package it, you can download it directly from the
+Next, you must edit your regression config file and add a paramter called
+SITE_NAME to identify the machine running the tests.  Ideally, it should
+contain something to identify yourself to whoever is viewing the test results
+as well as something to allow you to identify which machine is running the
+tests, ie
+Once you have cmake installed, you can perform one of two different kinds of
+runs to submit test results.  The most common kind will be Nightly runs.  A
+Nightly CTest backup will update the source directory (as defined by the
+BACULA_SOURCE setting in your config file) to the current version, run the
+specified list of tests, and submit all of the results to the server.  Note
+that all of the results in a given 24 hour period (starting at 9pm EST) are
+lumped together to appear as a single block, rather than each test showing up
+as a different run.
+The simplest way to trigger a nightly run is to use one of the two provided
+scripts.  The nightly-all script will run all non root tests, both tape and
+disk based, while the nightly-disk script will run only the disk based tests.
+Periodically, however, you may want to submit a single test separately from a
+weekly run.  This may be a test of a particular patch you're working on, or
+perhaps a new OS patch.  For these one-shot tests, you will want to manually
+run ctest in Experimental mode, something like:
+REGRESS_DEBUG=1 ctest -D Experimental -R all-non-root:auto-label-test
+The '-D Experimental' option tells ctest to submit the test results as
+Experimental instead of Nightly.  We reccomend you use the REGRESS_DEBUG
+environmental variable to ensure that any errors from the test are logged in
+the dashboard (all of the ctest wrapper scripts set it).  The '-R <pattern>'
+option gives ctest a regular expression.  Any tests with a name as defined in
+DartTestfile.txt that matches the pattern will be run.
+Note that you must have run ./scripts/do_sed at least once already in order to
+use Experimental mode.
+Updating and Building Within CTest
+Before each Nightly run, ctest will automatically update the BACULA_SOURCE
+directory, and submit these updates along with the test results.  Any
+Experimental runs will not.
+Before either type of run actually begins running tests, ctest will run the
+script scripts/update-ctest.  This script first compares the svn version of
+BUILD_SOURCE with that of the build/ directory.  If the two versions differ, or
+if the build/ directory does not exist, it will automatically run 'make setup'
+for you.
+Viewing the Dashboard
+You can view the dashboard at:
+Results will not be visible as soon as they are submitted to the server.
+Processing is currently done every 10 minutes, so you may have to wait up to 15
+minutes or so before your results show up.
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
Bacula-devel mailing list

This mailing list archive is a service of Copilot Consulting.