[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bacula-devel] Reporting of regression tests


Kern Sibbald wrote:
>> of tests by hand with a manually generated config.  I'll work a bit on
>> recreating more of the existing functionality, and report back when I have
>> a little more progress made.
> 
> Oh, nice.  Thanks :-)

Ok, here's a first pass.  At the moment, it basically does two things.  First,
it creates a test definition for each test that's currently used.  Each test
name is prefixed with the script it's currently run from, so the same groups
of tests can still be selected out and ran.  Second, it replaces the direct
calls to the test scripts with a ctest command that runs the same set of
commands.  Everything else (svn update, building, which commands the user
should run to start the tests, etc) should be unaffected.

For example, all commands that were previously from all-non-root-tests are
created with names prefixed with 'all-non-root:'.  The all-non-root-tests
script then has all of the 'tests/foo' commands replaced with a single command:

ctest -R all-non-root:

In addition to the diff, I've also included the quickie script I used to
create the initial DartTestfile.txt.in (the ctest test definition file).

>From here, the next steps will be to set it up to submit test results to a
dashboard, which requires a server to be setup, and also re-evaluate the
current set of shell scripts to see redundancies with ctest could be pruned out.

-- 
Frank Sweetser fs at wpi.edu  |  For every problem, there is a solution that
WPI Senior Network Engineer   |  is simple, elegant, and wrong. - HL Mencken
    GPG fingerprint = 6174 1257 129E 0D21 D8D4  E8A3 8E39 29E3 E2E8 8CEC
#!/bin/sh

for run in all-non-root-2tape all-non-root-tape all-non-root all-root all-tape-and-file ; do
    for test in `grep -h tests/ ${run}-tests | grep -v '^#' | sed -e 's,^.*tests/,,'` ; do
	echo ADD_TEST\(${run}:${test} \"@regressdir@/tests/${test}\"\)
    done
done
Property changes on: .
___________________________________________________________________
Name: svn:ignore
   - kerns.config
1
config
config.out
build
bin
test.out
weird-files
weird-files2
diff
tmp
working
Makefile
test1.out
time.out

   + kerns.config
1
config
config.out
build
bin
test.out
weird-files
weird-files2
diff
tmp
working
Makefile
test1.out
time.out
DartConfiguration.tcl
DartTestfile.txt
Testing


Index: all-non-root-tests
===================================================================
--- all-non-root-tests	(revision 6488)
+++ all-non-root-tests	(working copy)
@@ -6,73 +6,6 @@
 echo " " >>test.out
 echo "Start non-root disk tests"
 echo "Start non-root disk tests" >>test.out
-nice tests/auto-label-test
-nice tests/backup-bacula-test
-nice tests/bextract-test
-nice tests/big-vol-test
-nice tests/bscan-test
-nice tests/bsr-opt-test
-nice tests/compressed-test
-nice tests/compress-encrypt-test
-nice tests/concurrent-jobs-test
-nice tests/data-encrypt-test
-nice tests/encrypt-bug-test
-nice tests/fifo-test
-nice tests/backup-to-null
-nice tests/regexwhere-test
-nice tests/accurate-test
-# The following two can uses *lots* of disk space
-#   so they are normally turned off, but if you have
-#   at least two GB free, you can run them
-#nice tests/sparse-encrypt-test
-#nice tests/gigaslam-sparse-test
-nice tests/differential-test
-nice tests/four-concurrent-jobs-test
-nice tests/four-jobs-test
-nice tests/incremental-test
-nice tests/query-test
-nice tests/recycle-test
-nice tests/restore2-by-file-test
-nice tests/restore-by-file-test
-nice tests/restore-disk-seek-test
-#nice tests/runscript-test
-nice tests/six-vol-test
-nice tests/span-vol-test
-nice tests/maxvol-test
-nice tests/maxvol2-test
-nice tests/sparse-compressed-test
-nice tests/sparse-test
-nice tests/two-jobs-test
-nice tests/two-vol-test
-nice tests/verify-cat-test
-nice tests/verify-vol-test
-nice tests/weird-files2-test
-nice tests/weird-files-test
-nice tests/migration-job-test
-nice tests/migration-jobspan-test
-nice tests/migration-volume-test
-nice tests/migration-time-test
-nice tests/hardlink-test
-nice tests/tls-test
+ctest -R all-non-root:
 echo "End non-root disk tests"
 echo "End non-root disk tests" >>test.out
-
-# 
-# The following are Virtual Disk Autochanger tests
-echo " "
-echo " " >>test.out
-echo "Start non-root virtual disk autochanger tests"
-echo "Start non-root virtual disk autochanger tests" >>test.out
-#nice tests/three-pool-recycle-test
-nice tests/two-pool-test
-nice tests/fast-two-pool-test
-nice tests/two-volume-test
-nice tests/incremental-2disk
-nice tests/2drive-incremental-2disk
-nice tests/scratch-pool-test
-nice tests/2drive-concurrent-test
-nice tests/incremental-2media
-nice tests/three-pool-test
-nice tests/2drive-3pool-test
-echo "End non-root virtual disk autochanger tests"
-echo "End non-root virtual disk autochanger tests" >>test.out
Index: scripts/do_sed
===================================================================
--- scripts/do_sed	(revision 6488)
+++ scripts/do_sed	(working copy)
@@ -10,7 +10,10 @@
 dirport=${BASEPORT}
 fdport=`expr ${BASEPORT} '+' 1`
 sdport=`expr ${BASEPORT} '+' 2`
+os=`uname -s`
 
+dbengine=`echo ${WHICHDB} | sed -e 's/--with-//' -e 's/=.*//'`
+
 # Create sed command script
 echo "s%@sbindir@%${cwd}/bin%g" >${out}
 echo "s%@scriptdir@%${cwd}/bin%g" >>${out}
@@ -40,6 +43,11 @@
 echo "s%@dirport@%${dirport}%g" >>${out}
 echo "s%@sdport@%${sdport}%g" >>${out}
 echo "s%@fdport@%${fdport}%g" >>${out}
+echo "s%@regressdir@%${cwd}%g" >>${out}
+echo "s%@dbengine@%${dbengine}%g" >>${out}
+echo "s%@srcdir@%${BACULA_SOURCE}%g" >>${out}
+echo "s%@sitename@%${SITE_NAME}%g" >>${out}
+echo "s%@os@%${os}%g" >>${out}
 
 # process .in files with sed script
 sed -f ${out} ${cwd}/scripts/bacula-dir.conf.errors.in >${cwd}/scripts/bacula-dir.conf.errors
@@ -87,6 +95,9 @@
 sed -f ${out} ${cwd}/scripts/bacula-dir-2d.conf.in >${cwd}/scripts/bacula-dir-2d.conf
 sed -f ${out} ${cwd}/scripts/bconsole-2d.conf.in >${cwd}/scripts/bconsole-2d.conf
 
+sed -f ${out} ${cwd}/DartConfiguration.tcl.in >${cwd}/DartConfiguration.tcl
+sed -f ${out} ${cwd}/DartTestfile.txt.in >${cwd}/DartTestfile.txt
+
 # These files have no fancy sed stuff, so we just copy them over
 cp scripts/win32-bacula-sd-tape.conf.in scripts/win32-bacula-sd-tape.conf
 cp scripts/win32-bacula-fd.conf.in      scripts/win32-bacula-fd.conf
Index: all-root-tests
===================================================================
--- all-root-tests	(revision 6488)
+++ all-root-tests	(working copy)
@@ -6,9 +6,6 @@
 echo " " >>test.out
 echo "Start root tests"
 echo "Start root tests" >>test.out
-tests/dev-test-root
-tests/etc-test-root
-tests/lib-test-root
-tests/usr-tape-root
+nice ctest -R all-root:
 echo "End root tests"
 echo "End root tests" >>test.out
Index: DartConfiguration.tcl.in
===================================================================
--- DartConfiguration.tcl.in	(revision 0)
+++ DartConfiguration.tcl.in	(revision 0)
@@ -0,0 +1,4 @@
+SourceDirectory: @srcdir@
+BuildDirectory: @regressdir@
+Site: @sitename@
+BuildName: bacula-@os@-@dbengine@
Index: all-non-root-tape-tests
===================================================================
--- all-non-root-tape-tests	(revision 6488)
+++ all-non-root-tape-tests	(working copy)
@@ -10,21 +10,6 @@
 echo " " >>test.out
 echo "Start all non-root tape tests"
 echo "Start all non-root tape tests" >>test.out
-nice tests/ansi-label-1t1d-tape
-nice tests/backup-bacula-1t1d-tape
-#nice tests/btape-fill-full-tape
-nice tests/fixed-block-size-1t1d-tape
-nice tests/four-concurrent-jobs-1t1d-tape
-nice tests/four-jobs-1t1d-tape
-nice tests/incremental-1t1d-tape
-nice tests/relabel-1t1d-tape
-nice tests/restore-by-file-1t1d-tape
-nice tests/small-file-size-1t1d-tape
-nice tests/truncate-bug-1t1d-tape
-nice tests/two-pool-2t1da-tape
-nice tests/2drive-incremental-2t2da-tape
-nice tests/bscan-1t1d-tape
-nice tests/verify-vol-1t1d-tape
-#nice tests/vol-duration-2t1da-tape
+nice ctest -R all-non-root-tape:
 echo "End all non-root tape tests"
 echo "End all non-root tape tests" >>test.out
Index: all-non-root-2tape-tests
===================================================================
--- all-non-root-2tape-tests	(revision 6488)
+++ all-non-root-2tape-tests	(working copy)
@@ -4,7 +4,6 @@
 #
 echo "Start two tape tests"
 echo "Start two tape tests" >>test.out
-nice tests/two-volume-tape
-nice tests/incremental-2tape
+nice ctest -R all-non-root-2tape:
 echo "End two tape tests"
 echo "End two tape tests" >>test.out
Index: DartTestfile.txt.in
===================================================================
--- DartTestfile.txt.in	(revision 0)
+++ DartTestfile.txt.in	(revision 0)
@@ -0,0 +1,72 @@
+ADD_TEST(all-non-root-2tape:two-volume-tape "@regressdir@/tests/two-volume-tape")
+ADD_TEST(all-non-root-2tape:incremental-2tape "@regressdir@/tests/incremental-2tape")
+ADD_TEST(all-non-root-tape:ansi-label-1t1d-tape "@regressdir@/tests/ansi-label-1t1d-tape")
+ADD_TEST(all-non-root-tape:backup-bacula-1t1d-tape "@regressdir@/tests/backup-bacula-1t1d-tape")
+ADD_TEST(all-non-root-tape:fixed-block-size-1t1d-tape "@regressdir@/tests/fixed-block-size-1t1d-tape")
+ADD_TEST(all-non-root-tape:four-concurrent-jobs-1t1d-tape "@regressdir@/tests/four-concurrent-jobs-1t1d-tape")
+ADD_TEST(all-non-root-tape:four-jobs-1t1d-tape "@regressdir@/tests/four-jobs-1t1d-tape")
+ADD_TEST(all-non-root-tape:incremental-1t1d-tape "@regressdir@/tests/incremental-1t1d-tape")
+ADD_TEST(all-non-root-tape:relabel-1t1d-tape "@regressdir@/tests/relabel-1t1d-tape")
+ADD_TEST(all-non-root-tape:restore-by-file-1t1d-tape "@regressdir@/tests/restore-by-file-1t1d-tape")
+ADD_TEST(all-non-root-tape:small-file-size-1t1d-tape "@regressdir@/tests/small-file-size-1t1d-tape")
+ADD_TEST(all-non-root-tape:truncate-bug-1t1d-tape "@regressdir@/tests/truncate-bug-1t1d-tape")
+ADD_TEST(all-non-root-tape:two-pool-2t1da-tape "@regressdir@/tests/two-pool-2t1da-tape")
+ADD_TEST(all-non-root-tape:2drive-incremental-2t2da-tape "@regressdir@/tests/2drive-incremental-2t2da-tape")
+ADD_TEST(all-non-root-tape:bscan-1t1d-tape "@regressdir@/tests/bscan-1t1d-tape")
+ADD_TEST(all-non-root-tape:verify-vol-1t1d-tape "@regressdir@/tests/verify-vol-1t1d-tape")
+ADD_TEST(all-non-root:auto-label-test "@regressdir@/tests/auto-label-test")
+ADD_TEST(all-non-root:backup-bacula-test "@regressdir@/tests/backup-bacula-test")
+ADD_TEST(all-non-root:bextract-test "@regressdir@/tests/bextract-test")
+ADD_TEST(all-non-root:big-vol-test "@regressdir@/tests/big-vol-test")
+ADD_TEST(all-non-root:bscan-test "@regressdir@/tests/bscan-test")
+ADD_TEST(all-non-root:bsr-opt-test "@regressdir@/tests/bsr-opt-test")
+ADD_TEST(all-non-root:compressed-test "@regressdir@/tests/compressed-test")
+ADD_TEST(all-non-root:compress-encrypt-test "@regressdir@/tests/compress-encrypt-test")
+ADD_TEST(all-non-root:concurrent-jobs-test "@regressdir@/tests/concurrent-jobs-test")
+ADD_TEST(all-non-root:data-encrypt-test "@regressdir@/tests/data-encrypt-test")
+ADD_TEST(all-non-root:encrypt-bug-test "@regressdir@/tests/encrypt-bug-test")
+ADD_TEST(all-non-root:fifo-test "@regressdir@/tests/fifo-test")
+ADD_TEST(all-non-root:backup-to-null "@regressdir@/tests/backup-to-null")
+ADD_TEST(all-non-root:regexwhere-test "@regressdir@/tests/regexwhere-test")
+ADD_TEST(all-non-root:accurate-test "@regressdir@/tests/accurate-test")
+ADD_TEST(all-non-root:differential-test "@regressdir@/tests/differential-test")
+ADD_TEST(all-non-root:four-concurrent-jobs-test "@regressdir@/tests/four-concurrent-jobs-test")
+ADD_TEST(all-non-root:four-jobs-test "@regressdir@/tests/four-jobs-test")
+ADD_TEST(all-non-root:incremental-test "@regressdir@/tests/incremental-test")
+ADD_TEST(all-non-root:query-test "@regressdir@/tests/query-test")
+ADD_TEST(all-non-root:recycle-test "@regressdir@/tests/recycle-test")
+ADD_TEST(all-non-root:restore2-by-file-test "@regressdir@/tests/restore2-by-file-test")
+ADD_TEST(all-non-root:restore-by-file-test "@regressdir@/tests/restore-by-file-test")
+ADD_TEST(all-non-root:restore-disk-seek-test "@regressdir@/tests/restore-disk-seek-test")
+ADD_TEST(all-non-root:six-vol-test "@regressdir@/tests/six-vol-test")
+ADD_TEST(all-non-root:span-vol-test "@regressdir@/tests/span-vol-test")
+ADD_TEST(all-non-root:maxvol-test "@regressdir@/tests/maxvol-test")
+ADD_TEST(all-non-root:maxvol2-test "@regressdir@/tests/maxvol2-test")
+ADD_TEST(all-non-root:sparse-compressed-test "@regressdir@/tests/sparse-compressed-test")
+ADD_TEST(all-non-root:sparse-test "@regressdir@/tests/sparse-test")
+ADD_TEST(all-non-root:two-jobs-test "@regressdir@/tests/two-jobs-test")
+ADD_TEST(all-non-root:two-vol-test "@regressdir@/tests/two-vol-test")
+ADD_TEST(all-non-root:verify-cat-test "@regressdir@/tests/verify-cat-test")
+ADD_TEST(all-non-root:verify-vol-test "@regressdir@/tests/verify-vol-test")
+ADD_TEST(all-non-root:weird-files2-test "@regressdir@/tests/weird-files2-test")
+ADD_TEST(all-non-root:weird-files-test "@regressdir@/tests/weird-files-test")
+ADD_TEST(all-non-root:migration-job-test "@regressdir@/tests/migration-job-test")
+ADD_TEST(all-non-root:migration-jobspan-test "@regressdir@/tests/migration-jobspan-test")
+ADD_TEST(all-non-root:migration-volume-test "@regressdir@/tests/migration-volume-test")
+ADD_TEST(all-non-root:migration-time-test "@regressdir@/tests/migration-time-test")
+ADD_TEST(all-non-root:hardlink-test "@regressdir@/tests/hardlink-test")
+ADD_TEST(all-non-root:tls-test "@regressdir@/tests/tls-test")
+ADD_TEST(all-non-root:two-pool-test "@regressdir@/tests/two-pool-test")
+ADD_TEST(all-non-root:fast-two-pool-test "@regressdir@/tests/fast-two-pool-test")
+ADD_TEST(all-non-root:two-volume-test "@regressdir@/tests/two-volume-test")
+ADD_TEST(all-non-root:incremental-2disk "@regressdir@/tests/incremental-2disk")
+ADD_TEST(all-non-root:2drive-incremental-2disk "@regressdir@/tests/2drive-incremental-2disk")
+ADD_TEST(all-non-root:scratch-pool-test "@regressdir@/tests/scratch-pool-test")
+ADD_TEST(all-non-root:2drive-concurrent-test "@regressdir@/tests/2drive-concurrent-test")
+ADD_TEST(all-non-root:incremental-2media "@regressdir@/tests/incremental-2media")
+ADD_TEST(all-non-root:three-pool-test "@regressdir@/tests/three-pool-test")
+ADD_TEST(all-non-root:2drive-3pool-test "@regressdir@/tests/2drive-3pool-test")
+ADD_TEST(all-root:dev-test-root "@regressdir@/tests/dev-test-root")
+ADD_TEST(all-root:etc-test-root "@regressdir@/tests/etc-test-root")
+ADD_TEST(all-root:lib-test-root "@regressdir@/tests/lib-test-root")
+ADD_TEST(all-root:usr-tape-root "@regressdir@/tests/usr-tape-root")
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Bacula-devel mailing list
Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/bacula-devel


This mailing list archive is a service of Copilot Consulting.