[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bacula-devel] CTest testers needed

On Friday 29 February 2008 01.29:19 Frank Sweetser wrote:
> Kern Sibbald wrote:
> >>> which does all the regression tests including the tape tests, and it
> >>> turns on REGRESS_DEBUG so that the full output should be available.
> >>
> >> Heh.  I was about to suggest that, but I see you beat me to the punch =)
> >>
> > :-)
> >
> > I am interested in seeing the volume of data, because it might be
> > necessary to reduce it a bit.
> Looking at a couple of tests, it seems to be somewhere around 100-200k
> each. By default, it runs using a simple standalone database engine called
> derby from the apache project.  If necessary, though, I can move it over to
> something heavier duty such as mysql.  I believe that ctest can also be
> configured with different size limits for successful tests vs. failed
> tests.

OK.  I think we can also reduce the output.  In a number of cases, we have 
debug code enabled.  Another thing that I would like to do over time is to 
direct the output to a file, then if anything goes wrong print it all -- that 
could be a sort of intermediate option for the case where we are really 
confident if there are not problems.  This could well become the default with 
REGRESS_DEBUG=0.  Then users could determine exactly what they want.

> >> I guess the biggest question now, then, is what would you like the final
> >> hostname for the dashboard to be?
> >
> > Well, I think the official name should be dart.bacula.org or
> > regress.bacula.org.

> I would vote for regress, but either one works for me.  The
> DartConfiguration.tcl.in and README.ctest file would just have to be
> updated to point to wherever the final decision is.

OK, that is fine with me, and it is easier to remember than dart.bacula.org.  

> >> I can host it here, so aliasing whatever
> >> you'd like the final name to be to baculadart.wpi.edu will work.

OK, thanks, I'll do that today.

> >
> > Well, it would be really nice to have someone host it.  I suggest you
> > discuss it with Dan, because he offered to head up this project and host
> > it on his machine.  I hope the two of you can work it out to your
> > satisfaction.
> I've been chatting with Dan on IRC, and I don't think he has any problem
> with me hosting it (it's a big java app, and luckily I was able to easily
> get it running on Fedora 8.  I don't know if freebsd would go so
> smoothly...), but again either way works for me.

If Dan is happy with having at your place, I am too -- many thanks :-)

> > We might think if there is a way to restrict the input to Dart.  I have
> > no problem if any Bacula user has access to it and can submit data, but
> > it would be a bit of a pity if one day when we are doing final testing
> > for a release if some spammer came along and filled everything up with a
> > bunch of junk.  It is probably pretty unlikely but there do seem to be a
> > lot of weirdos out there ...
> The overall design of the application seems pretty resistant to it.  It's
> listening on a non-standard port, the only URL that takes unauthenticated
> input (test result submissions) isn't linked to on any pages, and it only
> takes well-formed XML.  That said, I'll go through the docs and see if it
> has provisions for securing itself.

Since it will be running on your machine, I will let you decide.  As I 
mentioned, I see no problem with it being open to any Bacula user to submit 
to -- we just need to setup a framework, which seems to be emerging 
nicely ...

> >> Once
> >> that's decided, I have one more patch with a quick README.ctest and a
> >> few more tweaks to send in, and then I'll purge out the testing data in
> >> the current dashboard.
> >
> > OK.
> Here's the patch.
>  - It moves the svn update responsibility from the update-ctest script back
> to ctest itself.  This has the advantage that ctest will log the updates it
> applies before each run.
>  - Adds a CTestCustom.cmake file.  This file adds options to ctest that
> ignore false errors on pgsql builds (warnings such as creating implicit
> indexes and database already existing), and increases the maximum log
> message size to 1M to hold the entire output of REGRESS_DEBUG tests.
>  - Remove the absolute path from /bin/true in the ConfigureCommand
> placeholder for systems that keep true somewhere else.
>  - Adds in a README.ctest file.
> One other warning - I'm going to take down and reprovision the virtual
> machine hosting the dashboard, either tonight or tomorrow.  In addition to
> clearing out all of the various testing runs, I'm also going to bump up the
> memory and disk space allocated to it to accommodate the dashboard and my
> own set of nightly regression runs.
> > Many thanks to you and Dan (and other participants) for working on this. 
> > It is a really nice addition to the project, and it will make it much
> > easier for us to get test results from various machines ...

Thanks for the patch.  I'll apply it today and notify the list when it is in.

On my side, I would like to do several things:
1. Make sure there is a set of scripts (perhaps the old ./all-....) that will 
run the tests without cmake and without dart.  This is for me to run lots of 
tests during development before committing that really should not be part of 
Dart.  Also, often, a test will stall here, and I find the summary output of 
ctest less useful than the Bacula output which has the start and end 
times ...

2. We have a number of unit tests, but I have never taken the time to run them 
and check the results.  However, now that we have cmake in the regress 
directory, I think it is time to put the unit tests into the regression 
suite.  This will probably take a bit more thought or a cmake file to 
actually build them ...

3. I looked through the ctest section of the manual, but didn't find anything 
that could set a time limit on a particular test -- that would be very useful 
because particularly for the multidrive tests, Bacula will sometimes stall, 
and I would prefer that after 10 mins the test is killed and ctest moves on 
to the other tests ...  it seems to me the easiest is to put a time limit in 
each of our scripts. 

> Glad to help out.  Given how much we depend on Bacula here, I simply think
> of it as enlightened self interest =)

Dart is going to be very useful to give everyone a quick global view of how 
well Bacula builds and runs on all the different platforms.

Best regards,


This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
Bacula-devel mailing list

This mailing list archive is a service of Copilot Consulting.