[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bacula-devel] Alternative DB proposal


On Tuesday 14 October 2008 10:06:22 Yuri Timofeev wrote:
> 2008/10/14 Kern Sibbald <kern@xxxxxxxxxxx>:
> > On Tuesday 14 October 2008 09:39:27 John Huttley wrote:
> >> So the modified version is actually a bit faster?
> >
> > That is what I understood too, but I wanted to get a confirmation.
> >
> > If it is indeed the case that the new case runs faster, it is indeed odd,
> > and I would say the tester has fallen into a very trap that is very
> > common in performance analysis.
>
> Of course, this is only the first test!
> I think that will soon be able to hold a series of tests.
> I just limiting the number of entries from 10M to 5M (very long wait)

Yes, clearly running something 10 times is not very practical if it takes 35 
hours each time, so the test size must be reduced, and you can reduce the 
number of runs from 10 to say 5.

However, what was not at all evident from your first post is that there are 
apparently subtle differences in schemas that I did not see and differences 
in the size of the data you were inserting -- and those could possibly 
explain a large (or even all)  the difference in timings.

>
> > That is to get reliable test results on a memory caching machine such as
> > Linux, you must run each test 10 times, and throw out the two with the
> > times that differ the most from the average, then re-compute the average
> > based on the remaining 8 samples which should not vary more than a few
> > percent.
> >
> > Another aid is to run one set of 10 tests, then the other set of 10
> > tests, then the first set of 10 tests again, and make sure the two runs
> > of the first set generate the same results (using the methodology
> > mentioned above). If you want to have a better measure of disk times, you
> > can sprinkle "sync" shell commands between each of the 10 test runs.
> >
> > Regards,
> >
> > Kern
> >
> >> Thats odd.
> >>
> >> are you running  64Bit?
> >> Whats your Ram?
> >> Version of Mysql?
> >>
> >>
> >> I'll run it on my system also.
> >>
> >> Regards,
> >>
> >> john
> >>
> >> Yuri Timofeev wrote:
> >> > Hi
> >> >
> >> > 2008/10/7 John Huttley <John@xxxxxxxxxxxxxxxxxx>:
> >> >> Hi
> >> >> My first test are complete.
> >> >> To aid in getting all the questions/answers and results in one place,
> >> >> I've made a wiki entry.
> >> >>
> >> >> http://wiki.bacula.org/doku.php?id=wiki:playground
> >> >
> >> > My first test are complete. ;)
> >> >
> >> > Summary:
> >> >
> >> > MySQL, ProLiant DL180G5, CentOS 5.2
> >> >
> >> > 1. Batch inserts 10M unique! records in table 'File' (DB schema not
> >> > changed) : 39 hours (23+16), DB file size 15,4 Gb
> >> > 2. Batch inserts 10M unique! records in table 'File' (NEW DB schema
> >> > see also create_mysql_test2.sh) : 35 hours (19+16), DB file size 15,2
> >> > Gb
> >> >
> >> > Source code attached.
> >> >
> >> > Please note that the length of time (16 hours) insert records from the
> >> > table 'batch' to the table 'File' the same.
> >> > This indicates the stability of results.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Full report:
> >> >
> >> > # ./test
> >> > Date: Oct 10 2008 11:27:37
> >> > Host: Localhost via UNIX socket
> >> > Use database: 'bacula_test'.
> >> > Temporary table 'batch' created.
> >> > Start filling tables 'batch'(temporary), 'Path', 'Filename' ...
> >> > End.
> >> > 10000000 records inserts.
> >> > *** CPUs time used : 375.880000
> >> > *** Elapsed time (wall clock) : 85334.000000
> >> >
> >> > Start inserts from temporary table 'batch' to 'File' ...
> >> > End inserts to 'File' ...
> >> > *** CPUs time used : 0.000000
> >> > *** Elapsed time (wall clock) : 58838.000000
> >> >
> >> >
> >> >
> >> >
> >> > # ./test2
> >> > Date: Oct 10 2008 11:27:37
> >> > Host: Localhost via UNIX socket
> >> > Use database: 'bacula_test2'.
> >> > Temporary table 'batch' created.
> >> > Start filling tables 'batch'(temporary), 'Path', 'Filename' ...
> >> > End.
> >> > 10000000 records inserts.
> >> > *** CPUs time used : 384.650000
> >> > *** Elapsed time (wall clock) : 71670.000000
> >> >
> >> > Start inserts from temporary table 'batch' to 'File' ...
> >> > End inserts to 'File' ...
> >> > *** CPUs time used : 0.000000
> >> > *** Elapsed time (wall clock) : 58564.000000
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > ----------------------------------------------------------------------
> >> >--
> >> >
> >> > ----------------------------------------------------------------------
> >> >--- This SF.Net email is sponsored by the Moblin Your Move Developer's
> >> > challenge Build the coolest Linux based applications with Moblin SDK &
> >> > win great prizes Grand prize is a trip for two to an Open Source event
> >> > anywhere in the world
> >> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> > ----------------------------------------------------------------------
> >> >--
> >> >
> >> > _______________________________________________
> >> > Bacula-devel mailing list
> >> > Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
> >> > https://lists.sourceforge.net/lists/listinfo/bacula-devel
> >
> > -------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Moblin Your Move Developer's
> > challenge Build the coolest Linux based applications with Moblin SDK &
> > win great prizes Grand prize is a trip for two to an Open Source event
> > anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Bacula-devel mailing list
> > Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
> > https://lists.sourceforge.net/lists/listinfo/bacula-devel



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-devel mailing list
Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/bacula-devel


This mailing list archive is a service of Copilotco.