[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bacula-devel] concurrent multidrive use


> Hello,
> 
> Le Wednesday 30 July 2008 21:05:12 Kern Sibbald, vous avez écrit :
> 
>>On Wednesday 30 July 2008 20:19:20 T. Horsnell wrote:
>>
>>>Regarding performance:
>>>We will be backing up a 16TB raid system to LTO4, both hosted by the same
>>>machine.
>>>dd'ing a 100GB file from our raid to /dev/null I can achive >
>>>400Mbytes/sec.
>>>
>>>dd'ing  the same file to /dev/nst0 with bs=256*512 I can get
>>>100Mbytes/sec. (This drops to 75 Mbytes/sec with a blocksize of 128*512
>>>...)
>>>I get a similar speed if I tar the file to tape with the same blocksize.
>>>
>>>
>>>The documented (streaming?) speed of the drive is 120Mbytes/sec.
>>
>>120MB/sec seems terribly slow for an LTO4 drive.
> 
> 
> It's certainly without compression (it's something like 80MB/s for LTO3). With 
> compression and good files (database, log, text, etc...) the speed can go to 
> 180 and maybe 200MB/s

This file was deliberately created full of random numbers to avoid 
compression (I dont know how to set my drives to 'no-compression' yet...)

> 
> 
>>>Backing up this file with Bacula (again with bs=256*512) I get
>>>70Mbytes/sec
> 
> 
> Try to make your tests with /dev/zero file, you will have the biggest 
> throughput.

You're telling me! A file of zero's compresses wonderfully, but is 
hardly realistic.

> 
> When the drive will write up to 180MB/s, try to change 
> "Minimum block size" and "Maximum block size" parameters (to something like 
> 256KB and 512KB like kern saids) and set "Maximum File Size=10G"

I did this with my dd and tar tests, but it made no further improvement. 
I assumed there would be no improvement with Bacula either, so didnt 
try. I didnt know about 'Maximum File Size', but it wouldnt have 
affected my 100GByte test.

> 
> The Maximum File Size write a file mark every 1G by default, this breaks the 
> data flow and reduce the throughput.
> 
> If the data stream isn't constant (with small files for example) your LTO4 can 
> became more slower than a LTO1. To avoid this you can use spooling.

I cant predict the eventual file-mix, so I may use spooling. My 
experiments so far have only been with the single 100Gbyte file.


> 
> 
>>Well, two comments here:
>>
>>1. You probably should at least double your Bacula buffer size to 256KB and
>>possibly up it to 512KB.

OK, I'll play.

>>
>>2. With an LTO3 tape drive it is possible with multiple simultaneous jobs
>>and very careful hardware tuning to get 150MB/sec through Bacula across a
>>network.  With a good hardware setup and an LTO4 even higher throughput
>>should be possible.  You may be limited by a hardware bottleneck somewhere
>>or possibly by the FD being unable to feed the SD fast enough ...
>>

There is *no* network involved. The filesystem being backed up and the 
tapechanger are both on the same box, so DIR, FD and SD all run on the 
same box.

>>
>>>(from Bacula's email:
>>>   FD Files Written: 1
>>>   SD Files Written: 1
>>>   FD Bytes Written: 100,000,000,000 (100.0 GB)
>>>   SD Bytes Written: 100,000,000,106 (100.0 GB)
>>>   Rate: 70028.0 KB/s)
>>>
>>>16Tbytes at 70Mbytes/sec is 63 hours
> 
> 
> If you define 2 jobs in // on two drives (with different pool), you can go up 
> to 240MB/s (without compression).

Yes, but as I said before, this means that each job would have to be 
given a part of the whole filesystem (true?), and the size of those 
parts may vary enormously, so one job may have very little to do whilst 
the other has a lot to do. So one drive would be mostly idle while the 
other is always busy.

As always, thanks everyone for your input.

Terry.
> 
> 
>>>Using two copies if this file, and backing each one up with two separate
>>>jobs
>>>onto two separate tapedrives achieves about the same rate *per drive*, so
>>>I think that for me it would be worthwhile.
>>>And for users who are trying to make use of a number of slower cheaper
>>>drives instead of fast drives in an expensive tapechanger, this would be
>>>even more beneficial.
>>
>>Perhaps, but even assuming the project were approved, you would need to
>>find someone to implement it ...
>>
>>
>>>I agree that such rates might not be achievable over a network.
>>>There is no network involved for our specific case,
>>
>>With Bacula there is *always* a network involved ...
>>
>>Regards,
>>
>>Kern
>>
>>
>>>but our emerging
>>>10GbE net would easily sustain multiple 70Mbyte/sec streams.
>>>
>>>Cheers,
>>>Terry
>>>
>>>
>>>>Hello,
>>>>
>>>>After having discussed this a bit on the list, and re-reading your note
>>>>here, I realize that yes splitting to separate tapes would be possible
>>>>on a file by file basis.  However, that feature would work much better
>>>>if the multiple threads for a given job were implemented, which is
>>>>already a project listed in the Projects file.
>>>>
>>>>Regards,
>>>>
>>>>Kern
>>>>
>>>>On Monday 21 July 2008 19:55:20 T. Horsnell wrote:
>>>>
>>>>>OK, thanks. And sorry to have pestered the development *and* the users
>>>>>list about this. I just wanted to be sure you understood what I meant.
>>>>>
>>>>>Actually, I would have said that striping was the process of spreading
>>>>>a single *file* across multiple drives simultaneously (just like
>>>>>disk-striping). To my mind, spreading the files of a single *job*
>>>>>across multiple drives doesnt mean that part of each *file* is written
>>>>>to multiple drives, but instead, that when the storage daemon is
>>>>>writing to a device which has been declared as an autochanger with
>>>>>multiple drives, it would take the next file from its input stream and
>>>>>write it to an idle drive in that autochanger.
>>>>>
>>>>>I guess if the storage-daemon scheme has one thread per tapedrive,
>>>>>it doesnt lend itself to this mode of operation.
>>>>>
>>>>>THanks again,
>>>>>Terry
>>>>>
>>>>>Kern Sibbald wrote:
>>>>>
>>>>>>Yes, sorry, I did not understand.  Bacula does not have the ability
>>>>>>to write a single job to multiple drives -- normally that is called
>>>>>>striping.  It is unlikely that it will be implemented any time in the
>>>>>>near future.
>>>>>>
>>>>>>On Monday 21 July 2008 18:27:43 T. Horsnell wrote:
>>>>>>
>>>>>>>Thank you for that quick reply, and once again, apologies for the
>>>>>>>interruption, but I dont want to split the backup across multiple
>>>>>>>jobs, (with part of the filesystem being handled by one job and part
>>>>>>>by the other), because however I make the split, the content (and
>>>>>>>hence the size) of each part of the filesystem will be continually
>>>>>>>changing (this is a 16TB multiuser raid system), and so one
>>>>>>>tapedrive may well be mostly idle whilst the other one is
>>>>>>>continually busy. So I want, if possible, to do this with a single
>>>>>>>job.
>>>>>>>
>>>>>>>Cheers,
>>>>>>>Terry
>>>>>>>
>>>>>>>Kern Sibbald wrote:
>>>>>>>
>>>>>>>>Hello,
>>>>>>>>
>>>>>>>>We generally do not supply support help on this list, but I will
>>>>>>>>give a few tips ...
>>>>>>>>
>>>>>>>>Bacula has been able to write to multiple drives simultaneously for
>>>>>>>>a very long time now -- many years and many versions.
>>>>>>>>
>>>>>>>>The simplest way to do it is to use different pools for each job.
>>>>>>>>
>>>>>>>>A not so satisfactory way of doing it is to use "Prefer Mounted
>>>>>>>>Volumes = no". I don't recommend this as it leads to many
>>>>>>>>operational problems.
>>>>>>>>
>>>>>>>>In general, if you are backing up raid disks, you should be able to
>>>>>>>>tune your hardware so that it will write approximately 150 MB/sec
>>>>>>>>with Bacula to an LTO3 drive, and so splitting jobs is not
>>>>>>>>generally necessary. Getting your hardware tuned to run at those
>>>>>>>>speeds is not easy and requires professional help.
>>>>>>>>
>>>>>>>>Best regards,
>>>>>>>>
>>>>>>>>Kern
>>>>>>>>
>>>>>>>>On Monday 21 July 2008 17:20:11 T. Horsnell wrote:
>>>>>>>>
>>>>>>>>>Apologies for pestering the developers list, but I cant determine
>>>>>>>>>from the user docs whether what I want to do is failing because
>>>>>>>>>I'm doing it wrongly, or simply that its not supported.
>>>>>>>>>
>>>>>>>>>I want to define a single job which will backup a single (big)
>>>>>>>>>raid filesystem to an autochanger which contains multiple
>>>>>>>>>tapedrives, and I want this job to use all the tapedrives
>>>>>>>>>simultaneously. This would seem to me to be a pretty standard
>>>>>>>>>requirement, but I cant get it to work with Bacula version 2.4.1
>>>>>>>>>
>>>>>>>>>Looking at the Storage Daemon section (6.4) in the new delvelopers
>>>>>>>>>docs for 2.5.2 (dated 20th July 2008 !) I see that this may not
>>>>>>>>>yet be possible:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>---cut---
>>>>>>>>>Today three jobs (threads), two physical devices each job writes
>>>>>>>>>to only one device:
>>>>>>>>>Job1 -> DCR1 -> DEVICE1
>>>>>>>>>Job2 -> DCR2 -> DEVICE1
>>>>>>>>>Job3 -> DCR3 -> DEVICE2
>>>>>>>>>
>>>>>>>>>To be implemented three jobs, three physical devices, but job1 is
>>>>>>>>>writing simultaneously to three devices:
>>>>>>>>>
>>>>>>>>>Job1 -> DCR1 -> DEVICE1
>>>>>>>>>       -> DCR4 -> DEVICE2
>>>>>>>>>       -> DCR5 -> DEVICE3
>>>>>>>>>Job2 -> DCR2 -> DEVICE1
>>>>>>>>>Job3 -> DCR3 -> DEVICE2
>>>>>>>>>---cut---
>>>>>>>>>
>>>>>>>>>Is what I want possible in 2.5.2, or should I wait?
>>>>>>>>>
>>>>>>>>>Cheers,
>>>>>>>>>Terry.
>>>>>>>>>


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-devel mailing list
Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/bacula-devel


This mailing list archive is a service of Copilotco.