[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bacula-devel] [Bacula-users] Virtual backup


On Monday 08 September 2008 10:49:31 Arno Lehmann wrote:
> Hi,
>
> 08.09.2008 10:33, Kern Sibbald wrote:
> > On Sunday 07 September 2008 03:04:21 Michael Heim wrote:
> >> Hi Kern,
> >>
> >> can't you use something like data spooling for this?
> >>
> >> First step: Make a virtFull to a temp spool file on disk
> >> Second step: Spool this to the destination
> >
> > Yes, interesting idea.  It could possibly help in some cases, but the
> > problem is that it doesn't scale well enough.  Take for example an
> > extreme example: someone has a backup to tape amounting to 10TB.  With
> > this, suggestion, it would be necessary to spool 10TB to disk then to
> > write it to tape.  Someone with this size of data will require a solution
> > that is tape to tape ...
>
> Definitely...
>
> > I think the best suggestion so far has been to add new code in Bacula
> > that uses the volume list (list of volumes to be read) to ensure that
> > none of those volumes are selected for writting.  That will provide a
> > quite reasonable means of ensuring there are no deadlocks -- providing
> > the user has two drives available.
>
> Which is a prerequisite for this sort of jobs anyway... the same with
> migration and copy jobs.

For the moment, Migration and Copy jobs *require* working with two different 
pools, which prevent the kind of deadlock I am worried about (there are lots 
of different kinds of problems that can come up).  This two pool requirement 
doesn't really pose a problem for Migration or Copy jobs, but does for 
Virtual backup.   However, if I remove the two pool requirement from Virtual 
backup, I will make sure that if at all possible the same applies to 
Migration and Copy jobs.

>
> Actually, If you implemented something like that, I believe the code
> could easily be used for virtual full, copy, and migration jobs. It
> would make all of them more resistant to deadlocks.

Yes.

>
> After building the list of needed volumes to read you could create a
> list of storage devices needed to read them. Then you can easily see
> if a device remains free to use for writing - if it doesn't, stop the
> job and print a message...

The list already exists in the code, which is nice.  Checking if there are 
enough devices free to use for writing is a bit more complicated -- not at 
all impossible, but a non-trivial algorithm.  What is not so nice, is that at 
the point the read volume list is built, ..., the job is already started, so 
it would be nice to detect these kinds of problems earlier, but that will 
have to wait for later.

Regards,

Kern

>
> Arno
>
> > Best regards,
> >
> > Kern
> >
> >> regards
> >> Michael
> >>
> >> Kern Sibbald wrote:
> >>> Hello,
> >>>
> >>> As many of you know Virtual Backup (consolidation, synthetic full, ...)
> >>> is a new feature that is implemented in the development trunk and
> >>> scheduled to be released later this year.  It essentially copies what
> >>> would be a "full current" restore to a new Volume thus creating an
> >>> virtual backup that can serve as a Full backup.  This has a lot of
> >>> advantages, particularly for sites with full backups that run long
> >>> times or for remote sites where the time to transmit a full backup is
> >>> excessive.
> >>>
> >>> The Virtual Backup feature works much like Migration and Copy.  It
> >>> reads from the required Volumes and writes to a Volume specified in the
> >>> pool as "Next Pool".  This ensures that the read and write Volumes are
> >>> different.
> >>>
> >>> Everything seems to work fine with the Virtual Backup.  However,
> >>> thinking about longer term operations, it has occurred to me that when
> >>> you want to make a second Virtual Backup things will become very
> >>> complicated.  First, the Virtual backup will want to read the previous
> >>> Virtual backup volume, and then if that Volume is not full, it will
> >>> want to write to the same Volume.  Even if the volume is full, you will
> >>> be in a situation where the Job will want to read and write to volumes
> >>> in the same pool.  In all those cases, there is no guarantee that there
> >>> will not be a deadlock situation (actually Bacula currently cancels any
> >>> job attempting to read and write from the same Storage device).
> >>>
> >>> I am not 100% sure what to do to resolve this issue.  I suppose one
> >>> could run a Migration job to "move" the Virtual Backup back to the Pool
> >>> from which it originally came, then the next Virtual Backup would work
> >>> fine (the same as the first one), but that sounds a bit kludgie.
> >>>
> >>> If anyone has any suggestions, I would appreciate to hear them. 
> >>> However, suggestions that require implementing significant amounts of
> >>> code or complex new algorithms such as deadlock detection won't be very
> >>> helpful since there is no time left to do such implementations between
> >>> now and release time.  In addition, deadlock detection won't help, what
> >>> we really need is deadlock resolution, and that is an even more
> >>> difficult subject.
> >>>
> >>> Best regards,
> >>>
> >>> Kern
> >>>
> >>>
> >>> -----------------------------------------------------------------------
> >>>-- This SF.Net email is sponsored by the Moblin Your Move Developer's
> >>> challenge Build the coolest Linux based applications with Moblin SDK&
> >>> win great prizes Grand prize is a trip for two to an Open Source event
> >>> anywhere in the world
> >>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >>> _______________________________________________
> >>> Bacula-users mailing list
> >>> Bacula-users@xxxxxxxxxxxxxxxxxxxxx
> >>> https://lists.sourceforge.net/lists/listinfo/bacula-users
> >>
> >> ------------------------------------------------------------------------
> >>- This SF.Net email is sponsored by the Moblin Your Move Developer's
> >> challenge Build the coolest Linux based applications with Moblin SDK &
> >> win great prizes Grand prize is a trip for two to an Open Source event
> >> anywhere in the world
> >> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> >> _______________________________________________
> >> Bacula-devel mailing list
> >> Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
> >> https://lists.sourceforge.net/lists/listinfo/bacula-devel
> >
> > -------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Moblin Your Move Developer's
> > challenge Build the coolest Linux based applications with Moblin SDK &
> > win great prizes Grand prize is a trip for two to an Open Source event
> > anywhere in the world
> > http://moblin-contest.org/redirect.php?banner_id=100&url=/
> > _______________________________________________
> > Bacula-users mailing list
> > Bacula-users@xxxxxxxxxxxxxxxxxxxxx
> > https://lists.sourceforge.net/lists/listinfo/bacula-users



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-devel mailing list
Bacula-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/bacula-devel


This mailing list archive is a service of Copilot Consulting.