[osg-users] [VPB] Getting started with parallel builds

[Robert Osfield]

Hi Glenn,

The most simple example would be (based on yours):

   osgdem --terrain -t imagery.tif -o out.ive

Becomes:

   vpbmaster --terrain -t imagery.tif -o output/out.ive

If you have a multi-core machine It'll automatically divide the job up
into tasks and then run them on all the cores you have available.

The above example takes advantage of the fact that vpbmaster uses the
same command line parser as osgdem, it also adds its own options too,
but those are for another round of testing.

Note the change of -o out.ive to -o output/out.ive is just done so
that the generated database is self contained in its own directory.
VPB will create the required directories if they aren't already there.

The vpbmaster run will generate a build_master.tasks and
build_master.source files at the start - this is effect the recipe for
the build, and also generate logs and tasks subdirectories contains
task files and log files.  I'd recommend spending some time having a
look at all the various files it generates as it'll give you a picture
of all the things that is it doing during a build.

As an appetizer, at the end of last week VPB completed its first
terrabyte build, I'm not sure of the latest build size but it was well
of 3TB whole earth model.  It took a week to build, running over a
network of 8 quad core machines.  Previously osgdem on its own would
struggle to get over a 10GB database, with some users leaving osgdem
for months in vain hope that it couldn't crash and burn.   With
vpbmaster now anyone with the data can now give Google Earth a good
run for it's money.

Robert.

On Wed, Mar 26, 2008 at 4:35 PM, Glenn Waldron <gwaldron at gmail.com> wrote:
> Robert et al,
>
> I am finally getting around to experimenting with the new VPB, and so far so
> good. Now I would like to try the new parallel build features and I'm
> looking for the simplest possible example. I've searched the archives and
> perused the source but I could still use a quick jump start.
>
> Given a simple build command like this:
>
>   osgdem --terrain -t imagery.tif -o out.ive
>
> How would I split that into a parallel build, say with 2 processes each
> building half the database. Not looking to do multiple machines or anything
> fancy just yet.
>
> Thanks for the tips!
> Glenn
>
> --
> Glenn Waldron : Pelican Mapping : http://pelicanmapping.com : 703-652-4791
> _______________________________________________
>  osg-users mailing list
>  osg-users at lists.openscenegraph.org
>  http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
>
>