[osg-users] Refactoring DatabasePager NeedToRemovestringflagging technique

Wojciech Lewandowski lewandowski at ai.com.pl
Mon Nov 23 06:34:20 PST 2009

Hi Robert,

My apologies for "SARCASM". I wanted to be provocative because former posts 
went unnoticed. Maybe I have gone too far...

But I would strongly defend merits of arguments in the post. You say we did 
wrong, but whats you recommendation on mixing many intersections with 
rendering ?  Its very common scenario. How  should we tackle it, if current 
approach is wrong ?

Second important issue for me is usage of _name in scene graph. I always 
expected that _name is reserved for users and its a normal rule in all Scene 
Graph implementations that libraries do not change it.  Names are ususlly 
used to identify certain portions of models and hook up the code properly. 
Thats something that provide standard linking mechanisms between artists and 
programmers works.

Please also note that we use cache becuse we otherwise were loading PageLOD 
files twice. Is it reasonable ?.  Even if we skip intersections,  other 
situations are also possible. For example we may have few highly detailed 
special PageLOD tiles with ariports which we want to preload and keep in 
memory for whole application runtime. So we modify readFileCallback to work 
for such cache and return these preloaded models each time thery are 
requested. Their names will be changed as well and "NeedToRemove" scheme 
stops working anyway.

With all due Respect ;-)
Wojtek Lewandowski

----- Original Message ----- 
From: "Robert Osfield" <robert.osfield at gmail.com>
To: "OpenSceneGraph Users" <osg-users at lists.openscenegraph.org>
Sent: Monday, November 23, 2009 2:59 PM
Subject: Re: [osg-users] Refactoring DatabasePager 
NeedToRemovestringflagging technique

Hi Wojtek,

On Mon, Nov 23, 2009 at 11:53 AM, Wojciech Lewandowski
<lewandowski at ai.com.pl> wrote:
> 3. I don't know if this was deliberate or not, but IntersectionVisitor
> in USE_HGHEST_LEVEL_OF_DETAIL mode does load highest level tiles 
> temporary.
> It does intersections and free them. It does not hook them up to their
> parents.

This is deliberate - if you start modifying the scene graph during the
Intersection traversal then you can't run it multi-threaded anymore.

There is also the issue of having to maintain a PageLOD with it's
children loaded from lowest res to highest res - you can't skip
intermediate children and just attach the highest rest one, keeping
the loaded subgraph local.

It's also worth mentioning that IntersectionVsitor itself does do any
loaded of data, it's callback that you attach to it that do it so
usual usage of IntersectionVisitor doesn't load highest level of
detail - it doesn't load anything.  Hence the

> 4. Above has this horrible effect that if some node was not already loaded
> by DatabasePager it will be constantly loaded and removed by
> IntersectionVisitor. SARCASM#1: But don't loose faith there is a solution.
> See next point.

Well if you want the IntersectionVisitor to happen asynchronously from
the DatabasePager loaded then you have to have keep the two separate,
or manage the integration very careful by using a
IntersectionReadCallback that is design to integrate the two.

Please note DatabaseCacheReadCallback is not designed to integrate
DatabasePager and IntersectionVisitor, it's a specific solution for
IntersectionVisition and is wholly parallel and independent to
DatabasePager and is meant to be this way.  IntersectionVisitor is
synchronous - you have to get the subgraph immediately, so the reading
of external tiles has to be done in the same thread.

> 5. We found out that we can use osgSim::DatabaseCacheReadCallback to
> mitigate former problem. This is an IntersectVisitor read callback that
> keeps internal cache to avoid repetitive loading and freeing. Well...this
> works but only for IntesectVisitor. DatabasePager does not know anything
> about this cache so when DatabasePager finally decides to load a tile it
> does it again, although IntersectionVisitor have already loaded it.
> SARCASM#2: But don't loose hope yet, because we found a solution to this 
> as
> well....
> 6. How to make sure DatabasePager sees the tile IntersectVisitor already
> loaded ? Its simple: We could use osgDB cache. So we started to load tiles
> with CACHE_NODES option and everything seemed to be fine....
> 7 But after some time we started to observe crashes and memory leaks. Long
> story about them is in my friend Pawel Ksiezopolski post "Re: [osg-users]
> PagedLOD experts?" from November 5th. Short story is that caching tiles
> does not free renamed "NeedToRemove" nodes, but keep them in memory so 
> with
> time some if them land int the scene again. When this happens, logic that
> was invented to remove not used nodes removes wrong ones leaving those 
> that
> should be removed. Hence we get PageLOD thrashing and memory leaks. Cool 
> ? (Yes its SARCASM#3)

Well if you do start mixing stuff that wasn't intended to be mixed in
the way you are mixing it well perhaps we shouldn't be too surprised
that problems eventually do ensue.

Also the SCARCASM stuff really isn't helpful and just lowers to tone
of communication.  Please stop it, it has no place on osg-users.  If
you have a technical point to make it without the extra crap.

> Now to conclusion:
> In my opinion PageLOD renaming is the reason our elaborated scheme failed. 
> I
> am both unhappy and glad that it happened because it clearly shows that
> Object::_name should not be modified by internal OSG methods. It should be
> never ever changed by OSG. Management of node name belongs to creators or
> users of the node.

Um.... your elaborate scheme illustrates a weakness in the flexibility
of the performance trick in DatabasePager, not anything about using
Object::_name.  It also illustrates problems your elaborate scheme.

The performance trick in DatabasePager is a bug because it just is
robust in the presence of multiple-viewpoints and mixing of threading
of loading.

> Thats why I am going to prepare a fix that will not rename nodes to 
> remove.
> But will instead drop their addresses into a set (or sorted vector) and
> will later us this set to test if node is marked for removal. It may be 
> not
> O(n) but O(n log n) but it will work at least. Is anyone preparing
> similar fix ?

Ironically, I originally did implement the code with a std::set<> but
went for using setName/getName() to make the code more efficient.  In
hindsight this optimization this may have been a mistake, but until
I've actually recreated the multi-viewpoint bug and dug down into
exactly what was going wrong I can't say for sure where just replace
it with std::set will fix all the problems.

In your case you will need to think more deeply about mixing
IntersectionVisitor and DatabasePager loading.  There aren't meant to
be mixed.

osg-users mailing list
osg-users at lists.openscenegraph.org

More information about the osg-users mailing list