[osg-users] Refactoring DatabasePager NeedToRemove stringflagging technique

Robert Osfield robert.osfield at gmail.com
Mon Nov 23 05:59:46 PST 2009


Hi Wojtek,

On Mon, Nov 23, 2009 at 11:53 AM, Wojciech Lewandowski
<lewandowski at ai.com.pl> wrote:
> 3. I don't know if this was deliberate or not, but IntersectionVisitor
> in USE_HGHEST_LEVEL_OF_DETAIL mode does load highest level tiles temporary.
> It does intersections and free them. It does not hook them up to their
> parents.

This is deliberate - if you start modifying the scene graph during the
Intersection traversal then you can't run it multi-threaded anymore.

There is also the issue of having to maintain a PageLOD with it's
children loaded from lowest res to highest res - you can't skip
intermediate children and just attach the highest rest one, keeping
the loaded subgraph local.

It's also worth mentioning that IntersectionVsitor itself does do any
loaded of data, it's callback that you attach to it that do it so
usual usage of IntersectionVisitor doesn't load highest level of
detail - it doesn't load anything.  Hence the
DatabaseCacheReadCallback...

> 4. Above has this horrible effect that if some node was not already loaded
> by DatabasePager it will be constantly loaded and removed by
> IntersectionVisitor. SARCASM#1: But don't loose faith there is a solution.
> See next point.

Well if you want the IntersectionVisitor to happen asynchronously from
the DatabasePager loaded then you have to have keep the two separate,
or manage the integration very careful by using a
IntersectionReadCallback that is design to integrate the two.

Please note DatabaseCacheReadCallback is not designed to integrate
DatabasePager and IntersectionVisitor, it's a specific solution for
IntersectionVisition and is wholly parallel and independent to
DatabasePager and is meant to be this way.  IntersectionVisitor is
synchronous - you have to get the subgraph immediately, so the reading
of external tiles has to be done in the same thread.

> 5. We found out that we can use osgSim::DatabaseCacheReadCallback to
> mitigate former problem. This is an IntersectVisitor read callback that
> keeps internal cache to avoid  repetitive loading and freeing. Well...this
> works but only for IntesectVisitor. DatabasePager does not know anything
> about this cache so when DatabasePager finally decides to load a tile it
> does it again, although IntersectionVisitor have already loaded it.
> SARCASM#2: But don't loose hope yet, because we found a solution to this as
> well....
> 6. How to make sure DatabasePager sees the tile IntersectVisitor already
> loaded ? Its simple: We could use osgDB cache. So we started to load tiles
> with CACHE_NODES option and everything seemed to be fine....
> 7 But after some time we started to observe crashes and memory leaks. Long
> story about them is in my friend Pawel Ksiezopolski post "Re: [osg-users]
> PagedLOD experts?" from  November 5th. Short story is that caching tiles
> does not free renamed "NeedToRemove" nodes, but keep them in memory so with
> time some if them land int the scene again. When this happens, logic that
> was invented to remove not used nodes removes wrong ones leaving those that
> should be removed. Hence we get PageLOD thrashing and memory leaks. Cool HUH
> ? (Yes its SARCASM#3)

Well if you do start mixing stuff that wasn't intended to be mixed in
the way you are mixing it well perhaps we shouldn't be too surprised
that problems eventually do ensue.

Also the SCARCASM stuff really isn't helpful and just lowers to tone
of communication.  Please stop it, it has no place on osg-users.  If
you have a technical point to make it without the extra crap.

> Now to conclusion:
> In my opinion PageLOD renaming is the reason our elaborated scheme failed. I
> am both unhappy and glad that it happened because it clearly shows that
> Object::_name should not be modified by internal OSG methods. It should be
> never ever changed by OSG.  Management of node name belongs to creators or
> users of the node.

Um.... your elaborate scheme illustrates a weakness in the flexibility
of the performance trick in DatabasePager, not anything about using
Object::_name.  It also illustrates problems your elaborate scheme.

The performance trick in DatabasePager is a bug because it just is
robust in the presence of multiple-viewpoints and mixing of threading
of loading.

> Thats why I am going to prepare a fix that will not rename nodes to remove.
>  But will instead drop their addresses into a set (or sorted vector) and
> will later us this set to test if node is marked for removal. It may be not
> O(n) but O(n log n) but it will work at least.  Is anyone preparing
> similar fix ?

Ironically, I originally did implement the code with a std::set<> but
went for using setName/getName() to make the code more efficient.  In
hindsight this optimization this may have been a mistake, but until
I've actually recreated the multi-viewpoint bug and dug down into
exactly what was going wrong I can't say for sure where just replace
it with std::set will fix all the problems.

In your case you will need to think more deeply about mixing
IntersectionVisitor and DatabasePager loading.  There aren't meant to
be mixed.

Robert.


More information about the osg-users mailing list