[osg-users] Refactoring DatabasePagerNeedToRemovestringflagging technique

[Wojciech Lewandowski]

Hi Robert,

>> How should we tackle it, if current
>> approach is wrong ?

[..]

> If you do want to mix the two then you need to ask questions about how
> you want to do intersections and how your thread them.  Do you do
> intersections in the main loop?  In a separate thread?  Do you run the
> intersections multi-threaded?  I.e. multiple intersections traversals
> at one time?  Should the intersection traversal wait to load external
> tiles to get the result, or should then just return the intersections
> for what is already loaded into memory.
> What approach you will want to use will depend upon all of this.

Our case is following: We mostly have to update camera position and some 
objects around camera. And ocassionaly we reposition a camera to new 
location in the world. We are doing our intersections in update traversal. I 
assume its allowed to use IntersectVisitor in update and load new tiles 
then. We load many objects in update traversal.

And believe it or not but our scheme was working. Cache kept nodes bit 
longer than DatabasePager wanted, but not used tiles were eventually freed. 
I know it did work, because we made quick dirty fix by renaming 
"NeedToRemove" nodes to empty string name when fetching PagedLODs back to 
activePageLOD list from cache (described in Pawel Ksiezopolski post I 
mentioned earlier). Unfortuantely this fix was not appropriate as elegant 
submission. Sharing  loaded PageLOD tiles through cache was actually working 
as some preload for DatabasePager. DatabasePager still could load whatever 
it wanted.

>> Second important issue for me is usage of _name in scene graph. I always
>> expected that _name is reserved for users and its a normal rule in all 
>> Scene
>>  Graph implementations that libraries do not change it. Names are ususlly
>> used to identify certain portions of models and hook up the code 
>> properly.
>> Thats something that provide standard linking mechanisms between artists 
>> and
>> programmers works.

> I agree, but... in this instance the DatabasePager's algorithm was
> about deleting a subgraph that would no longer have any role to play
> in the applications life so the changing of name should never have got
> outside that algorithm as the subgraph would be just deleted.  So it
> is in theory just a black box, how it does it's job shouldn't effect
> anything else. Alas in this case it looks like the algorithm in
> DatabasePager is flawed.

>> Please also note that we use cache becuse we otherwise were loading 
>> PageLOD
>> files twice. Is it reasonable ?.

> Using a cache to prevent a subgraph from being deleted defeats the
> load balancing that the DatabasePager will be attempting to do, so
> it's a dangerous thing to do - it's a recipe for relentless growth in
> memory usage.

Well yes, but in our case PagedLOD usage timeframe should be extended to 
period when the tiles were used for intersections and cache maybe in bit 
hacky way provided this prolonged life time. And use of cache is not direct 
cause of leaks becaus osgDB cache is freed when node ref_count reaches zero 
so when DatabasePager & IntersectionVisitor stopped using the tile it was 
effectively removed from the cache as well.
Memory leaks were the result of extra NeedToRemove nodes that appeared in 
activePageLOD lists when some nodes were resurected from Cache because 
camera moved over them again. I know its sophistry, but in some way, it was 
more effective than native DatabasePager management because it loaded nodes 
in no time while DatabasePager would have to load them from disk in this 
case ;-).

>> Even if we skip intersections, other
>> situations are also possible. For example we may have few highly detailed
>> special PageLOD tiles with ariports which we want to preload and keep in
>> memory for whole application runtime. So we modify readFileCallback to 
>> work
>> for such cache and return these preloaded models each time thery are
>> requested.

> Use of the cache in conjunction with a paged database should be used
> very sparingly and for only very specific types of assets.  It does
> also open the question of how DatabasePager should deal with such
> datasets, without lots of reflection on the issue I can't say.  I can
> say in the design and development of DatabasePager I have made the
> assumption that it'd be the master of the PagedLOD's and manage all
> reading and expiring, and not have code on the outside managing things
> in a parallel.

Well I thought that if PagedLOD class is public and offers public interface 
and methods we could use it as long as we do not hit compiler errors. 
Perhaps different set of classes should be created for use in more VPB 
specific closed way.

> It's worth noting that PagedLOD has settings that allow you to control
> what happens with expiry - so you can individually switch off the
> expiry.

I have not thought about it. I will have to learn what we can do using this 
mechanism, perhaps we could do something smarter.

Wojtek