[osg-users] osgViewer::GraphicsWindowWin32 stack overflow crash

Peter Amstutz peter.amstutz at tseboston.com
Wed Mar 4 11:15:20 PST 2009

Well, the reason why I was ask if this is a bug is that I don't quite 
understand how it could possibly work correctly for the case that the 
native window is not "owned" by GraphicsWindowWin32.  The code very 
deliberately sets up a situation the allows infinite recursion to occur 
as I described in my previous email, so either I'm missing the 
intention, there's something funny about the way I'm using it, or this 
is an actual problem.  Perhaps using osg as a user control embedded in a 
larger application is not typical usage compared to single window or 
full screen?  My desire for GraphicsWindowWin32 to work a bit 
differently (or at least be able to work around it) has more to do with 
not having the program crash than any strong opinion about what the 
"right" way it is.


Robert Osfield wrote:
> Hi Peter,
> Something not working in the way you'd like it to work isn't what I'd
> class as a bug, rather it's a wish list item.
> In terms of making classes more flexible so it can be used in the way
> you want to use it, please feel free to suggest amendments so that
> others can review them and comment. FYI, hints about setup can be
> passed via the GraphicsContext::Traits class, this could be explanded
> to tell the window setup code in GraphicsWindowWin32 to do things
> differently.  Another aspect to osgViewer is that you can inject
> custom events into the viewer  by calling the windows
> osgGA::EventQueue directly, I don't know if this is relevant in this
> case though as I'm not familiar with the GraphicsWindowWin32.cpp.
> Hopefully others with experience of working with
> GraphicsWindowWin32.cpp window inheritance can chip in.
> Robert.
> On Wed, Mar 4, 2009 at 6:03 PM, Peter Amstutz
> <peter.amstutz at tseboston.com> wrote:
>> First off, for reference I am using osg 2.6.1, but I checked osg 2.8 and it does not appear to be substantially different.
>> I am using osg to render into a window embedded into a larger Windows Forms application.  The architecture is that Windows events/message are delivered into the main GUI thread and rendering occurs in a separate thread.  Input events, changes to the scene graph etc and marshaled between threads as necessary.  For the purposes of my application, osg should not know or care about the windows event loop (unless someone can give a really compelling reason why it needs to).
>> So, osgViewer::GraphicsWindowWin32 permits initializing with an existing HWND.  All well and good.  However, where things start to go wrong is this part of initialization:
>> GraphicsWindowWin32.cpp line 1155
>>    if (!registerWindowProcedure())
>> This sets the window procedure for the window in question, even though osg does not "own" that window.  This then creats a problem with the window procedure itself:
>> GraphicsWindowWin32.cpp line 2050
>> LRESULT GraphicsWindowWin32::handleNativeWindowingEvent( HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam )
>> Specifically, at the bottom of the function is the following code:
>> GraphicsWindowWin32.cpp line 2339
>>  return _windowProcedure==0 ? ::DefWindowProc(hwnd, uMsg, wParam, lParam) :
>>                                 ::CallWindowProc(_windowProcedure, hwnd, uMsg, wParam, lParam);
>> Notice that if "_windowProcedure" is not null, it will call "_windowProcedure".  The problem is, _windowProcedure is initialized by registerWindowProcedure, and indirectly calls handleNativeWindowingEvent, which calls _windowProcedure, which calls handleNativeWindowingEvent, which calls _windowProcedure, which calls handleNativeWindowingEvent...  And so on until we get a stack overflow.
>> Presumably the purpose here was to pass on unhandled events to some other window procedure up the chain, except for the fact that there's no way to actually provide a custom window procedure with the current GraphicsWindowWin32 API.  Hmm.
>> There also isn't a way to tell GraphicsWindowWin32 to not mess around with the WindowProcedure in the first place.  Hmm.
>> The best ideas I've been able to come up with are to either modify osgViewer::GraphicsWindowWin32 and comment out the call to registerWindowProcedure(), or possible subclass it in my application and make registerWindowProcedure() a no-op.
>> While I appreciate that osgViewer is much better integrated into the overall library than Producer was integrated into osg 1.0, this seems to be one case where the Producer architecture of separating window initialization and event handling was actually more appropriate.
>> Any thoughts, and should this be filed as a bug?
>> Thanks,
>> Peter Amstutz
>> Senior Software Engineer
>> Technology Solutions Experts
>> Natick, MA
>> 01760
>> _______________________________________________
>> osg-users mailing list
>> osg-users at lists.openscenegraph.org
>> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
> _______________________________________________
> osg-users mailing list
> osg-users at lists.openscenegraph.org
> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

More information about the osg-users mailing list