[osg-users] osgViewer::GraphicsWindowWin32 stack overflow crash

Peter Amstutz peter.amstutz at tseboston.com
Wed Mar 4 10:03:11 PST 2009

First off, for reference I am using osg 2.6.1, but I checked osg 2.8 and it does not appear to be substantially different.

I am using osg to render into a window embedded into a larger Windows Forms application.  The architecture is that Windows events/message are delivered into the main GUI thread and rendering occurs in a separate thread.  Input events, changes to the scene graph etc and marshaled between threads as necessary.  For the purposes of my application, osg should not know or care about the windows event loop (unless someone can give a really compelling reason why it needs to).

So, osgViewer::GraphicsWindowWin32 permits initializing with an existing HWND.  All well and good.  However, where things start to go wrong is this part of initialization:

GraphicsWindowWin32.cpp line 1155
    if (!registerWindowProcedure()) 

This sets the window procedure for the window in question, even though osg does not "own" that window.  This then creats a problem with the window procedure itself:

GraphicsWindowWin32.cpp line 2050
LRESULT GraphicsWindowWin32::handleNativeWindowingEvent( HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam )

Specifically, at the bottom of the function is the following code:

GraphicsWindowWin32.cpp line 2339
  return _windowProcedure==0 ? ::DefWindowProc(hwnd, uMsg, wParam, lParam) :
                                 ::CallWindowProc(_windowProcedure, hwnd, uMsg, wParam, lParam);

Notice that if "_windowProcedure" is not null, it will call "_windowProcedure".  The problem is, _windowProcedure is initialized by registerWindowProcedure, and indirectly calls handleNativeWindowingEvent, which calls _windowProcedure, which calls handleNativeWindowingEvent, which calls _windowProcedure, which calls handleNativeWindowingEvent...  And so on until we get a stack overflow.

Presumably the purpose here was to pass on unhandled events to some other window procedure up the chain, except for the fact that there's no way to actually provide a custom window procedure with the current GraphicsWindowWin32 API.  Hmm.

There also isn't a way to tell GraphicsWindowWin32 to not mess around with the WindowProcedure in the first place.  Hmm.

The best ideas I've been able to come up with are to either modify osgViewer::GraphicsWindowWin32 and comment out the call to registerWindowProcedure(), or possible subclass it in my application and make registerWindowProcedure() a no-op.

While I appreciate that osgViewer is much better integrated into the overall library than Producer was integrated into osg 1.0, this seems to be one case where the Producer architecture of separating window initialization and event handling was actually more appropriate.

Any thoughts, and should this be filed as a bug?


Peter Amstutz
Senior Software Engineer
Technology Solutions Experts
Natick, MA

More information about the osg-users mailing list