[osg-users] osgViewer::GraphicsWindowWin32 stack overflow crash

Robert Osfield robert.osfield at gmail.com
Wed Mar 4 10:36:50 PST 2009

Hi Peter,

Something not working in the way you'd like it to work isn't what I'd
class as a bug, rather it's a wish list item.

In terms of making classes more flexible so it can be used in the way
you want to use it, please feel free to suggest amendments so that
others can review them and comment. FYI, hints about setup can be
passed via the GraphicsContext::Traits class, this could be explanded
to tell the window setup code in GraphicsWindowWin32 to do things
differently.  Another aspect to osgViewer is that you can inject
custom events into the viewer  by calling the windows
osgGA::EventQueue directly, I don't know if this is relevant in this
case though as I'm not familiar with the GraphicsWindowWin32.cpp.

Hopefully others with experience of working with
GraphicsWindowWin32.cpp window inheritance can chip in.


On Wed, Mar 4, 2009 at 6:03 PM, Peter Amstutz
<peter.amstutz at tseboston.com> wrote:
> First off, for reference I am using osg 2.6.1, but I checked osg 2.8 and it does not appear to be substantially different.
> I am using osg to render into a window embedded into a larger Windows Forms application.  The architecture is that Windows events/message are delivered into the main GUI thread and rendering occurs in a separate thread.  Input events, changes to the scene graph etc and marshaled between threads as necessary.  For the purposes of my application, osg should not know or care about the windows event loop (unless someone can give a really compelling reason why it needs to).
> So, osgViewer::GraphicsWindowWin32 permits initializing with an existing HWND.  All well and good.  However, where things start to go wrong is this part of initialization:
> GraphicsWindowWin32.cpp line 1155
>    if (!registerWindowProcedure())
> This sets the window procedure for the window in question, even though osg does not "own" that window.  This then creats a problem with the window procedure itself:
> GraphicsWindowWin32.cpp line 2050
> LRESULT GraphicsWindowWin32::handleNativeWindowingEvent( HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam )
> Specifically, at the bottom of the function is the following code:
> GraphicsWindowWin32.cpp line 2339
>  return _windowProcedure==0 ? ::DefWindowProc(hwnd, uMsg, wParam, lParam) :
>                                 ::CallWindowProc(_windowProcedure, hwnd, uMsg, wParam, lParam);
> Notice that if "_windowProcedure" is not null, it will call "_windowProcedure".  The problem is, _windowProcedure is initialized by registerWindowProcedure, and indirectly calls handleNativeWindowingEvent, which calls _windowProcedure, which calls handleNativeWindowingEvent, which calls _windowProcedure, which calls handleNativeWindowingEvent...  And so on until we get a stack overflow.
> Presumably the purpose here was to pass on unhandled events to some other window procedure up the chain, except for the fact that there's no way to actually provide a custom window procedure with the current GraphicsWindowWin32 API.  Hmm.
> There also isn't a way to tell GraphicsWindowWin32 to not mess around with the WindowProcedure in the first place.  Hmm.
> The best ideas I've been able to come up with are to either modify osgViewer::GraphicsWindowWin32 and comment out the call to registerWindowProcedure(), or possible subclass it in my application and make registerWindowProcedure() a no-op.
> While I appreciate that osgViewer is much better integrated into the overall library than Producer was integrated into osg 1.0, this seems to be one case where the Producer architecture of separating window initialization and event handling was actually more appropriate.
> Any thoughts, and should this be filed as a bug?
> Thanks,
> Peter Amstutz
> Senior Software Engineer
> Technology Solutions Experts
> Natick, MA
> 01760
> _______________________________________________
> osg-users mailing list
> osg-users at lists.openscenegraph.org
> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

More information about the osg-users mailing list