[osg-submissions] win32 and win64: OSGA working with files largerthan 4GiB

[Wojciech Lewandowski]

Robert,

Today I found and fixed nasty issue with OSGA appending under win64. 
Attached is newer OSGA_Archive.cpp version. Please skip my former submittal 
and use the file attached here.

I also corrected 2 notification messages for reading errors.

Cheers,
Wojtek Lewandowski



----- Original Message ----- 
From: "Wojciech Lewandowski" <lewandowski at ai.com.pl>
To: <osg-submissions at lists.openscenegraph.org>
Sent: Wednesday, October 10, 2007 1:24 AM
Subject: [osg-submissions] win32 and win64: OSGA working with files 
largerthan 4GiB


> Robert,
>
> I earlier wrote about my hassles with archives under Windows. I 
> implemented
> 64 bit binary compatible OSGA archive reader/writer using mixed
> stdio/iostream calls. But during this work I learned that it can be made 
> in
> much simpler way.
>
> Attached is result of this new attempt. I hope its appropriate for 
> inclusion
> into OSG codebase. It was compiled and tested with latest SVN OSG, Windows
> XP 32 bit and Windows Vista business 64 bit. OSG was built using VS 2005
> Express SP1 for 32 bit environment and VS 2005 Std for 64 bit.
> ---
> Solution description (there were two problems involved):
> ---
> Problem 1: implicit conversions beetween file positions and 32 bit int. 
> This
> could be considered a MS compiler bug because this 32 bit int was
> additionally implicitly converted to/from 64 bit. As far as I know 
> compiler
> is allowed to make only one implict conversion (but maybe this rule does 
> not
> refer to simple types).
>
> Its actually possible to address OSGA files above 4 GiB range using 32 bit
> windows iostreams. MS Iostreams in practice offer the same level of
> functionality as stdio functions. There are functions fsetpos and fgetpos 
> in
> stdio lib which use 64 bit file pointers (fpos_t). These functions are
> internally called by seekp( streampos ), seekg( streampos ), tellp(), and
> tellg() methods. So its also possible to change and retrieve file postions
> using iostream calls. But the problem lies in implicit handling of 
> streampos
> type.
>
> streampos type is actually a template class used as seekp, seekg parameter
> and returnd from tellp, tellg. Its capable of storing 64 bit file 
> pointers.
> But streampos can be also converted to/from simple type streamoff. It has
> proper constructor and cast operator. In Win 32 environment streamoff is
> defined as long (~32 bit int). So when seekp, and tellp arent used with
> exact streampos objects but OSGA_Archive::pos_type complier makes implicit
> casts to 32 bit int types loosing important bits of information.
>
> So above problem could be easily handled by making conversion calls
> explicit. My code defines 2 functions used to convert back and forth 
> beetwen
> 64 bit OSGA_Archive::pos_type and std::streampos objects:
>
> OSGA_Archive::pos_type ARCHIVE_POS( const std::streampos & pos );
> std::streampos STREAM_POS( OSGA_Archive::pos_type & pos );
>
> Rest of the OSGA implementation code was modified to call these 
> conversions
> explicitly with seekp, seekg, tellp, tellg.
>
> ---
> Problem 2: seekp and seekg have two variants. Only one of these variants 
> is
> actually 64 bit proof.
>
> When I solved my first problem and made use of explicit streampos 
> conversion
> functions, OSGA archive was able to read my example 11 GiB archive. But
> there were still problems with write and append. I found that the reason 
> for
> this was pair of seekp( 0, std::ios_base::end ) and tellp() calls. It 
> turned
> out that use of seekp, seekg( offset, direction ) function variants was
> setting file pos pointer to EOF when file was larger than 4GiB. But I
> noticed that one arg seekp, seekg ( streampos ) versions worked correctly.
> So the solution was to change OSGA write logic a little, and replace
> seekp( offset, direction ) with seekp( absolute_pos ) calls.
> I achieved this by modifing IndexBlock write method to record and restore
> file pos after IndexBlock was written. This modification has the effect 
> that
> put pointer is generally kept at the end of file, so there is no need to
> repostion to the end before writing the files. This allowed me to get rid 
> of
> those problematic seekp( 0, std::ios_base::end ) calls.
>
> There was one place where I could not easily get rid of seekp( 0,
> std::ios_base::end ). It was situation where existing OSGA was opened for
> appending. I resolved this by computing file length by finding max 
> position
> from index block and file block endings. Then I replaced former seekp( 0,
> std::ios_base::end ) with  seekp( STREAM_POS( found_file_length ).
> ---
>
> Description of these changes may sound bit hacky but in practice these 
> were
> fairly simple and straightforward modifications. I hope they pass your
> review. There is one complex preprocessor condition which I based on few
> lines taken from boost positioning.hpp. Boost licence does allow such
> reproduction. In case of problems this condition may be easily simplified 
> to
> windows only implementation.
>
> Regards,
> Wojtek Lewandowski
>


--------------------------------------------------------------------------------


> _______________________________________________
> osg-submissions mailing list
> osg-submissions at lists.openscenegraph.org
> http://lists.openscenegraph.org/listinfo.cgi/osg-submissions-openscenegraph.org
> 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: OSGA_Archive.cpp
Url: http://lists.openscenegraph.org/pipermail/osg-submissions-openscenegraph.org/attachments/20071010/56b27659/attachment-0001.asc