Issue with loading Union related archives

More FYI, rather than actual issues. When OpenGothic load *.vdf from file system, game crashes with exception:

 for(auto& i:archives)
    inst->gothicAssets.merge(phoenix::vdf_file::open(i.name), false); // buffer_underflow here

Callstack:

0x00007ff67c54ed76: dbg::call_stack<64u>::collect(unsigned int) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3cf1dc: CrashLog::dumpStack(char const*) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3cf738: terminateHandler() in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007fff739263f6: ZN10__cxxabiv111__terminateEPFvvE in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007fff73a22d43: ZSt9terminatev in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007fff73a2bf56: _cxa_throw in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007ff67c430819: phoenix::buffer::slice(unsigned long long, unsigned long long) const in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47a368: phoenix::vdf_entry::read(phoenix::buffer&, unsigned int) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47aa5b: phoenix::vdf_file::open(phoenix::buffer&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47ac87: phoenix::vdf_file::open(std::filesystem::__cxx11::path const&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3b7c39: Resources::loadVdfs(std::vector<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t> >, std::allocator<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t> > > > const&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe

I've added simple try-catch handler to workaround issue, so here is the list of files that are failed to load:

unable to load archive: "D:/Games/Gothic II/Data/$Templates$/asmcl.dll"
unable to load archive: "D:/Games/Gothic II/Data/Union.vdf"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZUNIONUTILS.DLL"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZBINKFIX.DLL"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZMOUSEFIX.DLL"

Proposed change:
Can you add new dedicated type of exception, if *.vdf file signature does not match VDF_SIGNATURE_G* ?
Thanks!

Hi, thanks I'll look at it. I might just implement a proper parsing routine if I can find documentation :)

proper parsing routine

Not for this: ZUNIONUTILS.DLL :)

[#52] vdfs: throw a custom exception if the VDFs signature is not known

@Try

@Try I've added the exception as requested. You can now catch phoenix::vdfs_signature_error for this specific error. Could you send over the VDF file that caused the error so I can investigate?

Could you send over the VDF file that caused the error so I can investigate?

Unfortunately can't. This bug was reported on worldofplayer.ru forum, so don't have the file on my side.
Lets assume for now that issue was fixed :)

I see. I'd still like to implement it so I'll keep this issue open :>

Bump: got the Union.vdf file.
Union.zip

Thanks, that helps out a lot!

I managed to find out, that it's a single entry in the VDF which is causing the problem: SYSTEM/ZBINKFIXOLD.DLL.

The entire archive looks like this:

_WORK
_WORK/DATA
_WORK/DATA/ANIMS
_WORK/DATA/ANIMS/_COMPILED
_WORK/DATA/ANIMS/_COMPILED/HUMANS_SKELETON-T_SPAWN.MAN
_WORK/DATA/ANIMS/_COMPILED/INVALID_SOURCE_FILE.MDL
_WORK/DATA/ANIMS/_COMPILED/INVALID_SOURCE_FILE.MDM
_WORK/DATA/ANIMS/_COMPILED/INVALID_SOURCE_FILE.MMB
_WORK/DATA/MESHES
_WORK/DATA/MESHES/_COMPILED
_WORK/DATA/MESHES/_COMPILED/INVALID_SOURCE_FILE.MRM
_WORK/DATA/SOUND
_WORK/DATA/SOUND/INVALID_SOURCE_FILE.OGG
_WORK/DATA/SOUND/INVALID_SOURCE_FILE.WAV
_WORK/DATA/TEXTURES
_WORK/DATA/TEXTURES/_COMPILED
_WORK/DATA/TEXTURES/_COMPILED/INVALID_SOURCE_FILE-C.TEX
SYSTEM
SYSTEM/ASMCL.DLL
SYSTEM/AUTORUN
SYSTEM/AUTORUN/MENU
SYSTEM/AUTORUN/MENU/ZUNIONMENU.D
SYSTEM/AUTORUN/UTILS
SYSTEM/AUTORUN/UTILS/ZUNIONTRIGGER.D
SYSTEM/AUTORUN/UTILS/ZUNIONVOB.D
SYSTEM/AUTORUN/ZBINKFIX.DLL
SYSTEM/AUTORUN/ZMOUSEFIX.DLL
SYSTEM/AUTORUN/ZPARSEREXTENDER.DLL
SYSTEM/AUTORUN/ZUNIONUTILS.DLL
SYSTEM/M3D
SYSTEM/M3D/MSSA3D.M3D
SYSTEM/M3D/MSSDS3D.M3D
SYSTEM/M3D/MSSDS3DH.M3D
SYSTEM/M3D/MSSDX7.M3D
SYSTEM/M3D/MSSEAX.M3D
SYSTEM/M3D/MSSRSX.M3D
SYSTEM/M3D/MSSSOFT.M3D
SYSTEM/OSTEAMWORKS.DLL
SYSTEM/STEAM_API.DLL
SYSTEM/ZBINKFIXOLD.DLL

That being said, there seems to be a lot of entropy in that file. When extracted as-is, all content is just random bytes and no structure can be found. The file tool reports a plain "data" format for all files in the archive.

I assume its some sort of compression or encryption but if it's compression I can't tell the type. I'll have to look into the Vdfs32g.dll Union ships with. That could take some time :)

Btw, I apologize for my absence this past week, I've been ill.

Btw, I apologize for my absence this past week, I've been ill.

Welcome back :D

When extracted as-is, all content is just random bytes and no structure can be found.

Yes, most likely it's compressed. Yet it's probably not worth it to chaise after compression algorithm - just as long as it's not crashing it's fine.

@Try

Hey @Try, can you confirm this is done?

Tested today on latest phoenix commit(36d8d5f) in master branch:

Load fine without Union.vdf, but with Union.vdf, game throws exception upon game loading:

loading error: buffer underflow at byte 6 while reading 225443840 additional bytes [context: slicing]
Stack-trace:
1   libstdc++-6!.cxa_throw                                                                                                                                                                               
2   phoenix::buffer::slice          buffer.cc          
3   phoenix::buffer::extract        buffer.hh          
4   phoenix::animation::parse       animation.cc       
5   Animation::Sequence::Sequence   animation.cpp      
6   std::construct_at<Animation...  stl_construct.h    
7   std::allocator_traits<std::...  alloc_traits.h     
8   std::vector<Animation::Sequ...  vector.tcc         
9   std::vector<Animation::Sequ...  vector.tcc         
10  Animation::loadMAN              animation.cpp      
11  Animation::Animation            animation.cpp      
12  Resources::implLoadAnimation    resources.cpp      
13  Resources::loadAnimation        resources.cpp      
14  Resources::implLoadMeshMain     resources.cpp      
15  Resources::implLoadMesh         resources.cpp      
16  Resources::loadMesh             resources.cpp      
17  Resources::loadSkeleton         resources.cpp      
18  AnimationSolver::load           animationsolver.cpp
19  MdlVisual::load                 mdlvisual.cpp      
20  Npc::load                       npc.cpp

Offended file, that been requested from archive: humans_skeleton-T_SPAWN.MAN

Right, that fails because the VDF is compressed and I don't handle that. Just loading and accessing the files is fine but trying to actually parse data from them (animations in this case) will fail unless the data is properly decompressed beforehand.

Application side, unfortunately is not aware of where compression been used. Is it possible to pretend, that compressed files are not in archive at all?

Maybe it would be possible but it would be unreliable. Afaik there is no way of telling if a file is stored as compressed or not so I would have to try loading the file to check whether it's compressed or not. Or I would have to try to detect the file type using binary chunk IDs (for example).

If you can't handle the error in a try-catch, I recommend just not loading the VDF at all.

If you can't handle the error in a try-catch, I recommend just not loading the VDF at all.

Yes, that's good solution, that I'm happy with. We probably need an api to test if VDF has compressed files.

If you need to figure out if the VDF is made for/with Union, you can try checking the version stored in the header (vdf.header.version). For original Gothic 1 & 2 files it's 0x50 while Union.vdf uses 0xa0 instead. That's probably not reliable though since any file made with Union tools is likely to have that version. It's also the only difference between original files and the Union ones as far as I can tell.

Implementing a check for compressed files is probably not in scope for phoenix unless it's a really simple check. But it looks like there is no easy way of checking that.

Just my 2 cents to this matter. The modified dll talks about zipped files, so shouldn't it be possible to get a rather reliable variant by first checking the VDF version and then the file for signatures/magic words/patterns? Very likely a open algorithm was used here and ZIP means Deflate many cases.
I learned that the hard way when I extracted stuff from unknown binary data without any success and then was told that the first 4 Bytes of my file in question were the magic pattern for a LZ4 compression. Boom! Got my file. If the algorithm is fixed, the whole thing could be way easier to accomplish. This list is far from complete, but maybe... Just one of these like my LZ4 example are enough for a quiick and precise way to detect compressed files. https://en.wikipedia.org/wiki/List_of_file_signatures
Of course this is absolutely irrelevant to compatibility of the original games in any language or version that ever showed up, but if mods are in scope in the far future, this might show up again.

I was thinking about that initially, but I noticed none of the files had an obvious header. I did some more investigating and I did find something the could be considered a header.

At offset 0x18, in compressed files there will be the byte sequence 00 00 78 da which corresponds with ZLIB Best Compression (no preset dictionary) according to the Wiki page you linked. I tried extracting that part of the file and I can say that it was a success.

So it actually seems to be possible to detect this compression, I just didn't see those four bytes being the same everywhere. There is some data before that too but I haven't figured out what it means yet. I'm attaching a ZIP containing the first 50 bytes of all files in Union.vdf for reference. I'll look into implementing a check for this soon-ish unless one of you would like to do that earlier :)

For now added workaround for OpenGothic: checking vdf header: f22560e1
Assuming that union is always version 160 :)

Hey, I've created a fork of ZenKit that is able to read and save Union's compressed VDF files, since I needed this for my own project. You can check the commit here: https://github.com/skejt23/ZenKit/commit/d8a1a0e43a1b2bc6fcdcd8bfd424a221864c0784.

I verified that I can read the compressed files from the aforementioned Union.vdf. For example, see:

This tool uses ZenKit under the hood.

I also checked that the VDFS tool (https://auronen.cokoliv.eu/gmc/zengin/tools/vdfs_tools/vdfs_tool/) can read VDF files saved as compressed by ZenKit, and it correctly qualifies them as "zipped".

That being said, I wasn't sure if you wanted to support a custom format in ZenKit that isn't present in the original VDF files. If you are interested in having this in ZenKit, let me know and I'll create a PR.

@skejt23

Hi @skejt23, that's very cool work! I was originally not really in favour of adding non-vanilla stuff into ZenKit but it has turned out that some of these things are very helpful to have around.

Consequently, I'd be happy to have zipped archives integrated into this upstream repo!

I'd just ask one thing: Could you make the miniz dependency and thus the zipped file logic optional? I'd like to keep the amount of dependencies as low as possible for people how don't need them. Just a CMake option and a couple IFDEFs should be cool :)

@Try

@Try does #124 solve your original problem?

@lmichaelis

Hi, @lmichaelis !

Unfortunately, haven't had a time to enable it yet. I've checked today build with withZK_ENABLE_ZIPPED_VDF=ON:

CMake Error at CMakeLists.txt:135 (add_library):
  add_library cannot create target "miniz" because another target with the
  same name already exists.  The existing target is a static library created
  in source directory

Basically zenkit fetches another copy of minizip causing build conflict

Navigation Menu

Repository navigation

Issue with loading Union related archives #52

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with loading Union related archives #52

Description

Activity

lmichaelis commented on Jan 24, 2023

Try commented on Jan 24, 2023

lmichaelis commented on Jan 28, 2023

Try commented on Jan 28, 2023

lmichaelis commented on Jan 28, 2023

Try commented on Feb 12, 2023

lmichaelis commented on Feb 13, 2023

Try commented on Feb 13, 2023

4 remaining items

lmichaelis commented on Feb 26, 2023

Try commented on Feb 26, 2023

lmichaelis commented on Feb 26, 2023

Try commented on Feb 26, 2023

lmichaelis commented on Feb 26, 2023

Try commented on Feb 26, 2023

lmichaelis commented on Mar 3, 2023

dreimer1986 commented on Mar 19, 2023

lmichaelis commented on Mar 20, 2023

Try commented on Jun 25, 2023

skejt23 commented on Feb 8, 2026

lmichaelis commented on Feb 14, 2026

lmichaelis commented on May 9, 2026

Try commented on May 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions