Usefull directory structure (during installation) for Mods

[BastianW]

This "idea" has not directly to do with game mechanics, its more the idea how mod support should be added to the game.

Abstact:
I would purpose a installation structure which would allow the community to add there own maps (including quests and such stuff) without changing the "default" map.

Description:
I would prefer the following directory structure (* = Folder, ** = Subfolder, *** = SubSubFolder):

*Installation
** Maps
*** Default

The game when started should then check if there is a default map in place, if yes, he should start with that one (So nobody will see the dialog for a new map when the game is delivered). If there is another folder in the maps directory, the game should offer us a way to download this map (maybe the game can show us some infos to the map [would prefer XML for that]).

Why is that needed?:
If maybe mod support is added (I hope it will) we could add our own maps to the "Maps" folder and could load them in the game (without destroying the original map, maybe we wish to play it again).

Why a own folder for the maps/quests?
The community (or I) could build up a "map loader program". The loader can be installed on the OS as separate program. If it will be started it can connect to a website (yes, I can build that website too) and can then check for new additional maps [For the programmer: I would prefer a webservice on the website].

Could we get a example from the loader mentioned?
Sure ... it would look something like the following one:





What has inXile from this idea?
You guys got a community who build new maps. We keep the game alive and you can sell new game licenses (or maybe later one new game updates). There will then not only one map which can be played, there would maybe thousand new maps which can be played.

Summary:
I think such a directory structure would be a biiiiiiig benefit for the community, if later one mod tools could be provided. If the game is build via that way (from the start) the community could then easily add maps and we can build installer so that other persons, who maybe have problems to extract mods to the correct folder(s). Such a map installer could also maybe used to keep the map updated (maybe the map developer added new content or fixed quests?).

I like the idea, where can I vote for it?
Do it here:
http://goo.gl/mod/1mSz

--------------------------------------------------------
I will collect here a summary the input from all the postings in that thread:

- Not every user has Admin rights. So the custom maps and the savegames must be saved somewhere, where the user has rights to write.

[BastianW]

As additional idea you could add two maps to the final game via that construct here. The default map and a kind of "tutorial" map. So if the user wish to play the tutorial map, where he can can learn how to use the game [e.g. a questmaker is explaining the user how to steal something for a ingame character], then this map could be loaded/selected by the user. And if the user didn´t wish that, he can just start with the default map.

Some additional infos here (as mentioned via PM, email, chat, ...)

Graphics
The core graphics from the default map shouldn´t be in the default map. They should be somewhere in the game folder itself. So if we need to add our own graphics for the map, we could add them in our map folder, or refer to the "core" graphics folder.



XML File
(should be in the map) [MapDescription.xml]

What we need are fields for:
** MapName
** Description
** Author
* MapPicture (Name from the picture for the map, or we can use a default map picture)
* MapVersion
* DownloadURL
* UpdateURL
* Additional tags which might be useful for the loader we build, but aren´t really needed for the game itself.


** = required for the ingame map loader
* = required for the map loader the community need to build

[Zeful]

This sounds like you want Titan Quests' system of directory-level mod support. In TQ (if you didn't know), mods could be added pretty easily, simply by creating a directory path in the game folder and then giving them either a unique name (for custom maps/games) or the name of an existing path/asset in the archives and it would load from there first. Texture and mesh modding was very simple, to the point that you could toggle such mods on and off while the game was running.

[BastianW]

Didn´t played TQ, sorry. But I does some mods for FO2. The problem here was, that you need to rewrite the complete game (ok, it wasn´t build to be modded) and we got a lot of questions then how to restore the game if the mod was played or tested. The only option then was to reinstall it (or create a backup from the modded files). So for Wasteland2 I would prefer a better way from the beginning .

The idea behind the loader itself is, that the community could build there own stuff and company like Nexusmods (http://www.nexusmods.com/) could build also maps without the need to get in contact to other community's.

But maybe some ideas could be taken then from TQ. So the wheel musn´t be completely reinvented here.

[Sxerks]

Custom maps should actually be "packaged" in zip files.

The in-game custom scenario loading menu would just list the info from the MapDescription.xml in each archive, it might not be necessary to unpack the zip to play.

*install directory
**source....and whatnot
**custom
***tutorial.zip
***my_map.zip
***my_really_long_named_file_that_is_cool.zip
****(inside zip)
****MapDescription.xml
****map
****objects(2d/3d?)
*****object textures
****dialog text

Level Editor - Custom Scenarios thread in "What to Include" Forum

[BastianW]

Hm, why a ZIP file? There is no way to read content from ZIP files without extracting them, so the game couldn´t use the content from a zip file so it must be extracted to a temp folder (= there would be a loading screen where the content is expanded, or the game said there is not enough HD to play it) or the local RAM (= which means some PCs might require a RAM upgrade to hold the content from the ZIP file in the RAM).

So I didn´t see here a benefit from a ZIP file, or is there a good reason for the idea to keep the stuff in a ZIP file which I didn´t see?

Edit:
A technique which we (I´m working for a developer company) used in the past was, that we store some non often used elements in a ZIP file. So the game didn´t use so much HD space during the installation. So if not played it saved disc space, and if you start it, it extracted some elements into the temp folder. BUT we learned that many user then have only 50MB free disk space, but the game required 150MB so they couldn´t play the game, and the game quit with a error message. I think in the current "modern" time there should be always enough HD space for such games. For the user it is quite more important to play the game without annoying loading sections (where the sub map is extracted to the temp folder).

Another problem what we saw when we give up the ZIP idea was, that many users who play game have a file transfer in place. Before they start the game they had enough space for the file transfer, so they started it. If they play the game the file transfer finished in the background, the game must then load (and extract a new zipped map) AND it crashed because there is not enough space on the HD. So we implemented a option in the game to constantly check the disk space we need. But that whas also annoying for the user, because if he run out they couldn´t play the new map they entered, because there is not enough free HD discspace to extract the "in house map". So we implemented a kind of intelligent extract option to the game. It monitored the space and delete map content which are no longer needed and extract the one which was needed (BUT it was VERY expensive due to the development hours wasted for that!).

So from my point I would really avoid to ZIP some content. For bigger games where you have a bigger budget you can implement some features which save the game before the user enter a new map, and if there is not enough space, the game wouldn´t crash, and you can also implement a monitoring task which constantly check if there is enough space for the game. But for games which such a small budget I would not implement any space monitoring functions.

[suz]

Hm, why a ZIP file? There is no way to read content from ZIP files without extracting them

Even the most basic game engines implement a VFS which usually automagically read archives.
Most zip libraries support unpacking to memory. All textures you load from HDD are loaded into memory either way, so there's no additional significant memory footprint.

Even a renderer - Ogre3D has on-the-fly decompression of models, textures and other assets from zip files.

And the advantage of zip files and proper VFS is obvious to anyone who's done modding - segregation of mods and easier conflict handling.

[BastianW]

All textures you load from HDD are loaded into memory either way, so there's no additional significant memory footprint.

hm, maybe I wasn´t clear. I didn´t really talk about memory footprints, more the technique which is needed to build something which can be used to extract the textures just in time and the money which it cost. If the engine support that its ok, but I wouldn´t wast money to implement that. It isn´t really needed, or is there any kind of benefit from compressed texture files?

In order to load the textures into the RAM (for the game to play) you need to extract the ZIP content somewhere (to the RAM or to the disc [I think everyone would agree here with me]). Then you need to decide if you keep the extracted content somewhere (on disk or RAM) or throw it away (because the content, the textures where loaded into RAM).

Maybe a example would help to understand what I mean:
If the PC has only 512 MB RAM (maybe the gamer is using a VMWare or a old PC) then you couldn´t keep all the textured the game use in RAM.

So: If the gamer visit a snow area, the snow textures are extracted (to RAM or disc) and then the snow textures are loaded into RAM. The gamer then visit a cave on the snow map.

So what happen here now? The RAM is to low to keep the cave AND the snow area in the RAM. So the game itself must remove the snow area textures from RAM and extract the cave textures (to RAM or disc) and load the cave textures into RAM. But what will happen with the extracted textures (snow and cave)? Will you keep them in the (RAM or disc)? What happen if the user visit another area (like a building in the cave) will you then extract another texture map?

Sure you can build a loader, that every time the user visit a new map, all old maps are removed from disc and RAM and the new map is loaded then. Or you must program a kind of KI which say: If the user visit a cave, he will come back very shortly, so I will not remove the snow map textures, in order to minimize the load time.

I really hope you got my point here. I´m not against ZIP files, I´m more concerned that there are a lot of "problems" which must be considered before it should be implemented. And because of the game budget I really wouldn´t implement a map loader which comes up with a kind of KI to extract some zipped textures.

So my question is still open. If you compress a map, whats the benefit from it? If we later one build our own maps, we wouldn´t allow the user to download each file. They can download a *.exe which has maybe a installer who put the map into the correct folder. So i couldn´t really see a benefit here from compressing the complete texture folder...

[suz]

All textures you load from HDD are loaded into memory either way, so there's no additional significant memory footprint.

hm, maybe I wasn´t clear. I didn´t really talk about memory footprints, more the technique which is needed to build something which can be used to extract the textures just in time and the money which it cost. If the engine support that its ok, but I wouldn´t wast money to implement that. It isn´t really needed, or is there any kind of benefit from compressed texture files?

This is way out of scope for a forum post but for the sake of understanding...

In order to load the textures into the RAM (for the game to play) you need to extract the ZIP content somewhere (to the RAM or to the disc [I think everyone would agree here with me]). Then you need to decide if you keep the extracted content somewhere (on disk or RAM) or throw it away (because the content, the textures where loaded into RAM).

Extracting to disk only gives additional IO overhead. You extract only relevant items to RAM. You don't throw anything away because you don't decompress stuff you don't need.

Maybe a example would help to understand what I mean:
If the PC has only 512 MB RAM (maybe the gamer is using a VMWare or a old PC) then you couldn´t keep all the textured the game use in RAM.

You don't need to. Only currently used textures are kept in RAM. If there's no demand for RAM then the cached resources used on last map are still there and you'll get them without loading once again, whether by engine caching or by OS caching.

So: If the gamer visit a snow area, the snow textures are extracted (to RAM or disc) and then the snow textures are loaded into RAM. The gamer then visit a cave on the snow map.

So what happen here now? The RAM is to low to keep the cave AND the snow area in the RAM. So the game itself must remove the snow area textures from RAM and extract the cave textures (to RAM or disc) and load the cave textures into RAM. But what will happen with the extracted textures (snow and cave)? Will you keep them in the (RAM or disc)? What happen if the user visit another area (like a building in the cave) will you then extract another texture map?

The same thing happens when you have demand for more RAM than you have, unused resources get unloaded, if that's not enough some memory gets paged out, if that's not enough then your OS kills offending applications.

Sure you can build a loader, that every time the user visit a new map, all old maps are removed from disc and RAM and the new map is loaded then. Or you must program a kind of KI which say: If the user visit a cave, he will come back very shortly, so I will not remove the snow map textures, in order to minimize the load time.

Unless the game engine or the OS decides they're low on memory the unused resources stay in RAM as cache.

I really hope you got my point here. I´m not against ZIP files, I´m more concerned that there are a lot of "problems" which must be considered before it should be implemented. And because of the game budget I really wouldn´t implement a map loader which comes up with a kind of KI to extract some zipped textures.

Because of the relatively low game budget they're going for an established game engine that most likely already provides all these features, in what particular way - your guess is as good as mine, however resource management is very high on game engine's priority and resource packing/caching/prefetching is pretty important.

So my question is still open. If you compress a map, whats the benefit from it? If we later one build our own maps, we wouldn´t allow the user to download each file. They can download a *.exe which has maybe a installer who put the map into the correct folder. So i couldn´t really see a benefit here from compressing the complete texture folder...

I already said, it provides a way to segregate mods.

Let's assume for a sec morrowind/oblivion/skyrim/FO3/FONV are not using archives. You download a mod, with ~200 files in zip or self extracting archive of any kind. To install you copy all files into the data directory overwriting game's vanilla resources. To uninstall the mod you now need to reinstall the game, fun stuff.

It doesn't really HAVE to be zip files, even just separate directory trees per mod would suffice to separate them enough, however modern engines support resource packing.

I would consider this to be the best way to segregate mods:

Using creation date as a method to prevent conflicts - oldest file overrides all earlier files.

[BastianW]

hm, could it be maybe that we speak from two different stuff here:

- a complete zipped map folder (including textures) [*.zip]
- Image compression formats [seeOpenGL:Image compression]

Image compression is a wounder full and great idea, and (as already mentioned) this is up to the engine and I´m not against that. But zipping a complete map folder (*.zip) is something which must be handled by the game itself then (not the engine). And I think the budget is to small to implement such a idea since it includes a kind of loader (as mentioned above).

The same thing happens when you have demand for more RAM than you have, unused resources get unloaded, if that's not enough some memory gets paged out, if that's not enough then your OS kills offending applications.

Yes they are paged out which then also caused some IO on the HD, similar like extracting the zipped texture folder to a temp folder. So I didn´t really see here a a benefit from the zipped map folder.

It doesn't really HAVE to be zip files, even just separate directory trees per mod would suffice to separate them enough, however modern engines support resource packing.

That was the idea from that topic ... a "Useful directory structure for mods" . So with or without ZIP support we are hopefully on the same side

But anyway ... at long as we get a structure which didn´t force the gamer to re-install the game to remove the mod I think everybody would be happy with that. With or without ZIP files ... But from my point of view including zipped textures/maps folders didn´t make sense with that small budge.

[suz]

hm, could it be maybe that we speak from two different stuff here:

No, I'm talking of a complete, segregated directory structure archived into a single file that's ready to be downloaded

budget is to small to implement such a idea since it includes a kind of loader (as mentioned above).

Again, they don't write engine from scratch, budget is irrelevant, most engines have their own resource loaders built in.

Yes they are paged out which then also caused some IO on the HD, similar like extracting the zipped texture folder to a temp folder. So I didn´t really see here a a benefit from the zipped map folder.

Paging has no relevance whether you compress files or not. No memory -> paging -> oomkiller, in both a bunch of files and one compressed file.

But anyway ... at long as we get a structure which didn´t force the gamer to re-install the game to remove the mod I think everybody would be happy with that. With or without ZIP files ... But from my point of view including zipped textures/maps folders didn´t make sense with that small budge.

Again, budget is irrelevant.

Bethesda's engine used in games listed above already has a loader that loads from either 1) Files from data folder or 2) From the *.bsa archives.
The problem with that method is that multiple mods that have conflicts (e.g. 2 mods that affect supermutant model) are resolved via file replacement. Meaning you can't simply switch between the mods but you need stuff like OBMM/FOMM and the likes.

[Sxerks]

First, from the original post, the Map loader should be built into the game and be fully functional at release(not an external program). It would be a simple in-game menu with a directory listing where you choose a map, it displays the map info and with the click of a button it then UNPACKs it into your profile under "loaded scenario", you can still play the original game, this also eliminates any problem with performance(ram IO....).

There is no need for a mod-downloader, as people will get packaged maps from whatever site they want and just copy them into the base "Custom Scenarios" folder.


There are really 2 types of mods, overrides and maps

Overrides- supersede data, like Baldur's Gate
*Base Overrides- the base folder override would supersede normal data across all profiles, used for custom fixes and whatnot.
*Profile Overrides- would only supersede data for that person, in the example above one person has a sound/music override and the other has a mesh/texture override.

Maps(Custom Scenarios)
*Custom Scenarios - where the downloaded packed (zipped) maps are stored.
*Loaded Scenario - where the chosen map is unpacked for ease of play

Game saves would be stored in separate folders in each profile

[BastianW]

The idea why the map uploader shouldn´t be part from the game itself is, that company's like NexusMods could add Wastland 2 to THERE "Mod Manager".

So every community could build there own map uploader if needed. Then such stuff like::

- Downloading the maps (resume, pause, ...)
- mod uninstallation
- updating content (maybe somebody release a new map)
- check if there is HD disc space is in place
- Check if the ZIP file is damaged
- Check if there is a naming conflict
- Enough Access to the extraction folder? e.g. if the user isn´t an Administrator, this might be a issue then.
- ...

can be handled by the specified "Mod Manager". And if one community wish to work with ZIP files they can without the need to implement a "Mod Manager".

And ingame you can then only select the map you wish to play. So less work for the game developer and more content for the game itself.

[Sxerks]

The "map uploader"???? The map installer should be part of the game, I would be disappointed in the devs if it wasn't.

You could still have a map downloader that does all the things you stated (except for un/installing), and it could just save it to the "Custom Scenarios" directory.

The game itself would be more efficient at dealing with the installed files than an external loader that would have to back-up files that are being replaced.


Also, looking at the Nexus site, you need to be logged in to download a file.

[Biomehanika]

The "map uploader"???? The map installer should be part of the game, I would be disappointed in the devs if it wasn't.

Hu? A map installer is part of the game? I thought they said that they would made the game "mod friendly". That didn´t include a ingame installer in my eyes. If they will include that, they must support mods as well. otherwiese it is only half half ... a map installer but no mod tools ...

I personally prefer a bigger world instead a ingame map installer (with all the strange features you need). If you need one, then you can write one. But please do not push the inXile developers to wast money for a map installer. I think thats defenetly something for the comunity. Games like Skyrim have also no ingame map installer but have strong mod support.

And why did you push that "Zip file to a special folder" idea so much!?!? I really hate that crappy idea in other games . For some games I play I need to download a 5 GB new map!!!! This takes hours here in the location where I live. And if the map developer fixed a minor issue ... I´m force to download the complete crap again, only that I have the complete file and the ingame installer would recorgnize it as an update?

So please do not force me to download 5 GB files with my dsl 3mbps connection only because of the ingame map installer. Not every one live in the countrys where you guy have such a big infrastructure that you can download 5 GB files in some seconds. For some people outside in the world this ZIP file would take hours and hours again. And if the file is almost finished the connection breaks ... and you start again by zero.

I would prefer to put the files by hand in the folders where they belong to. So if there is a minor fix, I could get that file by a friend and put that into the folder. Simple and easy ... Or if you need a new map, download the zip and then extract the complete zip into the maps folder. Everybody can do that, there is no installer needed which does nothing more then a extraft from a file which I put into a special folder, thats wasted from the money.

@Sexerks
oh and I´m using the NexusMod downloader and it is a really big help (at least the resume function is a big benefit for me), and sure you need a login to use there tool. The downloader is build only for the NexusMods comunity, they save there links for piraty . So nobody could point in other forum to the zip file on the NexusMods Server and the owner from the NexusMods website need to pay the traffic bill because every one is downloading the stuff from there server but didn´t visit the advertisements who pay the bills . The complete service they offer is payed via advertisements, thats why they force you to create a login ... I think thats ok ... you musn´t click on the advertisements in the forum if you didn´t like. You need only a login. Thats all.

[BastianW]

The "map uploader"???? The map installer should be part of the game, I would be disappointed in the devs if it wasn't.

They didn´t announced mod support yet (or did they?). So what I would have here (from the beginning) is a clear directory structure, where we can extract a zip file into the maps folder. Nothing more ... so if there would be no money for mod tools we could then build something as a community because the game has the structure we can use to build mods.

[Sxerks]

@Biomehanika
They were mainly talking about the amount of money and time it would take to create "modding TOOLS". A map installer, would take little time/money to implement and would be done by a coder not by level designers/scripters so it would have no effect on the expanded size of the world. By having an in-game installer, that would make it mod friendly, even if they don't release a full set of tools.

5GB map??? what are you playing??? is it mostly CG videos??? the largest I've ever installed was 500MB. I doubt a WL2 map would even be that large. I don't have a fast connection either and have downloaded mods multiple times as well. How a mod is released is really up to the individual makers, so if they make a small fix, that's great. If you look at the directory structure in the image I posted, you can see that a small fix can be added to a specific folder. Not everyone can(or knows how to) extract Zips into directories.

After you explained how the NexusMods comunity works, I like it even LESS.

@BastianW
They really haven't committed to anything yet, even the engine.

And, yes, a clear directory structure is important, but dumping files into the main folders is an old/backwards/sloppy way of dealing with mods(overwriting existing files).

[Zeful]

First, from the original post, the Map loader should be built into the game and be fully functional at release(not an external program). It would be a simple in-game menu with a directory listing where you choose a map, it displays the map info and with the click of a button it then UNPACKs it into your profile under "loaded scenario", you can still play the original game, this also eliminates any problem with performance(ram IO....).

There is no need for a mod-downloader, as people will get packaged maps from whatever site they want and just copy them into the base "Custom Scenarios" folder.


There are really 2 types of mods, overrides and maps

Overrides- supersede data, like Baldur's Gate
*Base Overrides- the base folder override would supersede normal data across all profiles, used for custom fixes and whatnot.
*Profile Overrides- would only supersede data for that person, in the example above one person has a sound/music override and the other has a mesh/texture override.

Maps(Custom Scenarios)
*Custom Scenarios - where the downloaded packed (zipped) maps are stored.
*Loaded Scenario - where the chosen map is unpacked for ease of play

Game saves would be stored in separate folders in each profile

This is literally how Titan's Quest handles mods in the broad strokes. For texture and mesh mods, you simply had to know where to place them in the file structure, because the program looked for non-archived versions of archive files first.

This image shows the file structure required, but the game will load those textures instead of the ones in the proprietary archives, and could be done for anything. Want a gun, as long as you have a model, texture, and animations and placed them properly to override the archive, you'd have a gun. It was a little arcane as you have to go into the game's file structure yourself to do this, but it worked well.

For custom maps, you simply had to go into the far more readily accessible My Documents/My Games folders where TQ held your save information, create a folder named "CustomMaps", and make a folder for each map with all the level information, the end result looking like:


Which allowed you to use the "Play Custom Game" option on the title screen:

Bringing you to this screen:

Where you could make characters for custom maps, legacy them across them, and import characters from the default campaign.

And the game came with a full set of tools to allow modding, including various compilers (presumably to make your own archives so you can cut down some on space the mod takes up, the Lilith Mod was 660megs and it's full 3d).

Is this what you want when talking about "Mod-friendly"?

[Biomehanika]

@Biomehanika
They were mainly talking about the amount of money and time it would take to create "modding TOOLS". A map installer, would take little time/money to implement and would be done by a coder not by level designers/scripters so it would have no effect on the expanded size of the world. By having an in-game installer, that would make it mod friendly, even if they don't release a full set of tools.

Ok, and the coder didn´t get then any money ... he is working for free? Sorry ... EVERY stuff which is added to the game would lead into a smaller world. Because the coder will get money from the Kickstarter budge too ... nobody works for free!

If you look at the directory structure in the image I posted, you can see that a small fix can be added to a specific folder. Not everyone can(or knows how to) extract Zips into directories.

But this is exactly needed. Not every one like to use a ingame installer and downloading many GBs of Zip files. So it must be able to do both: "AutoInstallation" and the "extraction from the complete zip file into the maps folder". So that comunitys like NexusMods could join the mod generations with "there" map downloader.

@Biomehanika
5GB map??? what are you playing??? is it mostly CG videos??? the largest I've ever installed was 500MB.

Oh dude ... 500 MB ... the time is almost over for that. Check out the Skyrim HD patch for example (it is 1,29 GB in size COMPRESSED!!!)

http://skyrim.nexusmods.com/downloads/f ... 07#content

After you explained how the NexusMods comunity works, I like it even LESS.

Only because of the registration option? wow ... so why didn´t you come up with your own webshare, where the complete world can put there mods onto (without registration oh and please without captcha)? You need to check then every file they upload by hand, if they didn´t abuse your host to store MP3s or other crap. I currently own a small frum and have 2 GB space für 20 Euro per months. In order to setup another comunity on your own I would say you need more space then 2 GB ... say 20 GB (Website, Forum & Downloads). And that would then (because it is easy to calculate) cost arround 200 Euro per months. And you would give that for free to the comunity? That would be nice, so please come up with that comunity and I will change from NexusMods to your one (but only because of the fileshare, I didn´t need your forum).

[suz]

would lead into a smaller world.

I increased my pledge in the last hours because we were so close to $3m, I'm sure I'm not the only one who has done so because of the mod kit alone.

But this is exactly needed. Not every one like to use a ingame installer and downloading many GBs of Zip files. So it must be able to do both: "AutoInstallation" and the "extraction from the complete zip file into the maps folder". So that comunitys like NexusMods could join the mod generations with "there" map downloader.

Dude are you even reading the posts?
Size of the archives has zero relation to how hard or easy it is to install. Nexus Mod Manager which you advertise so hard has been doing exactly that, download an entire 1.29GB archive all over again every update.

Oh dude ... 500 MB ... the time is almost over for that. Check out the Skyrim HD patch for example (it is 1,29 GB in size COMPRESSED!!!)

Textures are not a map.