Game Indexing and Metadata Enrichment
Game Indexing
The server watches the files directory (by default /files) and creates an index of all games, when changes occur. The indexing can also be triggered via API.
The indexing algorithm follows these steps:
-
It recursively scans all games in the files directory.
-
For each game:
- It checks if the exact same game already exists in the database, either based on the file path or a combination of the title and release year. If it does, the game is skipped.
- It checks if the game exists in the database but was previously deleted, again based on the file path or a combination of the title and release year. If so, it restores the game and updates its information with the current file's details, such as size, title, year, early access flag, path, and version.
- It checks if the game already exists in the database. If so, it updates the game's information with the current file's details.
- If the game does not exist in the database at all, it indexes the game.
Finally, a simple integrity check is performed to ensure that the games in the database also exist in the files directory, and vice versa.
Retrieving and Storing Game Information
Integration with RAWG
Matching Games
After adding games, the server attempts to find the corresponding game in the RAWG database. It looks for games on RAWG with similar titles and release years to establish a match. Once a match is found, it becomes the official mapping for that game, and any changes to the mapping must be done manually.
Disabling Integration for a single Game
To avoid GameVault looking up your game on RAWG and matching it (for example if it's your personal indie game or a game not available on RAWG), you can use the NO-CACHE-Flag (NC) in the file path, as explained here.
RAWG API Key
To use the RAWG integration feature, users can set up a RAWG API Key by configuring the RAWG_API_KEY environment variable. The API Key can be obtained from the RAWG Developer Portal. The free tier of the API should be sufficient for most users.
If you encounter limitations with your RAWG API Key, you have the option to upgrade your account on RAWG or contact us for assistance.
Probability Logs for Game Matching
When the SERVER_LOG_LEVEL environment variable is set to DEBUG, the server logs all probability information during the game matching process. These logs can be helpful in resolving any mapping issues with games.
Caching Game Information
Cached Metadata
The server automatically caches all metadata obtained from RAWG for each game. If a game is already cached, the cache is updated if necessary. The cached metadata includes detailed descriptions, screenshots, release dates, as well as information about developers, publishers, genres, tags, images, and stores.
GameVault relies heavily on RAWG integration to enhance the game data.
Cache Expiry
Cached metadata is stored in the database permanently and remains valid for a configurable period of time. The duration can be set using the RAWG_API_CACHE_DAYS environment variable, with a default validity period of 30 days. The purpose of caching is to reduce the number of RAWG API calls. If you encounter rate limits with your RAWG API Key, you can extend the validity duration. Game metadata does not change frequently, so longer validity periods are generally acceptable.
RAWG Platforms
The following table provides an overview of platforms supported by RAWG, along with their corresponding ID:
| ID | Platform | Description |
|---|---|---|
| 1 | Xbox One | Eighth-generation home video game console by Microsoft |
| 3 | iOS | Mobile operating system developed by Apple Inc. |
| 4 | PC | General-purpose computing platform |
| 5 | macOS | Operating system for Apple Macintosh computers |
| 6 | Linux | Open-source Unix-like operating system kernel |
| 7 | Nintendo Switch | Hybrid video game console by Nintendo |
| 8 | Nintendo 3DS | Handheld game console by Nintendo |
| 9 | Nintendo DS | Dual-screen handheld game console by Nintendo |
| 10 | Wii U | Eighth-generation home video game console by Nintendo |
| 11 | Wii | Seventh-generation home video game console by Nintendo |
| 12 | Neo Geo | Family of video game hardware and software |
| 13 | Nintendo DSi | Handheld game console by Nintendo |
| 14 | Xbox 360 | Seventh-generation home video game console by Microsoft |
| 15 | PlayStation 2 | Sixth-generation home video game console by Sony |
| 16 | PlayStation 3 | Seventh-generation home video game console by Sony |
| 17 | PSP | Handheld game console by Sony |
| 18 | PlayStation 4 | Eighth-generation home video game console by Sony |
| 19 | PS Vita | Handheld game console by Sony |
| 21 | Android | Mobile operating system based on the Linux kernel |
| 22 | Atari Flashback | Series of dedicated video game consoles |
| 23 | Atari 2600 | Video game console by Atari |
| 24 | Game Boy Advance | Handheld game console by Nintendo |
| 25 | Atari 8-bit | Series of 8-bit home computers by Atari |
| 26 | Game Boy | Handheld game console by Nintendo |
| 27 | PlayStation | Fifth-generation home video game console by Sony |
| 28 | Atari 7800 | Video game console by Atari |
| 31 | Atari 5200 | Video game console by Atari |
| 34 | Atari ST | Home computer by Atari |
| 41 | Apple II | Series of home computers by Apple |
| 43 | Game Boy Color | Handheld game console by Nintendo |
| 46 | Atari Lynx | Handheld game console by Atari |
| 49 | NES | Third-generation home video game console by Nintendo |
| 50 | Atari XEGS | Video game console by Atari |
| 55 | Classic Macintosh | Classic Personal computer by Apple |
| 74 | SEGA Master System | Third-generation home video game console by SEGA |
| 77 | Game Gear | Handheld game console by SEGA |
| 79 | SNES | Fourth-generation home video game console by Nintendo |
| 80 | Xbox | Sixth-generation home video game console by Microsoft |
| 83 | Nintendo 64 | Fifth-generation home video game console by Nintendo |
| 105 | GameCube | Sixth-generation home video game console by Nintendo |
| 106 | Dreamcast | Sixth-generation home video game console by SEGA |
| 107 | SEGA Saturn | Fifth-generation home video game console by SEGA |
| 111 | 3DO | Interactive multiplayer video game console |
| 112 | Jaguar | Fifth-generation home video game console by Atari |
| 117 | SEGA 32X | Add-on for the SEGA Genesis/Mega Drive |
| 119 | SEGA CD | Add-on for the SEGA Genesis/Mega Drive |
| 166 | Commodore / Amiga | Series of home computers and game consoles |
| 167 | Genesis | Fourth-generation home video game console by SEGA |
| 171 | Web | Games that can be played with a web browser |
| 186 | Xbox Series S/X | Ninth-generation home video game consoles by Microsoft |
| 187 | PlayStation 5 | Ninth-generation home video game console by Sony |
RAWG Stores
The following table provides an overview of stores supported by RAWG, along with their corresponding ID:
| ID | Store | Description |
|---|---|---|
| 1 | Steam | Digital distribution platform for PC gaming |
| 2 | Xbox Store | Microsoft's digital distribution platform |
| 3 | PlayStation Store | Sony's digital storefront for PlayStation games |
| 4 | App Store | Apple's digital distribution platform for iOS |
| 5 | GOG | DRM-free digital distribution platform |
| 6 | Nintendo Store | Nintendo's official online store |
| 7 | Xbox 360 Store | Microsoft's digital store for Xbox 360 games |
| 8 | Google Play | Google's digital distribution platform for Android |
| 9 | Itch.io | Indie game marketplace and distribution platform |
| 11 | EPIC Games | Digital distribution platform by Epic Games |
Alternative Database Integration
In the event that RAWG discontinues its service, GameVault will retain the latest metadata and work on integrating another video game database as a replacement.
Disabling RAWG Integration
To turn off all calls to the RAWG API, utilize the TESTING_RAWG_API_DISABLED environmental variable. Omitting the RAWG_API_KEY value will also disable RAWG integration.
Game Box Arts
GameVault utilizes a simple Google Image Search library to find suitable box arts for games. This is necessary because RAWG does not provide box arts directly.
Google Image Search
Finding Box Arts
GameVault uses the following algorithm to find box arts:
- Searches for
[GAME TITLE] site:steamgriddb.com -profile. - If no results are found, searches for
[GAME TITLE] site:www.pcgamingwiki.com. - If no results are found, searches for
[GAME TITLE] game box art. - Selects the first available image that matches the target aspect ratio 2:3.
Rate Limiting and Cooldown
When adding or updating a large number of games (more than 100), there might be rate limits imposed by Google on your IP address. In such cases, GameVault implements a cooldown mechanism to manage the rate limit.
The cooldown works as follows:
- The cooldown duration is determined by the
IMAGE_GOOGLE_API_RATE_LIMIT_COOLDOWN_IN_HOURSconfiguration value, expressed in hours. - If no box art images are found for multiple games, indicating a potential Google Image Search rate limit, the cooldown is activated for the specified duration.
- The cooldown prevents further box art searches until the cooldown duration has elapsed.
- Once the cooldown is activated, a warning message is logged, whenever gamevault tries to search for new boxarts, indicating the remaining cooldown time.
Disabling Google Image
If you rather set your box-arts yourself and want to completely disable the Google Image Search feature, you can use the TESTING_GOOGLE_API_DISABLED environment variable. This will block all Google requests.
Image Storage
GameVault stores all images on the filesystem under the images directory (by default /images).
For more information on GameVault's image management, refer to this page.