Skip to main content

Game Indexing and Information Retrieval

Game Indexing

The server watches the files directory (by default /files) and creates an index of all games, when changes occur. The indexing can also be triggered via API.

The indexing algorithm follows these steps:

  • It recursively scans all games in the files directory.

  • For each game:

    • It checks if the exact same game already exists in the database, either based on the file path or a combination of the title and release year. If it does, the game is skipped.
    • It checks if the game exists in the database but was previously deleted, again based on the file path or a combination of the title and release year. If so, it restores the game and updates its information with the current file's details, such as size, title, year, early access flag, path, and version.
    • It checks if the game already exists in the database. If so, it updates the game's information with the current file's details.
    • If the game does not exist in the database at all, it indexes the game.

Finally, a simple integrity check is performed to ensure that the games in the database also exist in the files directory, and vice versa.

Retrieving and Storing Game Information

Integration with RAWG

Matching Games

After adding games, the server attempts to find the corresponding game in the RAWG database. It looks for games on RAWG with similar titles and release years to establish a match. Once a match is found, it becomes the official mapping for that game, and any changes to the mapping must be done manually.

Disabling Integration for a single Game

To avoid GameVault looking up your game on RAWG and matching it (for example if it's your personal indie game or a game not available on RAWG), you can use the NO-CACHE-Flag (NC) in the file path, as explained here.

RAWG API Key

To use the RAWG integration feature, users can set up a RAWG API Key by configuring the RAWG_API_KEY environment variable. The API Key can be obtained from the RAWG Developer Portal. The free tier of the API should be sufficient for most users.

tip

If you encounter limitations with your RAWG API Key, you have the option to upgrade your account on RAWG or contact us for assistance.

Probability Logs for Game Matching

When the SERVER_LOG_LEVEL environment variable is set to DEBUG, the server logs all probability information during the game matching process. These logs can be helpful in resolving any mapping issues with games.

Caching Game Information

Cached Metadata

The server automatically caches all metadata obtained from RAWG for each game. If a game is already cached, the cache is updated if necessary. The cached metadata includes detailed descriptions, screenshots, release dates, as well as information about developers, publishers, genres, tags, images, and stores.

GameVault relies heavily on RAWG integration to enhance the game data.

Cache Expiry

Cached metadata is stored in the database permanently and remains valid for a configurable period of time. The duration can be set using the RAWG_API_CACHE_DAYS environment variable, with a default validity period of 30 days. The purpose of caching is to reduce the number of RAWG API calls. If you encounter rate limits with your RAWG API Key, you can extend the validity duration. Game metadata does not change frequently, so longer validity periods are generally acceptable.

RAWG Platforms

The following table provides an overview of platforms supported by RAWG, along with their corresponding ID:

IDPlatformDescription
1Xbox OneEighth-generation home video game console by Microsoft
3iOSMobile operating system developed by Apple Inc.
4PCGeneral-purpose computing platform
5macOSOperating system for Apple Macintosh computers
6LinuxOpen-source Unix-like operating system kernel
7Nintendo SwitchHybrid video game console by Nintendo
8Nintendo 3DSHandheld game console by Nintendo
9Nintendo DSDual-screen handheld game console by Nintendo
10Wii UEighth-generation home video game console by Nintendo
11WiiSeventh-generation home video game console by Nintendo
12Neo GeoFamily of video game hardware and software
13Nintendo DSiHandheld game console by Nintendo
14Xbox 360Seventh-generation home video game console by Microsoft
15PlayStation 2Sixth-generation home video game console by Sony
16PlayStation 3Seventh-generation home video game console by Sony
17PSPHandheld game console by Sony
18PlayStation 4Eighth-generation home video game console by Sony
19PS VitaHandheld game console by Sony
21AndroidMobile operating system based on the Linux kernel
22Atari FlashbackSeries of dedicated video game consoles
23Atari 2600Video game console by Atari
24Game Boy AdvanceHandheld game console by Nintendo
25Atari 8-bitSeries of 8-bit home computers by Atari
26Game BoyHandheld game console by Nintendo
27PlayStationFifth-generation home video game console by Sony
28Atari 7800Video game console by Atari
31Atari 5200Video game console by Atari
34Atari STHome computer by Atari
41Apple IISeries of home computers by Apple
43Game Boy ColorHandheld game console by Nintendo
46Atari LynxHandheld game console by Atari
49NESThird-generation home video game console by Nintendo
50Atari XEGSVideo game console by Atari
55Classic MacintoshClassic Personal computer by Apple
74SEGA Master SystemThird-generation home video game console by SEGA
77Game GearHandheld game console by SEGA
79SNESFourth-generation home video game console by Nintendo
80XboxSixth-generation home video game console by Microsoft
83Nintendo 64Fifth-generation home video game console by Nintendo
105GameCubeSixth-generation home video game console by Nintendo
106DreamcastSixth-generation home video game console by SEGA
107SEGA SaturnFifth-generation home video game console by SEGA
1113DOInteractive multiplayer video game console
112JaguarFifth-generation home video game console by Atari
117SEGA 32XAdd-on for the SEGA Genesis/Mega Drive
119SEGA CDAdd-on for the SEGA Genesis/Mega Drive
166Commodore / AmigaSeries of home computers and game consoles
167GenesisFourth-generation home video game console by SEGA
171WebGames that can be played with a web browser
186Xbox Series S/XNinth-generation home video game consoles by Microsoft
187PlayStation 5Ninth-generation home video game console by Sony

RAWG Stores

The following table provides an overview of stores supported by RAWG, along with their corresponding ID:

IDStoreDescription
1SteamDigital distribution platform for PC gaming
2Xbox StoreMicrosoft's digital distribution platform
3PlayStation StoreSony's digital storefront for PlayStation games
4App StoreApple's digital distribution platform for iOS
5GOGDRM-free digital distribution platform
6Nintendo StoreNintendo's official online store
7Xbox 360 StoreMicrosoft's digital store for Xbox 360 games
8Google PlayGoogle's digital distribution platform for Android
9Itch.ioIndie game marketplace and distribution platform
11EPIC GamesDigital distribution platform by Epic Games

Alternative Database Integration

In the event that RAWG discontinues its service, GameVault will retain the latest metadata and work on integrating another video game database as a replacement.

Disabling RAWG Integration

To turn off all calls to the RAWG API, utilize the TESTING_RAWG_API_DISABLED environmental variable. Omitting the RAWG_API_KEY value will also disable RAWG integration.

Game Box Arts

GameVault utilizes a simple Google Image Search library to find suitable box arts for games. This is necessary because RAWG does not provide box arts directly.

Finding Box Arts

GameVault uses the following algorithm to find box arts:

  1. Searches for "GAME-TITLE" inurl:steamgriddb.com.
  2. If no results are found, searches for "GAME-TITLE" game box art.
  3. Selects the first available image that matches the target aspect ratio.

Rate Limiting and Cooldown

When adding or updating a large number of games (more than 100), there might be rate limits imposed by Google on your IP address. In such cases, GameVault implements a cooldown mechanism to manage the rate limit.

The cooldown works as follows:

  • The cooldown duration is determined by the IMAGE_GOOGLE_API_RATE_LIMIT_COOLDOWN_IN_HOURS configuration value, expressed in hours.
  • If no box art images are found for multiple games, indicating a potential Google Image Search rate limit, the cooldown is activated for the specified duration.
  • The cooldown prevents further box art searches until the cooldown duration has elapsed.
  • Once the cooldown is activated, a warning message is logged, whenever gamevault tries to search for new boxarts, indicating the remaining cooldown time.

Disabling Google Image

If you rather set your box-arts yourself and want to completely disable the Google Image Search feature, you can use the TESTING_GOOGLE_API_DISABLED environment variable. This will block all Google requests.

Image Storage

Storing Box Arts

To ensure availability, performance, and efficient compression, GameVault stores the found box arts on the filesystem under the images directory (by default /images).

For more information on GameVault's image management, refer to this page.

Customizing Storage Path

The storage path for images can be customized by modifying the VOLUMES_IMAGES environment variable.