Asset Indexing¶

Asset indexing provides an optimized way to load asset representations into the asset system. With asset indexing, users have to spend significantly less time waiting for asset libraries to be loaded and ready to use.

Information about the asset like its asset meta-data (link), name and data-block type is extracted from asset files and stored in asset index files. Asset indexing can observe the content of asset libraries to perform minimal updates to the asset index files.

The asset index files are stored in a local cache of the user's operating system.

Glossary¶

Glossary used within the scope of this page.

Asset: Data-blocks that have been tagged by users to be an asset.
Asset File: Blend file which is located inside an asset library.
Asset Library: See Asset Library (Wiki).
Asset Index File: JSON file containing information limited to selecting and browsing about assets contained inside a single asset file.
Asset Catalog: See Asset Catalos.
Asset representation: See Asset Representation.
File Browser: Editor in blender used for navigating file system and finding data-blocks in blend files. Used for linking, appending, file open, import, save and asset browsing functionalities.
Data-block: See ID Datablocks (Wiki).

Logical Overview¶

Problem statement¶

Assets are stored in a blend file as part of an asset library. In order to browse through the assets of an asset library Blender has to parse each asset file in order to know which data-blocks have been tagged as assets. Doing this each time when browsing assets and having large asset libraries extracting the assets can take a long time.

Blender loads asset files sequentially in a background thread, which is a slow operation.
Asset libraries can be located on network shares and version controlled repositories.
Asset libraries can have a large number of files.
Asset files can be large is size.
Asset files can be compressed.
Asset files may not actually contain assets.

Solution¶

To reduce the time needed to browse asset libraries we want to extract the assets representation from the asset files and store them in the asset index. When the asset file isn’t changed, the asset index file would contain the required information needed to be able to reconstruct the [https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Back_End#Asset_Representation asset representation].

Requirements¶

Each asset file has an asset index file.

Each asset file has an asset index file even when the asset file has no assets. The reason is that asset files can be part of an asset library without containing assets. For example when an asset file contains materials which are linked from assets inside a different asset file, but those materials are not assets themselves. If there was no index, the file would be parsed every time.

Icons of assets are not part of the asset index file.

Compared to the rest of the asset representation, icons can use a lot of disk space. Blender already has an icon manager that utilizes the OS to store and cache icons. Therefore we chose not to include the icon inside the asset representation.

Empty asset files can be detected without opening the asset index file.

Opening, reading and parsing asset index files has latency. Detecting if there are actually asset representations stored inside the asset index file after parsing isn’t efficient. By checking the file size on disk, the asset indexer can already check if there are any asset representations inside the asset index file, removing the need for opening, reading and parsing the file.

Info

The asset index is implemented as a file indexer (FileIndexerType). Browsing assets shares logic with the file browser editor. File browser uses file indexers to read data from folders on a file-system, but also to list data-blocks located inside blend-files or data-blocks from asset libraries.

The usage of the File browser for asset listing and FileIndexerType is planned to be refactored. (See {{Phab|T999999}})

Asset library loading process¶

The sequence diagram shown above describes an overview control flow of the asset indexer. When the asset browser needs to know the asset representations that are inside an asset library it uses a FileList. The file list walks over the file inside the asset library and when an asset file is detected it invokes read_index on the AssetFileIndexer. read_index returns if the found asset representations (EntriesLoaded) or that the asset index file need to be updated for the given asset file.

If the asset file index requires an update, the file list extracts the asset representations from the asset file and invokes update_index. Update entries would create update or create the asset index file with the given asset representations.

Asset Index¶

When browsing assets, asset indexes are created when they don’t exist or are out of date. The asset browser does not access asset files directly, but uses asset indexes.

Storage¶

The asset index files are stored in the [https://docs.blender.org/manual/en/latest/advanced/blender_directory_layout.html#local-cache-directory local cache directory]. Within the local cache directory the indexes are stores using the following pattern: /asset-library-indices/<asset-library-hash>/<asset-index-hash>_<asset_file>.index.json.

asset-library-hash: Hash of the absolute file path of the asset library.
asset-index-hash: Hash of the absolute file path of the asset file.
asset_file: Filename of the asset file. Not used by Blender, but is added for discoverability convenience.

Content¶

{
  "version": <file version number>,
  "entries": [{
    "name": "<asset name>",
    "catalog_id": "<catalog_id>",
    "catalog_name": "<catalog_name>",
    "description": "<description>",
    "author": "<author>",
    "tags": ["<tag>"],
    "properties": [..]
  }]
}

version: the version of the asset index file. It is used to identify the structure of the content. Asset indexes with a different version than used by Blender would be regenerated. Blender 3.1-3.4 expect version attribute to be 1.). Later versions might require to change it.
entries: list of assets in the asset file related to the asset index.
entries[].name: ID name of the asset, including its ID type prefix. (OBSuzanne).
entries[].catalog_id: UUID of the catalog associated with the asset.
entries[].catalog_name: Name of the catalog associated with the asset.
entries[].description: Human readable description of the asset.
entries[].author: Information about the author of the asset.
entries[].tags: List of tags associated with this asset.
entries[].properties: List of ID Properties associated with this asset.

Performance¶

Performance is optimized for reading and parsing asset indexes.

Asset libraries can have blend files that doesn’t contain any assets. To reduce opening and parsing an asset index that doesn’t contain any entries the size on disk of the asset index is checked. Asset indexes smaller than MIN_FILE_SIZE_WITH_ENTRIES would not be parsed and assumed to contain no entries.
Asset indexes older than asset files are assumed to be out of date and should be regenerated. This can be determined without opening the asset index.
Asset indexes of asset files on network shares are stored on your local system to reduce latency. Although this could depends on how your OS has been configured.

Future expected changes¶

This section will contain a table with planned tasks and remark how it would change the asset indexing.

To be added

References¶

Technical overview of asset catalogs: https://wiki.blender.org/wiki/Source/Architecture/Asset_System/Catalogs.
The main source code is located at source/blender/editors/asset/intern/asset_indexer.cc and source/blender/editors/asset/ED_asset_indexer.h.
End user documentation about assets library system https://docs.blender.org/manual/en/latest/files/asset_libraries/introduction.html, https://docs.blender.org/manual/en/latest/files/asset_libraries/catalogs.html

Last update: 2023-06-12
Created: 2023-06-12