hoops_ai.storage

Quick Overview

Modules

hoops_ai.storage.datasetstorage

Classes

DataStorage()

MemoryStorage()

MLStorage()

DGLGraphStoreHandler()

LabelStorage(path_for_storing[, ...])

Class for encoding and decoding labels.

MetricStorage(store)

Abstract class defining the interface for storing machine learning metrics based on their type of data and visualization.

CADFileRetriever(storage_provider[, ...])

LocalStorageProvider(directory_path)

Functions

convert_storage(source_handler, dest_handler)

Generic converter that works with ANY DataStorage implementation.

Data Storage Module

The Storage module provides persistent storage solutions for CAD data, ML models, and analysis results. It offers a unified interface for various storage backends, optimized for the unique requirements of CAD data processing and machine learning workflows.

This module handles the efficient storage and retrieval of large-scale CAD datasets, encoded geometric data, trained ML models, and experimental results. It provides both high-performance options for production use and convenient formats for development and prototyping.

For storage architecture details and usage patterns, see the Data Storage Programming Guide.

class hoops_ai.storage.CADFileRetriever(storage_provider, formats=None, filter_pattern=None, use_regex=False)

Bases: object

Parameters:

  • storage_provider (StorageProvider)

  • formats (List[str] | None)

  • filter_pattern (str | None)

  • use_regex (bool)

get_file_list()

Returns the filtered list of CAD file paths.

Return type:

List[str]

class hoops_ai.storage.DataStorage

Bases: ABC

abstract close()

Handles any necessary cleanup or resource deallocation.

Return type:

None

compress_store()
Return type:

int

abstract format()

a string spcifying the concrete format for this storage.

Return type:

str

abstract get_file_path(data_key)

Retrieves the file path for a given data key.

Parameters:

data_key (str) – The data key.

Returns:

The file path corresponding to the data key.

Return type:

str

get_group_for_array(array_name)

Determines which group an array belongs to based on the schema.

Parameters:

array_name (str) – Name of the array

Returns:

Group name for the array, or None if not found in schema

Return type:

str

abstract get_keys()

Retrieves a list of all keys in the storage. :returns: A list of all keys in the storage. :rtype: list

Return type:

list

get_schema()

Retrieves the schema definition for this storage instance.

Returns:

The schema definition, or empty dict if no schema is set

Return type:

dict

abstract load_data(data_key)

Loads data associated with a specific key.

Parameters:

data_key (str) – The key of the data to load.

Returns:

The loaded data.

Return type:

Any

abstract load_metadata(key)

Loads metadata associated with a specific key.

Parameters:

key (str) – The metadata key.

Returns:

The loaded metadata value.

Return type:

Any

abstract save_data(data_key, data)

Saves data associated with a specific key.

Parameters:

  • data_key (str) – The key under which to store the data.

  • data (Any) – The data to store.

Return type:

None

abstract save_metadata(key, value)

Saves metadata as a key-value pair into the metadata JSON file. If the file doesn’t exist, it will be created.

Parameters:

  • key (str) – The metadata key.

  • value (Any) – The metadata value (bool, int, float, string, list, or array).

Return type:

None

set_schema(schema)

Sets a schema definition for this storage instance. The schema defines how arrays should be organized into groups for merging.

Parameters:

schema (dict) – Schema definition containing group and array specifications

Return type:

None

validate_data_against_schema(data_key, data)

Validates data against the stored schema if present.

Parameters:

  • data_key (str) – The key under which the data will be stored

  • data (Any) – The data to validate

Returns:

True if valid or no schema present, False if validation fails

Return type:

bool

class hoops_ai.storage.LocalStorageProvider(directory_path)

Bases: StorageProvider

Parameters:

directory_path (str)

list_files(extensions)

Retrieves CAD file paths from a local directory or a text file using pathlib.

Parameters:

extensions (List[str])

Return type:

List[str]

hoops_ai.storage.convert_storage(source_handler, dest_handler, verbose=False)

Generic converter that works with ANY DataStorage implementation.

This universal converter supports all storage types including: - OptStorage / ZarrStorage - Compressed binary format - JsonStorageHandler - Human-readable JSON format - MemoryStorage - In-memory storage (no files) - Custom implementations - Any class inheriting from DataStorage

The converter uses only the base DataStorage interface methods: - get_keys() or load_metadata() to discover data keys - load_data() to read data arrays - save_data() to write data arrays - save_metadata() to copy metadata

This makes it completely storage-agnostic and extensible.

Common Use Cases: - OptStorage → JSON: Decompress for inspection/debugging - JSON → OptStorage: Compress for performance/storage - MemoryStorage → OptStorage: Persist in-memory data to disk - OptStorage → MemoryStorage: Load encoded data for ML training

Parameters:

  • source_handler (DataStorage) – Source storage to read from (any type).

  • dest_handler (DataStorage) – Destination storage to write to (any type).

  • verbose (bool) – Print progress messages. Default False.

Returns:

None

Raises:

RuntimeError – If no data keys found or conversion fails.

Return type:

None

Examples

>>> # OptStorage → JSON (decompression)
>>> opt = OptStorage("flows/data_mining/abc123")
>>> json_storage = JsonStorageHandler("flows/json/abc123")
>>> convert_storage(opt, json_storage)

>>> # MemoryStorage → OptStorage (persist to disk)
>>> mem = MemoryStorage()
>>> mem.save_data("faces/positions", np.array([[0,0,0], [1,1,1]]))
>>> opt = OptStorage("output/saved_data")
>>> convert_storage(mem, opt)

>>> # JSON → MemoryStorage (load for processing)
>>> json_storage = JsonStorageHandler("data/abc123")
>>> mem = MemoryStorage()
>>> convert_storage(json_storage, mem)