hoops_ai.storage.OptStorage

class hoops_ai.storage.OptStorage(store_path, compress_extension='.data')

Bases: DataStorage

Handles storage and retrieval of data using Zarr, including:

  • Setting dimension names so that xarray can open each array.

  • Checking for NaNs before writing and before zipping.

  • Compressing into a .zip-based Zarr store.

  1. metadata.json is cached in memory (no read+write on every save_metadata call) and written to disk only when flush_metadata() is called (e.g., close() / compress_store()).

  2. file size tracking uses estimated size from input data (no os.walk/stat scanning).

Parameters:

  • store_path (str)

  • compress_extension (str)

close()

Deletes all sibling DirectoryStores with same hash. Thread-safe. Also flushes metadata cache before copying/deleting.

Return type:

None

compress_store()

Copies data from the DirectoryStore to a ZipStore, verifying no NaNs exist in raw arrays. If everything is OK, removes the DirectoryStore. Returns size of generated zip.

Return type:

int

create_store_in_group(store_path)
Parameters:

store_path (str)

Return type:

OptStorage

flush_metadata()

Write cached metadata to metadata.json once (if dirty).

Return type:

None

format()

Returns ‘zarr’ to match your original interface.

Return type:

str

get_file_path(data_key)

Returns the direct path for a particular data key (sub-directory or array).

Parameters:

data_key (str)

Return type:

str

get_group_for_array(array_name)

Determines which group an array belongs to based on the schema.

Parameters:

array_name (str) – Name of the array

Returns:

Group name for the array, or None if not found in schema

Return type:

str

get_keys()

Retrieves a list of all top-level keys in the storage.

Return type:

list

get_schema()

Retrieves the schema definition for this storage instance.

Returns:

The schema definition, or empty dict if no schema is set

Return type:

dict

get_store_group()
Return type:

StoreGroup

load_data(data_key)

Loads data from the Zarr store by key.

Parameters:

data_key (str)

Return type:

Any

load_metadata(key)

Loads custom metadata by key from cached metadata.

Parameters:

key (str)

Return type:

Any

save_data(data_key, data)

Saves data (numpy arrays, lists, dicts, scalars, strings) under data_key. Overwrites if that key already exists.

Parameters:
Return type:

None

save_metadata(key, value)

Saves custom metadata (JSON-serializable) to cached metadata (no disk I/O here).

Parameters:
Return type:

None

set_schema(schema)

Sets a schema definition for this storage instance. The schema defines how arrays should be organized into groups for merging.

Parameters:

schema (dict) – Schema definition containing group and array specifications

Return type:

None

validate_data_against_schema(data_key, data)

Validates data against the stored schema if present.

Parameters:

  • data_key (str) – The key under which the data will be stored

  • data (Any) – The data to validate

Returns:

True if valid or no schema present, False if validation fails

Return type:

bool