espnet2.utils.sized_dict.SizedDict

source

class espnet2.utils.sized_dict.SizedDict(shared: bool = False, data: dict | None = None)

Bases: MutableMapping

A dictionary-like class that keeps track of the total size of its items.

This class extends collections.abc.MutableMapping and allows for the creation of a dictionary that can either be shared among multiple processes or kept local to a single process. The SizedDict automatically computes and maintains the total size of its contents, making it useful for memory-sensitive applications.

shared

Indicates whether the dictionary is shared among processes.

• Type: bool

cache

The underlying dictionary that stores the key-value pairs.

• Type: dict

size

The total size in bytes of the dictionary’s contents.

• Type: int

• Parameters:

  • shared (bool) – If True, use a multiprocessing manager to allow sharing between processes. Defaults to False.
  • data (dict , optional) – Initial data to populate the dictionary. Defaults to an empty dictionary.

Examples

>>> my_dict = SizedDict()
>>> my_dict['a'] = [1, 2, 3]
>>> my_dict.size
64  # Size may vary based on the object
>>> my_dict['b'] = 'hello'
>>> my_dict.size
90  # Size may vary based on the object
>>> del my_dict['a']
>>> my_dict.size
48  # Size may vary based on the object

NOTE

The size calculation includes the size of keys, values, and their references. This class is especially useful when dealing with large datasets where memory consumption needs to be monitored.