# fiftyone.core.utils¶

Core utilities.

Classes:

 DynamicBatcher(iterable, target_latency_seconds) Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency. LazyModule(module_name[, callback]) Proxy module that lazily imports the underlying module the first time it is actually used. LoggingLevel(level[, logger]) Context manager that allows for a temporary change to the level of a logging.Logger. MonkeyPatchFunction(module_or_fcn, monkey_fcn) Context manager that temporarily monkey patches the given function. ProgressBar(*args, **kwargs) ResourceLimit(limit[, soft, hard, …]) Context manager that allows for a temporary change to a resource limit exposed by the resource package. SetAttributes(obj, **kwargs) Context manager that temporarily sets the attributes of a class to new values. UniqueFilenameMaker([output_dir, …]) A class that generates unique output paths in a directory.

Functions:

 Returns the available patterns that can be used by fill_patterns(). call_on_exit(callback) Registers the given callback function so that it will be called when the process exits for (almost) any reason compute_filehash(filepath[, method, chunk_size]) Computes the hash of the given file. deserialize_numpy_array(numpy_bytes[, ascii]) Loads a serialized numpy array generated by serialize_numpy_array(). Context manager that temporarily disables all progress bars. ensure_import(requirement_str[, …]) Verifies that the given requirement is installed and importable. ensure_package(requirement_str[, …]) Verifies that the given package is installed. ensure_tf([eager, error_level, error_msg]) Verifies that tensorflow is installed and importable. ensure_tfds([error_level, error_msg]) Verifies that tensorflow_datasets is installed and importable. ensure_torch([error_level, error_msg]) Verifies that torch and torchvision are installed and importable. extract_kwargs_for_class(cls, kwargs) Extracts keyword arguments for a class from the given dictionary of arguments. fill_patterns(string) Fills the patterns in in the given string. handle_error(error, error_level[, base_error]) Handles the error at the specified error level. indent_lines(s[, indent, skip]) Indents the lines in the given string. iter_batches(iterable, batch_size) Iterates over the given iterable in batches. iter_slices(sliceable, batch_size) Iterates over batches of the given object via slicing. justify_headings(elements[, width]) Justifies the headings in a list of (heading, content) string tuples by appending whitespace as necessary to each heading. lazy_import(module_name[, callback]) Returns a proxy module object that will lazily import the given module the first time it is used. load_xml_as_json_dict(xml_path) Loads the XML file as a JSON dictionary. parse_serializable(obj, cls) Parses the given object as an instance of the given eta.core.serial.Serializable class. pformat(obj[, indent, width, depth]) Returns a pretty string representation of the Python object. pprint(obj[, stream, indent, width, depth]) Pretty-prints the Python object. serialize_numpy_array(array[, ascii]) Serializes a numpy array. set_resource_limit(limit[, soft, hard, …]) Uses the resource package to change a resource limit for the current process. split_frame_fields(fields) Splits the given fields into sample and frame fields. stream_objects(objects) Streams the iterable of objects to stdout via less.
fiftyone.core.utils.extract_kwargs_for_class(cls, kwargs)

Extracts keyword arguments for a class from the given dictionary of arguments.

Parameters
• cls – a class

• kwargs – a dictionary of keyword arguments

Returns

• class_kwargs: a dictionary of keyword arguments for cls

• other_kwargs: a dictionary containing the remaining kwargs

Return type

a tuple of

fiftyone.core.utils.pprint(obj, stream=None, indent=4, width=80, depth=None)

Pretty-prints the Python object.

Parameters
• obj – the Python object

• stream (None) – the stream to write to. The default is sys.stdout

• indent (4) – the number of spaces to use when indenting

• width (80) – the max width of each line in the pretty representation

• depth (None) – the maximum depth at which to pretty render nested dicts

fiftyone.core.utils.pformat(obj, indent=4, width=80, depth=None)

Returns a pretty string representation of the Python object.

Parameters
• obj – the Python object

• indent (4) – the number of spaces to use when indenting

• width (80) – the max width of each line in the pretty representation

• depth (None) – the maximum depth at which to pretty render nested dicts

Returns

the pretty-formatted string

fiftyone.core.utils.split_frame_fields(fields)

Splits the given fields into sample and frame fields.

Frame fields are those prefixed by "frames.", and this prefix is removed from the returned frame fields.

Parameters

fields – a field, iterable of fields, or dict mapping field names to new field names

Returns

• a list or dict of sample fields

• a list or dict of frame fields

Return type

a tuple of

fiftyone.core.utils.stream_objects(objects)

Streams the iterable of objects to stdout via less.

The output can be interactively traversed via scrolling and can be terminated via keyboard interrupt.

Parameters

objects – an iterable of objects that can be printed via str(obj)

fiftyone.core.utils.indent_lines(s, indent=4, skip=0)

Indents the lines in the given string.

Parameters
• s – the string

• indent (4) – the number of spaces to indent

• skip (0) – the number of lines to skip before indenting

Returns

the indented string

fiftyone.core.utils.justify_headings(elements, width=None)

Justifies the headings in a list of (heading, content) string tuples by appending whitespace as necessary to each heading.

Parameters
• elements – a list of (heading, content) tuples

• width (None) – an optional justification width. By default, the maximum heading length is used

Returns

a list of justified (heading, content) tuples

fiftyone.core.utils.available_patterns()

Returns the available patterns that can be used by fill_patterns().

Returns

a dict mapping patterns to their replacements

fiftyone.core.utils.fill_patterns(string)

Fills the patterns in in the given string.

Use available_patterns() to see the available patterns that can be used.

Parameters

string – a string

Returns

a copy of string with any patterns replaced

fiftyone.core.utils.ensure_package(requirement_str, error_level=None, error_msg=None, log_success=False)

Verifies that the given package is installed.

This function uses pkg_resources.get_distribution to locate the package by its pip name and does not actually import the module.

Therefore, unlike ensure_import(), requirement_str should refer to the package name (e.g., “tensorflow-gpu”), not the module name (e.g., “tensorflow”).

Parameters
• requirement_str – a PEP 440 compliant package requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed

• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to use

• log_success (False) – whether to generate a log message if the requirement is satisifed

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_import(requirement_str, error_level=None, error_msg=None, log_success=False)

Verifies that the given requirement is installed and importable.

This function imports the specified module and optionally enforces any version requirements included in requirement_str.

Therefore, unlike ensure_package(), requirement_str should refer to the module name (e.g., “tensorflow”), not the package name (e.g., “tensorflow-gpu”).

Parameters
• requirement_str – a PEP 440-like module requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed

• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to use

• log_success (False) – whether to generate a log message if the requirement is satisifed

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_tf(eager=False, error_level=None, error_msg=None)

Verifies that tensorflow is installed and importable.

Parameters
• eager (False) – whether to require that TF is executing eagerly. If True and TF is not currently executing eagerly, this method will attempt to enable it

• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_tfds(error_level=None, error_msg=None)

Verifies that tensorflow_datasets is installed and importable.

Parameters
• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_torch(error_level=None, error_msg=None)

Verifies that torch and torchvision are installed and importable.

Parameters
• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.handle_error(error, error_level, base_error=None)

Handles the error at the specified error level.

Parameters
• error – an Exception instance

• error_level – the error level to use, defined as:

• 0 (-) – raise the error

• 1 (-) – log the error as a warning

• 2 (-) – ignore the error

• base_error – (optional) a base Exception from which to raise error

class fiftyone.core.utils.LoggingLevel(level, logger=None)

Bases: object

Context manager that allows for a temporary change to the level of a logging.Logger.

Example:

   import logging
import fiftyone.core.utils as fou

with fou.LoggingLevel(logging.CRITICAL):
# do things with all logging at CRITICAL

with fou.LoggingLevel(logging.ERROR, logger="fiftyone"):
# do things with FiftyOne logging at ERROR

Args:
level: the logging level to use, e.g., logging.ERROR
logger (None): a logging.Logger or the name of a logger. By
default, the root logger is used

fiftyone.core.utils.lazy_import(module_name, callback=None)

Returns a proxy module object that will lazily import the given module the first time it is used.

Example usage:

# Lazy version of import tensorflow as tf
tf = lazy_import("tensorflow")

# Other commands

# Now the module is loaded
tf.__version__

Parameters
• module_name – the fully-qualified module name to import

• callback (None) – a callback function to call before importing the module

Returns

a proxy module object that will be lazily imported when first used

class fiftyone.core.utils.LazyModule(module_name, callback=None)

Bases: module

Proxy module that lazily imports the underlying module the first time it is actually used.

Parameters
• module_name – the fully-qualified module name to import

• callback (None) – a callback function to call before importing the module

fiftyone.core.utils.load_xml_as_json_dict(xml_path)

Loads the XML file as a JSON dictionary.

Parameters

xml_path – the path to the XML file

Returns

a JSON dict

fiftyone.core.utils.parse_serializable(obj, cls)

Parses the given object as an instance of the given eta.core.serial.Serializable class.

Parameters
• obj – an instance of cls, or a serialized string or dictionary representation of one

• cls – a eta.core.serial.Serializable class

Returns

an instance of cls

fiftyone.core.utils.set_resource_limit(limit, soft=None, hard=None, warn_on_failure=False)

Uses the resource package to change a resource limit for the current process.

If the resource package cannot be imported, this command does nothing.

Parameters
• limit – the name of the resource to limit. Must be the name of a constant in the resource module starting with RLIMIT. See the documentation of the resource module for supported values

• soft (None) – a new soft limit to apply, which cannot exceed the hard limit. If omitted, the current soft limit is maintained

• hard (None) – a new hard limit to apply. If omitted, the current hard limit is maintained

• warn_on_failure (False) – whether to issue a warning rather than an error if the resource limit change is not successful

class fiftyone.core.utils.ResourceLimit(limit, soft=None, hard=None, warn_on_failure=False)

Bases: object

Context manager that allows for a temporary change to a resource limit exposed by the resource package.

Example:

   import resource
import fiftyone.core.utils as fou

with fou.ResourceLimit(resource.RLIMIT_NOFILE, soft=4096):
# temporarily do things with up to 4096 open files

Args:
limit: the name of the resource to limit. Must be the name of a
constant in the resource module starting with RLIMIT. See
the documentation of the resource module for supported values
soft (None): a new soft limit to apply, which cannot exceed the hard
limit. If omitted, the current soft limit is maintained
hard (None): a new hard limit to apply. If omitted, the current hard
limit is maintained
warn_on_failure (False): whether to issue a warning rather than an
error if the resource limit change is not successful

class fiftyone.core.utils.ProgressBar(*args, **kwargs)

Bases: eta.core.utils.ProgressBar

Methods:

 close(*args) Closes the progress bar. draw([force]) Draws the progress bar at its current progress. Pauses the progress bar so that other information can be printed. set_iteration(iteration[, suffix, draw]) Sets the current iteration. Starts the progress bar. update([count, suffix, draw]) Increments the current iteration count by the given value (default = 1).

Attributes:

 complete Whether the task is 100%% complete, or None if the progress bar has no total. elapsed_time The elapsed time since the task was started, or None if this progress bar is not timing. has_dynamic_width Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window. has_total Whether the progress bar has a total iteration count. is_capturing_stdout Whether stdout is being captured between calls to draw(). is_finalized Whether the progress bar is finalized, i.e., close() has been called. is_iterable Whether the progress bar is tracking an iterable. is_running Whether the progress bar is running, i.e., it is between start() and close() calls. iter_rate The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running. iteration The current iteration. progress The current progress, in [0, 1], or None if the progress bar has no total. quiet Whether the progress bar is in quiet mode (no printing to stdout). time_remaining The estimated time remaining for the task, or None if the progress bar has no total or is not timing. total The total iterations, or None if not available.
close(*args)

Closes the progress bar.

property complete

Whether the task is 100%% complete, or None if the progress bar has no total.

draw(force=False)

Draws the progress bar at its current progress.

Parameters

force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

property elapsed_time

The elapsed time since the task was started, or None if this progress bar is not timing.

property has_dynamic_width

Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property has_total

Whether the progress bar has a total iteration count.

property is_capturing_stdout

Whether stdout is being captured between calls to draw().

property is_finalized

Whether the progress bar is finalized, i.e., close() has been called.

property is_iterable

Whether the progress bar is tracking an iterable.

property is_running

Whether the progress bar is running, i.e., it is between start() and close() calls.

property iter_rate

The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property iteration

The current iteration.

pause()

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

property progress

The current progress, in [0, 1], or None if the progress bar has no total.

property quiet

Whether the progress bar is in quiet mode (no printing to stdout).

set_iteration(iteration, suffix=None, draw=True)

Sets the current iteration.

Parameters
• iteration – the new iteration

• suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”

• draw – whether to call draw() at the end of this method. By default, this is True

start()

Starts the progress bar.

property time_remaining

The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property total

The total iterations, or None if not available.

update(count=1, suffix=None, draw=True)

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters
• count – the iteration increment. The default is 1

• suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged

• draw – whether to call draw() at the end of this method. By default, this is True

class fiftyone.core.utils.DynamicBatcher(iterable, target_latency_seconds, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None)

Bases: object

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency.

The batch sizes emitted when iterating over this object are dynamically scaled such that the latency between next() calls is as close as possible to a specified target latency.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

total = int(1e7)
elements = range(total)

batches = fou.DynamicBatcher(elements, 0.1, max_batch_beta=2.0)

with fou.ProgressBar(total) as pb:
for batch in batches:
batch_size = len(batch)
print("batch size: %d" % batch_size)
pb.update(count=batch_size)

Parameters
• iterable – an iterable

• target_latency_seconds – the target latency between next() calls, in seconds

• init_batch_size (1) – the initial batch size to use

• min_batch_size (1) – the minimum allowed batch size

• max_batch_size (None) – an optional maximum allowed batch size

• max_batch_beta (None) – an optional lower/upper bound on the ratio between successive batch sizes

fiftyone.core.utils.disable_progress_bars()

Context manager that temporarily disables all progress bars.

class fiftyone.core.utils.UniqueFilenameMaker(output_dir=None, default_ext=None, ignore_exts=False)

Bases: object

A class that generates unique output paths in a directory.

This class provides a get_output_path() method that generates unique filenames in the specified output directory.

If an input filename is provided, the filename is maintained, unless a name conflict in output_dir would occur, in which case an index of the form "-%d" % count is appended to the base filename.

If no input filename is provided, an output filename of the form <output_dir>/<count><default_ext> is generated, where count is the number of files in output_dir.

If no output_dir is provided, then unique filenames with no base directory are generated.

Parameters
• output_dir (None) – a directory in which to generate output paths

• default_ext (None) – the file extension to use when generating default output paths

• ignore_exts (False) – whether to omit file extensions when checking for duplicate filenames

Methods:

 get_output_path([input_path, output_ext]) Returns a unique output path.
get_output_path(input_path=None, output_ext=None)

Returns a unique output path.

Parameters
• input_path (None) – an input path from which to derive the output path

• output_ext (None) – an optional output extension to use

Returns

the output path

fiftyone.core.utils.compute_filehash(filepath, method=None, chunk_size=None)

Computes the hash of the given file.

Parameters
• filepath – the path to the file

• method (None) – an optional hashlib method to use. If not specified, the builtin str.__hash__ will be used

• chunk_size (None) – an optional chunk size to use to read the file, in bytes. Only applicable when a method is provided. The default is 64kB. If negative, the entire file is read at once

Returns

the hash

fiftyone.core.utils.serialize_numpy_array(array, ascii=False)

Serializes a numpy array.

Parameters
• array – a numpy array-like

• ascii (False) – whether to return a base64-encoded ASCII string instead of raw bytes

Returns

the serialized bytes

fiftyone.core.utils.deserialize_numpy_array(numpy_bytes, ascii=False)

Loads a serialized numpy array generated by serialize_numpy_array().

Parameters
• numpy_bytes – the serialized numpy array bytes

• ascii (False) – whether the bytes were generated with the ascii == True parameter of serialize_numpy_array()

Returns

the numpy array

fiftyone.core.utils.iter_batches(iterable, batch_size)

Iterates over the given iterable in batches.

Parameters
• iterable – an iterable

• batch_size – the desired batch size, or None to return the contents in a single batch

Returns

a generator that emits tuples of elements of the requested batch size from the input

fiftyone.core.utils.iter_slices(sliceable, batch_size)

Iterates over batches of the given object via slicing.

Parameters
• sliceable – an object that supports slicing

• batch_size – the desired batch size, or None to return the contents in a single batch

Returns

a generator that emits batches of elements of the requested batch size from the input

fiftyone.core.utils.call_on_exit(callback)

Registers the given callback function so that it will be called when the process exits for (almost) any reason

Note that this should only be used from non-interactive scripts because it intercepts ctrl + c.

Covers the following cases: - normal program termination - a Python exception is raised - a SIGTERM signal is received

Parameters

callback – the function to execute upon termination

class fiftyone.core.utils.MonkeyPatchFunction(module_or_fcn, monkey_fcn, fcn_name=None, namespace=None)

Bases: object

Context manager that temporarily monkey patches the given function.

If a namespace is provided, all functions with same name as the function you are monkey patching that are imported (recursively) by the module_or_fcn module are also monkey patched.

Parameters
• module_or_fcn – a module or function

• monkey_fcn – the function to monkey patch in

• fcn_name (None) – the name of the funciton to monkey patch. Required iff module_or_fcn is a module

• namespace (None) – an optional package namespace

class fiftyone.core.utils.SetAttributes(obj, **kwargs)

Bases: object

Context manager that temporarily sets the attributes of a class to new values.

Parameters
• obj – the object

• **kwargs – the attribute key-values to set while the context is active