# fiftyone.core.utils¶

Core utilities.

Copyright 2017-2022, Voxel51, Inc.

Classes:

DynamicBatcher(iterable[, target_latency, …])

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency.

LazyModule(module_name[, callback])

Proxy module that lazily imports the underlying module the first time it is actually used.

LoggingLevel(level[, logger])

Context manager that allows for a temporary change to the level of a logging.Logger.

MonkeyPatchFunction(module_or_fcn, monkey_fcn)

Context manager that temporarily monkey patches the given function.

ProgressBar(*args, **kwargs)

class eta.core.utils.ProgressBar(total=None, show_percentage=True, show_iteration=True, show_elapsed_time=True, show_remaining_time=True, show_iter_rate=True, iters_str='it', start_msg=None, complete_msg=None, use_bits=False, use_bytes=False, max_width=None, num_decimals=1, max_fps=10, quiet=False)

Class for printing a self-updating progress bar to stdout that tracks the progress of an iterative process towards completion.

The simplest use of ProgressBar is to create an instance and then call it (__call__) with an iterable argument, which will pass through elements of the iterable when iterating over it, updating the progress bar each time an element is emitted.

Alternatively, ProgressBars can track the progress of a task towards a total iteration count, where the current iteration is updated manually by calling either update(), which will increment the iteration count by 1, or via set_iteration(), which lets you manually specify the new iteration status. By default, both methods will automatically redraw the bar. If manual control over drawing is required, you can pass draw=False to either method and then manually call draw() as desired.

It is highly recommended that you always invoke ProgressBars using their context manager interface via the with keyword, which will automatically handle calling start() and close() to appropriately initialize and finalize the task. Among other things, this ensures that stdout redirection is appropriately ended when an exception is encountered.

When start() is called on the a ProgressBar, an internal timer is started to track the elapsed time of the task. In addition, stdout is automatically cached between calls to draw() and flushed each time draw() is called, which means that you can freely mix print statements into your task without interfering with the progress bar. When you are done tracking the task, call the close() method to finalize the progress bar. Both of these actions are automatically taken when the bar’s context manager interface is invoked or when it is called with an iterable.

If you want want to full manual control over the progress bar, call start() to start the task, call pause() before any print statements, and call close() when the task is finalized.

ProgressBar can optionally be configured to print any of the following statistics about the task:

• the elapsed time since the task was started

• the estimated time remaining until the task completes

• the current iteration rate of the task, in iterations per second

• customized status messages passed via the suffix argument

Example (wrapping an iterator):

import time
import eta.core.utils as etau

with etau.ProgressBar() as pb:
for _ in pb(range(100)):
time.sleep(0.05)


Example (with print statements interleaved):

import time
import eta.core.utils as etau

with etau.ProgressBar(100) as pb:
while not pb.complete:
if pb.iteration in {25, 50, 75}:
print("Progress = %.2f" % pb.progress)

time.sleep(0.05)
pb.update()


Example (tracking a bit total):

import random
import time
import eta.core.utils as etau

with etau.ProgressBar(1024 ** 2, use_bits=True) as pb:
while not pb.complete:
pb.update(random.randint(1, 10 * 1024))
time.sleep(0.05)


Methods:

 close(*args) Closes the progress bar. draw([force]) Draws the progress bar at its current progress. pause() Pauses the progress bar so that other information can be printed. set_iteration(iteration[, suffix, draw]) Sets the current iteration. start() Starts the progress bar. update([count, suffix, draw]) Increments the current iteration count by the given value (default = 1).

Attributes:

 complete Whether the task is 100%% complete, or None if the progress bar has no total. elapsed_time The elapsed time since the task was started, or None if this progress bar is not timing. has_dynamic_width Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window. has_total Whether the progress bar has a total iteration count. is_capturing_stdout Whether stdout is being captured between calls to draw(). is_finalized Whether the progress bar is finalized, i.e., close() has been called. is_iterable Whether the progress bar is tracking an iterable. is_running Whether the progress bar is running, i.e., it is between start() and close() calls. iter_rate The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running. iteration The current iteration. progress The current progress, in [0, 1], or None if the progress bar has no total. quiet Whether the progress bar is in quiet mode (no printing to stdout). time_remaining The estimated time remaining for the task, or None if the progress bar has no total or is not timing. total The total iterations, or None if not available.
property is_iterable

Whether the progress bar is tracking an iterable.

property is_running

Whether the progress bar is running, i.e., it is between start() and close() calls.

property is_finalized

Whether the progress bar is finalized, i.e., close() has been called.

property is_capturing_stdout

Whether stdout is being captured between calls to draw().

property has_dynamic_width

Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property has_total

Whether the progress bar has a total iteration count.

property total

The total iterations, or None if not available.

property iteration

The current iteration.

property progress

The current progress, in [0, 1], or None if the progress bar has no total.

property complete

Whether the task is 100%% complete, or None if the progress bar has no total.

property elapsed_time

The elapsed time since the task was started, or None if this progress bar is not timing.

property time_remaining

The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property iter_rate

The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property quiet

Whether the progress bar is in quiet mode (no printing to stdout).

start()

Starts the progress bar.

close(*args)

Closes the progress bar.

update(count=1, suffix=None, draw=True)

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters
• count – the iteration increment. The default is 1

• suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged

• draw – whether to call draw() at the end of this method. By default, this is True

set_iteration(iteration, suffix=None, draw=True)

Sets the current iteration.

Parameters
• iteration – the new iteration

• suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”

• draw – whether to call draw() at the end of this method. By default, this is True

pause()

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

draw(force=False)

Draws the progress bar at its current progress.

Parameters

force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

ResourceLimit(limit[, soft, hard, …])

Context manager that allows for a temporary change to a resource limit exposed by the resource package.

ResponseStream(response[, chunk_size])

Wrapper around a requests.Response that provides a file-like object interface with read(), seek(), and tell() methods.

SetAttributes(obj, **kwargs)

Context manager that temporarily sets the attributes of a class to new values.

UniqueFilenameMaker([output_dir, rel_dir, …])

A class that generates unique output paths in a directory.

Functions:

 Returns the available patterns that can be used by fill_patterns(). call_on_exit(callback) Registers the given callback function so that it will be called when the process exits for (almost) any reason compute_filehash(filepath[, method, chunk_size]) Computes the hash of the given file. Converts a datetime.date or datetime.datetime to milliseconds since epoch. deserialize_numpy_array(numpy_bytes[, ascii]) Loads a serialized numpy array generated by serialize_numpy_array(). Context manager that temporarily disables all progress bars. ensure_import(requirement_str[, …]) Verifies that the given requirement is installed and importable. ensure_package(requirement_str[, …]) Verifies that the given package is installed. ensure_tf([eager, error_level, error_msg]) Verifies that tensorflow is installed and importable. ensure_tfds([error_level, error_msg]) Verifies that tensorflow_datasets is installed and importable. ensure_torch([error_level, error_msg]) Verifies that torch and torchvision are installed and importable. extract_kwargs_for_class(cls, kwargs) Extracts keyword arguments for the given class’s constructor from the given kwargs. extract_kwargs_for_function(fcn, kwargs) Extracts keyword arguments for the given function from the given kwargs. fill_patterns(string) Fills the patterns in in the given string. Returns the preferred multiprocessing context for the current OS. handle_error(error, error_level[, base_error]) Handles the error at the specified error level. indent_lines(s[, indent, skip]) Indents the lines in the given string. Determines whether the system is 32-bit. Determines whether the system is an ARM-based Mac (Apple Silicon). iter_batches(iterable, batch_size) Iterates over the given iterable in batches. iter_slices(sliceable, batch_size) Iterates over batches of the given object via slicing. justify_headings(elements[, width]) Justifies the headings in a list of (heading, content) string tuples by appending whitespace as necessary to each heading. lazy_import(module_name[, callback]) Returns a proxy module object that will lazily import the given module the first time it is used. load_xml_as_json_dict(xml_path) Loads the XML file as a JSON dictionary. normalize_path(path) Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary. normpath(path) Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes. parse_serializable(obj, cls) Parses the given object as an instance of the given eta.core.serial.Serializable class. pformat(obj[, indent, width, depth]) Returns a pretty string representation of the Python object. pprint(obj[, stream, indent, width, depth]) Pretty-prints the Python object. safe_relpath(path[, start, default]) A safe version of os.path.relpath that returns a configurable default value if the given path if it does not lie within the given relative start. serialize_numpy_array(array[, ascii]) Serializes a numpy array. set_resource_limit(limit[, soft, hard, …]) Uses the resource package to change a resource limit for the current process. split_frame_fields(fields) Splits the given fields into sample and frame fields. stream_objects(objects) Streams the iterable of objects to stdout via less. Converts a datetime.timedelta to milliseconds. Converts a timestamp (number of milliseconds since epoch) to a datetime.datetime.
fiftyone.core.utils.extract_kwargs_for_class(cls, kwargs)

Extracts keyword arguments for the given class’s constructor from the given kwargs.

Parameters
• cls – a class

• kwargs – a dictionary of keyword arguments

Returns

a tuple of

• class_kwargs: a dictionary of keyword arguments for cls

• other_kwargs: a dictionary containing the remaining kwargs

fiftyone.core.utils.extract_kwargs_for_function(fcn, kwargs)

Extracts keyword arguments for the given function from the given kwargs.

Parameters
• fcn – a function

• kwargs – a dictionary of keyword arguments

Returns

a tuple of

• fcn_kwargs: a dictionary of keyword arguments for fcn

• other_kwargs: a dictionary containing the remaining kwargs

fiftyone.core.utils.pprint(obj, stream=None, indent=4, width=80, depth=None)

Pretty-prints the Python object.

Parameters
• obj – the Python object

• stream (None) – the stream to write to. The default is sys.stdout

• indent (4) – the number of spaces to use when indenting

• width (80) – the max width of each line in the pretty representation

• depth (None) – the maximum depth at which to pretty render nested dicts

fiftyone.core.utils.pformat(obj, indent=4, width=80, depth=None)

Returns a pretty string representation of the Python object.

Parameters
• obj – the Python object

• indent (4) – the number of spaces to use when indenting

• width (80) – the max width of each line in the pretty representation

• depth (None) – the maximum depth at which to pretty render nested dicts

Returns

the pretty-formatted string

fiftyone.core.utils.split_frame_fields(fields)

Splits the given fields into sample and frame fields.

Frame fields are those prefixed by "frames.", and this prefix is removed from the returned frame fields.

Parameters

fields – a field, iterable of fields, or dict mapping field names to new field names

Returns

a tuple of

• sample_fields: a list or dict of sample fields

• frame_fields: a list or dict of frame fields

fiftyone.core.utils.stream_objects(objects)

Streams the iterable of objects to stdout via less.

The output can be interactively traversed via scrolling and can be terminated via keyboard interrupt.

Parameters

objects – an iterable of objects that can be printed via str(obj)

fiftyone.core.utils.indent_lines(s, indent=4, skip=0)

Indents the lines in the given string.

Parameters
• s – the string

• indent (4) – the number of spaces to indent

• skip (0) – the number of lines to skip before indenting

Returns

the indented string

fiftyone.core.utils.justify_headings(elements, width=None)

Justifies the headings in a list of (heading, content) string tuples by appending whitespace as necessary to each heading.

Parameters
• elements – a list of (heading, content) tuples

• width (None) – an optional justification width. By default, the maximum heading length is used

Returns

a list of justified (heading, content) tuples

fiftyone.core.utils.available_patterns()

Returns the available patterns that can be used by fill_patterns().

Returns

a dict mapping patterns to their replacements

fiftyone.core.utils.fill_patterns(string)

Fills the patterns in in the given string.

Use available_patterns() to see the available patterns that can be used.

Parameters

string – a string

Returns

a copy of string with any patterns replaced

fiftyone.core.utils.normpath(path)

Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.

Use this function when you need a version of os.path.normpath that converts \ to / on Unix.

Parameters

path – a path

Returns

the normalized path

fiftyone.core.utils.normalize_path(path)

Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.

Parameters

path – a path

Returns

the normalized path

fiftyone.core.utils.ensure_package(requirement_str, error_level=None, error_msg=None, log_success=False)

Verifies that the given package is installed.

This function uses pkg_resources.get_distribution to locate the package by its pip name and does not actually import the module.

Therefore, unlike ensure_import(), requirement_str should refer to the package name (e.g., “tensorflow-gpu”), not the module name (e.g., “tensorflow”).

Parameters
• requirement_str – a PEP 440 compliant package requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed

• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to use

• log_success (False) – whether to generate a log message if the requirement is satisifed

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_import(requirement_str, error_level=None, error_msg=None, log_success=False)

Verifies that the given requirement is installed and importable.

This function imports the specified module and optionally enforces any version requirements included in requirement_str.

Therefore, unlike ensure_package(), requirement_str should refer to the module name (e.g., “tensorflow”), not the package name (e.g., “tensorflow-gpu”).

Parameters
• requirement_str – a PEP 440-like module requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed

• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to use

• log_success (False) – whether to generate a log message if the requirement is satisifed

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_tf(eager=False, error_level=None, error_msg=None)

Verifies that tensorflow is installed and importable.

Parameters
• eager (False) – whether to require that TF is executing eagerly. If True and TF is not currently executing eagerly, this method will attempt to enable it

• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_tfds(error_level=None, error_msg=None)

Verifies that tensorflow_datasets is installed and importable.

Parameters
• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.ensure_torch(error_level=None, error_msg=None)

Verifies that torch and torchvision are installed and importable.

Parameters
• error_level (None) –

the error level to use, defined as:

• 0: raise error if requirement is not satisfied

• 1: log warning if requirement is not satisifed

• 2: ignore unsatisifed requirements

By default, fiftyone.config.requirement_error_level is used

• error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisifed

fiftyone.core.utils.handle_error(error, error_level, base_error=None)

Handles the error at the specified error level.

Parameters
• error – an Exception instance

• error_level – the error level to use, defined as:

• 0 (-) – raise the error

• 1 (-) – log the error as a warning

• 2 (-) – ignore the error

• base_error – (optional) a base Exception from which to raise error

class fiftyone.core.utils.LoggingLevel(level, logger=None)

Bases: object

Context manager that allows for a temporary change to the level of a logging.Logger.

Example:

   import logging
import fiftyone.core.utils as fou

with fou.LoggingLevel(logging.CRITICAL):
# do things with all logging at CRITICAL

with fou.LoggingLevel(logging.ERROR, logger="fiftyone"):
# do things with FiftyOne logging at ERROR

Args:
level: the logging level to use, e.g., logging.ERROR
logger (None): a logging.Logger or the name of a logger. By
default, the root logger is used

fiftyone.core.utils.lazy_import(module_name, callback=None)

Returns a proxy module object that will lazily import the given module the first time it is used.

Example usage:

# Lazy version of import tensorflow as tf
tf = lazy_import("tensorflow")

# Other commands

# Now the module is loaded
tf.__version__

Parameters
• module_name – the fully-qualified module name to import

• callback (None) – a callback function to call before importing the module

Returns

a proxy module object that will be lazily imported when first used

class fiftyone.core.utils.LazyModule(module_name, callback=None)

Bases: module

Proxy module that lazily imports the underlying module the first time it is actually used.

Parameters
• module_name – the fully-qualified module name to import

• callback (None) – a callback function to call before importing the module

fiftyone.core.utils.load_xml_as_json_dict(xml_path)

Loads the XML file as a JSON dictionary.

Parameters

xml_path – the path to the XML file

Returns

a JSON dict

fiftyone.core.utils.parse_serializable(obj, cls)

Parses the given object as an instance of the given eta.core.serial.Serializable class.

Parameters
• obj – an instance of cls, or a serialized string or dictionary representation of one

• cls – a eta.core.serial.Serializable class

Returns

an instance of cls

fiftyone.core.utils.set_resource_limit(limit, soft=None, hard=None, warn_on_failure=False)

Uses the resource package to change a resource limit for the current process.

If the resource package cannot be imported, this command does nothing.

Parameters
• limit – the name of the resource to limit. Must be the name of a constant in the resource module starting with RLIMIT. See the documentation of the resource module for supported values

• soft (None) – a new soft limit to apply, which cannot exceed the hard limit. If omitted, the current soft limit is maintained

• hard (None) – a new hard limit to apply. If omitted, the current hard limit is maintained

• warn_on_failure (False) – whether to issue a warning rather than an error if the resource limit change is not successful

class fiftyone.core.utils.ResourceLimit(limit, soft=None, hard=None, warn_on_failure=False)

Bases: object

Context manager that allows for a temporary change to a resource limit exposed by the resource package.

Example:

   import resource
import fiftyone.core.utils as fou

with fou.ResourceLimit(resource.RLIMIT_NOFILE, soft=4096):
# temporarily do things with up to 4096 open files

Args:
limit: the name of the resource to limit. Must be the name of a
constant in the resource module starting with RLIMIT. See
the documentation of the resource module for supported values
soft (None): a new soft limit to apply, which cannot exceed the hard
limit. If omitted, the current soft limit is maintained
hard (None): a new hard limit to apply. If omitted, the current hard
limit is maintained
warn_on_failure (False): whether to issue a warning rather than an
error if the resource limit change is not successful

class fiftyone.core.utils.ProgressBar(*args, **kwargs)
class ProgressBar(total=None, show_percentage=True, show_iteration=True, show_elapsed_time=True, show_remaining_time=True, show_iter_rate=True, iters_str='it', start_msg=None, complete_msg=None, use_bits=False, use_bytes=False, max_width=None, num_decimals=1, max_fps=10, quiet=False)

Class for printing a self-updating progress bar to stdout that tracks the progress of an iterative process towards completion.

The simplest use of ProgressBar is to create an instance and then call it (__call__) with an iterable argument, which will pass through elements of the iterable when iterating over it, updating the progress bar each time an element is emitted.

Alternatively, ProgressBars can track the progress of a task towards a total iteration count, where the current iteration is updated manually by calling either update(), which will increment the iteration count by 1, or via set_iteration(), which lets you manually specify the new iteration status. By default, both methods will automatically redraw the bar. If manual control over drawing is required, you can pass draw=False to either method and then manually call draw() as desired.

It is highly recommended that you always invoke ProgressBars using their context manager interface via the with keyword, which will automatically handle calling start() and close() to appropriately initialize and finalize the task. Among other things, this ensures that stdout redirection is appropriately ended when an exception is encountered.

When start() is called on the a ProgressBar, an internal timer is started to track the elapsed time of the task. In addition, stdout is automatically cached between calls to draw() and flushed each time draw() is called, which means that you can freely mix print statements into your task without interfering with the progress bar. When you are done tracking the task, call the close() method to finalize the progress bar. Both of these actions are automatically taken when the bar’s context manager interface is invoked or when it is called with an iterable.

If you want want to full manual control over the progress bar, call start() to start the task, call pause() before any print statements, and call close() when the task is finalized.

ProgressBar can optionally be configured to print any of the following statistics about the task:

• the elapsed time since the task was started

• the estimated time remaining until the task completes

• the current iteration rate of the task, in iterations per second

• customized status messages passed via the suffix argument

Example (wrapping an iterator):

import time
import eta.core.utils as etau

with etau.ProgressBar() as pb:
for _ in pb(range(100)):
time.sleep(0.05)


Example (with print statements interleaved):

import time
import eta.core.utils as etau

with etau.ProgressBar(100) as pb:
while not pb.complete:
if pb.iteration in {25, 50, 75}:
print("Progress = %.2f" % pb.progress)

time.sleep(0.05)
pb.update()


Example (tracking a bit total):

import random
import time
import eta.core.utils as etau

with etau.ProgressBar(1024 ** 2, use_bits=True) as pb:
while not pb.complete:
pb.update(random.randint(1, 10 * 1024))
time.sleep(0.05)


Methods:

 close(*args) Closes the progress bar. draw([force]) Draws the progress bar at its current progress. pause() Pauses the progress bar so that other information can be printed. set_iteration(iteration[, suffix, draw]) Sets the current iteration. start() Starts the progress bar. update([count, suffix, draw]) Increments the current iteration count by the given value (default = 1).

Attributes:

 complete Whether the task is 100%% complete, or None if the progress bar has no total. elapsed_time The elapsed time since the task was started, or None if this progress bar is not timing. has_dynamic_width Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window. has_total Whether the progress bar has a total iteration count. is_capturing_stdout Whether stdout is being captured between calls to draw(). is_finalized Whether the progress bar is finalized, i.e., close() has been called. is_iterable Whether the progress bar is tracking an iterable. is_running Whether the progress bar is running, i.e., it is between start() and close() calls. iter_rate The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running. iteration The current iteration. progress The current progress, in [0, 1], or None if the progress bar has no total. quiet Whether the progress bar is in quiet mode (no printing to stdout). time_remaining The estimated time remaining for the task, or None if the progress bar has no total or is not timing. total The total iterations, or None if not available.
property ProgressBar.is_iterable

Whether the progress bar is tracking an iterable.

property ProgressBar.is_running

Whether the progress bar is running, i.e., it is between start() and close() calls.

property ProgressBar.is_finalized

Whether the progress bar is finalized, i.e., close() has been called.

property ProgressBar.is_capturing_stdout

Whether stdout is being captured between calls to draw().

property ProgressBar.has_dynamic_width

Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property ProgressBar.has_total

Whether the progress bar has a total iteration count.

property ProgressBar.total

The total iterations, or None if not available.

property ProgressBar.iteration

The current iteration.

property ProgressBar.progress

The current progress, in [0, 1], or None if the progress bar has no total.

property ProgressBar.complete

Whether the task is 100%% complete, or None if the progress bar has no total.

property ProgressBar.elapsed_time

The elapsed time since the task was started, or None if this progress bar is not timing.

property ProgressBar.time_remaining

The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property ProgressBar.iter_rate

The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property ProgressBar.quiet

Whether the progress bar is in quiet mode (no printing to stdout).

ProgressBar.start()

Starts the progress bar.

ProgressBar.close(*args)

Closes the progress bar.

ProgressBar.update(count=1, suffix=None, draw=True)

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters
• count – the iteration increment. The default is 1

• suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged

• draw – whether to call draw() at the end of this method. By default, this is True

ProgressBar.set_iteration(iteration, suffix=None, draw=True)

Sets the current iteration.

Parameters
• iteration – the new iteration

• suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”

• draw – whether to call draw() at the end of this method. By default, this is True

ProgressBar.pause()

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

ProgressBar.draw(force=False)

Draws the progress bar at its current progress.

Parameters

force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

Methods:

 close(*args) Closes the progress bar. draw([force]) Draws the progress bar at its current progress. Pauses the progress bar so that other information can be printed. set_iteration(iteration[, suffix, draw]) Sets the current iteration. Starts the progress bar. update([count, suffix, draw]) Increments the current iteration count by the given value (default = 1).

Attributes:

 complete Whether the task is 100%% complete, or None if the progress bar has no total. elapsed_time The elapsed time since the task was started, or None if this progress bar is not timing. has_dynamic_width Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window. has_total Whether the progress bar has a total iteration count. is_capturing_stdout Whether stdout is being captured between calls to draw(). is_finalized Whether the progress bar is finalized, i.e., close() has been called. is_iterable Whether the progress bar is tracking an iterable. is_running Whether the progress bar is running, i.e., it is between start() and close() calls. iter_rate The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running. iteration The current iteration. progress The current progress, in [0, 1], or None if the progress bar has no total. quiet Whether the progress bar is in quiet mode (no printing to stdout). time_remaining The estimated time remaining for the task, or None if the progress bar has no total or is not timing. total The total iterations, or None if not available.
close(*args)

Closes the progress bar.

property complete

Whether the task is 100%% complete, or None if the progress bar has no total.

draw(force=False)

Draws the progress bar at its current progress.

Parameters

force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

property elapsed_time

The elapsed time since the task was started, or None if this progress bar is not timing.

property has_dynamic_width

Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property has_total

Whether the progress bar has a total iteration count.

property is_capturing_stdout

Whether stdout is being captured between calls to draw().

property is_finalized

Whether the progress bar is finalized, i.e., close() has been called.

property is_iterable

Whether the progress bar is tracking an iterable.

property is_running

Whether the progress bar is running, i.e., it is between start() and close() calls.

property iter_rate

The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property iteration

The current iteration.

pause()

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

property progress

The current progress, in [0, 1], or None if the progress bar has no total.

property quiet

Whether the progress bar is in quiet mode (no printing to stdout).

set_iteration(iteration, suffix=None, draw=True)

Sets the current iteration.

Parameters
• iteration – the new iteration

• suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”

• draw – whether to call draw() at the end of this method. By default, this is True

start()

Starts the progress bar.

property time_remaining

The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property total

The total iterations, or None if not available.

update(count=1, suffix=None, draw=True)

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters
• count – the iteration increment. The default is 1

• suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged

• draw – whether to call draw() at the end of this method. By default, this is True

class fiftyone.core.utils.DynamicBatcher(iterable, target_latency=0.2, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, return_views=False, progress=False, total=None)

Bases: object

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency.

The batch sizes emitted when iterating over this object are dynamically scaled such that the latency between next() calls is as close as possible to a specified target latency.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.DynamicBatcher(
elements, target_latency=0.1, max_batch_beta=2.0
)

for batch in batcher:
print("batch size: %d" % len(batch))

batcher = fou.DynamicBatcher(
elements,
target_latency=0.1,
max_batch_beta=2.0,
progress=True,
)

with batcher:
for batch in batcher:
print("batch size: %d" % len(batch))

Parameters
• iterable – an iterable

• target_latency (0.2) – the target latency between next() calls, in seconds

• init_batch_size (1) – the initial batch size to use

• min_batch_size (1) – the minimum allowed batch size

• max_batch_size (None) – an optional maximum allowed batch size

• max_batch_beta (None) – an optional lower/upper bound on the ratio between successive batch sizes

• return_views (False) – whether to return each batch as a fiftyone.core.view.DatasetView. Only applicable when the iterable is a fiftyone.core.collections.SampleCollection

• progress (False) – whether to render a progress bar tracking the consumption of the batches

• total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

fiftyone.core.utils.disable_progress_bars()

Context manager that temporarily disables all progress bars.

class fiftyone.core.utils.UniqueFilenameMaker(output_dir=None, rel_dir=None, default_ext=None, ignore_exts=False, ignore_existing=False, idempotent=True)

Bases: object

A class that generates unique output paths in a directory.

This class provides a get_output_path() method that generates unique filenames in the specified output directory.

If an input path is provided, its filename is maintained, unless a name conflict in output_dir would occur, in which case an index of the form "-%d" % count is appended to the filename.

If no input filename is provided, an output filename of the form <output_dir>/<count><default_ext> is generated, where count is the number of files in output_dir.

If no output_dir is provided, then unique filenames with no base directory are generated.

If a rel_dir is provided, then this path will be stripped from each input path to generate the identifier of each file (rather than just its basename). This argument allows for populating nested subdirectories in output_dir that match the shape of the input paths.

Parameters
• output_dir (None) – a directory in which to generate output paths

• rel_dir (None) – an optional relative directory to strip from each path. The path is converted to an absolute path (if necessary) via normalize_path()

• default_ext (None) – the file extension to use when generating default output paths

• ignore_exts (False) – whether to omit file extensions when checking for duplicate filenames

• ignore_existing (False) – whether to ignore existing files in output_dir for output filename generation purposes

• idempotent (True) – whether to return the same output path when the same input path is provided multiple times (True) or to generate new output paths (False)

Methods:

 get_output_path([input_path, output_ext]) Returns a unique output path. seen_input_path(input_path) Checks whether we’ve already seen the given input path.
seen_input_path(input_path)

Checks whether we’ve already seen the given input path.

Parameters

input_path – an input path

Returns

True/False

get_output_path(input_path=None, output_ext=None)

Returns a unique output path.

Parameters
• input_path (None) – an input path

• output_ext (None) – an optional output extension to use

Returns

the output path

fiftyone.core.utils.safe_relpath(path, start=None, default=None)

A safe version of os.path.relpath that returns a configurable default value if the given path if it does not lie within the given relative start.

Parameters
• path – a path

• start (None) – the relative prefix to strip from path

• default (None) – a default value to return if path does not lie within start. By default, the basename of the path is returned

Returns

the relative path

fiftyone.core.utils.compute_filehash(filepath, method=None, chunk_size=None)

Computes the hash of the given file.

Parameters
• filepath – the path to the file

• method (None) – an optional hashlib method to use. If not specified, the builtin str.__hash__ will be used

• chunk_size (None) – an optional chunk size to use to read the file, in bytes. Only applicable when a method is provided. The default is 64kB. If negative, the entire file is read at once

Returns

the hash

fiftyone.core.utils.serialize_numpy_array(array, ascii=False)

Serializes a numpy array.

Parameters
• array – a numpy array-like

• ascii (False) – whether to return a base64-encoded ASCII string instead of raw bytes

Returns

the serialized bytes

fiftyone.core.utils.deserialize_numpy_array(numpy_bytes, ascii=False)

Loads a serialized numpy array generated by serialize_numpy_array().

Parameters
• numpy_bytes – the serialized numpy array bytes

• ascii (False) – whether the bytes were generated with the ascii == True parameter of serialize_numpy_array()

Returns

the numpy array

fiftyone.core.utils.iter_batches(iterable, batch_size)

Iterates over the given iterable in batches.

Parameters
• iterable – an iterable

• batch_size – the desired batch size, or None to return the contents in a single batch

Returns

a generator that emits tuples of elements of the requested batch size from the input

fiftyone.core.utils.iter_slices(sliceable, batch_size)

Iterates over batches of the given object via slicing.

Parameters
• sliceable – an object that supports slicing

• batch_size – the desired batch size, or None to return the contents in a single batch

Returns

a generator that emits batches of elements of the requested batch size from the input

fiftyone.core.utils.call_on_exit(callback)

Registers the given callback function so that it will be called when the process exits for (almost) any reason

Note that this should only be used from non-interactive scripts because it intercepts ctrl + c.

Covers the following cases: - normal program termination - a Python exception is raised - a SIGTERM signal is received

Parameters

callback – the function to execute upon termination

class fiftyone.core.utils.MonkeyPatchFunction(module_or_fcn, monkey_fcn, fcn_name=None, namespace=None)

Bases: object

Context manager that temporarily monkey patches the given function.

If a namespace is provided, all functions with same name as the function you are monkey patching that are imported (recursively) by the module_or_fcn module are also monkey patched.

Parameters
• module_or_fcn – a module or function

• monkey_fcn – the function to monkey patch in

• fcn_name (None) – the name of the funciton to monkey patch. Required iff module_or_fcn is a module

• namespace (None) – an optional package namespace

class fiftyone.core.utils.SetAttributes(obj, **kwargs)

Bases: object

Context manager that temporarily sets the attributes of a class to new values.

Parameters
• obj – the object

• **kwargs – the attribute key-values to set while the context is active

fiftyone.core.utils.is_arm_mac()

Determines whether the system is an ARM-based Mac (Apple Silicon).

Returns

True/False

fiftyone.core.utils.is_32_bit()

Determines whether the system is 32-bit.

Returns

True/False

fiftyone.core.utils.get_multiprocessing_context()

Returns the preferred multiprocessing context for the current OS.

Returns

a multiprocessing context

fiftyone.core.utils.datetime_to_timestamp(dt)

Converts a datetime.date or datetime.datetime to milliseconds since epoch.

Parameters

dt – a datetime.date or datetime.datetime

Returns

the float number of milliseconds since epoch

fiftyone.core.utils.timestamp_to_datetime(ts)

Converts a timestamp (number of milliseconds since epoch) to a datetime.datetime.

Parameters

ts – a number of milliseconds since epoch

Returns

a datetime.datetime

fiftyone.core.utils.timedelta_to_ms(td)

Converts a datetime.timedelta to milliseconds.

Parameters

td – a datetime.timedelta

Returns

the float number of milliseconds

class fiftyone.core.utils.ResponseStream(response, chunk_size=64)

Bases: object

Wrapper around a requests.Response that provides a file-like object interface with read(), seek(), and tell() methods.

Source:

https://gist.github.com/obskyr/b9d4b4223e7eaf4eedcd9defabb34f13

Parameters
• response – a requests.Response

• chunk_size (64) – the chunk size to use to read the response’s content

Methods:

 read([size]) seek(position[, whence])
read(size=None)
seek(position, whence=0)
tell()