fiftyone.core.odm.database¶

Database utilities.

Copyright 2017-2022, Voxel51, Inc.
voxel51.com

Functions:

`aggregate`(collection, pipelines)	Executes one or more aggregations on a collection.
`bulk_write`(ops, coll[, ordered])	Performs a batch of write operations on a collection.
`count_documents`(coll, pipeline)
`delete_annotation_run`(name, anno_key[, dry_run])	Deletes the annotation run with the given key from the dataset with the given name.
`delete_annotation_runs`(name[, dry_run])	Deletes all annotation runs from the dataset with the given name.
`delete_brain_run`(name, brain_key[, dry_run])	Deletes the brain method run with the given key from the dataset with the given name.
`delete_brain_runs`(name[, dry_run])	Deletes all brain method runs from the dataset with the given name.
`delete_dataset`(name[, dry_run])	Deletes the dataset with the given name.
`delete_evaluation`(name, eval_key[, dry_run])	Deletes the evaluation run with the given key from the dataset with the given name.
`delete_evaluations`(name[, dry_run])	Deletes all evaluations from the dataset with the given name.
`drop_database`()	Drops the database.
`drop_orphan_collections`([dry_run])	Drops all orphan collections from the database.
`drop_orphan_run_results`([dry_run])	Drops all orphan run results from the database.
`establish_db_conn`(config)	Establishes the database connection.
`export_collection`(docs, json_path[, key, …])	Exports the collection to disk in JSON format.
`export_document`(doc, json_path)	Exports the document to disk in JSON format.
`get_async_db_conn`()	Returns an async connection to the database.
`get_collection_stats`(collection_name)	Sets stats about the collection.
`get_db_client`()	Returns a database client.
`get_db_conn`()	Returns a connection to the database.
`import_collection`(json_path)	Imports the collection from JSON on disk.
`import_document`(json_path)	Imports a document from JSON on disk.
`insert_documents`(docs, coll[, ordered])	Inserts a list of documents into a collection.
`list_collections`()	Returns a list of all collection names in the database.
`list_datasets`()	Returns the list of available FiftyOne datasets.
`stream_collection`(collection_name)	Streams the contents of the collection to stdout.
`sync_database`()	Syncs all pending database writes to disk.

fiftyone.core.odm.database.establish_db_conn(config)¶

Establishes the database connection.

If fiftyone.config.database_uri is defined, then we connect to that URI. Otherwise, a fiftyone.core.service.DatabaseService is created.

Parameters

config – a fiftyone.core.config.FiftyOneConfig

Raises

ConnectionError – if a connection to mongod could not be established
FiftyOneConfigError – if fiftyone.config.database_uri is not defined and mongod could not be found
ServiceExecutableNotFound – if fiftyone.core.service.DatabaseService startup was attempted, but mongod was not found in fiftyone.db.bin
RuntimeError – if the mongod found does not meet FiftyOne’s requirements, or validation could not occur

fiftyone.core.odm.database.aggregate(collection, pipelines)¶

Executes one or more aggregations on a collection.

Multiple aggregations are executed using multiple threads, and their results are returned as lists rather than cursors.

Parameters

collection – a pymongo.collection.Collection or motor.motor_tornado.MotorCollection
pipelines – a MongoDB aggregation pipeline or a list of pipelines

Returns

If a single pipeline is provided, a pymongo.command_cursor.CommandCursor or motor.motor_tornado.MotorCommandCursor is returned
If multiple pipelines are provided, each cursor is extracted into a list and the list of lists is returned

fiftyone.core.odm.database.get_db_client()¶

Returns a database client.

Returns: a pymongo.mongo_client.MongoClient

fiftyone.core.odm.database.get_db_conn()¶

Returns a connection to the database.

Returns: a pymongo.database.Database

fiftyone.core.odm.database.get_async_db_conn()¶

Returns an async connection to the database.

Returns: a motor.motor_tornado.MotorDatabase

fiftyone.core.odm.database.drop_database()¶: Drops the database.

fiftyone.core.odm.database.sync_database()¶: Syncs all pending database writes to disk.

fiftyone.core.odm.database.list_collections()¶

Returns a list of all collection names in the database.

Returns: a list of all collection names

fiftyone.core.odm.database.drop_orphan_collections(dry_run=False)¶

Drops all orphan collections from the database.

Orphan collections are collections that are not associated with any known dataset or other collections used by FiftyOne.

Parameters: dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.drop_orphan_run_results(dry_run=False)¶

Drops all orphan run results from the database.

Orphan run results are results that are not associated with any known dataset.

Parameters: dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.stream_collection(collection_name)¶

Streams the contents of the collection to stdout.

Parameters: collection_name – the name of the collection

fiftyone.core.odm.database.get_collection_stats(collection_name)¶

Sets stats about the collection.

Parameters: collection_name – the name of the collection
Returns: a stats dict

fiftyone.core.odm.database.count_documents(coll, pipeline)¶

fiftyone.core.odm.database.export_document(doc, json_path)¶

Exports the document to disk in JSON format.

Parameters

doc – a BSON document dict
json_path – the path to write the JSON file

fiftyone.core.odm.database.export_collection(docs, json_path, key='documents', num_docs=None)¶

Exports the collection to disk in JSON format.

Parameters

docs – an iteraable containing the documents to export
json_path – the path to write the JSON file
key ("documents") – the field name under which to store the documents
num_docs (None) – the total number of documents. If omitted, this must be computable via len(docs)

fiftyone.core.odm.database.import_document(json_path)¶

Imports a document from JSON on disk.

Parameters: json_path – the path to the document
Returns: a BSON document dict

fiftyone.core.odm.database.import_collection(json_path)¶

Imports the collection from JSON on disk.

Parameters: json_path – the path to the collection on disk
Returns: a BSON dict

fiftyone.core.odm.database.insert_documents(docs, coll, ordered=False)¶

Inserts a list of documents into a collection.

The _id field of the input documents will be populated if it is not already set.

Parameters

docs – the list of BSON document dicts to insert
coll – a pymongo collection instance
ordered (False) – whether the documents must be inserted in order

fiftyone.core.odm.database.bulk_write(ops, coll, ordered=False)¶

Performs a batch of write operations on a collection.

Parameters

ops – a list of pymongo operations
coll – a pymongo collection instance
ordered (False) – whether the operations must be performed in order

fiftyone.core.odm.database.list_datasets()¶

Returns the list of available FiftyOne datasets.

This is a low-level implementation of dataset listing that does not call fiftyone.core.dataset.list_datasets(), which is helpful if a database may be corrupted.

Returns: a list of Dataset names

fiftyone.core.odm.database.delete_dataset(name, dry_run=False)¶

Deletes the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Parameters

name – the name of the dataset
dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_annotation_run(name, anno_key, dry_run=False)¶

Deletes the annotation run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_annotation_run(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

name – the name of the dataset
anno_key – the annotation key
dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_annotation_runs(name, dry_run=False)¶

Deletes all annotation runs from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_annotation_runs(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

name – the name of the dataset
dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_brain_run(name, brain_key, dry_run=False)¶

Deletes the brain method run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_brain_run(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

name – the name of the dataset
brain_key – the brain key
dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_brain_runs(name, dry_run=False)¶

Deletes all brain method runs from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_brain_runs(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

name – the name of the dataset
dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_evaluation(name, eval_key, dry_run=False)¶

Deletes the evaluation run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_evaluation(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

name – the name of the dataset
eval_key – the evaluation key
dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_evaluations(name, dry_run=False)¶

Deletes all evaluations from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_evaluations(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

name – the name of the dataset
dry_run (False) – whether to log the actions that would be taken but not perform them