Treants¶

Treants are the core units of functionality of datreant. They function as specially marked directories with distinguishing characteristics. They are designed to be subclassed, with their functionality extendable with attachable Limbs.

The components documented here are those included within datreant.core.

Treant¶

The class datreant.core.Treant is the central object of datreant.core.

class datreant.core.Treant(treant, new=False, categories=None, tags=None)¶

The Treant: a Tree with a state file.

treant should be a base directory of a new or existing Treant. An existing Treant will be regenerated if a state file is found. If no state file is found, a new Treant will be created.

A Tree object may also be used in the same way as a directory string.

If multiple Treant state files are in the given directory, a MultipleTreantsError will be raised; specify the full path to the desired state file to regenerate the desired Treant in this case. It is generally better to avoid having multiple state files in the same directory.

Use the new keyword to force generation of a new Treant at the given path.

Parameters:

treant (str or Tree) – Base directory of a new or existing Treant; will regenerate a Treant if a state file is found, but will genereate a new one otherwise; may also be a Tree object
new (bool) – Generate a new Treant even if one already exists at the given location
categories (dict) – dictionary with user-defined keys and values; used to give Treants distinguishing characteristics
tags (list) – list with user-defined values; like categories, but useful for adding many distinguishing descriptors

abspath¶: Absolute path of self.path.

attach(*limbname)¶: Attach limbs by name to this Treant.

categories¶: Interface to categories.

children¶

A View of all files and directories in this Tree.

Includes hidden files and directories.

discover(dirpath='.', depth=None, treantdepth=None)¶

Find all Treants within given directory, recursively.

Parameters:	dirpath (string, Tree) – Directory within which to search for Treants. May also be an existing Tree. depth (int) – Maximum directory depth to tolerate while traversing in search of Treants. `None` indicates no depth limit. treantdepth (int) – Maximum depth of Treants to tolerate while traversing in search of Treants. `None` indicates no Treant depth limit.
Returns:	found – Bundle of found Treants.
Return type:	Bundle

draw(depth=None, hidden=False)¶

Print an ASCII-fied visual of the tree.

Parameters:	depth (int) – Maximum directory depth to display. `None` indicates no limit. hidden (bool) – If False, do not show hidden files; hidden directories are still shown if they contain non-hidden files or directories.

exists¶: Check existence of this path in filesystem.

filepath¶: Absolute path to the Treant’s state file.

glob(pattern)¶

Return a View of all child Leaves and Trees matching given globbing pattern.

Arguments:	pattern globbing pattern to match files and directories with

hidden¶: A View of the hidden files and directories in this Tree.

leaves¶

A View of the files in this Tree.

Hidden files are not included.

limbs¶: A set giving the names of this object’s attached limbs.

loc¶

Get Tree/Leaf at path relative to Tree.

Use with getitem syntax, e.g. .loc['some name']

Allowed inputs are: - A single name - A list or array of names

If directory/file does not exist at the given path, then whether a Tree or Leaf is given is determined by the path semantics, i.e. a trailing separator (“/”).

Using e.g. Tree.loc['some name'] is equivalent to doing Tree['some name']. .loc is included for parity with View and Bundle API semantics.

location¶

The location of the Treant.

Setting the location to a new path physically moves the Treant to the given location. This only works if the new location is an empty or nonexistent directory.

make()¶

Make the directory if it doesn’t exist. Equivalent to makedirs().

Returns:	This Tree.
Return type:	Tree

makedirs()¶

Make all directories along path that do not currently exist.

Returns:	tree this tree

name¶

The name of the Treant.

The name of a Treant need not be unique with respect to other Treants, but is used as part of Treant’s displayed representation.

parent¶: Parent directory for this path.

path¶: Treant directory as a pathlib2.Path.

relpath¶: Relative path of self.path from current working directory.

sync(other, mode='upload', compress=True, checksum=True, backup=False, dry=False, include=None, exclude=None, overwrite=False, rsync_path='/usr/bin/rsync')¶

Synchronize directories using rsync.

Parameters:	other (str or Tree) – Other end of the sync, can be either a path or another Tree. mode (str) – Either `"upload"` if uploading to other, or `"download"` if downloading from other

The other options are described in the datreant.core.rsync.rsync() documentation.

tags¶: Interface to tags.

treants¶

Bundle of all Treants found within this Tree.

This does not return a Treant for a bare state file found within this Tree. In effect this gives the same result as Bundle(self.trees).

treanttype¶: The type of the Treant.

tree¶: This Treant’s directory as a Tree.

trees¶

A View of the directories in this Tree.

Hidden directories are not included.

uuid¶

Get Treant uuid.

A Treant’s uuid is used by other Treants to identify it. The uuid is given in the Treant’s state file name for fast filesystem searching. For example, a Treant with state file:

'Treant.7dd9305a-d7d9-4a7b-b513-adf5f4205e09.h5'

has uuid:

'7dd9305a-d7d9-4a7b-b513-adf5f4205e09'

Changing this string will alter the Treant’s uuid. This is not generally recommended.

Returns:	uuid unique identifier string for this Treant

Tags¶

The class datreant.core.limbs.Tags is the interface used by Treants to access their tags.

class datreant.core.limbs.Tags(treant)¶

Interface to tags.

add(*tags)¶

Add any number of tags to the Treant.

Tags are individual strings that serve to differentiate Treants from one another. Sometimes preferable to categories.

Parameters:	tags (str or list) – Tags to add. Must be strings or lists of strings.

clear()¶: Remove all tags from Treant.

fuzzy(tag, threshold=80)¶

Get a tuple of existing tags that fuzzily match a given one.

Parameters:	tags (str or list) – Tag or tags to get fuzzy matches for. threshold (int) – Lowest match score to return. Setting to 0 will return every tag, while setting to 100 will return only exact matches.
Returns:	matches – Tuple of tags that match.
Return type:	tuple

remove(*tags)¶

Remove tags from Treant.

Any number of tags can be given as arguments, and these will be deleted.

Arguments:	tags Tags to delete.

Categories¶

The class datreant.core.limbs.Categories is the interface used by Treants to access their categories.

class datreant.core.limbs.Categories(treant)¶

Interface to categories.

add(categorydict=None, **categories)¶

Add any number of categories to the Treant.

Categories are key-value pairs that serve to differentiate Treants from one another. Sometimes preferable to tags.

If a given category already exists (same key), the value given will replace the value for that category.

Keys must be strings.

Values may be ints, floats, strings, or bools. None as a value will not the existing value for the key, if present.

Parameters:	categorydict (dict) – Dict of categories to add; keys used as keys, values used as values. categories (dict) – Categories to add. Keyword used as key, value used as value.

clear()¶: Remove all categories from Treant.

keys()¶

Get category keys.

Returns:	keys keys present among categories

remove(*categories)¶

Remove categories from Treant.

Any number of categories (keys) can be given as arguments, and these keys (with their values) will be deleted.

Parameters:	categories (str) – Categories to delete.

values()¶

Get category values.

Returns:	values values present among categories

Group¶

The class datreant.core.Group is a Treant with the ability to store member locations as a persistent Bundle within its state file.

class datreant.core.Group(treant, new=False, categories=None, tags=None)¶

A Treant with a persistent Bundle of other Treants.

treant should be a base directory of a new or existing Group. An existing Group will be regenerated if a state file is found. If no state file is found, a new Group will be created.

A Tree object may also be used in the same way as a directory string.

If multiple Treant/Group state files are in the given directory, a MultipleTreantsError will be raised; specify the full path to the desired state file to regenerate the desired Group in this case. It is generally better to avoid having multiple state files in the same directory.

Use the new keyword to force generation of a new Group at the given path.

Parameters:

treant (str or Tree) – Base directory of a new or existing Group; will regenerate a Group if a state file is found, but will genereate a new one otherwise; may also be a Tree object
new (bool) – Generate a new Group even if one already exists at the given location
categories (dict) – dictionary with user-defined keys and values; used to give Groups distinguishing characteristics
tags (list) – list with user-defined values; like categories, but useful for adding many distinguishing descriptors

abspath¶: Absolute path of self.path.

attach(*limbname)¶: Attach limbs by name to this Treant.

categories¶: Interface to categories.

children¶

A View of all files and directories in this Tree.

Includes hidden files and directories.

discover(dirpath='.', depth=None, treantdepth=None)¶

Find all Treants within given directory, recursively.

Parameters:	dirpath (string, Tree) – Directory within which to search for Treants. May also be an existing Tree. depth (int) – Maximum directory depth to tolerate while traversing in search of Treants. `None` indicates no depth limit. treantdepth (int) – Maximum depth of Treants to tolerate while traversing in search of Treants. `None` indicates no Treant depth limit.
Returns:	found – Bundle of found Treants.
Return type:	Bundle

draw(depth=None, hidden=False)¶

Print an ASCII-fied visual of the tree.

Parameters:	depth (int) – Maximum directory depth to display. `None` indicates no limit. hidden (bool) – If False, do not show hidden files; hidden directories are still shown if they contain non-hidden files or directories.

exists¶: Check existence of this path in filesystem.

filepath¶: Absolute path to the Treant’s state file.

glob(pattern)¶

Return a View of all child Leaves and Trees matching given globbing pattern.

Arguments:	pattern globbing pattern to match files and directories with

hidden¶: A View of the hidden files and directories in this Tree.

leaves¶

A View of the files in this Tree.

Hidden files are not included.

limbs¶: A set giving the names of this object’s attached limbs.

loc¶

Get Tree/Leaf at path relative to Tree.

Use with getitem syntax, e.g. .loc['some name']

Allowed inputs are: - A single name - A list or array of names

If directory/file does not exist at the given path, then whether a Tree or Leaf is given is determined by the path semantics, i.e. a trailing separator (“/”).

Using e.g. Tree.loc['some name'] is equivalent to doing Tree['some name']. .loc is included for parity with View and Bundle API semantics.

location¶

The location of the Treant.

Setting the location to a new path physically moves the Treant to the given location. This only works if the new location is an empty or nonexistent directory.

make()¶

Make the directory if it doesn’t exist. Equivalent to makedirs().

Returns:	This Tree.
Return type:	Tree

makedirs()¶

Make all directories along path that do not currently exist.

Returns:	tree this tree

members¶: Persistent Bundle for Groups.

name¶

The name of the Treant.

The name of a Treant need not be unique with respect to other Treants, but is used as part of Treant’s displayed representation.

parent¶: Parent directory for this path.

path¶: Treant directory as a pathlib2.Path.

relpath¶: Relative path of self.path from current working directory.

sync(other, mode='upload', compress=True, checksum=True, backup=False, dry=False, include=None, exclude=None, overwrite=False, rsync_path='/usr/bin/rsync')¶

Synchronize directories using rsync.

Parameters:	other (str or Tree) – Other end of the sync, can be either a path or another Tree. mode (str) – Either `"upload"` if uploading to other, or `"download"` if downloading from other

The other options are described in the datreant.core.rsync.rsync() documentation.

tags¶: Interface to tags.

treants¶

Bundle of all Treants found within this Tree.

This does not return a Treant for a bare state file found within this Tree. In effect this gives the same result as Bundle(self.trees).

treanttype¶: The type of the Treant.

tree¶: This Treant’s directory as a Tree.

trees¶

A View of the directories in this Tree.

Hidden directories are not included.

uuid¶

Get Treant uuid.

A Treant’s uuid is used by other Treants to identify it. The uuid is given in the Treant’s state file name for fast filesystem searching. For example, a Treant with state file:

'Treant.7dd9305a-d7d9-4a7b-b513-adf5f4205e09.h5'

has uuid:

'7dd9305a-d7d9-4a7b-b513-adf5f4205e09'

Changing this string will alter the Treant’s uuid. This is not generally recommended.

Returns:	uuid unique identifier string for this Treant

Members¶

The class datreant.core.limbs.MemberBundle is the interface used by a Group to manage its members. Its API matches that of datreant.core.Bundle.

class datreant.core.limbs.MemberBundle(treant)¶

Persistent Bundle for Groups.

abspaths¶

Return a list of absolute member directory paths.

Members that can’t be found will have path None.

Returns:	names list giving the absolute directory path of each member, in order; members that are missing will have path `None`

add(*treants)¶

Add any number of members to this collection.

Arguments:	treants treants to be added, which may be nested lists of treants; treants can be given as either objects or paths to directories that contain treant statefiles; glob patterns are also allowed, and all found treants will be added to the collection

attach(*agglimbname)¶: Attach agglimbs by name to this collection. Attaches corresponding limb to member Treants.

categories¶: Interface to categories.

clear()¶: Remove all members.

draw(depth=None, hidden=False)¶

Print an ASCII-fied visual of all member Trees.

Parameters:	depth (int) – Maximum directory depth to display. `None` indicates no limit. hidden (bool) – If False, do not show hidden files; hidden directories are still shown if they contain non-hidden files or directories.

filepaths¶

Return a list of member filepaths.

Members that can’t be found will have filepath None.

Returns:	names list giving the filepath of each member, in order; members that are missing will have filepath `None`

flatten(exclude=None)¶

Return a flattened version of this Bundle.

The resulting Bundle will have all members of any member Groups, without the Groups.

Parameters:	exclude (list) – uuids of Groups to leave out of flattening; these will not in the resulting Bundle.
Returns:	flattened – the flattened Bundle with no Groups
Return type:	Bundle

glob(pattern)¶

Return a View of all child Leaves and Trees of members matching given globbing pattern.

Parameters:	pattern (string) – globbing pattern to match files and directories with

globfilter(pattern)¶

Return a Bundle of members that match by name the given globbing pattern.

Parameters:	pattern (string) – globbing pattern to match member names with

limbs¶: A set giving the names of this collection’s attached limbs.

loc¶

Get a View giving Tree/Leaf at path relative to each Tree in collection.

Use with getitem syntax, e.g. .loc['some name']

Allowed inputs are: - A single name - A list or array of names

If directory/file does not exist at the given path, then whether a Tree or Leaf is given is determined by the path semantics, i.e. a trailing separator (“/”).

map(function, processes=1, **kwargs)¶

Apply a function to each member, perhaps in parallel.

A pool of processes is created for processes > 1; for example, with 40 members and ‘processes=4’, 4 processes will be created, each working on a single member at any given time. When each process completes work on a member, it grabs another, until no members remain.

kwargs are passed to the given function when applied to each member

Arguments:	function function to apply to each member; must take only a single treant instance as input, but may take any number of keyword arguments
Keywords:	processes how many processes to use; if 1, applies function to each member in member order
Returns:	results list giving the result of the function for each member, in member order; if the function returns `None` for each member, then only `None` is returned instead of a list

names¶

Return a list of member names.

Members that can’t be found will have name None.

Returns:	names list giving the name of each member, in order; members that are missing will have name `None`

relpaths¶

Return a list of relative member directory paths.

Members that can’t be found will have path None.

Returns:	names list giving the relative directory path of each member, in order; members that are missing will have path `None`

remove(*members)¶

Remove any number of members from the collection.

Arguments:	members instances or indices of the members to remove

searchtime¶

Max time to spend searching for missing members, in seconds.

Setting a larger value allows more time for the collection to look for members elsewhere in the filesystem.

If None, there will be no time limit. Use with care.

tags¶: Interface to aggregated tags.

treanttypes¶: Return a list of member treanttypes.

uuids¶

Return a list of member uuids.

Returns:	uuids list giving the uuid of each member, in order

view¶: Obtain a View giving the Tree for each Treant in this Bundle.