Treants

Treants are the core units of functionality of datreant. They function as specially marked directories with distinguishing characteristics. They are designed to be subclassed, with their functionality extendable with attachable Limbs.

The components documented here are those included within datreant.core.

Treant

The class datreant.core.Treant is the central object of datreant.core.

class datreant.core.Treant(treant, new=False, categories=None, tags=None)

The Treant: a Tree with a state file.

treant should be a base directory of a new or existing Treant. An existing Treant will be regenerated if a state file is found. If no state file is found, a new Treant will be created.

A Tree object may also be used in the same way as a directory string.

If multiple Treant state files are in the given directory, a MultipleTreantsError will be raised; specify the full path to the desired state file to regenerate the desired Treant in this case. It is generally better to avoid having multiple state files in the same directory.

Use the new keyword to force generation of a new Treant at the given path.

Parameters:
  • treant (str or Tree) – Base directory of a new or existing Treant; will regenerate a Treant if a state file is found, but will genereate a new one otherwise; may also be a Tree object
  • new (bool) – Generate a new Treant even if one already exists at the given location
  • categories (dict) – dictionary with user-defined keys and values; used to give Treants distinguishing characteristics
  • tags (list) – list with user-defined values; like categories, but useful for adding many distinguishing descriptors
abspath

Absolute path of self.path.

attach(*limbname)

Attach limbs by name to this Treant.

categories

Interface to categories.

children

A View of all files and directories in this Tree.

Includes hidden files and directories.

discover(dirpath='.', depth=None, treantdepth=None)

Find all Treants within given directory, recursively.

Parameters:
  • dirpath (string, Tree) – Directory within which to search for Treants. May also be an existing Tree.
  • depth (int) – Maximum directory depth to tolerate while traversing in search of Treants. None indicates no depth limit.
  • treantdepth (int) – Maximum depth of Treants to tolerate while traversing in search of Treants. None indicates no Treant depth limit.
Returns:

found – Bundle of found Treants.

Return type:

Bundle

draw(depth=None, hidden=False)

Print an ASCII-fied visual of the tree.

Parameters:
  • depth (int) – Maximum directory depth to display. None indicates no limit.
  • hidden (bool) – If False, do not show hidden files; hidden directories are still shown if they contain non-hidden files or directories.
exists

Check existence of this path in filesystem.

filepath

Absolute path to the Treant’s state file.

glob(pattern)

Return a View of all child Leaves and Trees matching given globbing pattern.

Arguments:
pattern

globbing pattern to match files and directories with

hidden

A View of the hidden files and directories in this Tree.

leaves

A View of the files in this Tree.

Hidden files are not included.

limbs

A set giving the names of this object’s attached limbs.

loc

Get Tree/Leaf at path relative to Tree.

Use with getitem syntax, e.g. .loc['some name']

Allowed inputs are: - A single name - A list or array of names

If directory/file does not exist at the given path, then whether a Tree or Leaf is given is determined by the path semantics, i.e. a trailing separator (“/”).

Using e.g. Tree.loc['some name'] is equivalent to doing Tree['some name']. .loc is included for parity with View and Bundle API semantics.

location

The location of the Treant.

Setting the location to a new path physically moves the Treant to the given location. This only works if the new location is an empty or nonexistent directory.

make()

Make the directory if it doesn’t exist. Equivalent to makedirs().

Returns:This Tree.
Return type:Tree
makedirs()

Make all directories along path that do not currently exist.

Returns:
tree

this tree

name

The name of the Treant.

The name of a Treant need not be unique with respect to other Treants, but is used as part of Treant’s displayed representation.

parent

Parent directory for this path.

path

Treant directory as a pathlib2.Path.

relpath

Relative path of self.path from current working directory.

sync(other, mode='upload', compress=True, checksum=True, backup=False, dry=False, include=None, exclude=None, overwrite=False, rsync_path='/usr/bin/rsync')

Synchronize directories using rsync.

Parameters:
  • other (str or Tree) – Other end of the sync, can be either a path or another Tree.
  • mode (str) – Either "upload" if uploading to other, or "download" if downloading from other

The other options are described in the datreant.core.rsync.rsync() documentation.

tags

Interface to tags.

treants

Bundle of all Treants found within this Tree.

This does not return a Treant for a bare state file found within this Tree. In effect this gives the same result as Bundle(self.trees).

treanttype

The type of the Treant.

tree

This Treant’s directory as a Tree.

trees

A View of the directories in this Tree.

Hidden directories are not included.

uuid

Get Treant uuid.

A Treant’s uuid is used by other Treants to identify it. The uuid is given in the Treant’s state file name for fast filesystem searching. For example, a Treant with state file:

'Treant.7dd9305a-d7d9-4a7b-b513-adf5f4205e09.h5'

has uuid:

'7dd9305a-d7d9-4a7b-b513-adf5f4205e09'

Changing this string will alter the Treant’s uuid. This is not generally recommended.

Returns:
uuid

unique identifier string for this Treant

walk(topdown=True, onerror=None, followlinks=False)

Walk through the contents of the tree.

For each directory in the tree (including the root itself), yields a 3-tuple (dirpath, dirnames, filenames).

Parameters:
  • topdown (Boolean, optional) – If False, walks directories from the bottom-up.
  • onerror (function, optional) – Optional function to be called on error.
  • followlinks (Boolean, optional) – If False, excludes symbolic file links.
Returns:

Wrapped scandir.walk() generator yielding datreant objects

Return type:

generator

Tags

The class datreant.core.limbs.Tags is the interface used by Treants to access their tags.

class datreant.core.limbs.Tags(treant)

Interface to tags.

add(*tags)

Add any number of tags to the Treant.

Tags are individual strings that serve to differentiate Treants from one another. Sometimes preferable to categories.

Parameters:tags (str or list) – Tags to add. Must be strings or lists of strings.
clear()

Remove all tags from Treant.

fuzzy(tag, threshold=80)

Get a tuple of existing tags that fuzzily match a given one.

Parameters:
  • tags (str or list) – Tag or tags to get fuzzy matches for.
  • threshold (int) – Lowest match score to return. Setting to 0 will return every tag, while setting to 100 will return only exact matches.
Returns:

matches – Tuple of tags that match.

Return type:

tuple

remove(*tags)

Remove tags from Treant.

Any number of tags can be given as arguments, and these will be deleted.

Arguments:
tags

Tags to delete.

Categories

The class datreant.core.limbs.Categories is the interface used by Treants to access their categories.

class datreant.core.limbs.Categories(treant)

Interface to categories.

add(categorydict=None, **categories)

Add any number of categories to the Treant.

Categories are key-value pairs that serve to differentiate Treants from one another. Sometimes preferable to tags.

If a given category already exists (same key), the value given will replace the value for that category.

Keys must be strings.

Values may be ints, floats, strings, or bools. None as a value will not the existing value for the key, if present.

Parameters:
  • categorydict (dict) – Dict of categories to add; keys used as keys, values used as values.
  • categories (dict) – Categories to add. Keyword used as key, value used as value.
clear()

Remove all categories from Treant.

keys()

Get category keys.

Returns:
keys

keys present among categories

remove(*categories)

Remove categories from Treant.

Any number of categories (keys) can be given as arguments, and these keys (with their values) will be deleted.

Parameters:categories (str) – Categories to delete.
values()

Get category values.

Returns:
values

values present among categories

Group

The class datreant.core.Group is a Treant with the ability to store member locations as a persistent Bundle within its state file.

class datreant.core.Group(treant, new=False, categories=None, tags=None)

A Treant with a persistent Bundle of other Treants.

treant should be a base directory of a new or existing Group. An existing Group will be regenerated if a state file is found. If no state file is found, a new Group will be created.

A Tree object may also be used in the same way as a directory string.

If multiple Treant/Group state files are in the given directory, a MultipleTreantsError will be raised; specify the full path to the desired state file to regenerate the desired Group in this case. It is generally better to avoid having multiple state files in the same directory.

Use the new keyword to force generation of a new Group at the given path.

Parameters:
  • treant (str or Tree) – Base directory of a new or existing Group; will regenerate a Group if a state file is found, but will genereate a new one otherwise; may also be a Tree object
  • new (bool) – Generate a new Group even if one already exists at the given location
  • categories (dict) – dictionary with user-defined keys and values; used to give Groups distinguishing characteristics
  • tags (list) – list with user-defined values; like categories, but useful for adding many distinguishing descriptors
abspath

Absolute path of self.path.

attach(*limbname)

Attach limbs by name to this Treant.

categories

Interface to categories.

children

A View of all files and directories in this Tree.

Includes hidden files and directories.

discover(dirpath='.', depth=None, treantdepth=None)

Find all Treants within given directory, recursively.

Parameters:
  • dirpath (string, Tree) – Directory within which to search for Treants. May also be an existing Tree.
  • depth (int) – Maximum directory depth to tolerate while traversing in search of Treants. None indicates no depth limit.
  • treantdepth (int) – Maximum depth of Treants to tolerate while traversing in search of Treants. None indicates no Treant depth limit.
Returns:

found – Bundle of found Treants.

Return type:

Bundle

draw(depth=None, hidden=False)

Print an ASCII-fied visual of the tree.

Parameters:
  • depth (int) – Maximum directory depth to display. None indicates no limit.
  • hidden (bool) – If False, do not show hidden files; hidden directories are still shown if they contain non-hidden files or directories.
exists

Check existence of this path in filesystem.

filepath

Absolute path to the Treant’s state file.

glob(pattern)

Return a View of all child Leaves and Trees matching given globbing pattern.

Arguments:
pattern

globbing pattern to match files and directories with

hidden

A View of the hidden files and directories in this Tree.

leaves

A View of the files in this Tree.

Hidden files are not included.

limbs

A set giving the names of this object’s attached limbs.

loc

Get Tree/Leaf at path relative to Tree.

Use with getitem syntax, e.g. .loc['some name']

Allowed inputs are: - A single name - A list or array of names

If directory/file does not exist at the given path, then whether a Tree or Leaf is given is determined by the path semantics, i.e. a trailing separator (“/”).

Using e.g. Tree.loc['some name'] is equivalent to doing Tree['some name']. .loc is included for parity with View and Bundle API semantics.

location

The location of the Treant.

Setting the location to a new path physically moves the Treant to the given location. This only works if the new location is an empty or nonexistent directory.

make()

Make the directory if it doesn’t exist. Equivalent to makedirs().

Returns:This Tree.
Return type:Tree
makedirs()

Make all directories along path that do not currently exist.

Returns:
tree

this tree

members

Persistent Bundle for Groups.

name

The name of the Treant.

The name of a Treant need not be unique with respect to other Treants, but is used as part of Treant’s displayed representation.

parent

Parent directory for this path.

path

Treant directory as a pathlib2.Path.

relpath

Relative path of self.path from current working directory.

sync(other, mode='upload', compress=True, checksum=True, backup=False, dry=False, include=None, exclude=None, overwrite=False, rsync_path='/usr/bin/rsync')

Synchronize directories using rsync.

Parameters:
  • other (str or Tree) – Other end of the sync, can be either a path or another Tree.
  • mode (str) – Either "upload" if uploading to other, or "download" if downloading from other

The other options are described in the datreant.core.rsync.rsync() documentation.

tags

Interface to tags.

treants

Bundle of all Treants found within this Tree.

This does not return a Treant for a bare state file found within this Tree. In effect this gives the same result as Bundle(self.trees).

treanttype

The type of the Treant.

tree

This Treant’s directory as a Tree.

trees

A View of the directories in this Tree.

Hidden directories are not included.

uuid

Get Treant uuid.

A Treant’s uuid is used by other Treants to identify it. The uuid is given in the Treant’s state file name for fast filesystem searching. For example, a Treant with state file:

'Treant.7dd9305a-d7d9-4a7b-b513-adf5f4205e09.h5'

has uuid:

'7dd9305a-d7d9-4a7b-b513-adf5f4205e09'

Changing this string will alter the Treant’s uuid. This is not generally recommended.

Returns:
uuid

unique identifier string for this Treant

walk(topdown=True, onerror=None, followlinks=False)

Walk through the contents of the tree.

For each directory in the tree (including the root itself), yields a 3-tuple (dirpath, dirnames, filenames).

Parameters:
  • topdown (Boolean, optional) – If False, walks directories from the bottom-up.
  • onerror (function, optional) – Optional function to be called on error.
  • followlinks (Boolean, optional) – If False, excludes symbolic file links.
Returns:

Wrapped scandir.walk() generator yielding datreant objects

Return type:

generator

Members

The class datreant.core.limbs.MemberBundle is the interface used by a Group to manage its members. Its API matches that of datreant.core.Bundle.

class datreant.core.limbs.MemberBundle(treant)

Persistent Bundle for Groups.

abspaths

Return a list of absolute member directory paths.

Members that can’t be found will have path None.

Returns:
names

list giving the absolute directory path of each member, in order; members that are missing will have path None

add(*treants)

Add any number of members to this collection.

Arguments:
treants

treants to be added, which may be nested lists of treants; treants can be given as either objects or paths to directories that contain treant statefiles; glob patterns are also allowed, and all found treants will be added to the collection

attach(*agglimbname)

Attach agglimbs by name to this collection. Attaches corresponding limb to member Treants.

categories

Interface to categories.

clear()

Remove all members.

draw(depth=None, hidden=False)

Print an ASCII-fied visual of all member Trees.

Parameters:
  • depth (int) – Maximum directory depth to display. None indicates no limit.
  • hidden (bool) – If False, do not show hidden files; hidden directories are still shown if they contain non-hidden files or directories.
filepaths

Return a list of member filepaths.

Members that can’t be found will have filepath None.

Returns:
names

list giving the filepath of each member, in order; members that are missing will have filepath None

flatten(exclude=None)

Return a flattened version of this Bundle.

The resulting Bundle will have all members of any member Groups, without the Groups.

Parameters:exclude (list) – uuids of Groups to leave out of flattening; these will not in the resulting Bundle.
Returns:flattened – the flattened Bundle with no Groups
Return type:Bundle
glob(pattern)

Return a View of all child Leaves and Trees of members matching given globbing pattern.

Parameters:pattern (string) – globbing pattern to match files and directories with
globfilter(pattern)

Return a Bundle of members that match by name the given globbing pattern.

Parameters:pattern (string) – globbing pattern to match member names with
limbs

A set giving the names of this collection’s attached limbs.

loc

Get a View giving Tree/Leaf at path relative to each Tree in collection.

Use with getitem syntax, e.g. .loc['some name']

Allowed inputs are: - A single name - A list or array of names

If directory/file does not exist at the given path, then whether a Tree or Leaf is given is determined by the path semantics, i.e. a trailing separator (“/”).

map(function, processes=1, **kwargs)

Apply a function to each member, perhaps in parallel.

A pool of processes is created for processes > 1; for example, with 40 members and ‘processes=4’, 4 processes will be created, each working on a single member at any given time. When each process completes work on a member, it grabs another, until no members remain.

kwargs are passed to the given function when applied to each member

Arguments:
function

function to apply to each member; must take only a single treant instance as input, but may take any number of keyword arguments

Keywords:
processes

how many processes to use; if 1, applies function to each member in member order

Returns:
results

list giving the result of the function for each member, in member order; if the function returns None for each member, then only None is returned instead of a list

names

Return a list of member names.

Members that can’t be found will have name None.

Returns:
names

list giving the name of each member, in order; members that are missing will have name None

relpaths

Return a list of relative member directory paths.

Members that can’t be found will have path None.

Returns:
names

list giving the relative directory path of each member, in order; members that are missing will have path None

remove(*members)

Remove any number of members from the collection.

Arguments:
members

instances or indices of the members to remove

searchtime

Max time to spend searching for missing members, in seconds.

Setting a larger value allows more time for the collection to look for members elsewhere in the filesystem.

If None, there will be no time limit. Use with care.

tags

Interface to aggregated tags.

treanttypes

Return a list of member treanttypes.

uuids

Return a list of member uuids.

Returns:
uuids

list giving the uuid of each member, in order

view

Obtain a View giving the Tree for each Treant in this Bundle.