inginious.common.filesystems package

Abstract class for filesystems providers.

class inginious.common.filesystems.FileSystemProvider(prefix)

Bases: object

Provides tools to access a given filesystem. The filesystem may be distant, and subclasses of FileSystemProvider should take care of doing appropriate caching.

abstract copy_from(src, dest_disk)

Copy the content of src into the on-disk folder dest_disk. If src is None, copy from the prefix.

abstract copy_to(src_disk, dest=None)

Copy the content of on-disk folder src_disk into dir dest. If dest is None, copy to the prefix.

abstract delete(filepath=None)

Delete a path recursively. If filepath is None, then the prefix will be deleted.

Parameters:

filepath (Optional[str]) – The prefix entry to delete.

Raises:

FileNotFoundError – If filepath points to a non-existing entry in the prefix.

abstract distribute(filepath, allow_folders=True)

Give information on how to distribute a file. Provides Zip files of folders. Can return: (“file”, mimetype, fileobj) where fileobj is an object-like file (with read()) and mimetype its mime-type. (“url”, None, url) where url is a url to a distant server which possess the file. (“invalid”, None, None) if the file cannot be distributed

abstract ensure_exists()

Ensure that the current prefix exists. If it is not the case, creates the directory.

Return type:

None

abstract exists(path=None)

Parameters:

path (Optional[str]) – A path to verify.

Return type:

bool

Returns:

True if the file at the given path exists. If the path is not given, then checks the existence of the prefix.

abstract from_subfolder(subfolder)

Parameters:

subfolder (str) – The prefix of the new FileSystemProvider.

Return type:

FileSystemProvider

Returns:

A new FileSystemProvider, with subfolder as prefix.

abstract get(filepath, timestamp=None)

Get the content of a file. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises:

FileNotFoundError – If the file does not exists or cannot be retrieved.

abstract get_fd(filepath, timestamp=None)

Returns a file descriptor. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises:

• FileNotFoundError – if the file does not exists or cannot be retrieved.

• IsADirectoryError – if filepath points to a directory.

Returns:

A file descriptor pointing to filepath.

abstract get_last_modification_time(filepath)

Get a timestamp representing the time of the last modification of the file at filepath

abstract classmethod get_needed_args()

Returns a list of arguments needed to create a FileSystemProvider. In the form {

“arg1”: (int, False, “description1”), “arg2: (str, True, “description2”)

}

The first part of the tuple is the type, the second indicates if the arg is mandatory Only int and str are supported as types.

abstract classmethod init_from_args(**args)

Given the args from get_needed_args, creates the FileSystemProvider

abstract list(folders=True, files=True, recursive=False)

List all the files/folder in this prefix. Folders are always ending with a ‘/’

Parameters:

• folders (bool) – Switch to list folders.

• files (bool) – Switch to list files.

• recursive (bool) – Switch to list recursively the prefix content.

Return type:

list

Returns:

The list of files/folders in the prefix.

abstract move(src, dest)

Move path src to path dest, recursively.

abstract put(filepath, content)

Write content in filepath

Submodules

inginious.common.filesystems.local module

class inginious.common.filesystems.local.LocalFSProvider(prefix)

Bases: FileSystemProvider

A FileSystemProvider that uses a real on-disk filesystem

copy_from(src, dest_disk)

Copy the content of src into the on-disk folder dest_disk. If src is None, copy from the prefix.

copy_to(src_disk, dest=None)

Copy the content of on-disk folder src_disk into dir dest. If dest is None, copy to the prefix.

delete(filepath=None)

Delete a path recursively. If filepath is None, then the prefix will be deleted.

Parameters:

filepath (Optional[str]) – The prefix entry to delete.

Raises:

FileNotFoundError – If filepath points to a non-existing entry in the prefix.

distribute(filepath, allow_folders=True)

Give information on how to distribute a file. Provides Zip files of folders. Can return: (“file”, mimetype, fileobj) where fileobj is an object-like file (with read()) and mimetype its mime-type. (“url”, None, url) where url is a url to a distant server which possess the file. (“invalid”, None, None) if the file cannot be distributed

ensure_exists()

Ensure that the current prefix exists. If it is not the case, creates the directory.

exists(path=None)

Parameters:

path (Optional[str]) – A path to verify.

Return type:

bool

Returns:

True if the file at the given path exists. If the path is not given, then checks the existence of the prefix.

from_subfolder(subfolder)

Parameters:

subfolder (str) – The prefix of the new FileSystemProvider.

Return type:

LocalFSProvider

Returns:

A new FileSystemProvider, with subfolder as prefix.

get(filepath, timestamp=None)

Get the content of a file. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises:

FileNotFoundError – If the file does not exists or cannot be retrieved.

get_fd(filepath, timestamp=None)

Returns a file descriptor. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises:

• FileNotFoundError – if the file does not exists or cannot be retrieved.

• IsADirectoryError – if filepath points to a directory.

Returns:

A file descriptor pointing to filepath.

get_last_modification_time(filepath)

Get a timestamp representing the time of the last modification of the file at filepath

classmethod get_needed_args()

Returns a list of arguments needed to create a FileSystemProvider. In the form {

“arg1”: (int, False, “description1”), “arg2: (str, True, “description2”)

}

The first part of the tuple is the type, the second indicates if the arg is mandatory Only int and str are supported as types.

classmethod init_from_args(location)

Given the args from get_needed_args, creates the FileSystemProvider

list(folders=True, files=True, recursive=False)

List all the files/folder in this prefix. Folders are always ending with a ‘/’

Parameters:

• folders (bool) – Switch to list folders.

• files (bool) – Switch to list files.

• recursive (bool) – Switch to list recursively the prefix content.

Return type:

list

Returns:

The list of files/folders in the prefix.

move(src, dest)

Move path src to path dest, recursively.

put(filepath, content)

Write content in filepath