inginious.common.filesystems package

Abstract class for filesystems providers.

class inginious.common.filesystems.FileSystemProvider(prefix)[source]

Bases: object

Provides tools to access a given filesystem. The filesystem may be distant, and subclasses of FileSystemProvider should take care of doing appropriate caching.

abstract copy_from(src, dest_disk)[source]

Copy the content of src into the on-disk folder dest_disk. If src is None, copy from the prefix.

abstract copy_to(src_disk, dest=None)[source]

Copy the content of on-disk folder src_disk into dir dest. If dest is None, copy to the prefix.

abstract delete(filepath=None)[source]

Delete a path recursively. If filepath is None, then the prefix will be deleted.

Parameters

filepath (Optional[str]) – The prefix entry to delete.

Raises

FileNotFoundError – If filepath points to a non-existing entry in the prefix.

abstract distribute(filepath, allow_folders=True)[source]

Give information on how to distribute a file. Provides Zip files of folders. Can return: (“file”, mimetype, fileobj) where fileobj is an object-like file (with read()) and mimetype its mime-type. (“url”, None, url) where url is a url to a distant server which possess the file. (“invalid”, None, None) if the file cannot be distributed

abstract ensure_exists()[source]

Ensure that the current prefix exists. If it is not the case, creates the directory.

Return type

None

abstract exists(path=None)[source]
Parameters

path (Optional[str]) – A path to verify.

Return type

bool

Returns

True if the file at the given path exists. If the path is not given, then checks the existence of the prefix.

abstract from_subfolder(subfolder)[source]
Parameters

subfolder (str) – The prefix of the new FileSystemProvider.

Return type

FileSystemProvider

Returns

A new FileSystemProvider, with subfolder as prefix.

abstract get(filepath, timestamp=None)[source]

Get the content of a file. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises

FileNotFoundError – If the file does not exists or cannot be retrieved.

abstract get_fd(filepath, timestamp=None)[source]

Returns a file descriptor. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises
  • FileNotFoundError – if the file does not exists or cannot be retrieved.

  • IsADirectoryError – if filepath points to a directory.

Returns

A file descriptor pointing to filepath.

abstract get_last_modification_time(filepath)[source]

Get a timestamp representing the time of the last modification of the file at filepath

abstract classmethod get_needed_args()[source]

Returns a list of arguments needed to create a FileSystemProvider. In the form {

“arg1”: (int, False, “description1”), “arg2: (str, True, “description2”)

}

The first part of the tuple is the type, the second indicates if the arg is mandatory Only int and str are supported as types.

abstract classmethod init_from_args(**args)[source]

Given the args from get_needed_args, creates the FileSystemProvider

abstract list(folders=True, files=True, recursive=False)[source]

List all the files/folder in this prefix. Folders are always ending with a ‘/’

Parameters
  • folders (bool) – Switch to list folders.

  • files (bool) – Switch to list files.

  • recursive (bool) – Switch to list recursively the prefix content.

Return type

list

Returns

The list of files/folders in the prefix.

abstract move(src, dest)[source]

Move path src to path dest, recursively.

abstract put(filepath, content)[source]

Write content in filepath

Submodules

inginious.common.filesystems.local module

class inginious.common.filesystems.local.LocalFSProvider(prefix)[source]

Bases: inginious.common.filesystems.FileSystemProvider

A FileSystemProvider that uses a real on-disk filesystem

copy_from(src, dest_disk)[source]

Copy the content of src into the on-disk folder dest_disk. If src is None, copy from the prefix.

copy_to(src_disk, dest=None)[source]

Copy the content of on-disk folder src_disk into dir dest. If dest is None, copy to the prefix.

delete(filepath=None)[source]

Delete a path recursively. If filepath is None, then the prefix will be deleted.

Parameters

filepath (Optional[str]) – The prefix entry to delete.

Raises

FileNotFoundError – If filepath points to a non-existing entry in the prefix.

distribute(filepath, allow_folders=True)[source]

Give information on how to distribute a file. Provides Zip files of folders. Can return: (“file”, mimetype, fileobj) where fileobj is an object-like file (with read()) and mimetype its mime-type. (“url”, None, url) where url is a url to a distant server which possess the file. (“invalid”, None, None) if the file cannot be distributed

ensure_exists()[source]

Ensure that the current prefix exists. If it is not the case, creates the directory.

exists(path=None)[source]
Parameters

path (Optional[str]) – A path to verify.

Return type

bool

Returns

True if the file at the given path exists. If the path is not given, then checks the existence of the prefix.

from_subfolder(subfolder)[source]
Parameters

subfolder (str) – The prefix of the new FileSystemProvider.

Return type

LocalFSProvider

Returns

A new FileSystemProvider, with subfolder as prefix.

get(filepath, timestamp=None)[source]

Get the content of a file. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises

FileNotFoundError – If the file does not exists or cannot be retrieved.

get_fd(filepath, timestamp=None)[source]

Returns a file descriptor. If timestamp is not None, it gives an indication to the cache that the file must have been retrieved from the (possibly distant) filesystem since the timestamp.

Raises
  • FileNotFoundError – if the file does not exists or cannot be retrieved.

  • IsADirectoryError – if filepath points to a directory.

Returns

A file descriptor pointing to filepath.

get_last_modification_time(filepath)[source]

Get a timestamp representing the time of the last modification of the file at filepath

classmethod get_needed_args()[source]

Returns a list of arguments needed to create a FileSystemProvider. In the form {

“arg1”: (int, False, “description1”), “arg2: (str, True, “description2”)

}

The first part of the tuple is the type, the second indicates if the arg is mandatory Only int and str are supported as types.

classmethod init_from_args(location)[source]

Given the args from get_needed_args, creates the FileSystemProvider

list(folders=True, files=True, recursive=False)[source]

List all the files/folder in this prefix. Folders are always ending with a ‘/’

Parameters
  • folders (bool) – Switch to list folders.

  • files (bool) – Switch to list files.

  • recursive (bool) – Switch to list recursively the prefix content.

Return type

list

Returns

The list of files/folders in the prefix.

move(src, dest)[source]

Move path src to path dest, recursively.

put(filepath, content)[source]

Write content in filepath