palladium package¶

Submodules¶

palladium.R module¶

palladium.cache module¶

The cache module provides caching utilities in order to provide faster access to data which is needed repeatedly. The disk cache (diskcache) which is primarily used during development when loading data from the local harddisk is faster than querying a remote database.

class palladium.cache.abstractcache(compute_key=None, ignore=False)¶

Bases: object

An abstract class for providing basic functionality for caching function calls. It contains the handling of keys used for caching objects.

palladium.cache.compute_key_attrs(attrs)¶

class palladium.cache.diskcache(*args, filename_tmpl=None, **kwargs)¶

Bases: palladium.cache.abstractcache

The disk cache stores results of function calls as pickled files to disk. Usually used during development and evaluation to save costly DB interactions in repeated calls with the same data.

Note: Should changes to the database or to your functions require you to purge existing cached values, then those cache files are found in the location defined in filename_tmpl.

class palladium.cache.picklediskcache(*args, filename_tmpl=None, **kwargs)¶

Bases: palladium.cache.diskcache

Same as diskcache, except that standard pickle is used instead of joblib’s pickle functionality.

::

palladium.config module¶

class palladium.config.ComponentHandler(config)¶: Bases: object

class palladium.config.Config¶

Bases: dict

A dictionary that represents the app’s configuration.

Tries to send a more user friendly message in case of KeyError.

class palladium.config.CopyHandler(configs)¶: Bases: object

class palladium.config.PythonHandler(config)¶: Bases: object

palladium.config.get_config(**extra)¶

palladium.config.initialize_config(**extra)¶

palladium.config.process_config(*configs, handlers0=<function _handlers_phase0>, handlers1=<function _handlers_phase1>, handlers2=<function _handlers_phase2>)¶

palladium.dataset module¶

DatasetLoader implementations.

class palladium.dataset.EmptyDatasetLoader¶

Bases: palladium.interfaces.DatasetLoader

This DatasetLoader can be used if no actual data should be loaded. Returns a (None, None) tuple.

class palladium.dataset.SQL(url, sql, target_column=None, ndarray=True, **kwargs)¶

Bases: palladium.interfaces.DatasetLoader

A DatasetLoader that uses pandas.io.sql.read_sql() to load data from an SQL database. Supports all databases that SQLAlchemy has support for.

class palladium.dataset.ScheduledDatasetLoader(impl, update_cache_rrule)¶

Bases: palladium.interfaces.DatasetLoader

A DatasetLoader that loads periodically data into RAM to make it available to the prediction server inside the process_store.

ScheduledDatasetLoader wraps another DatasetLoader class that it uses to do the actual loading of the data.

An update_cache_rrule is used to define how often data should be loaded anew.

This class’ read() read method never calls the underlying dataset loader. It will only ever fetch the data from the in-memory cache.

cache = {'process_metadata': {}}¶

::

class palladium.dataset.Table(path, target_column=None, ndarray=True, **kwargs)¶

Bases: palladium.interfaces.DatasetLoader

A DatasetLoader that uses pandas.io.parsers.read_table() to load data from a file or URL.

palladium.eval module¶

Utilities for testing the performance of a trained model.

palladium.eval.list(model_persister)¶

palladium.eval.list_cmd(argv=['-T', '-b', 'readthedocs', '-d', '_build/doctrees-readthedocs', '-D', 'language=en', '.', '_build/html'])¶

List information about available models.

Uses the ‘model_persister’ from the configuration to display a list of models and their metadata.

Usage:: pld-list [options]
Options:: -h –help Show this screen.

palladium.eval.test(dataset_loader_test, model_persister, scoring=None, model_version=None)¶

palladium.eval.test_cmd(argv=['-T', '-b', 'readthedocs', '-d', '_build/doctrees-readthedocs', '-D', 'language=en', '.', '_build/html'])¶

Test a model.

Uses ‘dataset_loader_test’ and ‘model_persister’ from the configuration to load a test dataset to test the accuracy of a trained model with.

Usage:

pld-test [options]

Options:

-h –help Show this screen.

`--model-version=<version>`
	The version of the model to be tested. If not specified, the newest model will be used.

palladium.fit module¶

palladium.interfaces module¶

Interfaces defining the behaviour of Palladium’s components.

class palladium.interfaces.CrossValidationGenerator¶

Bases: object

A CrossValidationGenerator provides train/test indices to split data in train and validation sets.

CrossValidationGenerator corresponds to the cross validation generator interface of scikit-learn.

class palladium.interfaces.DatasetLoader¶

Bases: object

A DatasetLoader is responsible for loading datasets for use in training and evaluation.

class palladium.interfaces.DatasetLoaderMeta(name, bases, attrs, **kwargs)¶: Bases: abc.ABCMeta

class palladium.interfaces.Model¶

Bases: Dummy

A Model can be fit() to data and can be used to predict() data.

Model corresponds to the estimators interface of scikit-learn.

Fit to data array *X* and possibly a target array *y*.

:return: self

Predict classes for data array *X* with shape n x m.

Some models may accept additional keyword arguments.

:return:
  A numpy array of length n with the predicted classes (for
  classification problems) or numeric values (for regression
  problems).

:raises:
  May raise a :class:`PredictError` to indicate that some
  condition made it impossible to deliver a prediction.

Predict probabilities for data array *X* with shape n x m.

:return:
  A numpy array of length n x c with a list class
  probabilities per sample.

:raises:
  :class:`NotImplementedError` if not applicable.

class palladium.interfaces.ModelPersister¶

Bases: object

Set the model with the given *version* to be the active
one.

Implies that any previously active model becomes inactive.

:param str version:
  The *version* of the model that's activated.

:raises:
  LookupError if no model with given *version* exists.

Delete the model with the given *version* from the
database.

:param str version:
  The *version* of the model that's activated.

:raises:
  LookupError if no model with given *version* exists.

List metadata of all available models.

:return:
  A list of dicts, with each dict containing information about
  one of the available models.  Each dict is guaranteed to
  contain the ``version`` key, which is the same version
  number that :meth:`ModelPersister.read` accepts for loading
  specific models.

List properties of :class:`ModelPersister` itself.

:return:
  A dictionary of key and value pairs, where both keys and
  values are of type ``str``.  Properties will usually include
  ``active-model`` and ``db-version`` entries.

Returns a :class:`Model` instance.

:param str version:
  *version* may be used to read a specific version of a model.
  If *version* is ``None``, returns the active model.

:return:
  The model object.

:raises:
  LookupError if no model was available.

Upgrade the underlying database to the latest version.

Newer versions of Palladium may require changes to the
:class:`ModelPersister`'s database.  This method provides an
opportunity to run the necessary upgrade steps.

It's the :class:`ModelPersister`'s responsibility to keep
track of the Palladium version that was used to create and
upgrade its database, and thus to determine the upgrade steps
necessary.

Persists a :class:`Model` and returns a new version number.

It is the :class:`ModelPersister`'s responsibility to annotate
the 'version' information onto the model before it is saved.

The new model will initially be inactive.  Use
:meth:`ModelPersister.activate` to activate the model.

:return:
  The new model's version identifier.

class palladium.interfaces.ModelPersisterMeta(name, bases, attrs, **kwargs)¶: Bases: abc.ABCMeta

exception palladium.interfaces.PredictError(error_message, error_code=-1)¶

Bases: Exception

Raised by Model.predict() to indicate that some condition made it impossible to deliver a prediction.

class palladium.interfaces.PredictService¶

Bases: object

Responsible for producing the output for the ‘/predict’ HTTP endpoint.

palladium.interfaces.annotate(obj, metadata=None)¶

palladium.julia module¶

class palladium.julia.AbstractModel(fit_func, predict_func, fit_kwargs=None, predict_kwargs=None, encode_labels=False)¶

Bases: palladium.interfaces.Model

::

class palladium.julia.ClassificationModel(fit_func, predict_func, fit_kwargs=None, predict_kwargs=None, encode_labels=False)¶: Bases: palladium.julia.AbstractModel

palladium.julia.make_bridge()¶

palladium.persistence module¶

ModelPersister implementations.

class palladium.persistence.CachedUpdatePersister(impl, update_cache_rrule=None, check_version=True)¶

Bases: palladium.interfaces.ModelPersister

A ModelPersister that serves as a caching decorator around another ~palladium.interfaces.ModelPersister object.

Calls to read() will look up a model from the global process_store, i.e. there is never any actual loading involved when calling read().

To fill the process_store cache periodically using the return value of the underlying ModelPersister‘s read method, a dictionary containing keyword arguments to dateutil.rrule.rrule may be passed. The cache will then be filled periodically according to that recurrence rule.

If no update_cache_rrule is used, the CachedUpdatePersister will call once and remember the return value of the underlying ModelPersister‘s read method during initialization.

.. py:attribute:: CachedUpdatePersister.cache

module: palladium.persistence

annotation: = {‘process_metadata’: {}}

::

::

::

::

class palladium.persistence.Database(url, poolclass=None, chunk_size=104857600, table_postfix='')¶

Bases: palladium.interfaces.ModelPersister

A ModelPersister that pickles models into an SQL database.

::

::

::

::

::

upgrade_steps = <palladium.persistence.UpgradeSteps object>¶

class palladium.persistence.DatabaseCLOB(url, poolclass=None, chunk_size=104857600, table_postfix='')¶

Bases: palladium.persistence.Database

A ModelPersister derived from Database, with only the slight difference of using CLOB instead of BLOB to store the pickle data.

Use when BLOB is not available.

::

class palladium.persistence.File(path)¶

Bases: palladium.persistence.FileLike

A ModelPersister that pickles models onto the file system, into a given directory.

::

class palladium.persistence.FileIO¶

Bases: palladium.persistence.FileLikeIO

::

class palladium.persistence.FileLike(path, io)¶

Bases: palladium.interfaces.ModelPersister

A ModelPersister that pickles models through file-like handles.

An argument io is used to access low level file handle operations.

::

::

::

upgrade_steps = <palladium.persistence.UpgradeSteps object>¶

class palladium.persistence.FileLikeIO¶

Bases: object

Used by FileLike to access low level file handle operations.

Test whether a path exists

For normal files, the implementation is:

```python
return os.path.exists(path)
```

Return a file handle

For normal files, the implementation is:

```python
return open(path, mode)
```

Remove a file

For normal files, the implementation is:

```python
os.remove(path)
```

class palladium.persistence.Rest(url, auth)¶

Bases: palladium.persistence.FileLike

::

class palladium.persistence.RestIO(auth)¶

Bases: palladium.persistence.FileLikeIO

::

class palladium.persistence.UpgradeSteps¶

Bases: object

::

palladium.server module¶

palladium.util module¶

Assorted utilties.

palladium.util.Partial(func, **kwargs)¶: Allows the use of partially applied functions in the configuration.

class palladium.util.PluggableDecorator(decorator_config_name)¶: Bases: object

class palladium.util.ProcessStore(*args, **kwargs)¶: Bases: collections.UserDict

class palladium.util.RruleThread(func, rrule, sleep_between_checks=60)¶

Bases: threading.Thread

Calls a given function in intervals defined by given recurrence rules (from datetuil.rrule).

palladium.util.apply_kwargs(func, **kwargs)¶: Call func with kwargs, but only those kwargs that it accepts.

palladium.util.args_from_config(func)¶: Decorator that injects parameters from the configuration.

palladium.util.get_metadata(error_code=0, error_message=None, status='OK')¶

palladium.util.memory_usage_psutil()¶: Return the current process memory usage in MB.

palladium.util.resolve_dotted_name(dotted_name)¶

palladium.util.run_job(func, **params)¶

palladium.util.session_scope(session)¶: Provide a transactional scope around a series of operations.

palladium.util.timer(log=None, message=None)¶

palladium.util.upgrade(model_persister, from_version=None, to_version=None)¶

palladium.util.upgrade_cmd(argv=['-T', '-b', 'readthedocs', '-d', '_build/doctrees-readthedocs', '-D', 'language=en', '.', '_build/html'])¶

Upgrade the database to the latest version.

Usage:

pld-ugprade [options]

Options:

`--from=<v>`	Upgrade from a specific version, overriding the version stored in the database.
`--to=<v>`	Upgrade to a specific version instead of the latest version.

-h –help Show this screen.

palladium.util.version_cmd(argv=['-T', '-b', 'readthedocs', '-d', '_build/doctrees-readthedocs', '-D', 'language=en', '.', '_build/html'])¶

Print the version number of Palladium.

Usage:: pld-version [options]
Options:: -h –help Show this screen.

palladium package¶

Submodules¶

palladium.R module¶

palladium.cache module¶

palladium.config module¶

palladium.dataset module¶

palladium.eval module¶

palladium.fit module¶

palladium.interfaces module¶

palladium.julia module¶

palladium.persistence module¶

palladium.server module¶

palladium.util module¶

palladium.wsgi module¶

Module contents¶