The mystic framework provides a collection of optimization algorithms and tools that allows the user to more robustly (and easily) solve hard optimization problems. All optimization algorithms included in mystic provide workflow at the fitting layer, not just access to the algorithms as function calls. mystic gives the user fine-grained power to both monitor and steer optimizations as the fit processes are running. Optimizers can advance one iteration with Step, or run to completion with Solve. Users can customize optimizer stop conditions, where both compound and user-provided conditions may be used. Optimizers can save state, can be reconfigured dynamically, and can be restarted from a saved solver or from a results file. All solvers can also leverage parallel computing, either within each iteration or as an ensemble of solvers.
Where possible, mystic optimizers share a common interface, and thus can be easily swapped without the user having to write any new code. mystic solvers all conform to a solver API, thus also have common method calls to configure and launch an optimization job. For more details, see mystic.abstract_solver. The API also makes it easy to bind a favorite 3rd party solver into the mystic framework.
Optimization algorithms in mystic can accept parameter constraints, either in the form of penalties (which “penalize” regions of solution space that violate the constraints), or as constraints (which “constrain” the solver to only search in regions of solution space where the constraints are respected), or both. mystic provides a large selection of constraints, including probabilistic and dimensionally reducing constraints. By providing a robust interface designed to enable the user to easily configure and control solvers, mystic greatly reduces the barrier to solving hard optimization problems.
>> Current Releasepathos is a framework for heterogeneous computing. It provides a consistent high-level interface for configuring and launching parallel computations across heterogeneous resources. pathos provides configurable launchers for parallel and distributed computing, where each launcher contains the syntactic logic to configure and launch jobs in an execution environment. Examples of launchers that plug into pathos are: a queue-less MPI-based launcher (in pyina), a ssh-based launcher (in pathos), and a multi-process launcher (in multiprocess).
pathos provides a consistent interface for parallel and/or distributed versions of map and apply for each launcher, thus lowering the barrier for users to extend their code to parallel and/or distributed resources. The guiding design principle behind pathos is that map and apply should be drop-in replacements in otherwise serial code, and thus switching to one or more of the pathos launchers is all that is needed to enable code to leverage the selected parallel or distributed computing resource. This not only greatly reduces the time to convert a code to parallel, but it also enables a single code-base to be maintained instead of requiring parallel, serial, and distributed versions of a code. pathos maps can be nested, thus hierarchical heterogeneous computing is possible by merely selecting the desired hierarchy of map and pipe (apply) objects.
The pathos framework is composed of several interoperating packages:
>> Visit our GitHub repository for the full pathos framework
- dill: a utility to serialize all of python
- pox: utilities for filesystem exploration and automated builds
- klepto: persistent caching to memory, disk, or database
- multiprocess: better multiprocessing and multithreading in python
- ppft: distributed and parallel python
- pyina: MPI parallel map and cluster scheduling
- pathos: graph management and execution in heterogeneous computing
The pathos package provides a few basic tools to make parallel and distributed computing more accessible to the end user. The goal of pathos is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring.
pathos provides methods for configuring, launching, monitoring, and controlling a service on a remote host. One of the most basic features of pathos is the ability to configure and launch a RPC-based service on a remote host. pathos seeds the remote host with the portpicker script, which allows the remote host to inform the localhost of a port that is available for communication.
Beyond the ability to establish a RPC service, and then post requests, is the ability to launch code in parallel. Unlike parallel computing performed at the node level (typically with MPI), pathos enables the user to launch jobs in parallel across heterogeneous distributed resources. pathoss provides distributed map and pipe algorithms, where a mix of local processors and distributed workers can be selected. pathos also provides a very basic automated load balancing service, as well as the ability for the user to directly select the resources.
The high-level pool.map interface, yields a map implementation that hides the RPC internals from the user. With pool.map, the user can launch their code in parallel, and as a distributed service, using standard python and without writing a line of server or parallel batch code.
RPC servers and communication in general is known to be insecure. However, instead of attempting to make the RPC communication itself secure, pathos provides the ability to automatically wrap any distributes service or communication in a ssh-tunnel. Ssh is a universally trusted method. Using ssh-tunnels, pathos has launched several distributed calculations on national lab clusters, and to date has performed test calculations that utilize node-to-node communication between several national lab clusters and a user’s laptop. pathos allows the user to configure and launch at a very atomistic level, through raw access to ssh and scp.
>> Current Release