Database¶
This package contains the utilities required to map the state of the resources managed by this API to the resources in a SQL-based database. Using this package along with SQLAlchemy should allow querying of all the resources in the database. This package alone DOES NOT guarantee that all the models will be self-consistent, as this codebase allows resources to span multiple storage systems. Since each type of resource is represented by a model class, the onus is on the developer to make sure that resources are consistent when loaded from multiple storage systems.
For example, if a Service has both a record in a MySQL database, and a
job_registration_schema stored on a MongoDB instance, the Service
model class would be responsible for ensuring that the MongoDB record and
the MySQL record agree.
Models¶
The models in this package use SQLAlchemy’s object-relational mapping (ORM)
capabilities to acquire a persistent representation in a relational database
. The API makes use of these model classes to obtain a representation of a
resource in the database, as well as to describe the relations between
models. For example, a Service has a one-to-many relationship between
Job``s, expressed by a ``FOREIGN KEY constraint in the relational database.
In order to create new database model classes, subclass
BASE, found in the declarative_base. Each persistent attribute
of a class must then map to a Column object provided by the database
schema. See the SQLAlchemy documentation
for more details about object-relational mapping.
Note
If any database model classes will be inheriting from other model classes, SQLAlchemy will need to be notified. An inheritance model will need to be specified so that SQLAlchemy knows which class to instantiate for which database record. The inheritance section in the SQLAlchemy documentation will give more data
Declarative Base¶
Defines a base class for creating mapped objects from SQLAlchemy. Any class that has persistent attributes should subclass this class.
-
topchef.database.models.declarative_base.BASE¶ alias of
Base
Service¶
- Contains a model for a TopChef service. A service is any operation that maps a
- set of parameters to a set of results. An example of a service could be an ODMR resonance experiment, as it takes a set of experiment parameters and outputs the result.
Job¶
Contains a model for a job. A job is matched to a particular service. The job’s parameters must be an instance of the service schema, and the result must be an instance of the result schema
Job Set¶
Models job sets
Schemas¶
The schema of a database is the set of all metadata that enables records to be consistently entered into that database. By “consistent”, what we mean is that any operation not resulting in an error maps the database from one valid state to another valid state. Unlike some no-SQL databases like MongoDB, relational databases require that a schema is defined before writing any records to the database. This module takes care of defining that schema, using abstractions provided by SQLAlchemy. Based on this schema and the particular database being used, SQLAlchemy can then generate the SQL code needed to perform a particular manipulation on a database model.
the interface for the database schema is defined in
abstract_database_schema. Each property there maps to a table in the
database.
Abstract Database Schema¶
Contains the interface for interacting with a SQLAlchemy database
schema. Each table in the schema should map to a property defined in this
interface. This is done to prevent accidental overwriting of tables. The
metadata property in this schema holds the instance of
sqlalchemy.MetaData
-
class
topchef.database.schemas.abstract_database_schema.AbstractDatabaseSchema[source]¶ Defines the interface for the database schema
-
job_sets¶ Returns: The table associated with job sets
-
jobs¶ Returns: The table used to write information about jobs
-
metadata¶ Returns: The metadata container used by SQLAlchemy to store metadata related to the DB schema
-
services¶ Returns: The table used to write information about services
-
Database Schema¶
JSON Type¶
SQL-based databases are strongly and statically typed. This means that every variable must have a type associated with it, that type must be defined in the schema, and the variable has that type for as long as this variable exists. This definition implies that “changing the type” of a variable (known as “casting”) involves executing a function that maps one variable and creates a new variable of a different type.
This module defines a backend-agnostic data type for storing JavaScript Object Notation (JSON) objects in a relational database. Increasingly, databases like MySQL and PostgreSQL are supporting JSON as a valid variable type in their schemas. Using a JSON type to store JSON instead of a String is a more elegant solution, as databases that support JSON make some effort of checking that the JSON placed into a relational database is at least syntactically correct. The type defined here will use a JSON type on MySQL and PostgreSQL. If these types are not available, it will default into storing the JSON as a string.
-
class
topchef.database.json_type.JSON(*args, **kwargs)[source]¶ Model class for the UUID type. impl represents the implementation to which this type will be forced when it is loaded from the DB
-
copy(*args, **kwargs) → topchef.database.json_type.JSON[source]¶ Parameters: - args – The arguments to the UUID constructor
- kwargs – The keyword arguments to the UUID constructor
Returns: A deep copy of this object
-
impl¶ alias of
VARCHAR
-
load_dialect_impl(dialect: <module 'sqlalchemy.dialects' from '/home/docs/checkouts/readthedocs.org/user_builds/topchef/envs/latest/lib/python3.5/site-packages/sqlalchemy/dialects/__init__.py'>) → typing.Union[sqlalchemy.dialects.postgresql.base.UUID, sqlalchemy.sql.sqltypes.VARCHAR][source]¶ SQLAlchemy wraps all database-specific features into dialects, which are then responsible for generating the SQL code for a specific DB type when loading in data.
load_dialect_implis called when CRUD (create, update, delete operations) needs to be done on the database. This method is responsible for telling SQLAlchemy how to configure the dialect to write this typeParameters: dialect – The loaded dialect Returns: The type descriptor for this type.
-
process_bind_param(value: typing.Union[dict, str, NoneType], dialect: <module 'sqlalchemy.dialects' from '/home/docs/checkouts/readthedocs.org/user_builds/topchef/envs/latest/lib/python3.5/site-packages/sqlalchemy/dialects/__init__.py'>) → typing.Union[str, NoneType][source]¶ Given a value and a dialect, determine how to serialize the type to the dialect
Note
Python3 will complain that int is not supported for this type. I want to ignore this if possible
Parameters: - value – The value to encode
- dialect – The dialect to which this will be encoded to
Returns: The value encoded in that dialect
-
process_result_value(value: typing.Union[str, NoneType], dialect: <module 'sqlalchemy.dialects' from '/home/docs/checkouts/readthedocs.org/user_builds/topchef/envs/latest/lib/python3.5/site-packages/sqlalchemy/dialects/__init__.py'>) → typing.Union[dict, NoneType][source]¶ Parameters: - value – The value to process from the SQL query
- dialect – The dialect to use for the processing
Returns: The value as a UUID
-
UUID Database Type¶
In a similar vein to the JSON column type defined in
topchef.database.json_type, this type provides a back-end
agnostic way of storing Universally Unique Identifiers (UUIDs) in the
database. If the database is a Postgres DB, then it will use Postgres’ UUID
data type to store the UUID. If such a type does not exist, then the UUID
will be stored as CHAR(32). This type represents a string that MUST have 32
characters.
This code was heavily inspired by SQLAlchemy’s UUID type implementation,
as defined in sqlalchemy.types.TypeDecorator.
-
class
topchef.database.uuid_database_type.UUID(*args, **kwargs)[source]¶ Model class for the UUID type. impl represents the implementation to which this type will be forced when it is loaded from the DB
-
copy(*args, **kwargs) → topchef.database.uuid_database_type.UUID[source]¶ Parameters: - args – The arguments to the UUID constructor
- kwargs – The keyword arguments to the UUID constructor
Returns: A deep copy of this object
-
impl¶ alias of
CHAR
-
load_dialect_impl(dialect: <module 'sqlalchemy.dialects' from '/home/docs/checkouts/readthedocs.org/user_builds/topchef/envs/latest/lib/python3.5/site-packages/sqlalchemy/dialects/__init__.py'>) → typing.Union[sqlalchemy.dialects.postgresql.base.UUID, sqlalchemy.sql.sqltypes.CHAR][source]¶ SQLAlchemy wraps all database-specific features into dialects, which are then responsible for generating the SQL code for a specific DB type when loading in data.
load_dialect_implis called when CRUD (create, update, delete operations) needs to be done on the database. This method is responsible for telling SQLAlchemy how to configure the dialect to write this typeParameters: dialect – The loaded dialect Returns: The type descriptor for this type.
-
process_bind_param(value: typing.Union[uuid.UUID, str, NoneType], dialect: <module 'sqlalchemy.dialects' from '/home/docs/checkouts/readthedocs.org/user_builds/topchef/envs/latest/lib/python3.5/site-packages/sqlalchemy/dialects/__init__.py'>) → typing.Union[str, NoneType][source]¶ Given a value and a dialect, determine how to serialize the type to the dialect
Note
Python3 will complain that int is not supported for this type. I want to ignore this if possible
Parameters: - value – The value to encode
- dialect – The dialect to which this will be encoded to
Returns: The value encoded in that dialect
-
process_result_value(value: typing.Union[str, NoneType], dialect: <module 'sqlalchemy.dialects' from '/home/docs/checkouts/readthedocs.org/user_builds/topchef/envs/latest/lib/python3.5/site-packages/sqlalchemy/dialects/__init__.py'>) → typing.Union[uuid.UUID, NoneType][source]¶ Parameters: - value – The value to process from the SQL query
- dialect – The dialect to use for the processing
Returns: The value as a UUID
-