This section details direct usage of the Engine, Connection, and related objects. Its important to note that when using the SQLAlchemy ORM, these objects are not generally accessed; instead, the Session object is used as the interface to the database. However, for applications that are built around direct usage of textual SQL statements and/or SQL expression constructs without involvement by the ORM’s higher level management services, the Engine and Connection are king (and queen?) - read on.
Recall from Engine Configuration that an Engine is created via the create_engine() call:
engine = create_engine('mysql://scott:tiger@localhost/test')The typical usage of create_engine() is once per particular database URL, held globally for the lifetime of a single application process. A single Engine manages many individual DBAPI connections on behalf of the process and is intended to be called upon in a concurrent fashion. The Engine is not synonymous to the DBAPI connect function, which represents just one connection resource - the Engine is most efficient when created just once at the module level of an application, not per-object or per-function call.
For a multiple-process application that uses the os.fork system call, or for example the Python multiprocessing module, it’s usually required that a separate Engine be used for each child process. This is because the Engine maintains a reference to a connection pool that ultimately references DBAPI connections - these tend to not be portable across process boundaries. An Engine that is configured not to use pooling (which is achieved via the usage of NullPool) does not have this requirement.
The engine can be used directly to issue SQL to the database. The most generic way is first procure a connection resource, which you get via the connect method:
connection = engine.connect()
result = connection.execute("select username from users")
for row in result:
print "username:", row['username']
connection.close()The connection is an instance of Connection, which is a proxy object for an actual DBAPI connection. The DBAPI connection is retrieved from the connection pool at the point at which Connection is created.
The returned result is an instance of ResultProxy, which references a DBAPI cursor and provides a largely compatible interface with that of the DBAPI cursor. The DBAPI cursor will be closed by the ResultProxy when all of its result rows (if any) are exhausted. A ResultProxy that returns no rows, such as that of an UPDATE statement (without any returned rows), releases cursor resources immediately upon construction.
When the close() method is called, the referenced DBAPI connection is returned to the connection pool. From the perspective of the database itself, nothing is actually “closed”, assuming pooling is in use. The pooling mechanism issues a rollback() call on the DBAPI connection so that any transactional state or locks are removed, and the connection is ready for its next usage.
The above procedure can be performed in a shorthand way by using the execute() method of Engine itself:
result = engine.execute("select username from users")
for row in result:
print "username:", row['username']Where above, the execute() method acquires a new Connection on its own, executes the statement with that object, and returns the ResultProxy. In this case, the ResultProxy contains a special flag known as close_with_result, which indicates that when its underlying DBAPI cursor is closed, the Connection object itself is also closed, which again returns the DBAPI connection to the connection pool, releasing transactional resources.
If the ResultProxy potentially has rows remaining, it can be instructed to close out its resources explicitly:
result.close()If the ResultProxy has pending rows remaining and is dereferenced by the application without being closed, Python garbage collection will ultimately close out the cursor as well as trigger a return of the pooled DBAPI connection resource to the pool (SQLAlchemy achieves this by the usage of weakref callbacks - never the __del__ method) - however it’s never a good idea to rely upon Python garbage collection to manage resources.
Our example above illustrated the execution of a textual SQL string. The execute() method can of course accommodate more than that, including the variety of SQL expression constructs described in SQL Expression Language Tutorial.
Bases: sqlalchemy.engine.base.Connectable
Provides high-level functionality for a wrapped DB-API connection.
Provides execution support for string-based SQL statements as well as ClauseElement, Compiled and DefaultGenerator objects. Provides a begin() method to return Transaction objects.
The Connection object is not thread-safe. While a Connection can be shared among threads using properly synchronized access, it is still possible that the underlying DBAPI connection may not support shared access between threads. Check the DBAPI documentation for details.
The Connection object represents a single dbapi connection checked out from the connection pool. In this state, the connection pool has no affect upon the connection, including its expiration or timeout state. For the connection pool to properly manage connections, connections should be returned to the connection pool (i.e. connection.close()) whenever the connection is not in use.
Construct a new Connection.
The constructor here is not public and is only called only by an Engine. See Engine.connect() and Engine.contextual_connect() methods.
Begin a transaction and return a Transaction handle.
Repeated calls to begin on the same Connection will create a lightweight, emulated nested transaction. Only the outermost transaction may commit. Calls to commit on inner transactions are ignored. Any transaction in the hierarchy may rollback, however.
Begin a nested transaction and return a Transaction handle.
Nested transactions require SAVEPOINT support in the underlying database. Any transaction in the hierarchy may commit and rollback, however the outermost transaction still controls the overall commit or rollback of the transaction of a whole.
Begin a two-phase or XA transaction and return a Transaction handle.
| Parameters: | xid – the two phase transaction id. If not supplied, a random id will be generated. |
|---|
Close this Connection.
Return True if this connection is closed.
Returns self.
This Connectable interface method returns self, allowing Connections to be used interchangably with Engines in most situations that require a bind.
The underlying DB-API connection managed by this Connection.
Returns self.
This Connectable interface method returns self, allowing Connections to be used interchangably with Engines in most situations that require a bind.
Create a Table or Index given an appropriate Schema object.
Detach the underlying DB-API connection from its connection pool.
This Connection instance will remain useable. When closed, the DB-API connection will be literally closed and not returned to its pool. The pool will typically lazily create a new connection to replace the detached connection.
This method can be used to insulate the rest of an application from a modified state on a connection (such as a transaction isolation level or similar). Also see PoolListener for a mechanism to modify connection state when connections leave and return to their connection pool.
Dialect used by this Connection.
Drop a Table or Index given an appropriate Schema object.
Executes the given construct and returns a ResultProxy.
The construct can be one of:
Set non-SQL options for the connection which take effect during execution.
The method returns a copy of this Connection which references the same underlying DBAPI connection, but also defines the given execution options which will take effect for a call to execute(). As the new Connection references the same underlying resource, it is probably best to ensure that the copies would be discarded immediately, which is implicit if used as in:
result = connection.execution_options(stream_results=True). execute(stmt)The options are the same as those accepted by sqlalchemy.sql.expression.Executable.execution_options().
Return True if a transaction is in progress.
A collection of per-DB-API connection instance properties.
Invalidate the underlying DBAPI connection associated with this Connection.
The underlying DB-API connection is literally closed (if possible), and is discarded. Its source connection pool will typically lazily create a new connection to replace it.
Upon the next usage, this Connection will attempt to reconnect to the pool with a new connection.
Transactions in progress remain in an “opened” state (even though the actual transaction is gone); these must be explicitly rolled back before a reconnect on this Connection can proceed. This is to prevent applications from accidentally continuing their transactional operations in a non-transactional state.
Return True if this connection was invalidated.
Reflect the columns in the given string table name from the database.
Executes and returns the first column of the first row.
The underlying result/cursor is closed after execution.
Execute the given function within a transaction boundary.
This is a shortcut for explicitly calling begin() and commit() and optionally rollback() when exceptions are raised. The given *args and **kwargs will be passed to the function.
See also transaction() on engine.
Bases: object
Interface for an object which supports execution of SQL constructs.
The two implementations of Connectable are Connection and Engine.
Connectable must also implement the ‘dialect’ member which references a Dialect instance.
Return a Connection object which may be part of an ongoing context.
Create a table or index given an appropriate schema object.
Drop a table or index given an appropriate schema object.
Bases: sqlalchemy.engine.base.Connectable, sqlalchemy.log.Identified
Connects a Pool and Dialect together to provide a source of database connectivity and behavior.
An Engine object is instantiated publically using the create_engine() function.
Return a new Connection object.
The Connection, upon construction, will procure a DBAPI connection from the Pool referenced by this Engine, returning it back to the Pool after the Connection.close() method is called.
Return a Connection object which may be part of some ongoing context.
By default, this method does the same thing as Engine.connect(). Subclasses of Engine may override this method to provide contextual behavior.
| Parameters: | close_with_result – When True, the first ResultProxy created by the Connection will call the Connection.close() method of that connection as soon as any pending result rows are exhausted. This is used to supply the “connectionless execution” behavior provided by the Engine.execute() method. |
|---|
Create a table or index within this engine’s database connection given a schema object.
Dispose of the connection pool used by this Engine.
A new connection pool is created immediately after the old one has been disposed. This new pool, like all SQLAlchemy connection pools, does not make any actual connections to the database until one is first requested.
This method has two general use cases:
- When a dropped connection is detected, it is assumed that all connections held by the pool are potentially dropped, and the entire pool is replaced.
- An application may want to use dispose() within a test suite that is creating multiple engines.
It is critical to note that dispose() does not guarantee that the application will release all open database connections - only those connections that are checked into the pool are closed. Connections which remain checked out or have been detached from the engine are not affected.
Driver name of the Dialect in use by this Engine.
Drop a table or index within this engine’s database connection given a schema object.
Executes the given construct and returns a ResultProxy.
The arguments are the same as those used by Connection.execute().
Here, a Connection is acquired using the contextual_connect() method, and the statement executed with that connection. The returned ResultProxy is flagged such that when the ResultProxy is exhausted and its underlying cursor is closed, the Connection created here will also be closed, which allows its associated DBAPI connection resource to be returned to the connection pool.
String name of the Dialect in use by this Engine.
Return a DB-API connection.
Given a Table object, reflects its columns and properties from the database.
Return a list of all table names available in the database.
| Parameters: |
|
|---|
Return a text() construct, bound to this engine.
This is equivalent to:
text("SELECT * FROM table", bind=engine)Execute the given function within a transaction boundary.
This is a shortcut for explicitly calling begin() and commit() and optionally rollback() when exceptions are raised. The given *args and **kwargs will be passed to the function.
The connection used is that of contextual_connect().
See also the similar method on Connection itself.
update the execution_options dictionary of this Engine.
For details on execution_options, see Connection.execution_options() as well as sqlalchemy.sql.expression.Executable.execution_options().
Wraps a DB-API cursor object to provide easier access to row columns.
Individual columns may be accessed by their integer position, case-insensitive column name, or by schema.Column object. e.g.:
row = fetchone()
col1 = row[0] # access via integer position
col2 = row['col2'] # access via name
col3 = row[mytable.c.mycol] # access via Column object.ResultProxy also handles post-processing of result column data using TypeEngine objects, which are referenced from the originating SQL statement that produced this result set.
Close this ResultProxy.
Closes the underlying DBAPI cursor corresponding to the execution.
Note that any data cached within this ResultProxy is still available. For some types of results, this may include buffered rows.
If this ResultProxy was generated from an implicit execution, the underlying Connection will also be closed (returns the underlying DBAPI connection to the connection pool.)
This method is called automatically when:
Fetch all rows, just like DB-API cursor.fetchall().
Fetch many rows, just like DB-API cursor.fetchmany(size=cursor.arraysize).
If rows are present, the cursor remains open after this is called. Else the cursor is automatically closed and an empty list is returned.
Fetch one row, just like DB-API cursor.fetchone().
If a row is present, the cursor remains open after this is called. Else the cursor is automatically closed and None is returned.
Fetch the first row and then close the result set unconditionally.
Returns None if no row is present.
Return the primary key for the row just inserted.
This only applies to single row insert() constructs which did not explicitly specify returning().
True if this ResultProxy is the result of a executing an expression language compiled expression.insert() construct.
When True, this implies that the inserted_primary_key attribute is accessible, assuming the statement did not include a user defined “returning” construct.
New in 0.6.7.
Return the current set of string keys for rows.
Return the primary key for the row just inserted.
Deprecated since version 0.6: Use ResultProxy.inserted_primary_key
Return last_inserted_params() from the underlying ExecutionContext.
See ExecutionContext for details.
Return last_updated_params() from the underlying ExecutionContext.
See ExecutionContext for details.
Return lastrow_has_defaults() from the underlying ExecutionContext.
See ExecutionContext for details.
return the ‘lastrowid’ accessor on the DBAPI cursor.
This is a DBAPI specific method and is only functional for those backends which support it, for statements where it is appropriate. It’s behavior is not consistent across backends.
Usage of this method is normally unnecessary; the inserted_primary_key attribute provides a tuple of primary key values for a newly inserted row, regardless of database backend.
Return postfetch_cols() from the underlying ExecutionContext.
See ExecutionContext for details.
True if this ResultProxy returns rows.
I.e. if it is legal to call the methods fetchone(), fetchmany() fetchall().
New in 0.6.7.
Return the ‘rowcount’ for this result.
The ‘rowcount’ reports the number of rows affected by an UPDATE or DELETE statement. It has no other uses and is not intended to provide the number of rows present from a SELECT.
Note that this row count may not be properly implemented in some dialects; this is indicated by supports_sane_rowcount() and supports_sane_multi_rowcount(). rowcount() also may not work at this time for a statement that uses returning().
Fetch the first column of the first row, and close the result set.
Returns None if no row is present.
Return supports_sane_multi_rowcount from the dialect.
Return supports_sane_rowcount from the dialect.
Proxy values from a single cursor row.
Mostly follows “ordered dictionary” behavior, mapping result values to the string-based column name, the integer position of the result in the row, as well as Column instances which can be mapped to the original Columns that produced this result set (for results that correspond to constructed SQL expressions).
Return True if this RowProxy contains the given key.
Return a list of tuples, each tuple containing a key/value pair.
Return the list of keys as strings represented by this RowProxy.
Note
This section describes how to use transactions when working directly with Engine and Connection objects. When using the SQLAlchemy ORM, the public API for transaction control is via the Session object, which makes usage of the Transaction object internally. See Managing Transactions for further information.
The Connection object provides a begin() method which returns a Transaction object. This object is usually used within a try/except clause so that it is guaranteed to rollback() or commit():
trans = connection.begin()
try:
r1 = connection.execute(table1.select())
connection.execute(table1.insert(), col1=7, col2='this is some data')
trans.commit()
except:
trans.rollback()
raiseThe Transaction object also handles “nested” behavior by keeping track of the outermost begin/commit pair. In this example, two functions both issue a transaction on a Connection, but only the outermost Transaction object actually takes effect when it is committed.
# method_a starts a transaction and calls method_b
def method_a(connection):
trans = connection.begin() # open a transaction
try:
method_b(connection)
trans.commit() # transaction is committed here
except:
trans.rollback() # this rolls back the transaction unconditionally
raise
# method_b also starts a transaction
def method_b(connection):
trans = connection.begin() # open a transaction - this runs in the context of method_a's transaction
try:
connection.execute("insert into mytable values ('bat', 'lala')")
connection.execute(mytable.insert(), col1='bat', col2='lala')
trans.commit() # transaction is not committed yet
except:
trans.rollback() # this rolls back the transaction unconditionally
raise
# open a Connection and call method_a
conn = engine.connect()
method_a(conn)
conn.close()Above, method_a is called first, which calls connection.begin(). Then it calls method_b. When method_b calls connection.begin(), it just increments a counter that is decremented when it calls commit(). If either method_a or method_b calls rollback(), the whole transaction is rolled back. The transaction is not committed until method_a calls the commit() method. This “nesting” behavior allows the creation of functions which “guarantee” that a transaction will be used if one was not already available, but will automatically participate in an enclosing transaction if one exists.
Represent a Transaction in progress.
The object provides rollback() and commit() methods in order to control transaction boundaries. It also implements a context manager interface so that the Python with statement can be used with the Connection.begin() method.
The Transaction object is not threadsafe.
The constructor for Transaction is private and is called from within the Connection.begin implementation.
Close this Transaction.
If this transaction is the base transaction in a begin/commit nesting, the transaction will rollback(). Otherwise, the method returns.
This is used to cancel a Transaction without affecting the scope of an enclosing transaction.
Commit this Transaction.
Roll back this Transaction.
The previous transaction example illustrates how to use Transaction so that several executions can take part in the same transaction. What happens when we issue an INSERT, UPDATE or DELETE call without using Transaction? The answer is autocommit. While many DBAPI implementation provide various special “non-transactional” modes, the current SQLAlchemy behavior is such that it implements its own “autocommit” which works completely consistently across all backends. This is achieved by detecting statements which represent data-changing operations, i.e. INSERT, UPDATE, DELETE, as well as data definition language (DDL) statements such as CREATE TABLE, ALTER TABLE, and then issuing a COMMIT automatically if no transaction is in progress. The detection is based on compiled statement attributes, or in the case of a text-only statement via regular expressions:
conn = engine.connect()
conn.execute("INSERT INTO users VALUES (1, 'john')") # autocommitsFull control of the “autocommit” behavior is available using the generative Connection.execution_options() method provided on Connection, Engine, Executable, using the “autocommit” flag which will turn on or off the autocommit for the selected scope. For example, a text() construct representing a stored procedure that commits might use it so that a SELECT statement will issue a COMMIT:
engine.execute(text("SELECT my_mutating_procedure()").execution_options(autocommit=True))Recall from the first section we mentioned executing with and without explicit usage of Connection. “Connectionless” execution refers to the usage of the execute() method on an object which is not a Connection. This was illustrated using the execute() method of Engine.
In addition to “connectionless” execution, it is also possible to use the execute() method of any Executable construct, which is a marker for SQL expression objects that support execution. The SQL expression object itself references an Engine or Connection known as the bind, which it uses in order to provide so-called “implicit” execution services.
Given a table as below:
meta = MetaData()
users_table = Table('users', meta,
Column('id', Integer, primary_key=True),
Column('name', String(50))
)Explicit execution delivers the SQL text or constructed SQL expression to the execute() method of Connection:
engine = create_engine('sqlite:///file.db')
connection = engine.connect()
result = connection.execute(users_table.select())
for row in result:
# ....
connection.close()Explicit, connectionless execution delivers the expression to the execute() method of Engine:
engine = create_engine('sqlite:///file.db')
result = engine.execute(users_table.select())
for row in result:
# ....
result.close()Implicit execution is also connectionless, and calls the execute() method on the expression itself, utilizing the fact that either an Engine or Connection has been bound to the expression object (binding is discussed further in Schema Definition Language):
engine = create_engine('sqlite:///file.db')
meta.bind = engine
result = users_table.select().execute()
for row in result:
# ....
result.close()In both “connectionless” examples, the Connection is created behind the scenes; the ResultProxy returned by the execute() call references the Connection used to issue the SQL statement. When the ResultProxy is closed, the underlying Connection is closed for us, resulting in the DBAPI connection being returned to the pool with transactional resources removed.
The “threadlocal” engine strategy is an optional feature which can be used by non-ORM applications to associate transactions with the current thread, such that all parts of the application can participate in that transaction implicitly without the need to explicitly reference a Connection. “threadlocal” is designed for a very specific pattern of use, and is not appropriate unless this very specfic pattern, described below, is what’s desired. It has no impact on the “thread safety” of SQLAlchemy components or one’s application. It also should not be used when using an ORM Session object, as the Session itself represents an ongoing transaction and itself handles the job of maintaining connection and transactional resources.
Enabling threadlocal is achieved as follows:
db = create_engine('mysql://localhost/test', strategy='threadlocal')The above Engine will now acquire a Connection using connection resources derived from a thread-local variable whenever Engine.execute() or Engine.contextual_connect() is called. This connection resource is maintained as long as it is referenced, which allows multiple points of an application to share a transaction while using connectionless execution:
def call_operation1():
engine.execute("insert into users values (?, ?)", 1, "john")
def call_operation2():
users.update(users.c.user_id==5).execute(name='ed')
db.begin()
try:
call_operation1()
call_operation2()
db.commit()
except:
db.rollback()Explicit execution can be mixed with connectionless execution by using the Engine.connect method to acquire a Connection that is not part of the threadlocal scope:
db.begin()
conn = db.connect()
try:
conn.execute(log_table.insert(), message="Operation started")
call_operation1()
call_operation2()
db.commit()
conn.execute(log_table.insert(), message="Operation succeeded")
except:
db.rollback()
conn.execute(log_table.insert(), message="Operation failed")
finally:
conn.close()To access the Connection that is bound to the threadlocal scope, call Engine.contextual_connect():
conn = db.contextual_connect()
call_operation3(conn)
conn.close()Calling close() on the “contextual” connection does not release its resources until all other usages of that resource are closed as well, including that any ongoing transactions are rolled back or committed.