Storage Layer
The Storage Layer takes care of all the data serialization functions. It uses SQLAlchemy to do much of the heavy lifting.
Theory of Operation
Assets have a name (uri) to uniquely identify them, and a lifetime (though it may be infinite) based on two date/times: active and expires. An Asset will only be considered for any operation between those two dates/times (unless the administrator is doing a time-shift, of which more elsewhere). All other data is stored as a dictionary, and this dictionary is pickled into the database by the storage layer.
This data is stored (more or less) twice: once in the Asset table, and then spread out into one or more records in the AssetData table. Keeping all the data in the Asset table allows for one-table key-based lookup and loading of assets from the database. Spreading it out allows us to add, remove and independently update types on an asset.
Each row in the AssetData table has a type (a foreign key into the AssetTypes table) and lifetime (like the Asset). It is by these lifetimes that data may be versioned; the lifetimes of AssetData rows for a particular Asset and type may not overlap. The data particular to each AssetType which makes up the asset is divided up (the type module names are the keys to each section in the asset's dictionary), and put into distinct AssetData rows. In most cases, the data a type defines is key/value stuff. That gets stored as a pickled dictionary and as a very simple XML document. Types which manage "raw" data (test, images, sound, video, etc) will store their data in the database in a blob field. If a type needs to store more than one variety of data, or more than one instance of data (say, an image and it's thumbnail), it can use two rows, discriminating them with a datakey field.
Finally, remember the bit about making the dict into XML, above? Well, that's not locked in stone. If a type is marked as "flattened" the storage layer will build a table mirroring the AssetData table, but including that type's fields, one column per field. Then the data is put into that table. The core AssetData table allows for rapid reconstruction of an asset via loading the pickles and updating the asset's dict. The flattened table allows for rapid searching, grouping, etc.
Key Functions
- StoreAsset - stores a new asset in the database
- UpdateAsset - update an existing asset's data
