Tutorial: Using Storm with Nevow

Note: There is work on integrating Storm with Twisted (storm/twisted-integration). Until then, the best way is probably to use deferToThread.

(Here is A possible approach with Nevow and Storm/twisted-integration)

Before a tutorial can be written on how to use storm in Nevow, someone needs to figure out ‘best practices for using Storm in asynchronous codebases.’ From a chat in #storm on freenode (11 July 2007):

[5:13pm] dialtone: storm API is fine, one just needs to see if: a) it's thread
                   safe and b) objects returned can be modified in a different
                   thread
[5:13pm] dialtone: and then committed in another one yet
[5:13pm] radix: dialtone: It is certainly thread-safe, but you should not use
                   the same Store in multiple threads
[5:14pm] dialtone: how do you do with deferToThread then?
[5:14pm] radix: dialtone: I don't know.
[5:14pm] radix: dialtone: Probably you don't load the objects until you get
                   into the other thread.
[5:14pm] dialtone: that's sort of a problem probably, unless there's an
                   alternative solution
[5:15pm] radix: dialtone: Nobody has figured out best practices for using Storm
                   in asynchronous codebases yet.
[5:15pm] dialtone: ouch :(
[5:15pm] radix: dialtone: (That is, assuming your application actually wants to
                   make the database operations asynchronous)
[5:16pm] dialtone: I can accept it being synchronous, but run in a different
                   thread using deferToThread
[5:16pm] radix: dialtone: for example, it would be very reasonable to use Storm
                   like Axiom: with SQLite, just blocking on database interaction
[5:16pm] dialtone: radix: yes well... but that comes with a lot of restrictions
                   in itself
[5:17pm] radix: dialtone: I even have a Twisted server which uses Storm to talk
                   to postgres in a totally blocking manner
[5:17pm] radix: but it's not a very typical server
[5:17pm] dialtone: sqlite doesn't really scale that well with high concurrency
                   and scaling a shared sqlite instance is pretty hard
[5:17pm] dotz: dialtone: What's wrong with 'ceate a high level API for data
                   manipulation and retrival, use some rpc mechanism, like pb,
                   and make the hi-level database backend another process'?
[5:18pm] dialtone: dotz: that' waaay more effort than what I'm currently doing
[5:18pm] radix: dialtone: I'm sure there are some clever things that can be done
                   to use Storm in asynchronous contexts, and I'm fairly interested
                   in learning about them
[5:18pm] dotz: dialtone: and what are you currently doing?
[5:18pm] dialtone: as I said: I'm using the query builder from sqlalchemy to build
                   and execute queries that return resultsets with a dict-like API
[5:19pm] dialtone: I simply write the queries and use the dict-like APi in nevow
[5:19pm] dialtone: and I run the functions in a separate thread with deferToThread
[5:20pm] dialtone: radix: maybe writing a store class that keeps track of which
                   threads owns which connection and then reuse the connection when
                   needed
[5:20pm] dialtone: this way you could share the store object and run queries
                   independently of the context
[5:20pm] radix: dialtone: maybe. I'm not sure it'd be that simple.