DirectorySessionStore.py

#

Store sessions in individual files within a directory. RJLRJL: Note that python 3.3. change fcntl to return OSError instead of IOError

RJLRJL: 19th August 2018 There is a problem with the existing code (adopted from the python 2 version), which leads to an EOFError: Ran out of input

The code in ::save_session() does f = open(filename, ‘wb’) which immediately makes the file zero bytes long. You can try this out in 2 terminals with one doing (where s is some dummy class with s.id as the file-name):-

>>> import pickle
>>> pickle.dump(s, f, 4)
>>> f.close()
>>> f = open(s.id, 'wb')

If in the other terminal you do:-

>>> f = open(s.id, 'rb')
>>> o = pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
EOFError: Ran out of input
>>>

This is not entirely unexpected BUT the code in load_session()…

f = open(filename, 'rb')
fcntl.flock(f.fileno(), fcntl.LOCK_SH)

…can get the shared lock (LOCK_SH) after save_session() performs the open() but BEFORE save_session() gets a chance to get the exclusive lock.

f = open(filename, 'wb')
fcntl.flock(f.fileno(), fcntl.LOCK_EX)

(You can try it by quickly refreshing a browser calling a Quixote server.)

What happens appears to be:-

save_session() opens the file to write  f = open(filename, 'wb')
load_session() opens the file to read   f = open(filename, 'rb')
load_session() asks for and GETS a shared lock [fcntl.LOCK_SH]
save_session() asks for an exclusive lock BUT gets blocked by the shared lock
load_session() tries to load the object and gets zero bytes. It then closes the
    file, allowing save_session() to proceed.

All that we need to do, now that we know that save_session() truncates the file and then waits for an exclusive lock, is have load_session() check for a zero-sized file. If it has one, then save_session() has just created (or re-created) it and we should let go and try again.

Addendum: It turns out that, during testing, one can get at EOFError from pickle anyway, so testing for that was added too.

import sys

if sys.version_info < (3,4,0):
    sys.stderr.write("You need python 3.4.0 or later to run this script\n")
    exit(1)


import fcntl, os, os.path
from pickle import dump, load
from session3.store.SessionStore import SessionStore
import time

SLEEPY_TIME = 0.1
#

Store sessions in individual files within a directory.

class DirectorySessionStore(SessionStore):
#
    is_multiprocess_safe = False  # Needs file locking; OS-specific.
    is_thread_safe = False        # Needs file locking or synchronization.
#

RJLRJL for Python3 we now use the highest protocol at time of writing, being protocol 4 (it was 2)

    pickle_protocol = 4
#

init takes a directory name, with an option to create it if it’s not already there.

    def __init__(self, directory, create=False):
#
        directory = os.path.abspath(directory)
#

make sure the directory exists:

        if not os.path.exists(directory):
            if create:
                os.mkdir(directory)
            else:
                raise OSError("error, '%s' does not exist." % (directory,))
#

is it actually a directory?

        if not os.path.isdir(directory):
            raise OSError("error, '%s' is not a directory." % (directory,))
        
        self.directory = directory
#

Build the filename from the session ID.

    def _make_filename(self, id):
#
        return os.path.join(self.directory, id)
#

Pickle the session and save it into a file.

    def save_session(self, session):
#
        filename = self._make_filename(session.id)
        f = open(filename, 'wb')
#

We wait at the following statement until we get an exclusive lock Note that load_session() can sometimes jump in here before we get the lock (the naughty thing) but it will get a zero-sized file (wb mode truncates the file)

        fcntl.flock(f.fileno(), fcntl.LOCK_EX)
        try:
            dump(session, f, self.pickle_protocol)
        finally:
            fcntl.flock(f.fileno(), fcntl.LOCK_UN)
            f.close()
#

Load the pickled session from a file.

    def load_session(self, id, default=None):
#
        filename = self._make_filename(id)
        finished = False
        while not finished:
            try:
                f = open(filename, 'rb')
#

Sometimes we get the following lock AFTER save_session() has created the file but BEFORE it has locked it. If so, we’ll have a zero-sized file (hence the loop, BTW).

                fcntl.flock(f.fileno(), fcntl.LOCK_SH)
                if os.stat(f.fileno()).st_size == 0:
                    fcntl.flock(f.fileno(), fcntl.LOCK_UN)
                    f.close()
#

wait around for a bit…

                    time.sleep(SLEEPY_TIME)
                else:
                    try:
                        obj = load(f)
#

Don’t be tempted to move this into a finally

                        fcntl.flock(f.fileno(), fcntl.LOCK_UN)
                        f.close()
                        finished = True             
                    except EOFError:
#

Sometimes we’ll also get EOFError from pickle anyway, so we might as well trap for that too…

                        fcntl.flock(f.fileno(), fcntl.LOCK_UN)
                        f.close()
                        time.sleep(SLEEPY_TIME)
                        
            except OSError:
                obj = default
                finished = True

        return obj
#

Delete the session file.

    def delete_session(self, session):
#
        
        filename = self._make_filename(session.id)
        os.unlink(filename)
#

Delete all sessions that have not been modified for N minutes.

This method is never called by the session manager. It’s for your application maintenance program; e.g., a daily cron job.

DirectorySessionStore.delete_old_sessions returns a tuple:

(n_deleted, n_remaining)
    def delete_old_sessions(self, minutes):
#
        
        deleted = 0
        remaining = 0
        for sess_id in os.listdir(self.directory):
            pth = self._make_filename(sess_id)
            mtime = os.stat(pth).st_mtime
            inactive_for = (time.time() - mtime) / 60.0
            
            if inactive_for > minutes:
                os.unlink(pth)
                deleted += 1
            else:
                remaining += 1
                
        return (deleted, remaining)