Production server code changes

Since merging the local code from the production server to SVN, I've been working on best practices to get new code up and running on the production server. There are two main problems to solve here:

1. How to restart the web server with the new source code
2. Getting a new SVN commit into the productions server.

Reloading Source Code

Previously it was thought that the only way to restart the web server and cause new code changes to take effect was to restart the root apache server process. The problem with this is that it requires intervention for a system administrator with root access, and affects all websites being hosted by the server. As it turns out though, a mechanism is already in place for each website's server to be run as a daemon process under the control of a non-root user. If this process is killed, the root apache server will take note and restart it automatically, which fully reloads all the source code. The options for reloading source code are discussed in more detail here:

http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode

After reading the section on reloading through the daemon process, I was able to come up with a single command to reload the website's source code:

kill `ps -u community | grep apache2 | cut -d ' ' -f 1`

This command is somewhat brittle due to the output formatting of the ps command. The basic idea is to find an apache2 process launched by the community user and kill it. A new apache2 process should be immediately restarted by the root apache daemon.

Moving code from SVN to the server

The main issue here is that the server does not have the SVN client installed. In addition, the code requires certain modifications in order to run properly in production, such as credentials to the MySQL server (why they are hardcoded is another issue entirely). What I've done for the moment is to create two branches of the code under Community/development and Community/production. Using SVN's merge mechanism, a code change to Community/development can be merged into Community/production and immediately be ready for the production server. To get it there, producing a tarball of the Community/production/community_csdt tree and then using scp to move it to the production server was my current method for a while.

Then I decided it would be more reliable to have the production server pull code directly from SVN rather than dealing with tarballs and scp all the time. I have requested the system administrator for an install of the SVN client, but in the meantime I also decided to compile SVN from scratch to install it under ~/modwsgi/csdt-env. I will not elaborate on all the difficulties involved in building SVN from scratch except to say that the programmer of the Neon library who called the SSLv2_server_method() function has lost an astonishing amount of my respect.

The end result is that ~/modwsgi/csdt-env/community_csdt is now an SVN working copy of the production branch and can be updated directly updated using

~/modwsgi/csdt-env/bin/svn up

Project Delete Button

After the Edit button was replaced with the description editor box, there remained the Delete project button to implement. There didn't seem to be any existing function for deleting projects, so I began looking into how they are created in the first place and collect a list of actions to undo. The following functions are involved in creating a new project:

src/views/upload_view.py function uploadXML

src/models/upload/upload.py function addProject

src/models/upload/upload.py function updateProject

src/models/upload/upload.py function createJNLP

Looking though all this code and manually browsing around the resulting MySQL database, I compiled a list of things that a thorough Delete function would have to do to remove a project:

        1. remove its row in the SQL projects table

            1.1 remove related rows in the SQL project_memberships table

            1.2 remove related rows in the SQL project_ratings table

            1.3 remove related rows in the SQL project_comments table

                1.4 remove related rows in the SQL project_comments_ratings table

        2. delete its xml file from uploads/projects/<type>/xml/

        3. delete its jnlp file from uploads/projects/<type>/jnlp/

I coded this into a function called deleteProject in

src/models/accounts/admin/account_project.py,

which can be invoked by requesting the page

/accounts/<owner>/admin/projects/delete?proj_id=<project ID>.

This link was added to templates/user.projects.mako

and was tested several times.

MySQL string bug

In the process of creating new projects, I discovered a bug which manifests as a server failure if there is an apostrophe (or single quote) in the name or description of the project. This is because that character is the string delimiter for MySQL commands and needs to be somehow escaped or encoded before being sent to MySQL. I have not done this yet.

MySQL usage errors

There are still frequent SQL errors. They fall into two categories as far as I can see:

1. Commands out of sync
2. Empty set returned

The second type results from the Python query result being an empty Python tuple instead of the Python None object.

Both of these errors show up at various SQL query calls in the code, and never repeatably at any particular call.

Uploads directory/package mystery

There is a mysterious difference between the development and production versions. In my development version, uploaded projects and logs show up in the python module directory under env/lib/python2.7/site-packages/community_csdt-0.0-py2.7.egg. In the production server, however, they show up in the source directory ~/modwsgi/csdt-env/community_csdt rather than ~/modwsgi/csdt-env/lib/python2.7/site-packages/community_csdt-0.0-py2.7.egg/community_csdt.

Somehow past developers have managed to convince Python that the module is installed at the source directory. This has some benefits and some drawbacks. On one hand, the uploads directory is not wiped out when the project is reloaded. On the other hand, messing with the source directory is messing with the live website, and becomes much more dangerous.

Thumbnails

Now for our feature presentation. My approach to enabling thumbnails for the community projects is two-fold.

1. modify the pCSDT Core to generate a thumbnail and embed it in the XML file when it is saved.
2. modify the Python upload function to extract this thumbnail and display it on the website.

I've more or less got phase 1 working. I combined code from the screenshot function jButtonShotActionPerformed at GUI.java line 1618 and placed it in the SaveDocument function in GUI.java line 1366, along with some image scaling code found on stack overflow and modified for speed. After that I just added another property like the background image (called it thumbnail image), and voila. The File->Save button creates an XML file with two base-64 encoded images, one is the background and the other is the thumbnail. The only thing left to do now is get Python to read the XML, decode the string and write out the image file. Hopefully most of this will be in its standard library.