Production server code changes

Since merging the local code from the production server to SVN, I've been working on best practices to get new code up and running on the production server. There are two main problems to solve here:

1. How to restart the web server with the new source code
2. Getting a new SVN commit into the productions server.

Reloading Source Code

Previously it was thought that the only way to restart the web server and cause new code changes to take effect was to restart the root apache server process. The problem with this is that it requires intervention for a system administrator with root access, and affects all websites being hosted by the server. As it turns out though, a mechanism is already in place for each website's server to be run as a daemon process under the control of a non-root user. If this process is killed, the root apache server will take note and restart it automatically, which fully reloads all the source code. The options for reloading source code are discussed in more detail here:

http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode

After reading the section on reloading through the daemon process, I was able to come up with a single command to reload the website's source code:

kill `ps -u community | grep apache2 | cut -d ' ' -f 1`

This command is somewhat brittle due to the output formatting of the ps command. The basic idea is to find an apache2 process launched by the community user and kill it. A new apache2 process should be immediately restarted by the root apache daemon.

Moving code from SVN to the server

The main issue here is that the server does not have the SVN client installed. In addition, the code requires certain modifications in order to run properly in production, such as credentials to the MySQL server (why they are hardcoded is another issue entirely). What I've done for the moment is to create two branches of the code under Community/development and Community/production. Using SVN's merge mechanism, a code change to Community/development can be merged into Community/production and immediately be ready for the production server. To get it there, producing a tarball of the Community/production/community_csdt tree and then using scp to move it to the production server was my current method for a while.

Then I decided it would be more reliable to have the production server pull code directly from SVN rather than dealing with tarballs and scp all the time. I have requested the system administrator for an install of the SVN client, but in the meantime I also decided to compile SVN from scratch to install it under ~/modwsgi/csdt-env. I will not elaborate on all the difficulties involved in building SVN from scratch except to say that the programmer of the Neon library who called the SSLv2_server_method() function has lost an astonishing amount of my respect.

The end result is that ~/modwsgi/csdt-env/community_csdt is now an SVN working copy of the production branch and can be updated directly updated using

~/modwsgi/csdt-env/bin/svn up

Project Delete Button

After the Edit button was replaced with the description editor box, there remained the Delete project button to implement. There didn't seem to be any existing function for deleting projects, so I began looking into how they are created in the first place and collect a list of actions to undo. The following functions are involved in creating a new project:

src/views/upload_view.py function uploadXML

src/models/upload/upload.py function addProject

src/models/upload/upload.py function updateProject

src/models/upload/upload.py function createJNLP

Looking though all this code and manually browsing around the resulting MySQL database, I compiled a list of things that a thorough Delete function would have to do to remove a project:

        1. remove its row in the SQL projects table

            1.1 remove related rows in the SQL project_memberships table

            1.2 remove related rows in the SQL project_ratings table

            1.3 remove related rows in the SQL project_comments table

                1.4 remove related rows in the SQL project_comments_ratings table

        2. delete its xml file from uploads/projects/<type>/xml/

        3. delete its jnlp file from uploads/projects/<type>/jnlp/

I coded this into a function called deleteProject in

src/models/accounts/admin/account_project.py,

which can be invoked by requesting the page

/accounts/<owner>/admin/projects/delete?proj_id=<project ID>.

This link was added to templates/user.projects.mako

and was tested several times.

MySQL string bug

In the process of creating new projects, I discovered a bug which manifests as a server failure if there is an apostrophe (or single quote) in the name or description of the project. This is because that character is the string delimiter for MySQL commands and needs to be somehow escaped or encoded before being sent to MySQL. I have not done this yet.

MySQL usage errors

There are still frequent SQL errors. They fall into two categories as far as I can see:

1. Commands out of sync
2. Empty set returned

The second type results from the Python query result being an empty Python tuple instead of the Python None object.

Both of these errors show up at various SQL query calls in the code, and never repeatably at any particular call.

Uploads directory/package mystery

There is a mysterious difference between the development and production versions. In my development version, uploaded projects and logs show up in the python module directory under env/lib/python2.7/site-packages/community_csdt-0.0-py2.7.egg. In the production server, however, they show up in the source directory ~/modwsgi/csdt-env/community_csdt rather than ~/modwsgi/csdt-env/lib/python2.7/site-packages/community_csdt-0.0-py2.7.egg/community_csdt.

Somehow past developers have managed to convince Python that the module is installed at the source directory. This has some benefits and some drawbacks. On one hand, the uploads directory is not wiped out when the project is reloaded. On the other hand, messing with the source directory is messing with the live website, and becomes much more dangerous.

Thumbnails

Now for our feature presentation. My approach to enabling thumbnails for the community projects is two-fold.

1. modify the pCSDT Core to generate a thumbnail and embed it in the XML file when it is saved.
2. modify the Python upload function to extract this thumbnail and display it on the website.

I've more or less got phase 1 working. I combined code from the screenshot function jButtonShotActionPerformed at GUI.java line 1618 and placed it in the SaveDocument function in GUI.java line 1366, along with some image scaling code found on stack overflow and modified for speed. After that I just added another property like the background image (called it thumbnail image), and voila. The File->Save button creates an XML file with two base-64 encoded images, one is the background and the other is the thumbnail. The only thing left to do now is get Python to read the XML, decode the string and write out the image file. Hopefully most of this will be in its standard library.

Merged production code into SVN

Previously several of the contributions to the website existed only as manual edits to the code on the production server, and were not backed up in the SVN repository.

I simply grabbed a tarball of the production server code, used scp to move it to my laptop, and looked carefully at the diff between the two trees.

Some of the differences were production-specific configuration (e.g. SQL credentials). It would greatly streamline development if such configuration could be in a file selected at runtime rather than hardcoded into the Python source.

I found some functionality in src/views/accounts_view.py: getFriends, getComments, etc. These have been safely merged into the SVN repository after being tested on my laptop. The src/views/register_view.py file had all mention of reCAPTCHA commented out, which I also merged into SVN. I did the same for several stylistic changes in the static files (JavaScript and CSS) and some of the

Mako templates.

In summary, the SVN repository now has all the functions of the production server except for production-specific configuration.

P.S. - I also looked at the code on the test server and found

no changes

Remove Edit Button

A user's projects table shows a column for editing and a column for deleting. Both of these contain buttons that take you back to the home page because they're not implemented. I previously implemented (or rather resurrected) the project description editor, so that the edit button column should be removed. In order to remove the edit button, however, I had to trace through most of the code in templates/user.projects.mako. This is because the table is not declared in a single coherent block of code, rather it is defined piece-by-piece, with each piece expecting the columns to be laid out a certain way. Some of the side effects of removing a column were hard to find: clicking on a row triggers JavaScript code that retrieves table values using jqGrid's getCell function with hardcoded column IDs. This means that a bunch of column IDs have to be manually decremented when a column is removed.

Remove Duplicate Project Table

I realized that the project table shows up in two places: a dedicated page for the project table and the user profile page. In the implementation, the code from the projects page was copy-pasted into the profile page. It will work better from an implementation standpoint to only keep the dedicated projects table page, which is what I've done in the SVN repository. If Prof. Eglash approves I will put this up in the production server.

Projects Wipeout Warning

The file docs/installation/pyramid.installation.txt now mentions that when the server is reinstalled, the uploads directory is wiped out and should be backed up before reinstallation.

Move Description Editor

The description now shows up above the description editor box, which makes the editor's purpose more clear.

Software Quality Issue: Copy-Paste

I frequently find large duplicate blocks of code throughout both the Python source and Mako templates. Future developers are advised as follows:

1. Do not copy and paste code.

2. Be aware that code is duplicated. If you change something, look for a duplicate somewhere. If you find it, combine them and then add your changes (this happened to me while removing the Edit button).

First Issue Fixed - Return to Profile

The easiest issue, which is bringing users back to their profile page after they upload a project, was fixed by simply changing the link in uploaded.mako.

Next Issue - Edit Description

While I was looking for a way to implement users changing their descriptions, I stumbled across some dormant code that is supposed to do it ! At the high level, it shows the user a text input window above the description in the single project display page, with a save button to change the description. It also worked ! (after I changed the python if statement that controls it to use newer information).

One Step Forward, Two Steps Back

While those issues are fixed on my local install, some things are still rather funny. There are two new issues that have shown up:

1. Java applet loads... sometimes

I'm not sure how to determine when and why the Java applet loads successfully. Its only loaded once after I visited the page that hosts the CSDTs. Still, this is probably affected by issue #2.

Update: the non-loading was just caused by my project files being erased when I reinstall the website Python egg. It works fine with a newly uploaded project.

2. MySQLdb: you can't run this command now

There was an error in the server's MySQL calls along these lines:

ProgrammingError: (2014, "Commands out of sync; you can't run this command now")

Googling this showed me several possible causes, and I think this one applies to my case:
http://stackoverflow.com/questions/11583083/python-commands-out-of-sync-you-cant-run-this-command-now
It seems our custom class database.py does not close the cursor.
Yep, after changing the database class to close the cursor I don't see this error anymore.

Summary

3 issues were fixed: returning to profile after upload, editing of project descriptions, database cursor closing.

What Now?

I should push these changes to the servers ASAP, will discuss the best method for this. I would ideally like the servers to checkout and update from SVN directly. Already I know that the code on the servers has changes not in SVN.

This post is a summary of the past week of work on the Community CSDT website.
The project has been going on for a while, and I recently joined in to help.
Following the great example of Haiqiong's blog on his work
http://csdt-lih9.blogspot.com
I will try to make this a blog that is useful for future developers to read.
The source code for the website is in this (restricted) SVN repository:
https://subversion.hss.rpi.edu/svn/csdt_src
All references to source files in this blog will be relative to that directory

My background is low-level C programming for supercomputers, so I will present a slightly different (inexperienced) perspective on this Python/Java/etc website. As a developer, my priorities are as follows:

1. understand the code
2. compile / run the code
3. improve the code

Understanding

This particular project is somewhat lacking in resource for understanding the code. What I have done so far is to read through most of the Python source code, which I understood more after reading this

http://me.veekun.com/blog/2011/07/14/pyramid-traversal-almost-useful/

which describes the overall way that the Pyramid framework calls Python code based on which URL is requested.

The Community CSDT website is built as follows: the Python Pyramid framework is the core of the system, serving content based on the Python classes defined. The user information is stored in a MySQL database, and the main content being served is the actual CSDT Java applets.

Compiling and Running

I was able to compile and run the website on my laptop using the instructions at

Community/docs/installation/pyramid.installation.txt

Well... almost. I ran into a couple of issues (which I later found that Haiqiong also ran into and mentioned in the blog). I added comments to the installation text file so that future developers do not need to read blogs to compile their code.

Right now, the setup I have on my laptop does not show the thumbnails or applets, I think there is still some copying I have to do from pCSDT to Community.

Improving the Code

There were several issues I explored before even compiling the code for myself.

Presenting them will serve as a work log and a depth-first explanation of how the code works.

First, the website allows new users to register and has an email-confirmation system. That means registration is broken into two steps: ask the user for her name and email, send a link for part 2 to the email provided, then when the link is clicked continue with part 2. To be more precise, once the user has provided a name and email, that information is encrypted into a long string which becomes the query in the URL for part 2. This URL is long, so it is then shortened using TinyURL.

There were reports of this TinyURL not working, here is what I observed:

• The very first time I tried to register, clicking the TinyURL brought me to a page saying TinyURL couldn't find the destination.
• I got the true URL from the log file of the web server, and pasted that into my browser. that worked.
• I pasted the true URL into the TinyURL website. This returned me the same TinyURL as before, which means it was registered the first time. Now following the TinyURL works
• After that, I tried to reproduce the failure again but couldn't. I registered with different names and emails from different computers.

So that issue is on hold until it can be replicated again. One solution would be to avoid TinyURL altogether. We use it because the true URL is long, which is because the name and email are encoded in the query string. Maybe we can keep the name and email as session data (in a cookie or in the MySQL database), and just send a URL with some kind of session id as the email confirmation.

Second, when viewing a user's list of projects, the Edit and Delete buttons just link to the home page instead of doing anything. Simple enough, they have not been implemented. the links in 

Community/src/community_csdt/community_csdt/templates/user.profile.mako:85

should be changed when there is code for editing and deleting.

Third, after a project is uploaded, we would like users to be taken back to their profile page instead of the home page. This should be easy to do by simply changing one line:

Community/src/community_csdt/community_csdt/templates/uploaded.mako:16

Now What?

I have an almost complete setup on my laptop, so I will try to solve some of the issues mentioned and then push them to the test and development servers before next Wednesday.