Merged production code into SVN

Previously several of the contributions to the website existed only as manual edits to the code on the production server, and were not backed up in the SVN repository.

I simply grabbed a tarball of the production server code, used scp to move it to my laptop, and looked carefully at the diff between the two trees.

Some of the differences were production-specific configuration (e.g. SQL credentials). It would greatly streamline development if such configuration could be in a file selected at runtime rather than hardcoded into the Python source.

I found some functionality in src/views/accounts_view.py: getFriends, getComments, etc. These have been safely merged into the SVN repository after being tested on my laptop. The src/views/register_view.py file had all mention of reCAPTCHA commented out, which I also merged into SVN. I did the same for several stylistic changes in the static files (JavaScript and CSS) and some of the

Mako templates.

In summary, the SVN repository now has all the functions of the production server except for production-specific configuration.

P.S. - I also looked at the code on the test server and found

no changes

Remove Edit Button

A user's projects table shows a column for editing and a column for deleting. Both of these contain buttons that take you back to the home page because they're not implemented. I previously implemented (or rather resurrected) the project description editor, so that the edit button column should be removed. In order to remove the edit button, however, I had to trace through most of the code in templates/user.projects.mako. This is because the table is not declared in a single coherent block of code, rather it is defined piece-by-piece, with each piece expecting the columns to be laid out a certain way. Some of the side effects of removing a column were hard to find: clicking on a row triggers JavaScript code that retrieves table values using jqGrid's getCell function with hardcoded column IDs. This means that a bunch of column IDs have to be manually decremented when a column is removed.

Remove Duplicate Project Table

I realized that the project table shows up in two places: a dedicated page for the project table and the user profile page. In the implementation, the code from the projects page was copy-pasted into the profile page. It will work better from an implementation standpoint to only keep the dedicated projects table page, which is what I've done in the SVN repository. If Prof. Eglash approves I will put this up in the production server.

Projects Wipeout Warning

The file docs/installation/pyramid.installation.txt now mentions that when the server is reinstalled, the uploads directory is wiped out and should be backed up before reinstallation.

Move Description Editor

The description now shows up above the description editor box, which makes the editor's purpose more clear.

Software Quality Issue: Copy-Paste

I frequently find large duplicate blocks of code throughout both the Python source and Mako templates. Future developers are advised as follows:

1. Do not copy and paste code.

2. Be aware that code is duplicated. If you change something, look for a duplicate somewhere. If you find it, combine them and then add your changes (this happened to me while removing the Edit button).

First Issue Fixed - Return to Profile

The easiest issue, which is bringing users back to their profile page after they upload a project, was fixed by simply changing the link in uploaded.mako.

Next Issue - Edit Description

While I was looking for a way to implement users changing their descriptions, I stumbled across some dormant code that is supposed to do it ! At the high level, it shows the user a text input window above the description in the single project display page, with a save button to change the description. It also worked ! (after I changed the python if statement that controls it to use newer information).

One Step Forward, Two Steps Back

While those issues are fixed on my local install, some things are still rather funny. There are two new issues that have shown up:

1. Java applet loads... sometimes

I'm not sure how to determine when and why the Java applet loads successfully. Its only loaded once after I visited the page that hosts the CSDTs. Still, this is probably affected by issue #2.

Update: the non-loading was just caused by my project files being erased when I reinstall the website Python egg. It works fine with a newly uploaded project.

2. MySQLdb: you can't run this command now

There was an error in the server's MySQL calls along these lines:

ProgrammingError: (2014, "Commands out of sync; you can't run this command now")

Googling this showed me several possible causes, and I think this one applies to my case:
http://stackoverflow.com/questions/11583083/python-commands-out-of-sync-you-cant-run-this-command-now
It seems our custom class database.py does not close the cursor.
Yep, after changing the database class to close the cursor I don't see this error anymore.

Summary

3 issues were fixed: returning to profile after upload, editing of project descriptions, database cursor closing.

What Now?

I should push these changes to the servers ASAP, will discuss the best method for this. I would ideally like the servers to checkout and update from SVN directly. Already I know that the code on the servers has changes not in SVN.

This post is a summary of the past week of work on the Community CSDT website.
The project has been going on for a while, and I recently joined in to help.
Following the great example of Haiqiong's blog on his work
http://csdt-lih9.blogspot.com
I will try to make this a blog that is useful for future developers to read.
The source code for the website is in this (restricted) SVN repository:
https://subversion.hss.rpi.edu/svn/csdt_src
All references to source files in this blog will be relative to that directory

My background is low-level C programming for supercomputers, so I will present a slightly different (inexperienced) perspective on this Python/Java/etc website. As a developer, my priorities are as follows:

1. understand the code
2. compile / run the code
3. improve the code

Understanding

This particular project is somewhat lacking in resource for understanding the code. What I have done so far is to read through most of the Python source code, which I understood more after reading this

http://me.veekun.com/blog/2011/07/14/pyramid-traversal-almost-useful/

which describes the overall way that the Pyramid framework calls Python code based on which URL is requested.

The Community CSDT website is built as follows: the Python Pyramid framework is the core of the system, serving content based on the Python classes defined. The user information is stored in a MySQL database, and the main content being served is the actual CSDT Java applets.

Compiling and Running

I was able to compile and run the website on my laptop using the instructions at

Community/docs/installation/pyramid.installation.txt

Well... almost. I ran into a couple of issues (which I later found that Haiqiong also ran into and mentioned in the blog). I added comments to the installation text file so that future developers do not need to read blogs to compile their code.

Right now, the setup I have on my laptop does not show the thumbnails or applets, I think there is still some copying I have to do from pCSDT to Community.

Improving the Code

There were several issues I explored before even compiling the code for myself.

Presenting them will serve as a work log and a depth-first explanation of how the code works.

First, the website allows new users to register and has an email-confirmation system. That means registration is broken into two steps: ask the user for her name and email, send a link for part 2 to the email provided, then when the link is clicked continue with part 2. To be more precise, once the user has provided a name and email, that information is encrypted into a long string which becomes the query in the URL for part 2. This URL is long, so it is then shortened using TinyURL.

There were reports of this TinyURL not working, here is what I observed:

• The very first time I tried to register, clicking the TinyURL brought me to a page saying TinyURL couldn't find the destination.
• I got the true URL from the log file of the web server, and pasted that into my browser. that worked.
• I pasted the true URL into the TinyURL website. This returned me the same TinyURL as before, which means it was registered the first time. Now following the TinyURL works
• After that, I tried to reproduce the failure again but couldn't. I registered with different names and emails from different computers.

So that issue is on hold until it can be replicated again. One solution would be to avoid TinyURL altogether. We use it because the true URL is long, which is because the name and email are encoded in the query string. Maybe we can keep the name and email as session data (in a cookie or in the MySQL database), and just send a URL with some kind of session id as the email confirmation.

Second, when viewing a user's list of projects, the Edit and Delete buttons just link to the home page instead of doing anything. Simple enough, they have not been implemented. the links in 

Community/src/community_csdt/community_csdt/templates/user.profile.mako:85

should be changed when there is code for editing and deleting.

Third, after a project is uploaded, we would like users to be taken back to their profile page instead of the home page. This should be easy to do by simply changing one line:

Community/src/community_csdt/community_csdt/templates/uploaded.mako:16

Now What?

I have an almost complete setup on my laptop, so I will try to solve some of the issues mentioned and then push them to the test and development servers before next Wednesday.