Codesquare is a project mainly written in JAVASCRIPT and JAVA, it's free.
This Repository contains both Front and Back-End code for the CodeSquare Jive Web Application.
CODE DEPENDENCIES [CD] BUILD/MODIFACATION INFO [BMI] INFO AVAILABLE FOR DETERMINING BADGES [IADB] ADDING BADGES [AB] GIT COMMIT PROCESSING MECHANISMS [GCPM] BASIC BADGE PROCESSING MECHANISMS [BBPM] MAPRDUCE COMPLEX BADGE PROCESSING MECHANISMS [MCBPM] FRONT-END MECHANISMS [FEM]
This code requires a few dependencies to be put into place. There must be two configuration files in the bin folder of the Tomcat 7 folder that is running the application servlets. There must be a configuration file for HBase configurations, and one for the HDFS configurations. You must also confirm the location of the configuration files for the map reduces in the toolbox.java class. There is also a url that needs to be edited in the map reduce's toolbox.java that uses a servlet to post the users activity stream. The configurations were extracted from the main code base to conceal sensitive information that is needed for CodeSquare to run. You will see that the update hook (update.secondary) is not complete because that also has sensitive information. The last thing that should be verified is the path a json file containing the badge name, description, and icon url is stored, in the ServletTools.java in the method getBadgeInfo.
The Front-End Code can be found in CodeSquareApp although if you plan on modifying the code and want to see your changes, you should first ask a Jive Web Master to give you access to the CodeSquare Repo on JiveApps and then you should clone the repo in a separate folder outside this git folder using the jiveapps clone codesquare terminal command. It is also recommended that you read all the documentation found in developers.jivesoftware.com if you plan on making permanent changes to the app.
The Back-End Code can be found in the CodeSquareJava folder, although a config folder has been omitted from this repo in this repo to prevent the knowledge of private Jive URLs and other sensitive company-specific infrastructure data.
Needless to say, MapReduce Code used in our app can be found in the MapReduce folder but I'm saying this just in case it wasn't clear.
Both the CodeSquare Back-End Code and the MapReduces each have their own build.xml and two scripts exists in each folder to automatically build these two projects those scripts are run automatically when someone pushes to this Repository. IT IS STRONGLY RECOMMENDED THAT YOUR CODE IS BATTLE-TESTED BEFORE PUSHING TO THE REPO AND ALSO THAT YOU KEEP A COPY OF THE PREVOIUSLY WORKING WAR FILE ON THE TOMCAT & BEFORE REPLACING IT WITH THE NEWLY MADE ONE
There are two places in our app where information can be used to determine new badges. The first place in when the commits are being processed on the fly in BackEndServlet.java, where basic badges are determined. Basic Badges consist of holiday badges, number of commit badges, and various other badges. Information for these badges are stored in the Hbase under the qualifier name "Info" in the "EmpBadges" table. Each user has their boss email, # of consecutive commits, jive id, last commit date, name, newBadges, number of bugs(currently not implemented), and number of commits. Also each awarded badge with have a key, value pair in this format "Badge:'insert badge number here'", 'unix time stamp of first received' an example would be "Badge:1", "1313509878".
The second place were information can be used is in the HDFS where the map reduces grab commit data from. The naming scheme of the folders in the HDFS is the push date of the commit. For example, a push date of 10/24/2011, at 10:25am would be 2011/10/24/10/25/. The commit is stored as a .txt file where the name of the file is the sha1 commit hash and the contents are in the the following format:
CommId EmpEmail seconds_since_1970 time_zone numFilesChanged filesChanged numInsertions numDeletions message example: 3fd3fh345g [email protected] 2109345234 GMT-7 3 [
It is very easy to install a new badge to the system. There are few things that must be done to add a new badge. The first thing is you must add the badge name, description, and icon url to the "badgedescriptions" file that is inside the user's home folder where the tomcat 7 is running Codesquare.war. The file "badgedescriptions is formatted in json and is fairly straight forward to edit. The second part is that an image and a thumbnail must be pushed up to the image folder on codesquareapp git repository. The image must be 300x300 pixels and named in the following way: 'badgenumber.png' ex 4.png. The thumbnail must be put in images/thumbnails/ sized at 100x100 pixels and named 'badgenumberTH.png,' ex. 5TH.png. The last thing, also the hardest, is you must determine how the badge is going to be achieved. If it is a badge that does not require a map reduce, then it should be added to basicbadges.java where it processes simple badges, such as date or time badges. You can look at the methods in basicbadges.java as an example of how to give a user a badge. If it is a badge that requires a map reduce, then the map reduce will have to run by our jerkins program at a specific time. The map reduce should output to the hbase, with the user email as the row identifier. The badge should be added to under the 'Badge' qualifier, badge number column and should include the unix time stamp of when it was earned as the value of the cell. No, other code changes should be necessary. We designed the app so that new badges could be added to the system with out any major changes to the code, except the code that determines if a user earned the badge. All the front end app is able to handle an increase in badges, as it is receives all the information from the backend.
We learned many things about git in building this application for Jive. First off, it is way different than SVN. Adding code to a central repository requires the user to do two things. He/she must commit the change to their local repository, then he/she must push those commits up to either their current branch or the main repository. We took the approach of using an update hook to retrieve the new commits coming into the central git server. We felt this was the only option, because there is no way to determine which commits are new by looking at the git log, because two week old commits can be pushed and would be stuck in to log as two week old commits, not put at the top of the log. Since many of our badges used map reduces that were based on when you committed code, we were forced to use the push date as the sorting date for our map reduces. If we had used the commit date for the map reduces we would have had to re-run map reduces when ever some committed code that fell into a batch of commits that had already been processed by a map reduce. Then you would run into the decision on whether or not to take away and re-reward badges when map reduces are rerun. We were able to avoid all those questions by using push date for processing map reduces. When you commit to your local repo, your code does not affect others. Only when you push your code, does it affect your co-workers. It made much more sense for us to use push date for running map reduces, and commit date for basic badges.
The update hook used the two sha1 commit hash ids that identified the change delta. The two ids were are given to any update hook, along with the branch that the commits were pushed to. The hook uses those id's to do a query on the log with using the pretty print and stats options. Then it uses a curl command to post the data alone with the unix time stamp of the push to the backend servlet where it is processed for basic badges and stored into the HDFS.
The basic badge processing is done in the back end servlet. The git update hook preforms a curl command that posts a large json. That json contains a json object for each commit in that specific user's push. The back end servlet receives the commits and stores them in a local variable, and closes the servlet connection so it is not held during the processing. The back end servlet calls basicbadges.java, which takes each of those commits and makes a commit object(commit.java) out of each of them. A user object(user.java) is also created from data pulled form the hbase (Hbasetools.java). Next basicbadges.java writes each commit to the HDFS and determines the basic badges that the user has earned. Once all the commits are processed it preforms one put to the hbase. That put updates the new badges, and updated user info. In total, there is only one get and one put to the hbase for each time the hook calls the backend servlet.
A Jenkins server currently found on BigDataDev1 runs a script that ssh's into the HadoopDev007 cluster, where MapRedHaoop.jar is located and runs the appropriate MapReduce command depending on whether a minute, hourly, daily, or weekly MapReduce is to be run. MapRedHaoop.jar is also found in the MapReduces folder in our repo and it can be re-created by simply doing 'ant hadoopjar' in a terminal assuming you are in the MapReduces folder. This builds all the class files using the source files in the src folder and also imports the binaries it requires from the lib folder and then creates the hadoop jar containing all these binaries.
Toolbox.java contains an array of static methods used by the MapReduces such as getting an HBase or HDFS configuration object to use when connecting and manipulate the Hbase or hdfs. The badge_*.java class names each represent the main running class for the badge specified. (Ex. badge_14_15 is the main MapReduce running the passes that handle the processing for Badge 14 and Badge 15). The Passes that pertain to a certain badge are categorized in folders and the shared passes folder contains intermediate passes used by more than one MapReduce. Badges 14 and 15 are hourly MapReduces taking in the location to an hour folder. Badges 20, 24 are daily MapReduces taking in the location to a day folder. Badges 21,22,23 take in 7 locations of day folders which represent a week to run it's processing on. Badge 25 takes in two minute folders since it's a sliding window badge (since two people committing within 30 seconds can happen in two different minute folders). The Descriptions of the badges can be found in the Jive Intern Summer Class of 2011 brewspace place.
The front end is a jive app that, just like all jive apps, have a home view and a canvas view. The home view sends a request to the AppSevlet that contains the user's email, name, id and boss email. The canvas view sends a request with only the user's email. The servlet returns a json object that contains the obtained badges. Everything presented in the front end is dynamically built from that json object, so there is no modifying that needs to be done on the front end to support new badges. The app canvas view also contains a compare tab and a brag tab. The compare tab retrieves the other users badges by sending a request to the AppServlet with person the user is trying to view along with another argument compare set to false. The brag tab enables the user to select a badge and post it to their notifications, as well as the activity stream.