Challenge Overview
Challenge Requirements
In this challenge, we want to update the TC Online Review / Auto Pilot to handle a new review type called Peer Review. The main Peer Review process (assign reviewers, fill scorecards) will be handled outside the Online Review, but the phase management and final score aggregation / ranking submissions will still be done in Online Review.
1) Please refer to Create a Peer Review Challenge - Supply - Confluence.pdf on how to setup an existing draft CODE challenge into a Peer Review Challenge. You need to use TopCoder Direct on your VM to setup a new draft CODE challenge and following the steps in the document (i.e. run SQL scripts to do neccessary changes).
2) The peer review process itself is done outside the Online Review, see the Peer_Review_Screenshots.zip in the documents section.
3) After doing 1), the challenge will only have reg/submission/review phases, we want you to update the Online Review / Auto Copilot for Peer Review Challenge
- A peer review challenge has the project_info with type 79 (Review Type) set to 'PEER'
- Review phase of peer review challenge should only be started when the review start time is reached && there is at least one submission.
- Review phase start will not check the Review Number Phase Criteria and Whether the required reviewer number is met.
- Review phase of peer review in Online Review should not insert any review / resource related records, these will all be handled outside the Online Review.
- Review phase should only be closed when the review end time is reached && there are at least 5 reviews committed for each submission. Note that if reviews number is enough but end time is not reached, the Review phase should not be closed.
- Please have new phase start and end templates for the Review phase of peer review challenge. Update the configuration and creating new email notification templates. (for the supported template fields, check the handler implementation or ask in the forum).
- Before the review phase of peer review closes, it should call the score aggregation API to aggregate all the review scores. The scoring REST API is not ready yet, but please leave a placeholder method which we can add in the score aggregation logic which calls the a REST API later.
IMPORTANT: Remember to also update and test the AutoPilot, not just the Online Review application.
4) For Online Review project details page of Peer Review, please hide the Review/Appeals tab. You may also need to update the corresponding struts2 actions, so it does not throw any error.
5) For testing the implementation, you will manually insert peer review records for each submission. Please refer to attached ap-review-microservice-dev.zip and Scoring Service - Supply - Confluence.pdf on how to manually simulate the peer review process.
6) You will find there are plenty of places in the Online Review that you may have to update to work correctly. Please do a thorough testing and code review to make sure that everything works. And also make sure that the introduce of peer review handling does not break any existing challenge types with normal review.
Challenge Inputs:
Please find all the mentioned documents in the challenge - documents section.
Code Base:
The SVN repo for Online Review/Auto Pilot is in: https://coder.topcoder.com/tcs/clients/cronos/applications/online_review/trunk (Revision 248952)
Please send an email to support@topcoder.com with your TopCoder username, the challenge name/link and the SVN URL to request the read-only acesss.
Virtual Machine
VM specific information is found here: http://apps.topcoder.com/wiki/display/projects/Direct+VM and http://www.topcoder.com/wiki/display/docs/VM+Image+2.5. Upon registration as a submitter or reviewer you will need to request a VM based on the new TopCoder VM image. To request your image, please use the forum. Before requesting your VM, you need to ensure that you have an SSH key created and in your member profile. Instructions to do so are here: http://www.topcoder.com/wiki/display/projects/Generate+SSH+Key, and instructions to connect afterwards are here: http://www.topcoder.com/wiki/display/projects/Connect+Using+SSH+Key.
If you prefer to deploy Online Review locally instead of using a VM, please see this guide:http://apps.topcoder.com/wiki/display/docs/Online Review Setup Guide
Final Submission Guidelines
- All the updated source codes, configurations against the Online Review code base.
- A deployment guide with detailed verification steps.