Module Assembly - reporting tool refactor and load user json data to Postgres

Key Information

Register
Submit
The challenge is finished.

Challenge Overview

In this challenge, we want to refactor the TopCoder reporting tool and add the ability to extracted user json data into Postgres database.

Requirements Details

Refactoring

  • Use the root package com.topcoder.utitlities.reporting instead
    • Put all the model classes into package com.topcoder.utitlities.reporting.model
    • Put all the extractor classes into package com.topcoder.utitlities.reporting.extractor. Remove the word BigQuery in class names and make them generalized.
    • Put big query specific classes into com.topcoder.utitlities.reporting.bigquery, you can create sub package under it if you want.
    • For conf and schema, please re-organize to put Big Query specific files into sub folder bigquery. For common files, rename or reorgize to remove the word big query.

Load to Postgres.

  • We want to load the user data into postgres first. You need to define the table schema and create a new table user in postgres. The table schema should be similar to the google big query user table schema, make sure to choose the appropriate date type.
  • Implement the postgres user loader under package com.topcoder.utitlities.reporting���.postgres
    • The loader should load the extracted user json data into the user table created above.
    • Before each loading, clear all the data in that table.
  • Define job configuration similar to big query and implement an utility to read the jobs config and do the extract and load job.
  • The utility should be able to hook up throught ant like big query utility.

Environment

  • JDK 1.6 or above
  • Postgres version 9.3.3

VM Environment

You need a TopCoder VM to work on this challenge, you will mainly use the informix database on VM and if you choose to run data loader to load the warehouse data, you will need to run the loaders under account 'tc'. Please request the VM in the challenge forum.

Information about VM can be found below:

VM specific information is found here: http://www.topcoder.com/wiki/display/docs/VM+Image+2.5. Upon registration as a submitter or reviewer you will need to request a VM based on the new TopCoder Cockpit/Direct image. To request your image, please use the forum. Before requesting your VM, you need to ensure that you have an SSH key created and in your member profile. Instructions to do so are here: http://www.topcoder.com/wiki/display/projects/Generate+SSH+Key, and instructions to connect afterwards are here: http://www.topcoder.com/wiki/display/projects/Connect+Using+SSH+Key.

Source Repo

  • Please use the latest commit in branch master : https://github.com/appirio-tech/tc1-batch-reporting as work base


Final Submission Guidelines

  • The added / changes files or a patch.
  • The deployment guide with verification details.

ELIGIBLE EVENTS:

2015 topcoder Open

REVIEW STYLE:

Final Review:

Community Review Board

Approval:

User Sign-Off

SHARE:

ID: 30046791