Key Information

Register
Submit
The challenge is finished.

Challenge Overview

Project Overview

Create 3 (almost same) time-based background worker applications on Heroku platform to act as connector bridges to 3 vendor services (Xero, Deputy and Vend).  This Assembly will demonstrate how records created in Vendors can be automatically published to message broker.

Competition Task Overview

We had run one PoC challenge earlier, you can get concept from that but be care of many difference in this spec.

Your task is to create 3 separate heroku apps for the 3 different vendor services (Xero, Deputy and Vend), the 3 apps are almost same, see following required features for all:

1. Architecture

1.1 Use Heroku and Node.js as main platform, refer PoC winning submission code.

1.2 No web UI but only one background worker to act all.

Note: For Heroku Free Plan, only can run one dyno, So
You can config Procfile with "worker: node worker.js"
Then run "heroku ps:scale web=0 worker=1" command to start the worker.


1.3 OAuth are all long term access

Xero: Private Applications Token (Enable Payroll API for this organisation)
Deputy: Permanent Custom Token
Vend: No OAuth, just use Basic HTTP Authorization


1.4 Config by env variables like PoC code done.

1.5 All data transferred(Poll, Publish) should be JSON.

1.6 Logging all request url/header(except common and Authorization, only log what you set, such as "If-Modified-Since" for Xero API)/body, response code/body, error and exception into console(logplex)

1.7 Use Heroku Toolbel commandline to start/stop worker and view logs. document to deployment guide.

 

2. Worker

The worker will poll data periodically from vendor service to get new records and publish them to broker queue.

2.1 Functions in worker can be scheduled by "setInterval".

Note: interval configuration default value should be reasonable for API rate limits such as Xero is 1000/24hrs, Vend is 350s, no limits in deputy but set it more than 2 mins.


2.2 Poll data from API

2.2.1 Only Poll new data and don't publish repeat to queue. So each request should carry one timestamp (different format and position in APIs, see 2.2.3). ignore timestamp in first request to get all existing records.

Note: Using latest timestamp to request, the result maybe include previous latest record, if so, you need remember latest record id to avoid publish repeat to broker queue.
If not and you confirmed all date not repeat, you can ignore this comment.


2.2.2 Integrate which object from which Vendor, should be generic and config in one JSON file(not env variables). such as
xero:[receipts],
xero.payroll:[employees,timesheets]
These just mockup here, detail format is open to you.

Note you don't need handle all kinds of objects in APIs, just handle 2.2.3 objects, try best to reuse request handler and response handler in single Vendor integration.


2.2.3 Required objects
See attachment APIs.PDF

2.2.4 Keep state(timestamp, last object id if required?...) to postgresql to avoid broken when dyno restart. should check and read when worker dyno start.

Note: when worker start, should check both postgresql(queue-timestamp mapping) and object configuration JSON. 
If one queue/object exist in postgresql but not in configuration, remove it from postgresql and delete its queue.

 

2.3 Publish data to broker queue

2.3.1 Use heroku CloudAMQP add-on as broker. There's only one single broker for the 3 apps.

Following two approaches both are acceptable:
1. Create 1 new heroku app, add CloudAMQP add-on (Note this is free but requires verfication by creditcard)  to it, get CLOUDAMQP_URL for using in other 3 apps.

2. Use CloudAMQP directly which not require creditcard, see detail about how to in forum.


2.3.2 One kind of object one Queue, Queue is named by "vendor:object type" format such as "xero:receipts", "xero.payroll:employees". Don't delete/recreate queue like PoC.

2.3.3 One object one message, don't publish collections which got from poll directly.

2.3.4 All datetime format should be converted to standard ISO-8601 format(2014-07-23T08:39:14Z) when publish to queue.

This requirement is deleted, Just publish original object json into queue and don't make convert. note this's different with PoC done.

2.3.5 Don't publish repeat record(except updated records) to queue, see 2.2.1 Note.

2.3.6 Web UI to view data stored in the broker queue: Use CloudAMQP dashboard(Note it's very slow, need wait after clicking) to handle the queue on broker, Need document details in deployment guide like PoC done. If you use CloudAMQP directly, need describe clearly.

 

3. Setup Test Data

Xero: https://www.xero.com/ (Create Australian organisation)
Deputy: http://www.deputy.com/
Vend: http://www.vendhq.com/

You need register free trial on these websites and setup all corresponding data which required for upper functions verification. Don't use API to setup test data like which in PoC Deployment guide.

If you meet any issue to setup date, please ask in forum. 

 

 

4. Deployment guide

Should be detailed about setup/config/validate the requirements.
Each app includes own deployment guide

 

Documentation Provided

Document Name Document Description                                
APIs.PDF API details
PoC_ff2.zip PoC Source Code

 



Final Submission Guidelines

Submission Deliverables

A complete list of deliverables can be viewed in the TopCoder Assembly competition Tutorial at: http://apps.topcoder.com/wiki/display/tc/Assembly+Competition+Tutorials 

Below is an overview of the deliverables:

  • 3 separate apps per requirement.
  • Each includes a complete and detailed deployment documented explaining how to deploy the application including configuration information.

Final Submission

For each member, the final submission should be uploaded to the Online Review Tool.

Development Environment

Heroku
Nodejs

REVIEW STYLE:

Final Review:

Community Review Board

Approval:

User Sign-Off

SHARE:

ID: 30044674