EPA Android App and PM - Back End and Data Management Module Architecture

1. Project Overview

The EPA is a U.S. federal government agency devoted to safeguarding the environment. One of the EPA's great concerns is the proliferation of cyanobacterial harmful blooms (cyanoHABs) in the nation's lakes. The following resources provide information on what cyanoHABs are and how they threaten the environment.

• EPA: Cyanobacterial Harmful Blooms (CyanoHABs)
• University of North Carolina: Cyanobacteria [video]

The TopCoder project on cyanoHABs aims to develop an algorithm that will be deployed in an Android app with mapping and data visualization capabilities. The app will inform local and federal policy makers about locations where bloom events are likely to occur, allowing them to concentrate their efforts in those areas.

2. Contest Overview

Welcome to a very exciting and one of the most important architecture contest for EPA Android App project !!!

In this contest, we are looking for you to design the module architecture for complete back-end system of EPA Android App. We want you to use all the available information architecture in form of wireframes, system architecture and application requirements specification to develop this module architecture. We will also provide access to the prototype and front-end that have been already developed. Finally, we will also provide you component design for a component that is required to be embedded into the backend.

 

We have two major modules in this Android App server system and following are the short description for each of them:

1.) EPA Android App Back-End: This module will house all the back-end operations of the app. We envision it to be hosted on a server like Amazon EC2. The backed will contain a computation-intensive prediction algorithm which is going to be developed using Marathon match. The details of algorithm will not be available for this contest. The module will also be responsible for aggregation analysis to be done using the available data which will be sent to this module from the data management module explained next. This module will have very heavy computation, complete automation and very less data processing.

Some key points to consider while designing architecture for this module:

- It needs to be flexible to fit the prediction algorithm that comes out of the Marathon Match.

- This module will receive data only from data management module. It can either do it by polling a shared folder where the data will already be stored by data management module once it processes it or it can directly get the data from data management module.

- This module will send notifications and push image data (jpgs and pngs) to app front-end. It will also send results from algorithm and aggregation analysis to app front-end and also to the admin website front-end.

- This module is always expected to receive data in the form of csv format (to be used for analysis and algorithm) and light-weight images to be pushed to front-end.

- This module is required to be performance conscious.

2.) EPA Android App Data Management Module: This module will be responsible for doing all the data-processing tasks. There is a lot of data available in different formats - mostly geo-spatial data and hence it is required to be preprocessed before wither sending it to front-end or sending to the above back-end module for algorithm analysis. All the validation, conversion and standardization of data will be done in this module. In addition to other tasks, this module will also contain a component that will be used for data standardization. We have provided design document of the component. This module will have a lot of data processing and I/O intensive tasks.

Some key points to consider while designing architecture for this module:

- It needs to fit the component which will be used for data standardization.

- As mentioned above, this module will receive data in different formats (mostly tif and xls files) and this module will convert them into csv files based on some constraints. These constraints will be covered by the component but that component will be embedded as a part of this module.

- This module will also validate the data that is received and see if that data is useful. For example, on cannot get any Geospatial information from jpg images. The images must be in GeoTiff format to be useful for data extraction.

- This module will receive data only from admin website. It can either do it by polling a shared folder where the data will already be stored by admin website or it can directly get the data from admin website - storing in shared folder is preferable option.

- This module will also convert heavy tif images into light-weight jpg/images after the component has already extracted data from those images. These converted light weight images will then be sent to backend which will then push it to front-end for display.

- This module is also required to be performance conscious, specifically I/O performance.

3. Data Details

We have two types of input data:

• Image Data : GeoTiff images which has geo-encoded information
• Text Data: Excel files containing data.

The data is in very large amounts. The images will always be GeoTiff. The data management module will have the component which will read all this data and convert into single canonical structure. The component design document will provide information about this process.

4. Technology Overview

1. The Android App front-end has been developed for Android 4.1-4.4 support using android developer tools. It also uses Google Maps Android API v2, AChartEngine 1.1.0 and ProgressWheel.

2. The Data Standardization component (to be embedded in data management module) is developed in Java.

3. The Admin Website is a CMS developed using Django framework 1.6.1, Python 2.7x and PIL 1.1.7.

4. We are open to use of any technology for the back-end and data management module as far as it does not limit any specific OS requirements. Please remember that the backend will in-house algorithm obtained from Marathon Match, hence the architecture needs to be flexible to include that algorithm.

5. We plan to host the back-end and data management module along with the admin website on a single (same) server - ex. Amazon EC2 Server.

6. Open source software resources are welcome, but they must have third party support services available. Please ask in forum when in doubt.

7.  You are allowed to use any open source DB, data storage mechanisms for storing data at various stages.

8. Image data is being read using gdal Library. You are welcome to suggest/use other libraries but remember the component for data conversion uses this library

9. Please note that the developers are not allowed to use any component from TC Catalog for this contest.

5. Resources Provided

The following resources have been provided in the forums. You will be able to access it after registration:

1.) Conceptualization Document

2.) EPA Android App Front-End and EPA Admin Website Wireframes

3.) EPA Android App Front-End and EPA Admin Website Prototype

4.) EPA Android App Data Standardization Design Document.

5.) System Design Specification

6.) System Architecture TCUML

7.) Application Requirements Specification and Use Case TCUML

Please Note: In case of discrepancies, the System Architecture and ARS takes precedence over all other resources as they are the most update one. Please follow them and if there is still confusion, please ask in forums.
,

Final Submission Guidelines