JOIN
Get Time
long_comps_topcoder  Problem Statement
Contest: Spacenet Round 2 *TCO17*
Problem: BuildingDetector2

Problem Statement

    

Prize Distribution

              Prize             USD
  1st                          $6,000
  2nd                          $3,000
  3rd                          $1,500
  Best F-score, Las Vegas      $1,000
  Best F-score, Paris*         $1,000
  Best F-score, Shanghai       $1,000
  Best F-score, Khartoum       $1,000
  Early Incentive**            $1,000
  Total Prizes                $15,500

* The winning entry on the Paris data set must score higher than 400,000, which is above a score derived solely from OpenStreetMaps (OSM).

** To win the Early Incentive your solution must be the first to reach a threshold of 400,000 for the average F-score of all the cities.

Overview of changes from round 1

This contest is the second edition of the SpaceNet building footprint detection challenge. Most of the problem statement is identical to the specification used in the first challenge. The most important differences are the following:

  • There are much more data, featuring 4 cities
  • There are multiple new imagery formats (PAN, MUL, RGB-PanSharpen, MUL-PanSharpen)
  • The quality of the building footprint annotations has been improved
  • The length of the competition is longer (2 months)
  • Usage of external data and pretrained models are explicitly allowed
  • Source code and algorithm description of round 1 winners is available
  • Teams are allowed (up to 5 people)
  • The contest is not rated
  • Organizations can participate but will not be eligible for prize money
  • All winning solutions must be submitted as docker containers with all preprocessing and postprocessing steps

There are other, less significant differences, so if you participated in the first contest please read this document carefully to make sure you are aware of changes in the details.

Background

The commercialization of the geospatial industry has led to an explosive amount of data being collected to characterize our changing planet. One area for innovation is the application of computer vision and deep learning to extract information from satellite imagery at scale. DigitalGlobe, CosmiQ Works, and NVIDIA have partnered to release the SpaceNet data set to the public to enable developers and data scientists.

Today, map features such as roads, building footprints, and points of interest are primarily created through manual techniques. We believe that advancing automated feature extraction techniques will serve important downstream uses of map data. For example, this type of map data is needed for planning large-scale international events, as well as humanitarian and disaster response as recently observed in Haiti to aid the response to Hurricane Matthew. Furthermore, we think that solving this challenge is an important stepping stone to unleashing the power of advanced computer vision algorithms applied to a variety of remote sensing data applications in both the public and private sector.

Objective

Can you help us automate mapping? In this challenge, competitors are tasked with finding automated methods for extracting map-ready building footprints from high-resolution satellite imagery. Moving towards more accurate fully automated extraction of buildings will help bring innovation to computer vision methodologies applied to high-resolution satellite imagery, and ultimately help create better maps where they are needed most.

Your task will be to extract polygonal areas that represent buildings from satellite images. The polygons your algorithm returns will be compared to ground truth data, the quality of your solution will be judged by the combination of precision and recall. See Scoring for details.

Input Files

Satellite images

Four types of images are available for the target areas:

  1. PAN: panchromatic (single channel, 16-bit grayscale, ~30 cm resolution)
  2. MUL: 8-band multi-channel (8*16-bit, ~1.2m resolution). This is the equivalent of the 8-band images from the first competition.
  3. RGB-PanSharpen: pan-sharpened version of Red-Green-Blue bands from the multispectral product (3 channels, 3*16-bit, ~30 cm resolution). This is the equivalent of the 3-band images from the first competition. This is formed by using the PAN image to interpolate 3 bands of the MUL dataset to increase the resolution of the Red, Green and Blue bands.
  4. MUL-PanSharpen: pan-sharpened version of MUL (8 channels, 8*16 bit, ~30 cm resolution)

The images were collected by the DigitalGlobe Worldview-3 satellite over Las Vegas, Paris, Shanghai and Khartoum. The training data contains more than 10,000 images, each image covers 200m x 200m on the ground. Worldview-3 is sensitive to light in a wide range of wavelengths, see the specification of the frequency bands here. This extended range of spectral bands allows Worldview-3 imagery to be used to classify the material that is being imaged.

Images are provided in GeoTiff format. Since most image viewer applications will not be able to display 16-bit images and 8-band images, we provide a visualizer tool that you can use to look at the data. See the Visualizer description for details.

Building footprints

The location and shape of known buildings are referred to as ‘ground truth’ in this document. These data are described in CSV files using the following format:

ImageId,BuildingId,PolygonWKT_Pix,PolygonWKT_Geo

img1,1,"POLYGON ((103.7 205.4 0,107.8 201.9 0,
100.5 203.2 0,94.4 208.0 0,93.0 215.7 0,92.1 226.1 0,89.6 228.8 0,
95.0 233.9 0,92.4 236.6 0,95.4 239.8 0,116.7 221.0 0,116.7 216.1 0,
103.7 205.4 0))","POLYGON ((-43.681699199999969 -22.981289 0, ... 
[truncated for brevity]...))"

img2,-1,POLYGON EMPTY, POLYGON EMPTY

img339,1,"POLYGON ((213 269 0,184 221 0,
130 249 0,154 293 0,162 288 0,165 292 0,169 290 0,180 285 0,
213 269 0),(151 253 0,177 242 0,189 263 0,164 275 0,151 253 0))"
,"POLYGON ((...[truncated for brevity]...))"

(The sample above contains 4 lines of text, extra line breaks are added only for readability.)

  • ImageId is a string that uniquely identifies the image.
  • BuildingId is an integer that identifies a building in the image, it is unique only within an image. A special id of -1 is used to mean that there are no buildings in the image.
  • PolygonWKT_Pix specifies the points of the shape that represents the building in Well Known Text format. Only the POLYGON object type of the WKT standard is supported. If BuildingId is -1 then the POLYGON EMPTY construct is used in this column. (See line #3 for an example above.) The coordinate values represent pixels, the origin of the coordinate system is at the top left corner of the image, the first coordinate is the x value (positive is right), the second coordinate is the y value (positive is down), the third coordinate is always 0. Note that the polygons must be closed: the first and last entries in their point list must be the same.
  • PolygonWKT_Geo specifies the points of the same shape in geographical coordinates in {latitude, longitude, 0} triplets.

Usually a building is defined by a single polygon that represents the exterior boundary edge of its shape. This is sometimes not enough, see building #6 on image AOI_5_Khartoum_img1308 for an example shape that contains a hole. In such cases two (or more) polygons are needed, the first one always defines the exterior edge, the second (third, fourth, etc) defines the interior rings (holes). (See line #4 above for an example of the required syntax in this case.)

The PolygonWKT_Pix coordinates refer to the PAN images. Note that because of the different resolutions, the coordinates of the same object on the MUL images are different.

Notes

  • The way of representing holes is different from how the WKT standard specifies them: the standard mandates that the points of a polygon must be enumerated in an anti-clockwise order if the polygon represents a shape with positive area (exterior rings), and in a clockwise order if the polygon represents a shape with negative area (interior rings). However, what appears clockwise in the latitude/longitude based geographic coordinate system appears anti-clockwise in the image space where the y axis points downwards, so in the hope of causing less confusion we chose a simpler approach: the first polygon is always positive, the rest are always negative.
  • Some of the 8-band images may be erroneous, they may be either empty (all pixel values are 0) or contain unrealistically large values in some of the bands. Your algorithm should be able to handle these and similar errors.
  • The ground truth data was created manually and its quality is much better than it was in the first challenge. Nevertheless, as in all real life problems, it may contain errors. Also you may annotate building footprints differently than how the annotators did.
  • The task of finding building footprints is different from finding the outline of the building’s shape on an image. In most cases the difference is not significant, but you should be aware of this difference, especially when a high building is viewed at an angle.

Downloads

Input files are available for download from the spacenet-dataset AWS bucket. A separate guide is available that details the process of obtaining the data. See also this page for description of the Spacenet AWS data and download instructions.

Note that the bucket holds much more data than what is required for this challenge. The minimal list of data files that you should download is the following:

  • spacenet_sample.tar.gz (200 MB)
  • AOI_2_Vegas/AOI_2_Vegas_Train.tar.gz (25.6 GB)
  • AOI_2_Vegas/AOI_2_Vegas_Test_public.tar.gz (8.4 GB)
  • AOI_3_Paris/AOI_3_Paris_Train.tar.gz (5.7 GB)
  • AOI_3_Paris/AOI_3_Paris_Test_public.tar.gz (1.9 GB)
  • AOI_4_Shanghai/AOI_4_Shanghai_Train.tar.gz (25.1 GB)
  • AOI_4_Shanghai/AOI_4_Shanghai_Test_public.tar.gz (8.3 GB)
  • AOI_5_Khartoum/AOI_5_Khartoum_Train.tar.gz (5.0 GB)
  • AOI_5_Khartoum/AOI_5_Khartoum_Test_public.tar.gz (1.6 GB)

Where

  • spacenet_sample.tar.gz contains a few sample images from all 4 cities. Use this if you want to get familiar with the data without having to download any of the large files. Also this is the data you should use when making example submissions.
  • AOI_<n>_<city>_Train.tar.gz is the training set that belongs to a given city. It contains imagery in the 4 formats described above and also contains building footprints in a CSV file, see the summaryData/ folder.
  • AOI_<n>_<city>_Test_public.tar.gz is the testing set that belongs to a given city. It contains imagery but does not contain building footprints.

A sample submission file that scores non-0 on the spacenet_sample dataset is available here.

A note on image ids and image names

The format of an imageId is

AOI_<n>_<city>_img<i>

where the <n>_<city> part can take one of the following values: 2_Vegas, 3_Paris, 4_Shanghai, 5_Khartoum. <i> is a 1-based integer that is unique for an image within a set that belongs to a city. Images within the PAN, MUL, etc folders have names like PAN_<imageId>.tif, MUL_<imageId>.tif, etc.

Output File

Your output must be a CSV file with almost identical format to the building footprint definition files.

ImageId,BuildingId,PolygonWKT_Pix,Confidence

Your output file may or may not include the above header line. The rest of the lines should specify the buildings your algorithm extracted, one per line.

The required fields are:

  • ImageId is a string that uniquely identifies the image.
  • BuildingId is an integer that identifies a building in the image, it should be unique within an image and must be positive unless the special id of -1 is used. -1 must be used to signal that there are no buildings in the image.
  • PolygonWKT_Pix specifies the points of the shape that represents the building you found. The format is exactly the same as given above in the Input files section. Important to know that the coordinates must be given in the scale of the PAN images. So if you find a building that has a corner at (40, 20) on the PAN image and (10, 5) on the corresponding MUL image then your output file should have a (40 20 0) coordinate triplet listed in the shape definition.
  • Confidence is a positive real number, higher numbers mean you are more confident that this building is indeed present. See the details of scoring for how this value is used.

Your output must be a single file with .csv extension. Optionally the file may be zipped, in which case it must have .zip extension. The file must not be larger than 150MB and must not contain more than 2 million lines.

Functions

This match uses the result submission style, i.e. you will run your solution locally using the provided files as input, and produce a CSV or ZIP file that contains your answer.

In order for your solution to be evaluated by Topcoder’s marathon system, you must implement a class named BuildingDetector2, which implements a single function: getAnswerURL(). Your function will return a String corresponding to the URL of your submission files. You may upload your files to a cloud hosting service such as Dropbox or Google Drive, which can provide a direct link to the file.

To create a direct sharing link in Dropbox, right click on the uploaded file and select share. You should be able to copy a link to this specific file which ends with the tag "?dl=0". This URL will point directly to your file if you change this tag to "?dl=1". You can then use this link in your getAnswerURL() function.

If you use Google Drive to share the link, then please use the following format: "https://drive.google.com/uc?export=download&id=" + id

Note that Google has a file size limit of 25MB and can’t provide direct links to files larger than this. (For larger files the link opens a warning message saying that automatic virus checking of the file is not done.)

You can use any other way to share your result file, but make sure the link you provide opens the filestream directly, and is available for anyone with the link (not only the file owner), to allow the automated tester to download and evaluate it.

An example of the code you have to submit, using Java:

public class BuildingDetector2 {
  public String getAnswerURL() {
    //Replace the returned String with your submission file’s URL
    return "https://drive.google.com/uc?export=download&id=XYZ1";
  }
}

Keep in mind that your complete code that generates these results will be verified at the end of the contest if you achieve a score in the top six of the average F-score for all cities or in the top two of the highest F-score per city, as described later in the "Requirements to Win a Prize" section, i.e. participants will be required to provide fully automated executable software to allow for independent verification of software performance and the metric quality of the output data.

Scoring

A full submission will be processed by the Topcoder Marathon test system, which will download, validate and evaluate your submission file.

Any malformed or inaccessible file, or one that exceeds the maximum file size (150 MB) or the maximum number of lines (2 million) will receive a zero score.

If your submission is valid, your solution will be scored using the following algorithm.

  1. Sort the building polygons you returned in decreasing order of confidence.
  2. For each polygon find the best matching one from the set of the ground truth polygons. Loop over the truth polygons, and:
    1. Skip this truth polygon if it was already matched with another solution polygon.
    2. Skip this truth polygon if it belongs to a different ImageId than the solution polygon.
    3. Otherwise calculate the IOU (Intersection over Union, Jaccard index) of the two polygons.
    4. Note the truth polygon which has the highest IOU score if this score is higher than 0.5. Call this the ‘matching’ polygon.
  3. If there is a matching polygon found above, increase the count of true positives by one (TP).
  4. If there is no matching polygon found, increase the count of false positives by one (FP).
  5. When all solution polygons are processed then for each truth polygon that left unmatched increase the count of false negatives by one (FN).

The precision and recall of your algorithm are defined as

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

The F-score of your algorithm is defined as 0 if either Precision or Recall is 0. Otherwise:

F_score = 2 * Precision * Recall / (Precision + Recall)

Your overall score is calculated as 1000000 * F_score.

Note that your returned confidence values are used only to sort the building footprints you extract so that buildings with higher confidence values are tried to be matched first. The exact values don’t matter, they only establish a ranking among the buildings.

Note also that due to clipping building footprints at image chip boundaries very small buildings may appear. As it is not realistic to expect that your algorithm can find such buildings, all buildings with area less than 20 (measured in pixels on the PAN imagery) will be ignored both in the ground truth and in your solution.

For the exact algorithm of the registration and scoring see the visualizer source code.

Example submissions can be used to verify that your chosen approach to upload submissions works. The tester will verify that the returned String contains a valid URL, its content is accessible, i.e. the tester is able to download the file from the returned URL. If your file is valid, it will be evaluated, and precision, recall and F-score values will be available in the test results. The example evaluation is based on a small subset of the training data, the contents of the spacenet_sample.tar.gz file is used for this purpose.

Full submissions must contain in a single file all the extracted building polygons that your algorithm found in all images of the AOI_<n>_<city>_Test_public folders, i.e. all images of all 4 cities should be processed into a single submission file.

Final Scoring

The top six competitors for average F-score across all cities and the top two for the highest F-score per city, according the provisional scores, will be invited to the final testing round. In the past such final testing of 'submit data' type contests was done on AWS VM instances where contestants deployed an executable version of their systems which was given new test data to determine the final rankings. This time the process will be different. If your score qualifies you as a top 6 average F-score per city or top 2 F-score per city competitor, you will have to submit a Docker container that the organizers can use to score your solution. The exact details of this process will be announced not later than 2 weeks before the end of the provisional testing phase. The most important requirements for your final submission will be the following:

  • It should work out of the box: should contain or should be able to obtain all dependencies (data, libraries and 3rd party software).
  • It must be able to work on raw SpaceNet data, i.e. it should download or mount from the local filesystem the training and testing data from AWS and do all the preprocessing and training steps that are necessary to reproduce your complete work flow.
  • It must include both a training and an inference script.
  • You may use prebuilt data for testing, even encouraged to do so if training takes more than 24 hours. However, if training was done on SpaceNet data then organizers would like to have the ability to replicate the training: your training script must be able to reproduce your model if given more time.
  • The allowed time limit for testing is 8 hours. The final testing data will be similar in size and in content to the provisional testing data.
  • Your submission must contain full source code even if your scripts execute prebuilt binaries.

Your solution will be subjected to three tests:

First, your solution will be validated, i.e. we will check if it produces the same output file as your last submission, using the same input files used in this contest. Note that this means that your solution must not be improved further after the provisional submission phase ends. (We are aware that it is not always possible to reproduce the exact same results. E.g. if you do online training then the difference in the training environments may result in different number of iterations, meaning different models. Also you may have no control over random number generation in certain 3rd party libraries. In any case, the results must be statistically similar, and in case of differences you must have a convincing explanation why the same result can not be reproduced.)

Second, your solution will be tested against a set of new image files. The number and size of these new set of images will be similar to the one you downloaded as testing data. Also the scene content will be similar.

Third, the resulting output from the steps above will be validated and scored. The final rankings will be based on this score alone.

Competitors who fail to provide their solution as expected will receive a zero score in this final scoring phase, and will not be eligible to win prizes.

Additional Resources

  • A visualizer is available here that you can use to test your solution locally. It displays your extracted building footprints, the expected ground truth, and the difference of these two. It also calculates precision, recall and f-scores so it serves as an offline tester. (But note that the visualizer does not enforce the limits on allowed file size and nuber of lines.)
  • The SpaceNet Challenge Asset Library contains plenty of reading material and tools that make it easier to get started.
  • See the algorithm description and source code of the winning entries from the first contest here.

General Notes

  • This match is NOT rated.

  • Teaming is allowed. Topcoder members are permitted to form teams for this competition. After forming a team, Topcoder members of the same team are permitted to collaborate with other members of their team. To form a team, a Topcoder member may recruit other Topcoder members, and register the team by completing this Topcoder Teaming Form. Each team must declare a Captain. All participants in a team must be registered Topcoder members in good standing. All participants in a team must individually register for this Competition and accept its Terms and Conditions prior to joining the team. Team Captains must apportion prize distribution percentages for each teammate on the Teaming Form. The sum of all prize portions must equal 100%. The minimum permitted size of a team is 1 member, the maximum permitted team size is 5 members. Only team Captains may submit a solution to the Competition. Topcoder members participating in a team will not receive a rating for this Competition. Notwithstanding Topcoder rules and conditions to the contrary, solutions submitted by any Topcoder member who is a member of a team but is not the Captain of the team may be deleted and is ineligible for award. The deadline for forming teams is 11:59pm ET on the 21th day following the date that Registration & Submission opens as shown on the Challenge Details page. Topcoder will prepare a Teaming Agreement for each team that has completed the Topcoder Teaming Form, and distribute it to each member of the team. Teaming Agreements must be electronically signed by each team member to be considered valid. All Teaming Agreements are void, unless electronically signed by all team members by 11:59pm ET of the 28th day following the the date that Registration & Submission opens as shown on the Challenge Details page. Any Teaming Agreement received after this period is void. Teaming Agreements may not be changed in any way after signature.

    The registered teams will be listed in the contest forum thread titled "Registered Teams".

  • Organizations such as companies may compete as one competitor, but they will not be eligible for a prize. However, organizations can place in the standings, if they release their implementation code and methods.

  • Relinquish - Topcoder is allowing registered competitors or teams to "relinquish". Relinquishing means the member will compete, and we will score their solution, but they will not be eligible for a prize. This is to allow contractors to compete. Once a person or team relinquishes, we post their name to a forum thread labeled "Relinquished Competitors". Relinquishers must submit their implementation code and methods to maintain leaderboard status.

  • In this match you may use any programming language and libraries, including commercial solutions, provided Topcoder is able to run it free of any charge. You may also use open source languages and libraries, with the restrictions listed in the next section below. If your solution requires licenses, you must have these licenses and be able to legally install them in a testing VM (see "Requirements to Win a Prize" section). Submissions will be deleted/destroyed after they are confirmed. Topcoder will not purchase licenses to run your code. Prior to submission, please make absolutely sure your submission can be run by Topcoder free of cost, and with all necessary licenses pre-installed in your solution. Topcoder is not required to contact submitters for additional instructions if the code does not run. If we are unable to run your solution due to license problems, including any requirement to download a license, your submission might be rejected. Be sure to contact us right away if you have concerns about this requirement.

  • You may use open source languages and libraries provided they are equally free for your use, use by another competitor, or use by the client.

  • If your solution includes licensed software (e.g. commercial software, open source software, etc), you must include the full license agreements with your submission. Include your licenses in a folder labeled "Licenses". Within the same folder, include a text file labeled "README" that explains the purpose of each licensed software package as it is used in your solution.

  • External data sets and pre-trained networks are allowed for use in the competition provided the following are satisfied:

    • The pre-trained network dataset is unencumbered with any legal restriction that conflicts its use in the competition.
    • The data source or data used to train the pre-trained network is defined in the submission description.
    • The external data source must be declared in the competition forum in the first 45 days of the competition to be eligible in a final solution. References and instructions on how to obtain are valid declarations (for instance in the case of license restrictions).If you want to use a certain external data source, post a question in the forum thread titled "Requested Data Sources". Contest stakeholders will verify the request and if the use of the data source is approved then it will be listed in the forum thread titled "Approved Data Sources".
  • Use the match forum to ask general questions or report problems, but please do not post comments and questions that reveal information about the problem itself or possible solution techniques.

  • The stakeholders of this competition are especially interested in how the presence of 8-band imagery improves the quality of results. They believe the extra multispectral bands provide extra information that your algorithms can incorporate in their decisions even better than how a trained human could.

Requirements to Win a Prize

Early incentive

To encourage early participation a bonus prize will be awarded to the first participant who reaches a threshold of 400,000 on the average F-score over all 4 cities. In order to receive this bonus prize you must package the source codes you used to generate the submission that passes this threshold, and within two days you must send it to walrus71@copilots.topcoder.com and tim@copilots.topcoder.com so that we can verify that your submission was generated algorithmically. We won't try to run your code at this point, so you don't have to package data files, model files or external libraries, this is just a superficial check to see whether your system looks convincingly automatized.

Final prizes

In order to receive a final prize, you must do all the following:

Achieve a score in the top three of the average F-score of all cities, or the highest F-score per city, according to system test results. See the "Final scoring" section above.

Once the final scores are posted and winners are announced, the prize winner candidates (on overall F-score and on individual city F-score) have 7 days to submit a report outlining their final algorithm explaining the logic behind and steps to its approach. You will receive a template that helps creating your final report.

If you place in a prize winning rank but fail to do any of the above, then you will not receive a prize, and it will be awarded to the contestant with the next best performance who did all of the above.

 

Definition

    
Class:BuildingDetector2
Method:getAnswerURL
Parameters:
Returns:String
Method signature:String getAnswerURL()
(be sure your method is public)
    
 

Examples

0)
    
Test case 1

This problem statement is the exclusive and proprietary property of TopCoder, Inc. Any unauthorized use or reproduction of this information without the prior written consent of TopCoder, Inc. is strictly prohibited. (c)2010, TopCoder, Inc. All rights reserved.