×
Namespaces

Variants
Actions
(Difference between revisions)

HERE Maps API - Converting any data file to KML

From Nokia Developer Wiki
Jump to: navigation, search
jasfox (Talk | contribs)
m (Jasfox - Add link.)
jasfox (Talk | contribs)
m (Jasfox - links)
(12 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
{{Abstract|This article explains how to read address data from an arbitrary file format, and create a KML file for display on a map.}}  
 
{{Abstract|This article explains how to read address data from an arbitrary file format, and create a KML file for display on a map.}}  
 
{{SeeAlso|
 
{{SeeAlso|
*   [http://api.maps.nokia.com/ Nokia Maps API]  
+
* [http://developer.here.net/javascript_api Nokia Maps API]  
 
* [[Nokia Maps API - How to create a KML data file]]
 
* [[Nokia Maps API - How to create a KML data file]]
 
* [[Nokia Maps API - How to display KML file data on the map]]
 
* [[Nokia Maps API - How to display KML file data on the map]]
* [http://api.maps.nokia.com/2.1.1/playground/?example=kmlfile Loading a KML file example]
+
* [http://developer.here.net/apiexplorer/examples/api-for-js/data-visualization/map-with-interactive-kml-objects.html Loading a KML file example]
* [http://api.maps.nokia.com/2.1.1/playground/?example=search Search example]
+
* [http://developer.here.net/apiexplorer/examples/api-for-js/places-search/search-by-address.html Search example]
 
}}
 
}}
 
{{ArticleMetaData
 
{{ArticleMetaData
|sourcecode= [[Media:AnyFormatToKMLExample.zip]]  
+
|sourcecode= [[Media:AnyFormatToKMLExample.zip|AnyFormatToKMLExample.zip]]  
 
|installfile= <!-- Link to installation file (e.g. [[Media:The Installation File.sis]]) -->
 
|installfile= <!-- Link to installation file (e.g. [[Media:The Installation File.sis]]) -->
|devices= Firefox 9.0.1
+
|devices= Firefox , Internet Explorer, Google Chrome, Opera
 
|sdk= <!-- SDK(s) built and tested against (e.g. [http://linktosdkdownload/ Nokia Qt SDK 1.1]) -->
 
|sdk= <!-- SDK(s) built and tested against (e.g. [http://linktosdkdownload/ Nokia Qt SDK 1.1]) -->
 
|platform= Web Browser
 
|platform= Web Browser
 
|devicecompatability= <!-- Compatible devices e.g.: All* (must have internal GPS) -->
 
|devicecompatability= <!-- Compatible devices e.g.: All* (must have internal GPS) -->
|dependencies= <!-- Any other/external dependencies e.g.: Google Maps Api v1.0 -->
+
|dependencies=Nokia Maps 2.2.3
 
|signing=<!-- Signing requirements - empty or one of: Self-Signed, DevCert, Manufacturer -->
 
|signing=<!-- Signing requirements - empty or one of: Self-Signed, DevCert, Manufacturer -->
 
|capabilities= <!-- Capabilities required by the article/code example (e.g. Location, NetworkServices. -->
 
|capabilities= <!-- Capabilities required by the article/code example (e.g. Location, NetworkServices. -->
Line 36: Line 36:
 
Keyhole Markup Language (KML) is an XML notation for geographic applications. The advantages of using KML are numerous, and have been listed in a [[Nokia_Maps_:_Converting_from_JavaScript_to_KML| previous article]].  A typical enterprise may wish to add some markers representing addresses onto a map for their website, but  without necessarily learning too much about geocoding or KML. It is likely that the address data they have is already held in a file or spreadsheet somewhere.
 
Keyhole Markup Language (KML) is an XML notation for geographic applications. The advantages of using KML are numerous, and have been listed in a [[Nokia_Maps_:_Converting_from_JavaScript_to_KML| previous article]].  A typical enterprise may wish to add some markers representing addresses onto a map for their website, but  without necessarily learning too much about geocoding or KML. It is likely that the address data they have is already held in a file or spreadsheet somewhere.
  
This example aims to take the pain out of creating a KML dataset. It aims to take any data format and attempt to the locate addresses given by specified fields from the file. The addresses are then transformed into KML <Placemarks> with associated <name> and <description> elements taken from other fields from the same record. The generated KML data can be inspected and edited using the editor example given [[Nokia Maps API - How to create a KML data file|here]] and then displayed using the code from the [[Nokia Maps API - How to display KML file data on the map|How to display KML data]] example. In summary, this article demonstrates a real world use of the geocoding service and is an example of making sequential asynchronous JavaScript calls  to obtain longitude and latitude.
+
This example aims to take the pain out of creating a KML dataset. It aims to take any data format and attempt to locate addresses given by specified fields from the file. The addresses are then transformed into KML {{Icode|&lt;Placemark&gt;}} elements with associated {{Icode|&lt;name&gt;}} and {{Icode|&lt;description&gt;}} elements taken from other fields from the same record. The generated KML data can be inspected and edited using the editor example given [[Nokia Maps API - How to create a KML data file|here]] and then displayed using the code from the [[Nokia Maps API - How to display KML file data on the map|How to display KML data]] example. In summary, this article demonstrates a real world use of the geocoding service and is an example of making sequential asynchronous JavaScript calls  to obtain longitude and latitude.
 
+
  
 
==  Defining the issue ==
 
==  Defining the issue ==
Line 68: Line 67:
 
</code>
 
</code>
  
 +
=== Tab Separated Variable Format ===
 +
Tab Separated Variable (TSV) Format is similiar to CSV, with each colum separated by whitespace as shown below:
 +
<code>
 +
serial name unused address_street address_city address_district description
 +
13556 Millenium Stage Something 401 Bay Drive New York This is an example
 +
3243 The Chambers Something else 53 Bothwell Street Watford Hertfordshire Some more text here
 +
9954 4th Dimension More data 226 Myrtle Ave Toronto
 +
8645 E.K.B.R, Even more data Invalidenstrasse 117 Berlin Hier gibts etwas
 +
</code>
 
=== Other proprietary formats ===
 
=== Other proprietary formats ===
 
The [http://www.rightmove.co.uk/ps/pdf/guides/V3TestFile.blm Right Move] Estate Agent Data Format is a proprietary format popular amongst estate agents in the United Kingdom. A simplified extract for the properties defined above would look something the data below. This has been chosen as an alternative to illustrate the problem.
 
The [http://www.rightmove.co.uk/ps/pdf/guides/V3TestFile.blm Right Move] Estate Agent Data Format is a proprietary format popular amongst estate agents in the United Kingdom. A simplified extract for the properties defined above would look something the data below. This has been chosen as an alternative to illustrate the problem.
Line 110: Line 118:
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
!  !! CSV !! Right Move
+
!  !! CSV !! Tab Delimited !! Right Move
 
|-
 
|-
| '''Start of Header Indicator'''|| blank|| #DEFINITION#\n
+
| '''Start of Header Indicator'''|| blank|| blank|| #DEFINITION#\n
 
|-
 
|-
| '''Start of Data Indicator''' || \n|| #DATA#\n
+
| '''Start of Data Indicator''' || \n|| \n|| #DATA#\n
 
|-
 
|-
| '''Record Separator'''|| \n|| |\n\n
+
| '''Record Separator'''|| \n|| \n|| |\n\n
 
|-
 
|-
| '''Field Separator'''|| , || ^
+
| '''Field Separator'''|| , || \t  || ^
 
|}
 
|}
  
{{Note| The character sequence '''\n''' may be used to indicate a new line }}
+
{{Note| <br/>
 +
The character sequence '''\n''' may be used to indicate a new line  
 +
* The character sequence '''\t''' may be used to indicate a tabbed white space
 +
}}
  
Each record will be transformed into a KML <Placemark>. Obviously the decision as to  which fields to add to the KML will depend on the data provided. The following KML elements are supported:
+
Each record will be transformed into a KML {{Icode|&lt;Placemark&gt;}}. Obviously the decision as to  which fields to add to the KML will depend on the data provided. The following KML elements are supported:
  
* ID Field:  This translates to the <Placemark> id attribute, a unique indicator for each marker.
+
* ID Field:  This translates to the {{Icode|&lt;Placemark&gt;}} id attribute, a unique indicator for each marker.
* Style URL Field:    This translates to the<styleUrl> within the <Placemark> - it can be used to change the appearance of the marker by addding an associated <Style>
+
* Style URL Field:    This translates to the {{Icode|&lt;styleUrl&gt;}} within the {{Icode|&lt;Placemark&gt;}} - it can be used to change the appearance of the marker by addding an associated {{Icode|&lt;Style&gt;}}
* Description Fields:  This translates to the <description>within the <Placemark>, it is able to support HTML tags such as < b > or < h2 >
+
* Description Fields:  This translates to the {{Icode|&lt;description&gt;}} within the {{Icode|&lt;Placemark&gt;}}, it is able to support HTML tags such as {{Icode|&lt;b&gt;}} or {{Icode|&lt;h2&gt;}}
* Name Fields:  This translates to the <name> within the <Placemark>, it must be plain text
+
* Name Fields:  This translates to the {{Icode|&lt;name&gt;}} within the {{Icode|&lt;Placemark&gt;}}, it must be plain text
 
* Address: This will hold the address for which the geocode attempt was successful. See addressing attempts for details.
 
* Address: This will hold the address for which the geocode attempt was successful. See addressing attempts for details.
 
<br>
 
<br>
By default Nokia Maps will display the <name> and the <description> of the marker in an infobox when clicked. This may be styled by defining an associated <BalloonStyle>, defining a <Style> is not within the scope of this article.
+
By default Nokia Maps will display the {{Icode|&lt;name&gt;}} and the {{Icode|&lt;description&gt;}} of the marker in an infobox when clicked. This may be styled by defining an associated {{Icode|&lt;BalloonStyle&gt;}}, defining a {{Icode|&lt;Style&gt;}} is not within the scope of this article.
  
 
All these variables must be initialised, and read in from the form as shown:
 
All these variables must be initialised, and read in from the form as shown:
Line 166: Line 177:
 
     // These fields will make up the <styleURL> of the <Placemark>  
 
     // These fields will make up the <styleURL> of the <Placemark>  
 
     styleURLFields = document.getElementById('styleURLFields').value.split(fieldSep);     
 
     styleURLFields = document.getElementById('styleURLFields').value.split(fieldSep);     
 +
 
 +
  // Convert up to two \t into tabs
 +
  fieldSep = fieldSep.replace("\\t", "\t").replace("\\t", "\t");
 
  </code>
 
  </code>
  
 
=== Geocoding ===
 
=== Geocoding ===
A search manager is required to process the address of each element.  We need to wait for the searchManager to finish (by observing the'' state'' attribute) and then add the marker to the map if found.
+
A call to {{Icode|nokia.places.search.manager.geoCode()}}  is required to process the address of each element.  We need to wait for the function to finish (by adding  the {{Icode|onSearchComplete}} as a callback function) and then add the marker to the map if found. The methods for adding the marker and holding the state of the KML data are the same as in the [[Nokia Maps API - How to create a KML data file|How to create a KML data file]] example, and have been removed from the code snippet for clarity. It is important to notice that the '''next''' call to {{Icode|doNextGeocode()}} is made once the previous search has finished, hence each record will be processed sequentially.
The methods for adding the marker and holding the state of the KML data hare the same as in the [[Nokia Maps API - How to create a KML data file|How to create a KML data file]] example, and have been removed from the code snippet for clarity.It is important to notice that the '''next''' call to ''doNextGeocode()'' is made once the previous search has finished, hence each record will be processed sequentially.
+
 
<code javascript>
 
<code javascript>
 
// Search Manager taken directly from playground examples.
 
// Search Manager taken directly from playground examples.
  var searchManager = new nokia.maps.search.Manager();
+
var onSearchComplete = function (data, requestStatus) {    
  searchManager.addObserver("state", function (observedManager, key, value) {
+
      if (requestStatus == "ERROR") {
        // If the search  has finished we can process the result.
+
        // Try again with geocoding the same data, using alternate fields
        if (value == "finished") {
+
        // to define the address.
                // Geocode is successful so add a marker .. code removed for clarity
+
          lat.innerHTML= "Not Found: " + address;
                addMarker(markerData);
+
          addressingAttempt++;
                // Center on the new marker and start to process the next record.
+
          if (addressingAttempt ==  addressFields.length){
                map.setCenter(observedManager.locations[0].displayPosition);
+
            // There are no more fall back addressing options.
                addressingAttempt = 0;
+
            // Move on to the next record regardless.
                currentRecord++;
+
              addressingAttempt = 0;
              } else {
+
              currentRecord++;
                // Try again with geocoding the same data, using alternate address fields
+
          }
                // to define the address.
+
      } else {
                  lat.innerHTML= "Not Found: " + address;
+
     
                  addressingAttempt++;
+
        var markerData = new Object();
                  if (addressingAttempt ==  addressFields.length){
+
        // Since we have an address we can add the current data to the map
                    // There are no more fall back addressing options.
+
        // as the addressing data has been found to be valid.
                    // Move on to the next record regardless.
+
        markerData.coords = data.location.position;
                      addressingAttempt = 0;
+
         
                      currentRecord++;
+
       
                  }
+
        markerData.id = id;
              }
+
        markerData.title = map.objects.getLength() + 1;
              // Find the next address, either a new record or using new address fields.
+
        markerData.description =  description.trim(); 
              doNextGeoCode();
+
        markerData.name = name.trim(); 
                           
+
        markerData.address = address.trim(); 
} else if (value == "failed") {
+
        markerData.styleURL = styleURL.trim();           
    // Geocoding has failed.
+
        addMarker(markerData);
            lat.innerHTML = "The search request failed.";              
+
        // Center on the new marker and start to process the next record.
}
+
        map.setCenter(data.location.position);
+
        addressingAttempt = 0;
});
+
    currentRecord++;
</code>
+
    lat.innerHTML= "Found: " + address;
 +
      }
 +
      // Find the next address, either a new record or using new address fields.
 +
      doNextGeoCode();
 +
}</code>
  
''doNextGeoCode()'' kicks off the geocoding process, the current record is split into fields, and the address, description, name etc are calculated and held as global variables. The search manager is called after each request (which then chains back to ''doNextGeoCode()'' to process the next record. If no further addressing strategies can be tried, the record is not added - the code could be altered here to alert the user if necessary. Once we have completed all the records, the KML is generated using the ''saveMapObjects(map)'' method taken directly from the [[Nokia Maps API - How to create a KML data file|How to create a KML data file]] example.  
+
{{Icode|doNextGeoCode()}} kicks off the geocoding process, the current record is split into fields, and the address, description, name etc are calculated and held as global variables. The search manager is called after each request (which then chains back to {{Icode|doNextGeoCode()}} to process the next record. If no further addressing strategies can be tried, the record is not added - the code could be altered here to alert the user if necessary. Once we have completed all the records, the KML is generated using the {{Icode|saveMapObjects(map)}} method taken directly from the [[Nokia Maps API - How to create a KML data file|How to create a KML data file]] example.  
  
 
<code javascript>   
 
<code javascript>   
 
   // Obtains the Longitude and Latitude of the next record
 
   // Obtains the Longitude and Latitude of the next record
   // Based upon the data in the chosen fields of that recod.
+
   // Based upon the data in the chosen fields of that record.
 
   function doNextGeoCode(){
 
   function doNextGeoCode(){
 
 
Line 225: Line 242:
 
// Assuming we have an address to try, we should geocode it.
 
// Assuming we have an address to try, we should geocode it.
 
if (address != "" ){                             
 
if (address != "" ){                             
searchManager.geocode(address);
+
nokia.places.search.manager.geoCode({
 +
                searchTerm : address,
 +
    onComplete:  onSearchComplete
 +
    });
 
} else {
 
} else {
 
// Otherwise we need to try another addressing strategy.
 
// Otherwise we need to try another addressing strategy.
Line 261: Line 281:
  
 
=== Addressing Strategies ===
 
=== Addressing Strategies ===
The quality of the address data in an arbitrary data set  is by definition unknown, and may not match the geocode data held by Nokia. It will be necessary to attempt to mitigate this, by attempting to form a recognised address from different sections of each record. Several text boxes are displayed offering a variety of addressing strategies. These should hold the field separated header names used to build up the address. When the address is created using the ''getFieldsFromDefinition()'' method each field corresponding to a header element is added in turn. This means you can try various addressing strategies from the most specific to the least specific. For example using the fields defined in the example CSV file:
+
The quality of the address data in an arbitrary data set  is by definition unknown, and may not match the geocode data held by Nokia. It will be necessary to attempt to mitigate this, by attempting to form a recognised address from different sections of each record. Several text boxes are displayed offering a variety of addressing strategies. These should hold the field separated header names used to build up the address. When the address is created using the {{Icode|getFieldsFromDefinition()}} method each field corresponding to a header element is added in turn. This means you can try various addressing strategies from the most specific to the least specific. For example using the fields defined in the example CSV file:
 
<code>
 
<code>
 
  serial, name , unused,address_street, address_city, address_district, description
 
  serial, name , unused,address_street, address_city, address_district, description
Line 270: Line 290:
 
</code>
 
</code>
 
The following addressing strategy would be used:
 
The following addressing strategy would be used:
#  address_street,address_city,address_district
+
{{Icode|address_street,address_city,address_district}}
# address_city,address_district
+
# {{Icode|address_city,address_district}}
#  address_district
+
{{Icode|address_district}}
  
 
Note that each field must be separated by the standard field separator - in the case above a comma. If the geocoding service recognizes the full address, the marker is added, otherwise a more general strategy is used. It would also be possible to alter the order of the fields (e.g. put house number before or after street) or to add in hard coded field, since if an element of the addressing strategy does not correspond to a field in the header record it is added directly to the addressing strategy. For example, if you have a list of cities in Spain, the following  addressing strategy would avoid placing "Toledo" in Toledo, Ohio, USA:
 
Note that each field must be separated by the standard field separator - in the case above a comma. If the geocoding service recognizes the full address, the marker is added, otherwise a more general strategy is used. It would also be possible to alter the order of the fields (e.g. put house number before or after street) or to add in hard coded field, since if an element of the addressing strategy does not correspond to a field in the header record it is added directly to the addressing strategy. For example, if you have a list of cities in Spain, the following  addressing strategy would avoid placing "Toledo" in Toledo, Ohio, USA:
  
#  address_city,Spain
+
{{Icode|address_city,Spain}}
  
  
Line 313: Line 333:
 
</code>
 
</code>
  
''getFieldsFromDefinition()'' is also used to build up the data to be held in the other KML elements. Most of these elements will hold simple ASCII text. The <description> element is the only element which is capably of holding formatted HTML. As such it is possible to use the same "hard coded field" method in the description definition to add HTML directly to the <description>. For example to add an image to the <description> following description definition could be used:
+
{{Icode|getFieldsFromDefinition()}} is also used to build up the data to be held in the other KML elements. Most of these elements will hold simple ASCII text. The {{Icode|&lt;description&gt;}} element is the only element which is capably of holding formatted HTML. As such it is possible to use the same "hard coded field" method in the description definition to add HTML directly to the {{Icode|&lt;description&gt;}}. For example to add an image to the {{Icode|&lt;description&gt;}} following description definition could be used:
  
<img src="http://www.example/path_to.image/^'''IMAGE_FIELD'''^"/>^'''SOME_OTHER_DESC'''
+
{{Icode|&lt;img src&#61;"http://www.example/path_to.image/^'''IMAGE_FIELD'''^"/&gt;^'''SOME_OTHER_DESC'''}}
  
 
Where '''IMAGE_FIELD''' and '''SOME_OTHER_DESC''' are header definitions, the field separator is a circumflex ^ and the HTML is hard coded into the description definition.
 
Where '''IMAGE_FIELD''' and '''SOME_OTHER_DESC''' are header definitions, the field separator is a circumflex ^ and the HTML is hard coded into the description definition.
Line 321: Line 341:
 
==  Working Example - Right Move ==
 
==  Working Example - Right Move ==
  
The working example may be found in the ''geocode.html'' file in the attached  [[Media:AnyFormatToKMLExample.zip|Source Code]] . The initial field definitions are set up for the Right Move data format.
+
The working example may be found in the {{Icode|geocode.html}} file in the attached  [[Media:AnyFormatToKMLExample.zip|Source Code]] . The initial field definitions are set up for the Right Move data format.
The number of bedrooms has been chosen to  define the <styleUrl> used for each marker.
+
The number of bedrooms has been chosen to  define the {{Icode|&lt;styleUrl&gt;}} used for each marker.
  
 
[[File:EstateAgent.png]]
 
[[File:EstateAgent.png]]
Line 329: Line 349:
 
Once the data is generated, it may need to be edited using a [[Nokia Maps API - How to display KML file data on the map|KML Editor]]
 
Once the data is generated, it may need to be edited using a [[Nokia Maps API - How to display KML file data on the map|KML Editor]]
  
The  following <Style> - changing the colour of the marker  was manually added to the generated KML file after processing the file. The colour of the marker is different for one, two, three, four and five bedroom properties.
+
The  following {{Icode|&lt;Style&gt;}} - changing the colour of the marker  was manually added to the generated KML file after processing the file. The colour of the marker is different for one, two, three, four and five bedroom properties.
 
<code xml>
 
<code xml>
 
<Style id='1'><!-- i.e.  A style for properties with 1 bedroom-->
 
<Style id='1'><!-- i.e.  A style for properties with 1 bedroom-->
Line 342: Line 362:
 
</code>
 
</code>
  
The final result may be seen by loading the ''EstateAgents.kml'' file using the code from the [http://api.maps.nokia.com/2.1.1/playground/?example&#61;kmlfile Developer Playground] KML example. The file ''EstateAgentsKML.html'' is based upon this. It also includes right click routing for convenience.
+
The final result may be seen by loading the {{Icode|EstateAgents.kml}} file using the code from the [http://api.maps.nokia.com/2.1.1/playground/?example&#61;kmlfile Developer Playground] KML example. The file {{Icode|EstateAgentsKML.html}} is based upon this. It also includes right click routing for convenience.
  
 
{{Note| in order to load a KML file successfully, the generated KML file should be hosted on the same domain as the JavaScript or the results may be unpredictable. Some browsers will automatically prohibit cross-domain access.
 
{{Note| in order to load a KML file successfully, the generated KML file should be hosted on the same domain as the JavaScript or the results may be unpredictable. Some browsers will automatically prohibit cross-domain access.
For example, if you are hosting at  ''example.com'', the final line of the JavaScript to load the KML will need to be:  
+
For example, if you are hosting at  {{Icode|example.com}}, the final line of the JavaScript to load the KML will need to be:  
 
<code javascript>
 
<code javascript>
 
kml.parseKML("http://example.com/" + "generated_kml_data_file.kml")
 
kml.parseKML("http://example.com/" + "generated_kml_data_file.kml")
 
</code>
 
</code>
and both the KML loading HTML and the  generated_kml_data_file.kml should be placed on  ''http://example.com/''
+
and both the KML loading HTML and the  generated_kml_data_file.kml should be placed on  {{Icode|http://example.com/}}
 
}}
 
}}
  
=== Known Limitations ===
+
=== Encoding the extended character set ===
When loading mapping data via KML, the data to be displayed must be URL encoded, there are difficulties in parsing non-standard ASCII characters in any XML based format, especially when the data is not held within a '''CDATA''' section.(i.e.  for all elements except from the <description>). The tool does not ''escape()'' its data, since HTML is allowed within the Description field. Where necessary the data should be sanitized after KML generation, this could be done using '''find''' ... '''replace''' on the text. A few examples are shown below.
+
Under the strictest definition of  KML, the data in the file must not contain any characters which lie beyond the simple ascii character set (0-127). This presents problems for the following:
 +
* Any language which does not use the roman alphabet (e.g Russian, Chinese)
 +
* Any language which uses accents on its characters such as e.g German ('''&auml;''' '''&ouml;''' '''&uuml;''' '''&szlig;''') or Polish ( '''&#261;''' '''&#263;''' '''&#281;''' '''&#322;''' '''&#324;''' '''&#243;''' '''&#347;''' '''&#378;''' '''&#380;''')
 +
* Any situation which requires currrency symbols such as  '''&yen;'''  '''&euro;''' '''&pound;'''
 +
*  Also the use of the single '''&#39;''' and double quote marks'''"''' and the ampersand symbol '''&'''  have special meaning in KML.
  
 +
In order to create well formed KML, the data must be modified to use the unicode character codes or HTML codes for the extended character set, examples are given below:
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
! Example Data !! Escaped format
+
! Example Data !! Escaped format for &lt;description&gt; !! Escaped format for other fields.
 
|-
 
|-
| Sa'''ő''' Paulo || Sa'''&amp;otilde;''' Paulo
+
| S'''ã'''o Paulo || S'''&amp;atilde;'''o Paulo  || S'''&amp;amp;atilde;'''o Paulo
 
|-
 
|-
| K'''ö'''penicker Stra'''ß'''e|| K'''&amp;ouml;'''penicker Stra'''&amp;szlig;'''er
+
| '''&#21271;&#20140;''' || '''&amp;#21271;&amp;#20140;''' || '''&amp;amp;#21271;&amp;amp;#20140;'''
 
|-
 
|-
|  '''''' 500,000 || '''&amp;pound;''' 500,000
+
K'''ö'''penicker Stra'''ß'''e|| K'''&amp;ouml;'''penicker Stra'''&amp;szlig;'''e  || K'''&amp;amp;ouml;'''penicker Stra'''&amp;amp;szlig;'''e
 
|-
 
|-
| '''&''' || '''&amp;amp;'''
+
| ul. Stefana  Wyszy'''&#324;'''skiego 23 <br/>
 +
65-536 Zielona G'''ó'''ra
 +
||
 +
ul. Stefana Wyszy'''&amp;#324;'''skiego 23<br/>   
 +
65-536 Zielona G'''&amp;#243;'''ra
 +
||
 +
ul. Stefana Wyszy'''&amp;amp;#324;'''skiego 23<br/>
 +
65-536 Zielona G'''&amp;amp;#243;'''ra
 +
|-
 +
|  '''£''' 500,000 || '''&amp;pound;''' 500,000  || '''&amp;amp;pound;''' 500,000
 +
|-
 +
| '''&''' || '''&amp;amp;'''  || '''&amp;amp;amp;'''
 
|}
 
|}
  
HTML names for further examples may be found [http://www.ascii.cl/htmlcodes.htm here]
+
A library function for transforming extended characters into KML readable equivalents has been added and is called on the data of each element prior to creating the KML data set so that the overall KML file remains well formed. The data is singularly encoded within the CDATA section and doubly encoded outside of the CDATA section. In order to maintain the readability of the file  (and to reduce the size), the encoding only takes place when an extended character is found. This function could be replaced with a {{Icode|String.prototype}} but not all browsers will support it.
 +
 
 +
<code javascript>
 +
markerData.id = toUnicode("&amp;#",map.objects.get(i).$data.id);
 +
markerData.latitude = map.objects.get(i).coordinate.latitude;
 +
markerData.longitude = map.objects.get(i).coordinate.longitude;
 +
markerData.description =  toUnicode("&#", map.objects.get(i).$data.description);
 +
markerData.name = toUnicode("&amp;#",map.objects.get(i).$data.name);
 +
markerData.address = toUnicode("&amp;#",map.objects.get(i).$data.address);
 +
</code>
 +
<code javascript>
 +
function toUnicode (prefix, input){
 +
var output = "";
 +
var splitInput = input.split("");
 +
for (var i = 0; i < splitInput.length; i++){
 +
var currentChar = splitInput[i];
 +
// Encode any extended character plus &
 +
if (currentChar.charCodeAt()> 128 ||  currentChar.charCodeAt()== 38  || currentChar.charCodeAt()== 39 ) {
 +
output = output +  prefix + currentChar.charCodeAt() + ";";
 +
} else {
 +
output = output + currentChar;
 +
}
 +
}
 +
return output;
 +
}
 +
</code>
  
 
== Summary ==
 
== Summary ==
 
It should be possible to generate KML data for a map from an arbitrary data set, and once it is in a standard format it becomes an elementary exercise to  display  the data using standard techniques.
 
It should be possible to generate KML data for a map from an arbitrary data set, and once it is in a standard format it becomes an elementary exercise to  display  the data using standard techniques.

Revision as of 18:06, 3 January 2013

This article explains how to read address data from an arbitrary file format, and create a KML file for display on a map.

Article Metadata
Code ExampleTested with
Devices(s): Firefox , Internet Explorer, Google Chrome, Opera
Compatibility
Platform(s): Web Browser
Dependencies: Nokia Maps 2.2.3
Article
Keywords: Nokia Maps, JavaScript, KML
Created: jasfox (18 Jan 2012)
Last edited: jasfox (03 Jan 2013)

Contents

Introduction

Keyhole Markup Language (KML) is an XML notation for geographic applications. The advantages of using KML are numerous, and have been listed in a previous article. A typical enterprise may wish to add some markers representing addresses onto a map for their website, but without necessarily learning too much about geocoding or KML. It is likely that the address data they have is already held in a file or spreadsheet somewhere.

This example aims to take the pain out of creating a KML dataset. It aims to take any data format and attempt to locate addresses given by specified fields from the file. The addresses are then transformed into KML <Placemark> elements with associated <name> and <description> elements taken from other fields from the same record. The generated KML data can be inspected and edited using the editor example given here and then displayed using the code from the How to display KML data example. In summary, this article demonstrates a real world use of the geocoding service and is an example of making sequential asynchronous JavaScript calls to obtain longitude and latitude.

Defining the issue

For a typical data file, the first line in the file will be a header line which defines the fields in the records below. Each field will be separated from the next by some arbitrary separator character. For a CSV file for example, the separator character will be a comma (,) for other data formats it may be a space( ), a pipe (|) or some other character. At the end of the header line there will be some form of terminating character (typically a new line). Thereafter each subsequent line of data will hold a record of separated fields of data, with each field being associated to the definition of the field held in the header.

For example the following spreadsheet

serial name unused address_street address_city address_district description
13556 Millenium Stage Something 401 Bay Drive New York This is an example
3243 The Chambers Something else 53 Bothwell Street Watford Hertfordshire Some more text here
9954 4th Dimension More data 226 Myrtle Ave Toronto
8645 E.K.B.R. Even more data Invalidenstrasse 117 Berlin Hier gibts etwas

Comma Separated Variable Format

Comma Separated Variable (CSV) Format is a simple de facto standard for spreadsheet data, the table above would be represented as follows:

 serial, name , unused,address_street, address_city, address_district, description
13556, Millenium Stage, Something, 401 Bay Drive,New York, This is an example
3243, The Chambers, Something else, 53 Bothwell Street, Watford,Hertfordshire, Some more text here
9954, 4th Dimension, More data, 226 Myrtle Ave , Toronto,,
8645, E.K.B.R, Even more data, Invalidenstrasse 117, Berlin,, Hier gibts etwas

Tab Separated Variable Format

Tab Separated Variable (TSV) Format is similiar to CSV, with each colum separated by whitespace as shown below:

serial	 name 	 unused	address_street	 address_city	 address_district	 description
13556 Millenium Stage Something 401 Bay Drive New York This is an example
3243 The Chambers Something else 53 Bothwell Street Watford Hertfordshire Some more text here
9954 4th Dimension More data 226 Myrtle Ave Toronto
8645 E.K.B.R, Even more data Invalidenstrasse 117 Berlin Hier gibts etwas

Other proprietary formats

The Right Move Estate Agent Data Format is a proprietary format popular amongst estate agents in the United Kingdom. A simplified extract for the properties defined above would look something the data below. This has been chosen as an alternative to illustrate the problem.

#HEADER#
Version : 3
EOF : '^'
EOR : '|'
 
Property Count : 4
Generated Date : 19-May-2010 12:29
 
#DEFINITION#
AGENT_REF^NAME^SOME_OTHER_FIELD^ADDRESS_1^ADDRESS_2^ADDRESS_3^DESCRIPTION|
 
#DATA#
13556^Millenium Stage^ Something^401 Bay Drive^New York^This is an example|
 
3243^The Chambers^ Something else^ 53 Bothwell Street^Watford^Hertfordshire^Some more text here|
 
9954^4th Dimension^ More data^ 226 Myrtle Ave^Toronto^^|
 
8645^E.K.B.R^Even more data^Invalidenstrasse 117^Berlin^^Hier gibts etwas|
#END#

It can be seen that to parse data from an arbitrary file, the problem can be split into several parts:

  • Deciding where the header line starts - there may be no preamble, but potentially the data could have a prefix or some white space before it.
  • Deciding what the field and line terminators are
  • Deciding where the data starts - there may be no gaps between the header line and the data, but potentially there could be a some other extraneous information.
  • Deciding which fields are needed in the KML - Some columns of data will be irrelevant, empty or not used.
  • Deciding which fields form the address to geocode. Obviously the names of the field headers could differ as well.
  • Deciding what to do if the geocoding fails. The quality of the address data may be poor, or cover various addressing standards. Should the house number come before or after the street name for example?

GeoCode - a KML generator

The attached Source Code , reads an arbitrary data format from a text box, and discovers the locations specified on a map. The data is then displayed in KML format.


Initialisation

The following values need to be used for the formats described above:

CSV Tab Delimited Right Move
Start of Header Indicator blank blank #DEFINITION#\n
Start of Data Indicator \n \n #DATA#\n
Record Separator \n \n \n\n
Field Separator , \t ^

Note.pngNote: 

  • The character sequence \n may be used to indicate a new line
  • The character sequence \t may be used to indicate a tabbed white space

Each record will be transformed into a KML <Placemark>. Obviously the decision as to which fields to add to the KML will depend on the data provided. The following KML elements are supported:

  • ID Field: This translates to the <Placemark> id attribute, a unique indicator for each marker.
  • Style URL Field: This translates to the <styleUrl> within the <Placemark> - it can be used to change the appearance of the marker by addding an associated <Style>
  • Description Fields: This translates to the <description> within the <Placemark>, it is able to support HTML tags such as <b> or <h2>
  • Name Fields: This translates to the <name> within the <Placemark>, it must be plain text
  • Address: This will hold the address for which the geocode attempt was successful. See addressing attempts for details.


By default Nokia Maps will display the <name> and the <description> of the marker in an infobox when clicked. This may be styled by defining an associated <BalloonStyle>, defining a <Style> is not within the scope of this article.

All these variables must be initialised, and read in from the form as shown:

   var headerStart , dataStart, lineSep, fieldSep, addressFields,  descriptionFields,  nameFields ,  idFields, styleURLFields;
  var dataInput = document.getElementById('dataInput').value;
 
headerStart = document.getElementById('headerStart').value; //"#DEFINITION#\n" for Right Move.
headerStart = headerStart.replace("\\n", "\n").replace("\\n", "\n"); // Convert up to two \n into carriage returns
dataStart = document.getElementById('dataStart').value; // "#DATA#\n" for Right Move.
dataStart = dataStart.replace("\\n", "\n").replace("\\n", "\n"); // Convert up to two \n into carriage returns
lineSep = document.getElementById('lineSep').value;// ; "|\n\n" for Right Move.
lineSep = lineSep.replace("\\n", "\n").replace("\\n", "\n"); // Convert up to two \n into carriage returns
 
fieldSep = document.getElementById('fieldSep').value;
addressFields = new Array();
 
// Each of these address strategies will be tried in turn.
addressFields.push(document.getElementById('addressAttempt1').value.split(fieldSep));
addressFields.push(document.getElementById('addressAttempt2').value.split(fieldSep));
addressFields.push(document.getElementById('addressAttempt3').value.split(fieldSep));
addressFields.push(document.getElementById('addressAttempt4').value.split(fieldSep));
 
// Any fields added here will be appended to the <address> element.
descriptionFields = document.getElementById('descriptionFields').value.split(fieldSep);
// Any fields added here will be appended to the <name> element.
nameFields = document.getElementById('nameFields').value.split(fieldSep);
// This field will be the id of the <Placemark> element.
idFields = document.getElementById('idField').value.split(fieldSep);
// These fields will make up the <styleURL> of the <Placemark>
styleURLFields = document.getElementById('styleURLFields').value.split(fieldSep);
 
// Convert up to two \t into tabs
fieldSep = fieldSep.replace("\\t", "\t").replace("\\t", "\t");

Geocoding

A call to nokia.places.search.manager.geoCode() is required to process the address of each element. We need to wait for the function to finish (by adding the onSearchComplete as a callback function) and then add the marker to the map if found. The methods for adding the marker and holding the state of the KML data are the same as in the How to create a KML data file example, and have been removed from the code snippet for clarity. It is important to notice that the next call to doNextGeocode() is made once the previous search has finished, hence each record will be processed sequentially.

// Search Manager taken directly from playground examples.
var onSearchComplete = function (data, requestStatus) {
if (requestStatus == "ERROR") {
// Try again with geocoding the same data, using alternate fields
// to define the address.
lat.innerHTML= "Not Found: " + address;
addressingAttempt++;
if (addressingAttempt == addressFields.length){
// There are no more fall back addressing options.
// Move on to the next record regardless.
addressingAttempt = 0;
currentRecord++;
}
} else {
 
var markerData = new Object();
// Since we have an address we can add the current data to the map
// as the addressing data has been found to be valid.
markerData.coords = data.location.position;
 
 
markerData.id = id;
markerData.title = map.objects.getLength() + 1;
markerData.description = description.trim();
markerData.name = name.trim();
markerData.address = address.trim();
markerData.styleURL = styleURL.trim();
addMarker(markerData);
// Center on the new marker and start to process the next record.
map.setCenter(data.location.position);
addressingAttempt = 0;
currentRecord++;
lat.innerHTML= "Found: " + address;
}
// Find the next address, either a new record or using new address fields.
doNextGeoCode();
}

doNextGeoCode() kicks off the geocoding process, the current record is split into fields, and the address, description, name etc are calculated and held as global variables. The search manager is called after each request (which then chains back to doNextGeoCode() to process the next record. If no further addressing strategies can be tried, the record is not added - the code could be altered here to alert the user if necessary. Once we have completed all the records, the KML is generated using the saveMapObjects(map) method taken directly from the How to create a KML data file example.

  // Obtains the Longitude and Latitude of the next record
// Based upon the data in the chosen fields of that record.
function doNextGeoCode(){
 
if (currentRecord < data.length){
 
var dataRecord = data[currentRecord].splitCSV(fieldSep);
 
address = getFieldsFromDefinition(addressFields[addressingAttempt], headers, dataRecord );
description = getFieldsFromDefinition(descriptionFields, headers, dataRecord );
name = getFieldsFromDefinition(nameFields, headers, dataRecord );
id = getFieldsFromDefinition(idFields, headers, dataRecord );
styleURL = getFieldsFromDefinition(styleURLFields, headers, dataRecord);
 
// Assuming we have an address to try, we should geocode it.
if (address != "" ){
nokia.places.search.manager.geoCode({
searchTerm : address,
onComplete: onSearchComplete
});
} else {
// Otherwise we need to try another addressing strategy.
addressingAttempt++;
if (addressingAttempt == addressFields.length){
// Since we have run out of addressing strategies,
// try the next record.
addressingAttempt = 0;
currentRecord++;
}
doNextGeoCode();
}
} else {
// We can generate the KML
saveMapObjects(map)
}

The splitting of each data record into fields is achieved by a standard library for splitting up texts.

String.prototype.splitCSV = function(sep) {
for (var foo = this.split(sep = sep || ","), x = foo.length - 1, tl; x >= 0; x--) {
if (foo[x].replace(/"\s+$/, '"').charAt(foo[x].length - 1) == '"') {
if ((tl = foo[x].replace(/^\s+"/, '"')).length > 1 && tl.charAt(0) == '"') {
foo[x] = foo[x].replace(/^\s*"|"\s*$/g, '').replace(/""/g, '"');
} else if (x) {
foo.splice(x - 1, 2, [foo[x - 1], foo[x]].join(sep));
} else foo = foo.shift().split(sep).concat(foo);
} else foo[x].replace(/""/g, '"');
}
return foo;
};

Addressing Strategies

The quality of the address data in an arbitrary data set is by definition unknown, and may not match the geocode data held by Nokia. It will be necessary to attempt to mitigate this, by attempting to form a recognised address from different sections of each record. Several text boxes are displayed offering a variety of addressing strategies. These should hold the field separated header names used to build up the address. When the address is created using the getFieldsFromDefinition() method each field corresponding to a header element is added in turn. This means you can try various addressing strategies from the most specific to the least specific. For example using the fields defined in the example CSV file:

 serial, name , unused,address_street, address_city, address_district, description
13556, Millenium Stage, Something, 401 Bay Drive,New York, This is an example
3243, The Chambers, Something else, 53 Bothwell Street, Watford,Hertfordshire, Some more text here
9954, 4th Dimension, More data, 226 Myrtle Ave , Toronto,,
8645, E.K.B.R, Even more data, Invalidenstrasse 117, Berlin,, Hier gibts etwas

The following addressing strategy would be used:

  1. address_street,address_city,address_district
  2. address_city,address_district
  3. address_district

Note that each field must be separated by the standard field separator - in the case above a comma. If the geocoding service recognizes the full address, the marker is added, otherwise a more general strategy is used. It would also be possible to alter the order of the fields (e.g. put house number before or after street) or to add in hard coded field, since if an element of the addressing strategy does not correspond to a field in the header record it is added directly to the addressing strategy. For example, if you have a list of cities in Spain, the following addressing strategy would avoid placing "Toledo" in Toledo, Ohio, USA:

  1. address_city,Spain


addressFields = new Array();
 
// Each of these address strategies will be tried in turn.
addressFields.push(document.getElementById('addressAttempt1').value.split(fieldSep));
addressFields.push(document.getElementById('addressAttempt2').value.split(fieldSep));
addressFields.push(document.getElementById('addressAttempt3').value.split(fieldSep));
addressFields.push(document.getElementById('addressAttempt4').value.split(fieldSep));
function getFieldsFromDefinition(definition, headerFields, dataRecord ){
var result = "";
for (var defFieldCount = 0; defFieldCount < definition.length ; defFieldCount++){
for (var headerFieldCount = 0; headerFieldCount < headerFields.length ; headerFieldCount++){
 
if (headerFields[headerFieldCount] == definition [defFieldCount]){
if (headerFieldCount <= dataRecord.length){
result = result + dataRecord[headerFieldCount] + " ";
}
break;
}
if (headerFieldCount == headerFields.length - 1){
if ( headerFieldCount <= dataRecord.length){
result = result + definition [defFieldCount];
}
}
}
 
}
return result.trim();
}

getFieldsFromDefinition() is also used to build up the data to be held in the other KML elements. Most of these elements will hold simple ASCII text. The <description> element is the only element which is capably of holding formatted HTML. As such it is possible to use the same "hard coded field" method in the description definition to add HTML directly to the <description>. For example to add an image to the <description> following description definition could be used:

<img src="http://www.example/path_to.image/^IMAGE_FIELD^"/>^SOME_OTHER_DESC

Where IMAGE_FIELD and SOME_OTHER_DESC are header definitions, the field separator is a circumflex ^ and the HTML is hard coded into the description definition.

Working Example - Right Move

The working example may be found in the geocode.html file in the attached Source Code . The initial field definitions are set up for the Right Move data format. The number of bedrooms has been chosen to define the <styleUrl> used for each marker.

EstateAgent.png


Once the data is generated, it may need to be edited using a KML Editor

The following <Style> - changing the colour of the marker was manually added to the generated KML file after processing the file. The colour of the marker is different for one, two, three, four and five bedroom properties.

<Style id='1'><!-- i.e.  A style for properties with 1 bedroom-->
<IconStyle>
<Icon>
<href>http://www.developer.nokia.com/Community/Wiki/images/0/02/000000.png</href>
</Icon>
</IconStyle>
<BalloonStyle><text><![CDATA[<h2>$[name]</h2><p>$[description]</p>]]></text></BalloonStyle>
</Style>
... etc...

The final result may be seen by loading the EstateAgents.kml file using the code from the Developer Playground KML example. The file EstateAgentsKML.html is based upon this. It also includes right click routing for convenience.

Note.pngNote: in order to load a KML file successfully, the generated KML file should be hosted on the same domain as the JavaScript or the results may be unpredictable. Some browsers will automatically prohibit cross-domain access.

For example, if you are hosting at example.com, the final line of the JavaScript to load the KML will need to be:

kml.parseKML("http://example.com/" + "generated_kml_data_file.kml")
and both the KML loading HTML and the generated_kml_data_file.kml should be placed on http://example.com/

Encoding the extended character set

Under the strictest definition of KML, the data in the file must not contain any characters which lie beyond the simple ascii character set (0-127). This presents problems for the following:

  • Any language which does not use the roman alphabet (e.g Russian, Chinese)
  • Any language which uses accents on its characters such as e.g German (ä ö ü ß) or Polish ( ą ć ę ł ń ó ś ź ż)
  • Any situation which requires currrency symbols such as ¥ £
  • Also the use of the single ' and double quote marks" and the ampersand symbol & have special meaning in KML.

In order to create well formed KML, the data must be modified to use the unicode character codes or HTML codes for the extended character set, examples are given below:

Example Data Escaped format for <description> Escaped format for other fields.
São Paulo S&atilde;o Paulo S&amp;atilde;o Paulo
北京 &#21271;&#20140; &amp;#21271;&amp;#20140;
Köpenicker Straße K&ouml;penicker Stra&szlig;e K&amp;ouml;penicker Stra&amp;szlig;e
ul. Stefana Wyszyńskiego 23

65-536 Zielona Góra

ul. Stefana Wyszy&#324;skiego 23
65-536 Zielona G&#243;ra

ul. Stefana Wyszy&amp;#324;skiego 23
65-536 Zielona G&amp;#243;ra

£ 500,000 &pound; 500,000 &amp;pound; 500,000
& &amp; &amp;amp;

A library function for transforming extended characters into KML readable equivalents has been added and is called on the data of each element prior to creating the KML data set so that the overall KML file remains well formed. The data is singularly encoded within the CDATA section and doubly encoded outside of the CDATA section. In order to maintain the readability of the file (and to reduce the size), the encoding only takes place when an extended character is found. This function could be replaced with a String.prototype but not all browsers will support it.

markerData.id = toUnicode("&amp;#",map.objects.get(i).$data.id);
markerData.latitude = map.objects.get(i).coordinate.latitude;
markerData.longitude = map.objects.get(i).coordinate.longitude;
markerData.description = toUnicode("&#", map.objects.get(i).$data.description);
markerData.name = toUnicode("&amp;#",map.objects.get(i).$data.name);
markerData.address = toUnicode("&amp;#",map.objects.get(i).$data.address);
function toUnicode (prefix, input){
var output = "";
var splitInput = input.split("");
for (var i = 0; i < splitInput.length; i++){
var currentChar = splitInput[i];
// Encode any extended character plus &
if (currentChar.charCodeAt()> 128 || currentChar.charCodeAt()== 38 || currentChar.charCodeAt()== 39 ) {
output = output + prefix + currentChar.charCodeAt() + ";";
} else {
output = output + currentChar;
}
}
return output;
}

Summary

It should be possible to generate KML data for a map from an arbitrary data set, and once it is in a standard format it becomes an elementary exercise to display the data using standard techniques.

227 page views in the last 30 days.