Accessing and Mapping Geospatial Data in ArcGIS Online

A vast array of geospatial data is available on the web, most notably from open data portals provided by agencies at all levels of government.

Because of the diversity of ways that geospatial data can be distributed, some understanding of the architecture of web maps and the techniques for processing geospatial data is needed to be able to effectively use that data in ArcGIS Online.

This tutorial will cover some basic techniques on how to acquire data from online sources for mapping in the ArcGIS Online Map Viewer.

ArcGIS Online Map Architecture

When working with computers, an architecture is "the manner in which the components of a computer or computer system are organized and integrated" (Merriam-Webster 2022).

Since most GIS endeavors involve some measure of collaboration, most architectures separate components in the client-server model. (Techopedia 2020).

The ArcGIS Online Map Viewer is a web app that runs in a client after being downloaded from a server. The app then accesses additional services from the server to render the map in a browser window.

  1. The user enters the URL or clicks on a button to request a map from the server.
  2. The server responds with the map app and a map layout (JSON data) that contains information about the content of the map and what services should be used to access data for mapping.
  3. The map app makes additional requests from the server(s) to get data for mapping.
  4. The map app renders the map data in the browser.
  5. As the user interacts with the map app, the map app may request additional data from the servers.
Figure
ArcGIS Online Architecture

The separation of the service (the data) from the map layout (the visualization) has the following advantages.

Open vs. Proprietary Data

Open data is "data that can be freely used, re-used and redistributed by anyone - subject only, at most, to the requirement to attribute and sharealike" (Open Knowledge Foundation 2018).

Open data is free in two different ways:

A fundamental value of open data is interoperability. Interoperability is "the ability of diverse systems and organizations to work together" and intermix different datasets (Open Knowledge Foundation 2018).

The concept of open is often contrasted with proprietary. Proprietary data is data where access is controlled, usually to subscribers who pay to license use of that data.

Figure
Open vs. Proprietary

All data requires effort (and, therefore, money) to capture and maintain. So while data may be available to you at no direct cost, there are still costs associated with free data. Ultimately somebody has to pay to create, maintain and distribute that data.

Despite the ideal of perfectly interoperable systems built on open data, the current reality is that free and open geospatial data is commonly made available across a variety of different types of website in a variety of different formats.

Unlike proprietary data which is carefully processed for specific uses by paying customers, open data organization varies widely and open data often needs tedious cleanup. Although estimates vary and are primarily anecdotal, a common view is that data scientists spend 80% of their time finding, cleaning, and reorganizing data, and only 20% of their time doing meaningful data analysis (Ruiz 2017).

Open Data Portals

Many organizations, notably government agencies, make data available to the general public through open data portals, which are websites that permit easy navigation through available data sets, and provide capabilities for examining, visualizing, and/or downloading those data sets.

Open data portals use content management systems (CMS) to organize and provide access to their different data sets.

ArcGIS Hub is a proprietary CMS which is tightly integrated with ArcGIS Online and ArcGIS Server software and which is commonly used by US government agencies.

Figure
ArcGIS Hub (Spokane County GIS Data Catalog)

A common open source CMS used by government agencies for disseminating geospatial and non-geospatial data is Socrata.

Figure
Socrata (City of Chicago open data portal)

Large organizations commonly maintain their own custom CMS. For example, the US Census Bureau CMS is designed specifically provide easy exploration and access of the wide variety of different social and economic data maintained by the agency.

Figure
US Census Bureau open data portal

Mapping Data from Services

Feature Services

Feature services are streams of vector data (points, lines, and polygons) that can be accessed by GIS software directly through application programmer interfaces (APIs).

Feature services are accessed through REST endpoints, which are named based on the type of communications protocol they use (representational state transfer). A REST endpoint is a URL to a service on the server that includes subfolders and parameters needed to identify a specific service.

This is an example REST endpoint URL for the US Energy Information Administration's power plants feature service:

https://services7.arcgis.com/FGr1D95XCGALKXqM/arcgis/rest/services/Power_Plants_Testing/FeatureServer/0/

When using a feature service, the data is downloaded on demand as the user navigates around the map. The mapping app sends a request to the feature service, and the service responds with the requested data.

Figure
Open feature service architecture

Mapping with Open Feature Services

Open feature services are feature services provided to the public free of charge. Open feature services provide significant advantages over embedding the data as part of the map:

With this flexibility comes some disadvantages:

An example of a collection of open feature services is the US Energy Information Administration's U.S. Energy Atlas. This video demonstrates how to create an ArcGIS Online map using an open feature service.

  1. Go to the website containing a link to the feature service (US Energy Atlas, Power Plants).
  2. Find the web page with the feature service URL (I want to use this, View API Resources, GeoService).
  3. Create a new map in ArcGIS Online.
  4. Add layer and Add layer from URL using the feature service URL. If the URL has a query string at the end, remove it.
  5. Save the map under a meaningful name (Minn 2024 Power Plants).
  6. Adjust Share map as needed.
  7. Copy the URL from the location bar to share the map.
Creating an ArcGIS Online map using a national map feature service

Mapping with Proprietary Feature Services

Proprietary feature services are provided by organizations that restrict access to those services, usually to subscribers who pay to license use of that data.

Perhaps the most notable example of a proprietary feature service in the ESRI world is ESRI's Living Atlas of the World, which is a collection of geospatial data feature services that can be accessed in ArcGIS Pro or in ArcGIS Online. Some of the services in the Living Atlas are freely accessible to the general public, while other are proprietary and accessible only to licensed users.

Figure
Living Atlas map architecture

In addition to the general advantages listed above for open feature services, proprietary services like the Living Atlas contain a wide variety of useful proprietary data that is not available from open data sources.

However, with those advantages come some disadvantages:

To create a map using a layer from the Living Atlas:

  1. Start a new Map in ArcGIS Online.
  2. Click Add, Browse layers, and Living Atlas.
  3. Search for the desired data (median household income).
  4. If needed, click the title to view item details. The type of service will be listed at the top along with a description of the contents. Feature services are listed as Feature Service or Feature Layer.
  5. On the ellipsis (...) beside the layer, select Show Properties and change the Blending to Multiply so you can see the underlying base map through the features.
  6. Save the map under a meaningful name (Minn 2023 Income).
  7. Adjust Share map as needed (Everyone).
  8. Copy the URL from the location bar to share the map.
Creating an ArcGIS Online map using a layer from ESRI's Living Atlas

Mapping with Tile Services

A tile service provides map images divided into 256 x 256 pixel map tiles that the browser or mobile map app stitches together into a complete map image for the viewer. This technique for delivering web maps was first developed for Google Maps in 2005 and ultimately became a ubiquitous technique for web mapping (Forrest 2021).

With a tiled web map, the map app client makes a request for tiles that cover a specific geographic are at a given level of detail (zoom level) and the server responds with the tiles needed to cover that area. Tiles are pre-rendered and stored in a cache. The map app only needs to request new tiles as the user scrolls to new areas or zooms to new levels of detail.

Tiled web maps (also called slippy maps) load quickly and allow facile user interaction. The downside of tiled web maps is that the developer is dependent on the design choices of the tile provider and has no control over symbology or level of detail. Also, map tiles are designed for use on web maps and will be fuzzy if printed to higher-resolution paper or posters.

ArcGIS Online uses map tiles for most of their base maps and many of the layers in the Living Atlas.

Users can publish tile services from ArcGIS Pro.

Figure
Tile service + custom feature service architecture
Figure
Map tiles loading in a Google Map

Tile services are commonly used for data that changes infrequently, for base maps that have complex symbology, or for raster data that does not have discrete vector features. In the Living Atlas, you can identify whether a layer is a tile service by viewing the item information and looking for Imagery Layer or Raster Layer.

This video demonstrates adding the USA NAIP Imagery: NDVI imagery layer from the Living Atlas. Note that when the map is shared, a warning is issued that this is a proprietary layer that will be visible only to users who are logged into an ArcGIS Online account.

Identifying and adding a tile service to an ArcGIS Online map

Custom Feature Services

Publishing a feature service is the process of making data available as a feature service. The publication process can involve uploading data, setting symbology, defining interactive capabilities (like pop-ups), and restricting access with permissions.

ESRI's cloud environment, ArcGIS Online, provides users with the capability to publish their own data as feature services. The data is kept on the same server (or set of servers) and within the same organization as the map. Feature services can be published through the ArcGIS Online web app or from the ArcGIS Pro desktop software.

Most of the examples given below involve creating custom ArcGIS Online feature services, and then using those feature services to create web maps.

Figure
Custom feature service architecture

The advantages of custom feature services over proprietary or open feature services include:

However, with freedom comes responsibility:

Geospatial Data File Formats

If you want to make an ArcGIS Online map of open data, and either that data is not available through a feature service, or you do not want to use that feature service, you will need to download the data and create your own feature service.

The easiest data to work with in GIS is data that is provided in a format specifically designed for geospatial data.

Shapefiles

The shapefile (PDF) is a proprietary ESRI format that is supported by a wide variety of software from other vendors, and as such has become, probably, the most common format for distributing geospatial data through open data portals.

While the age of the shapefile format is reflected in its numerous limitations (most notably a column name length limit of 10 characters), this format is supported by a wide variety of GIS software and is a safe format that is still commonly used for distributing open geospatial data by government agencies.

The term shapefile is a misnomer since a shapefile is actually a collection of at least three (and usually more) separate files that store the locational data, the characteristics associated with those locations, and other information about the data. When copying a shapefile, it is critical that you include all of these different files, or the file will either not open or be missing important components.

Some common files associated with a shapefile include (listed by the file extension):

For convenience, the separate files in a shapefile are usually compressed into a single .zip archive file for distribution on websites and servers.

Figure
Multiple physical files associated with a single shapefile

Creating a Map from a Shapefile

.zip files of shapefiles can be downloaded from data portals and uploaded to ArcGIS Online to create custom feature services that can then be imported into maps.

For example, the Chicago Open Data Portal uses the Socrata CMS, and geospatial data sets usually have an export option for shapefiles.

  1. Acquire the data: Download the zipped shapefile from the portal (Park Boundaries).
  2. Store the data: Create the feature service.
  3. Communicate: Create the map.
Creating a feature service from a zipped shapefile downloaded from a Socrata data portal

ArcGIS Hub Shapefiles

Another content management system commonly used by city governments for their open data portals is ArcGIS Hub.

Because ArcGIS Hub is implemented with ArcGIS Enterprise server software, ArcGIS Hub sites will often be configured to give you the option to use a feature service or download a data file (usually a shapefile).

For this example, we use a shapefile from the Rochester (NY) Open Data Portal for abandoned buildings slated for demolition because of safety or health threats (Demolition Case Parcels: Live).

  1. Acquire the data: Download the zipped shapefile from the portal.
  2. Store the data: Create the feature service.
  3. Communicate: Create the map.
Creating a feature service from a zipped shapefile downloaded from an ArcGIS Hub data portal

Metadata

Metadata is data about your data, such as a description of what the data set contains, where the data originally came from, when the data was captured and/or published, and who the person and/or agency is who published the data.

Metadata is important so that users of your data can know whether the data is trustworthy and know how old the data is.

Because metadata is not a directly integrated part of the processing of geospatial data, it is commonly neglected, which leave data sets that are useless or untrustworthy because there is no metadata indicating where the data came from or how old it is.

When you create a feature service in ArcGIS Online, you are often prompted at creation time for metadata. If you need to update feature service metadata or if you run a tool that assigns nondescriptive metadata, you can view the item information page for your feature service and edit fields as needed.

This video demonstrates changing the title, summary, description, and credits information on a feature layer item information page.

Updating metadata for an ArcGIS Online feature service

Other Common Geospatial Data Formats

Although the shapefile is the most reliable format to use when you have a choice, geospatial data files are made available in other formats that may be useful depending on your application.

Tabular Data

Some data portals provide data that has a mappable geospatial component, but only provide it in table form.

The comma-separated variable (CSV) format is a text format that arranges data in rows, with column cells separated by commas. CSV files can be thought of as spreadsheets, and they are commonly edited using spreadsheet software like Microsoft Excel, but CSV files do not preserve formatting information.

CSV files can be used for geospatial data with columns of latitude and longitude associated on each row with specific attributes at those latitudes and longitudes. Most CSV files can only represent points, so for lines (like roads) and areas (like neighborhoods or census tracts) you need to save data in a specialized geospatial data file format like the shapefile.

Figure
A CSV file with latitudes and longitudes opened for editing in Excel

Mapping CSV Files with Latitudes and Longitudes

Point data in data portals like Socrata is sometimes made available as tables with latitudes and longitudes rather than in geospatial data formats like shapefiles.

Figure
Creating a custom feature service from a table with latitudes and longitudes

For this example, we create a feature class of locations with radiation producing equipment in New York City from the NYC Open Data Socrata portal.

  1. Acquire the data: Download the CSV for Excel file from the portal.
  2. Store the data: Create the feature service.
  3. Communicate: Create the map.
Creating a feature service from a point CSV file downloaded from a Socrata data portal

Mapping CSV Files by Geocoding

Locations in a CSV file can also be specified with location names like street addresses, city names, country names, etc. The process of converting from location names to latitude/longitude is called geocoding. Because place names are often inexact, geocoding always involves some level of uncertainty, and geocoded points will often be misplaced and require cleaning if the spatial accuracy of your data is important.

When your table contains place names, you can map the table as a bubble map (graduated symbols) by geocoding the place names into latitudes and longitudes. Geocoding is an imperfect process and if you have something like a country code that can be joined more accurately, that may be a better option.

Figure
Geocoding

For example, the Food and Agriculture Organization of the United Nations (FAO) collects a vast array of open country-level agricultural data and makes it available to the public through their FAOSTAT web portal.

This example uses FAO data for almond production by country. While almonds are delicious and nutritious (THC School of Public Health 2023) growing almonds is very water-intensive (Fulton, Norton, and Shilling 2019), so knowing where almonds are grown can point us to areas where agriculture may have a detrimental water footprint.

The only geospatial components the FAO tables contain are country names and FAO country codes, so one approach to bubble mapping is to geocode by country name to create points that can be used for bubble maps.

  1. Acquire the data: Download the table and open it in a spreadsheet program like Excel.
  2. Process the data:
  3. Store the data: Create the feature service.
  4. Communicate: Create the map.
Creating a feature service from a geocoded table

Mapping CSV Choropleths Using Attribute Joins

Geocoding only converts place names to points, so if you want to create a choropleth with tabular data, you will need to join it to a layer of polygons based on a common attribute.

A join is a common database operation where two data sets are connected to form a single data set. An attribute join connects two datasets based on common key values.

Figure
Attribute joins

One challenge is that attribute joins need to have a variables in both layers that match exactly. Variations in country names and abbreviations can cause gaps in the data because of failure to match and if you have standardized codes like ISO3 country codes, those are preferred.

Figure
Creating a custom feature service with an attribute join
  1. Acquire the data: Download the table and open it in a spreadsheet program like Excel. For this example we will use annual primary energy consumption for EU countries.
  2. Process the data:
  3. Store the data: Create an ArcGIS Online table.
  4. Process the data: Join the data table and polygons to create a new feature service.
  5. Communicate: Create the map.
Creating a feature service from a joined table

Mapping HTML Tables

Data is sometimes made available on websites as HTML tables on web pages. As with the tabular data above, it is possible to copy that data into CSV files and geocode it for mapping.

For example, Wikipedia pages commonly contain tables of data with place names that you can import into ArcGIS Pro. However, note that Wikipedia is an online encyclopedia and the data in Wikipedia usually comes from some other original source. Before using data copied from Wikipedia, you may want to look at the references for the original data source and use that instead so you can be assured of the most current and accurate data.

This example uses data for the notorious 1876 US Presidential Election, a closely-contested contest which was settled with the Compromise of 1877 that ended post-civil-war reconstruction and began the Jim Crow era.

  1. Process the data:
  2. Store the data: Create the feature service.
  3. Communicate: Create the map
Creating a feature service from an geocoded HTML table

Mapping Manually Entered Data

In some situations where governments or organizations either do not have the resources or the ideological inclination to support open data portals, data may only be available as tables in Portable Document Format (PDF) files. Even in places with open data portals, some agencies may make their data only available as PDF files, especially historic documents.

While tables can sometimes be copied directly from PDF files into a CSV file for import into GIS, PDF is a format designed for print and the data must either be copied with a significant amount of clean up, or in the worst cases may have to be entirely entered by hand. However, if you need the data and you can't find another electronic source, you may not have a choice.

For example, the US Census Bureau has published a variety of statistical abstracts since the late 19th century that summarize data about the US. The older documents are scanned into PDF files. This video shows manual entry of data about the number of miles of railroad in operation in 1877 by state, which is then geocoded for mapping in ArcGIS Online.

Creating a feature service from a geocoded manually-entered table