Geospatial Data Services in ArcGIS Pro

If you are working alone on a simple research task, keeping everything in a project on your desktop computer may be adequate. However, if you are collaborating with other people in an organization or want to make your data available to the general public, you need a system architecture designed for serving data.

This tutorial will give a basic overview of how ArcGIS Pro is integrated with services from ArcGIS Online, ArcGIS Enterprise, and other online services.

Services

The Client-Server Model

The client-server model of system architecture is fundamental to contemporary internet-based communication.

A challenge with using services is that they require a reliable internet connect to access data on the servers. While this is generally not a major issue in urban areas in the developed world, this can necessitate designing systems that can work in a standalone mode when operating in rural areas or areas in the Global South with limited connectivity.

The client-server model on the Internet

Servers

Server computers are powerful computers with large numbers of CPU cores, RAM, and storage compared to desktop computers.

A virtual server is a server that shares a single physical hardware server with other virtual servers. Server virtualization uses hardware efficiently by enabling smaller virtual machines to be provisioned and removed quickly and without the difficulty and expense of having to purchase, install, and maintain separate physical hardware for each server.

Racks of servers (Slovinsky 2022 via Wikimedia)

The Cloud

The cloud is a collection of servers in one or more data centers that offer a variety of services to clients via the internet. Examples of cloud services include Microsoft 365, Google's suite of apps, and ArcGIS Online.

While the cloud looks to a client like a single server providing different services, in reality those services can be provided by different virtual servers in server farms located around the world to improve speed and provide backup when server hardware fails.

Cloud architecture

Local vs. Server

When working with a desktop GIS program like ArcGIS Pro, you have a choice whether to keep your data locally on the desktop computer or keep the data on a server. ArcGIS Pro is tightly integrated with ArcGIS Online, making it possible to easily store your data in the cloud. But there are often times when it is more appropriate to keep your data on your desktop machine.

Local Machine Server
Working alone Working with a group or sharing data with the general public
Working only on one desktop machine Working with data on multiple different machines and / or on shared devices (like remote desktops or computer labs)
Temporarily exploring new data that you are not sure you will keep Working with data that you will be using in the future
Heavy use of high volumes of data in a desktop program (such as remote sensing analysis) Light use of a limited volume of data.
Traditional "small" GIS data "Big" data analysis requiring high-performance computing resources
Personal data that only you need to see Data that will be shared within an organization and/or with the general public
Unimportant data that can easily be replaced if something bad happens to your machine Mission-critical data that needs to be protected and backed-up by IT professionals

Uniform Resource Locators (URLs)

Services on the internet are accessed through uniform resource locators (URLs). The structure of a URL is similar to a file path, although it contains additional components.

Parts of a URL

Protocol Name

A URL starts with a protocol name or scheme.

Fully Qualified Domain Name

A fully-qualified domain name (FQDN) identifies a server.

In URLs, FQDNs follow the protocol name and consist of three parts:

Subdomains

Complex organizations often attach different subdomains to the front of the FQDN to represent different departments or services within an organization represented by the top-level domain.

The www subdomain has historically been used to specify a web server for internet pages within a domain, but www is commonly omitted in contemporary web server configurations.

The following are examples of subdomain names within the illinois.edu domain.

https://dres.illinois.edu/

https://housing.illinois.edu/

https://www.admissions.illinois.edu/

Subfolders and Resource Names

Optionally, the FQDN in a URL can be followed by subfolder names and/or resource names.

Query Strings

When URLs point to services or dynamic files created from database content, the URLs often contain additional request information with query strings.

A query string is an optional portion of a URL following a question mark (?) that is used to pass additional information to the server.

Parameters are separate chunks of information in a query string that are separated by ampersands (&).

In this case, the query string specifies an identification number for the desired web map.

https://univofillinois.maps.arcgis.com/apps/mapviewer/index.html?webmap=c71a6e7ac2c643d9a25d87de2f07f182

ArcGIS Servers

ArcGIS Online

ArcGIS Online is ESRI's cloud environment that provides a suite of services and web apps for processing and sharing geospatial data.

An example ArcGIS Online content page

Organizations pay for ArcGIS Online with a negotiated license subscription charge, plus additional charges for credits that are used when services are used. Users are given credit quotas that protect organization from excessive charges if a user intentionally or unintentionally incurs credit charges for unexpectedly high use of ArcGIS Online services.

ArcGIS Online credit utilization report

Web maps hosted in ArcGIS Online often draw on multiple services, such as a tile service for the base map and a feature service for points, lines, or polygons drawn over the base map.

Serving a web map with multiple services from ArcGIS Online

ArcGIS Online is tightly integrated with ESRI's flagship desktop software, ArcGIS Pro, facilitating seamless data processing and sharing of data with collaborators and the general public.

Adding data stored in ArcGIS Online to a map in ArcGIS Pro

ArcGIS Enterprise

ArcGIS Enterprise is a suite of server software from ESRI that allows organizations to provide services on their own physical or virtual servers in the same manner as ArcGIS Online.

An ArcGIS Enterprise installation has four components:

ArcGIS Enterprise can be installed on Windows Server (IIS) or Linux (Apache) servers, as well a virtual cloud servers from providers like Amazon Web Services (AWS) or Microsoft Azure.

Figure
ArcGIS Enterprise architecture

ArcGIS Online vs. ArcGIS Enterprise

The choice of whether an organization should use ArcGIS Online cloud service or ArcGIS Enterprise on their own server depends on the needs and limitations of the organization.

ArcGIS Online ArcGIS Enterprise
Operational costs are assessed in credits, which can be expensive with organization serving or processing large volumes of data. Operational costs include capital costs of building, maintaining, and housing servers, as well as the labor costs for IT staff to operate and maintain the servers.
Usage credits support ESRI staff that handle technical maintenance of servers. The organization needs to hire specialized IT staff to maintain the servers.
Limited to a single ArcGIS Online data store Ability to use enterprise (geo)databases
Cloud security dependent on ESRI personnel Data is secured in-house to satisfy legal or organizational policy requirements
Deleted items cannot be recovered Deleted items can be recovered if deleted since the last backup

ArcGIS Services

A wide variety of different geospatial data services are available from ArcGIS Online and ArcGIS Enterprise servers that can be used to create map layers in ArcGIS Online interactive web maps and ArcGIS Pro static maps.

Feature Services

Feature services provide clients with access to vector features in a server geodatabase through REST endpoints.

A REST endpoint is a URL to a service on the server that includes subfolders and parameters needed to identify a specific feature service. The acronym REST comes from the type of communications protocol these endpoints use: representational state transfer.

This is an example REST endpoint URL for the US Fish and Wildlife Service's wetlands feature service:

https://www.fws.gov/wetlandsmapservice/rest/services/Wetlands/MapServer

ESRI provides a cloud environment called ArcGIS Online that enables users to publish feature classes as feature services.

With this flexibility comes some disadvantages:

This is an example of a REST endpoint URL for a feature service published ArcGIS Online.

https://services.arcgis.com/GL0fWlNkwysZaKeV/arcgis/rest/services/Minn_2023_World_Bank_Indicators/FeatureServer

ArcGIS Online provides the ArcGIS Online web app that users can access in a web browser or on a mobile device to perform basic web map and feature service creation, and management of data and services in the ArcGIS Online environment.

Feature services

Image Services

Image services provide clients with the ability to access raster and image data from a server geodatabase.

Image services supply data as tiles, which are map images chopped up into smaller 256 x 256 pixel images that the browser or mobile map app stitches together into a complete map image for the viewer.

Tiled web maps (also called slippy maps) load quickly and allow facile user interaction.

This technique for quickly and efficiently delivering web maps was first developed for Google Maps in 2005, and tiles have become a ubiquitous technique for web mapping (Forrest 2021).

Map tiles loading in a Google Map

Tiled maps have disadvantages.

Different types of image services available in ArcGIS Online and ArcGIS Enterprise include:

Tile services

A number of different tiled imagery layers are available in ArcGIS Online through the Living Atlas. The layers are identified as Imagery Layer or Tile Layer in the search results.

Living Atlas imagery layers in the ArcGIS Online Map Viewer

Vector Tile Services

Vector tiles are tiles created as vector graphics rather than raster images, so they consume less bandwidth, can be expanded without getting fuzzy, and (sometimes) can be styled on the client side.

Vector tile services may be more appropriate than feature services or image tile services in some situations.

The Charted Territory vector tile base map from the ESRI Living Atlas

Using Services in ArcGIS Pro

Feature Service Endpoints

A portal is "a website serving as a guide or point of entry to the World Wide Web and usually including a search engine or a collection of links to other sites arranged especially by topic" (Merriam-Webster 2022).

Government agencies often make their geospatial data available through their open data portals. These portals occasionally provide links to feature services that are accessible to the general public, although they more commonly simply provide links to download the data as files.

You can find these open data portals by Googling "name_of_place open data" or "name_of_place gis data."

Although a variety of software exists to implement data portals, one package commonly used by city, county, and state governments is ESRI's ArcGIS Hub.

In this example, the Alabama GeoHub is implemented using ArcGIS Hub, which is obvious from the look of the site as well as the arcgis.com at the end of the URL.

We will use their layer of hospital locations.

  1. Acquire: Copy the GeoServer link from the API dropdown. API stands for application program interface as these links are commonly used by programmers when they are creating web maps.
  2. Start: Create a new map in ArcGIS Pro and give it a meaningful name.
  3. Add: In ArcGIS Pro, on the Map -> Add Data dropdown, select Data From Path and paste in the endpoint URL.
  4. Symbolize: Adjust the symbology appropriately.
  5. Present: Create a print layout
    • Add a legend (if needed).
    • Export the map for printing or insertion as a figure.
Creating a map using a public feature service

Feature Service Download

Feature service endpoints can also be passed as the Input Features to the Export Features tool to export the contents of a feature service into a new feature class in your project database.

Downloading features in this manner can be unreliable, especially if you are downloading hundreds or thousands of features. If a link to a downloadable file is available, that will generally be a better option.

Exporting features from a feature service

The REST endpoint URL for an feature service in an organizational portal can be found on the information page for the desired feature service layer. This example imports features from the Minn 2015-2019 ACS Counties feature service in the University of Illinois ArcGIS Online organization, with a filter limiting the counties to Illinois.

Exporting features from a portal feature service

ESRI's Living Atlas

Services can be protected so that they can only be used within a specific organization, or by people pay for access to those services.

ESRI's Living Atlas of the World is a collection of geospatial data feature services that can be accessed in ArcGIS Pro or in ArcGIS Online. Some of the services are open to the general public, while other are proprietary and accessible only to licensed users.

The video below shows how to add a Living Atlas layer to a map in ArcGIS Pro:

  1. Start: Create a new map in ArcGIS Pro and give it a meaningful name.
  2. Add: On your map, select Map -> Add Data, and navigate to the Portal -> Living Atlas.
  3. Symbolize: Adjust the symbology if desired.
  4. Present: Create a print layout.
Creating a map using a Living Atlas feature service
Living Atlas feature layer: ACS Median Household Income Variables - Boundaries
Living Atlas imagery layer: USA NAIP Imagery - Natural Color
Living Atlas vector tile layer: OpenStreetMap

Image Service Download

Although Living Atlas image services are primarily intended for display, in some cases it is possible to export portions of the image data to raster files for further use in ArcGIS Pro.

For this example we demonstrate download of a portion of the USA NAIP Imagery: Natural Color Living Atlas layer covering Peoria County, Illinois to a GeoTIFF file.

  1. Add the desired image service layer to your map.
  2. Zoom so the extent of your map display covers the desired area for export.
  3. Right click on the image layer and select Data and Export Raster tool
  4. Click Export.
  5. Remove the ArcGIS Online tile layer and leave only the new raster layer on the map.
Exporting a portion of an Living Atlas image layers to the project geodatabase

If you have polygon(s) to which you want to clip the raster:

  1. Add the desired image service layer to your map.
  2. Zoom so the extent of your map display covers the desired area for export.
  3. Right click on the image layer and select Data and Export Raster tool
  4. Click Export.
  5. Remove the ArcGIS Online tile layer and leave only the new raster layer on the map.
  6. You may need to Refresh your Catalog view to see the raster in your project geodatabase.
Exporting a clipped section of an Living Atlas image layers to the project geodatabase

User-Created Services

Publishing service is the process of making data available as a service from software like ArcGIS Pro.

Although publishing in ArcGIS Pro is usually performed with a handful of mouse clicks and name entries, behind the scenes the publication process can involve uploading data, setting symbology, defining interactive capabilities (like pop-ups), and restricting access with permissions.

For this example, we use addresses of FedEx locations in Chambana, IL that were geocoded from a CSV file as described later in this document.

  1. Add the feature class as a layer to your map and adjust the symbology to whatever you want the default to be when someone adds the new service to a map.
  2. Edit the metadata for the layer to provide a meaningful title, summary, description, and credit for the original source of the data.
  3. Right click the layer and click Share -> Share As Web Layer.
  4. If you get a warning that, Allow assignment of unique numeric IDs for sharing web layers is disabled in Map Properties, right click on the Map in the Contents pane, and on the General tab, check Allow assignment of unique numeric IDs for sharing web layers.
  5. You can adjust the sharing of your service in ArcGIS Online so that it is is visible only to you, only to people in your organization, or to everyone.
  6. You can also verify that it works in a web map by opening the service in a new ArcGIS Online map viewer.
Publishing a user-created layer

The Digital Dark Ages

The Dark Ages was a period of European history between around 500 - 1,000 AD when there was little urban life. Accordingly, there is very little written information that survives from that period of time (Encyclopedia Britannica, 2021).

In the developed world, we capture and store almost everything that can be stored: security video, electronic communications, smartphone photos of events momentous and trivial.

Almost none of that data will survive us.

Although storage becomes cheaper every year, technology changes every year. Data must be migrated to new storage media and modes of expression, or it is lost to physical degradation or technological obsolescence. It is likely that only a small fraction of the data from our time will migrated, and eventually almost all of it will be lost, leaving future generations with only a small fraction of the vast body of information we have accumulated in our time.

Data in the cloud never has a permanent physical home. The cloud is a performance and requires constant flows of capital and resources to stay in operation. Changes in the economics of the cloud will necessitate loss of some of that data.

Which data will be lost to time? Will this period be as "dark" to future historians as medieval Europe is to present day historians?

Contrast the impermanence of the digital with papyrus text from 2500 BC or clay tablets from as far back as 3300 BC.

Mesopotamian Cuneiform Tablet, ca. 2000 BC (Musée de Mariemont via Wikipedia)

While security camera video from an ATM where there has been no criminal activity may not be something that should outlive us, your grandchildren may want to see some of those thousands of baby pictures that you took of your son in the first year of his life. You should plan accordingly.

Me and my father, ca. 1966