OSGeo - Powering SDIs globally - GSDI 2009 Paper 272
OSGeo - Powering SDIs globally
This paper introduces to the activities of the Open Source Geospatial community and its role in supporting spatial data infrastructures around the world. Open Source software is used pervasively throughout all domains of IT ranging from operating systems to web applications. The Open Source spatial domain has grown over the past years producing many good software packages, some of which have developed into reliable core components that are used to build a strong and comprehensive spatial data infrastructure. The natural use of standards as a means to connect and interoparate with other packages has helped to build a strong community of professionals from all domains.
Introduction to OSGeo
The Open Source Geospatial Foundation (OSGeo) is a not-for-profit organization founded in 2006 whose mission is to support and promote the collaborative development of open geospatial technologies and data. The foundation provides a collaboration environment, organizational and legal support to the broader open source geospatial community. It serves as an independent legal entity to which community members can contribute code, funding and other resources, secure in the knowledge that their contributions will be maintained for public benefit. To ensure proper governance
OSGeo also serves as an outreach and advocacy organization for the Open Source geospatial community. It provides a common forum for users, businesses and developers of Open Source geospatial software. OSGeo operates a comprehensive development infrastructure for its projects ensuring highest quality software and improving cross-project collaboration.
The foundation's projects are all freely available and useable under a license certified by the Open Source Initiative (OSI). Open Source software projects can apply to go through the OSGeo Incubation process and become an official OSGeo Project.
The Incubation process
The purpose of the OSGeo incubation process is to ensure that projects follow the basic official OSGeo guidelines on Open Source software development.
This means that they:
- have a successfully operating open and collaborative development community
- have clear IP oversight of the code base of the project
- adopt the OSGeo principles and operating principles
- are mentored through the incubation process
All these different types of software and styles of governance have to adhere to the same basic Open Source principals as set up by OSGeo. Instead of inventing all of these principals from scratch the bascis have been gleaned and from successful Open Source initiatives like the Apache Foundation. To be able to account for the specialities of the existing diversity of projects they have been adapted and honed to suit the need of asa diverse community as that of the geospatial realm. The resulting guidelines specify that during incubation and after:
- Projects should manage themselves, striving for consensus and encouraging participation from all contributors - from beginning users to advanced developers
- Contributors are the scarce resource and successful projects court and encourage them
- Projects are encouraged to adopt open standards and collaborate with other OSGeo projects
- Projects are responsible for reviewing and controlling their code bases to insure the integrity of the open source baselines.
A designated mentor from the OSGeo Incubation Committee accompanies the project through the processes and helps to solve any problems. Completing the Incubation Process can take several months of work depending on the size of the project, legal background of the code base. When all issues that were identified as critical to a healthy project have been addressed the mentor proposes the project for graduation. Then the Incubation Committee consisting of currently 18 members from different backgrounds goes through the documentation of the project's incubation and brings up any open ends to ensure highest quality. At the next Board Meeting the directors of OSGeo have a formal vote to graduate the project after which it becomes an official OSGeo project.
All the details of the OSGeo Incubation Process are available at the OSGeo Wiki website at: http://wiki.osgeo.org/wiki/Incubation_Committee
The OSGeo SDI Stack
This paper will focus on the OSGeo SDI software stack as listed on the web site under Web Mapping (http://www.osgeo.org/). This stack is complemented by spatial Open Source components like PostGIS (http://www.postgis.org/) that are not (yet) part of the OSGeo stack and non-spatial software like operating systems, web servers (http://www.apache.org/) and databases (http://www.postgres.org/).
deegree supplies the building blocks of a Spatial Data Infrastructure, while implementing the standards of the Open Geospatial Consortium (OGC) and ISO/TC 211. The Java-based deegree Framework is the most extensive implementation of OGC/ISO standards in the field of Free Software.
deegree comprises five product groups of which three are directly related to SDI:
- deegree Web Services: the base components of any SDI; including WMS, WFS, WCS, CS-W, Gazetteer, WPS, WCTS, WTS/WPVS, security services like WAS and WSS, WMPS (deegree-specific) and owsWatch (service monitoring).
- deegree iGeoPortal: the portal framework of the deegree project. iGeoPortal has a modular structure and comprises so far the modules map, gazetteer, catalogue, security, data and 3D.
- deegree iGeoSecurity: to keep your SDI secure
Mapbender is an SDI back office to manage, monitor, catalog and visualize services. It contains a framework to deploy web based SDI applications and gives access to the OSGeo SDI stack. End-user interfaces can be highly focused with only the minimally required functionality or full-fledged applications with security proxying, WFS-T digitizing with auto snapping and more.
Comprehensive web based management interfaces for the administration of the SDI allow to maintain and categorize map and feature services and grant access to individuals, groups and other services. Adherence to standardized services, such as WMS and transactional WFS, allow to take advantage of interoperable services from various server platforms.
- Software and management services for mapping of OGC web services architectures
- Provides a data model and interfaces for displaying, navigating and querying OGC compliant map services
- Authentication and authorization services
- Security proxy functionality
- Management interfaces for user, group and service administration
MapGuide Open Source
- Windows and Linux, Apache and IIS, and multiple browser support
- Interactive map viewing
- Quality cartographic output
- Built-in storage of XML resource documents
- Uniform access of raster/vector data using Feature Data Objects (FDO) API
- Flexible application development: PHP, .NET, Java
- Extensive server-side APIs
- Fast, scalable, secure server platform
MapServer is an Open Source development environment for building spatially-enabled web mapping applications and services. It is fast, flexible, reliable and can be integrated into just about any GIS environment. Originally developed at the University of Minnesota, MapServer is now an OSGeo project and is maintained by developers around the world.
MapServer runs on all major operating systems and will work with almost any web server. MapServer features MapScript, a powerful scripting environment that supports many popular languages including PHP, Python, Perl, C# and Java. Using MapScript makes it fast and easy to build complex geospatial web applications.
- Supports industry standard data formats and spatial databases
- On-the-fly feature classification
- Sophisticated rule-based labeling
- On-the-fly projection for both raster and vector data
- Provides a wide variety of spatial and attribute-based queries
- Supports popular Open Geospatial Consortium (OGC) standards including WMS, WFS and WCS
- Leverages best-of-breed open source geospatial technologies such as GDAL/OGR, PostGIS and PROJ.4
- Integrates with popular front-end environments such as ka-Map, Chameleon, Mapbender, OpenLayers and Cartoweb
As a framework, OpenLayers is intended to separate map tools from map data so that all the tools can operate on all the data sources. This separation breaks the proprietary silos that earlier GIS revolutions have taught civilization to avoid. The mapping revolution on the public Web should benefit from the experience of history.
- Support for a variety of data sources
- Support for displaying geographic features, with markers and pop-ups
- Easy build configuration, designed to help build OpenLayers into other applications
The data layer is the foundation of any SDI. It can be built from a wide variety of spatial data formats using the GDAL/OGR library or stored in the object relational database PostgreSQL with the spatial library PostGIS. Both types of data store are supported by most web mapping server components. Additionally both GDAL/OGR and PostGIS can also be used to manipulate and convert spatial data directly.
The Geospatial Data Abstraction Library (GDAL/OGR) is a cross platform C++ translator library for raster and vector geospatial data formats that is released under an X/MIT style Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single abstract data model to the calling application for all supported formats. It also comes with a variety of useful commandline utilities for data translation and processing.
GDAL supports over 50 raster formats, and OGR over 20 vector formats.
It provides the primary data access engine for many applications including MapServer, GRASS, QGIS, and OpenEV. It is also utilized by packages such as OSSIM, Cadcorp SIS, FME, Google Earth, VTP, Thuban, ILWIS, MapGuide and ArcGIS. GDAL/OGR is the most widely used geospatial data access library.
Professional support options are available for commercial software developers desiring assistance integrating and extending GDAL, or dig in yourself and join the development team!
- Library access from Python, Java, C#, Ruby, VB6 and Perl
- Vector data model closely aligned with OGC Simple Features
- Coordinate system engine built on PROJ.4 and OGC Well Known Text coordinate system descriptions
- Utilities for data translation, image warping, subsetting, and various other common tasks
- Highly efficiency raster data access, taking advantage of tiling and overviews
- Support for large files - larger than 4GB
PostGIS adds support for geographic objects to the PostgreSQL object-relational database. In effect, PostGIS "spatially enables" the PostgreSQL server, allowing it to be used as a backend spatial database for geographic information systems (GIS), much like ESRI's SDE or Oracle's Spatial extension. PostGIS follows the OpenGIS "Simple Features Specification for SQL" (OGC SFSQL) and has been certified as compliant with the "Types and Functions" profile.
PostGIS has been developed by Refractions Research as a project in Open Source spatial database technology and is released under the GNU General Public License. The roadmap includes full topology support, raster support, networks and routing, three dimensional surfaces, curves and splines and other features.
- The Open Geopstial Consortium (OGC) standards for "Simple Features" (2D Geometries)
- The Open GIS Consortium (OGC) markup language for "Simple Features" known as Well Known Text (WKT) and Well Known Binary (WKB) representations of data.
- OGC SQL functions: standards for managing spatial tables using SQL language
- PostGIS extends support for geometry types EWKB, EWKT and Canonical Forms
- PostGIS adds support for embedding the SRID (projection information) and for 3 and 4D geometries with Z and M values, respectively.
- Other Types, Progress on Raster Support, Proposed X3D types: TIN and COLLADA for Web Output
- PostGIS enables powerful spatial analysis and editing of geometries
- PostGIS can be extended by scripting with an array of languages
- Client-side extensions exist to access spatial data in PostgreSQL for the programming languages, Python, Java (own class model, JTS, read-only support for Java2D, PHP and Ruby
The architecture of any SDI can be depicted in simple diagrams with three sections, the data store, the service and the client level. The diagram below additionally differentiates between the geospatial services (software) and the interfaces (standards) as developed and defined by the OGC and normed by ISO. Taking simplicity to be the prime goal of any general architecture design is a good idea because complexity emerges by itself.
Image source, slide one with friendly permission by Jeroen Ticheler: http://geonetwork-opensource.org/download/SDI-Architecture.ppt/view
To make it short, SDIs run on the Web. It's application protocol is HTTP hiding the underlying transport layers. Simple enough. But there is an ongoing discussion around the buzz words SOA, ROA, REST, RESTful and SOAP that need to be put into context to make sense, this time having clarity in mind as a prime goal.
The messaging protocol SOAP exposes arbitrary non-uniform software procedures through messaging between service endpoints. This allows to connect software procedures via the global network by piggy packing messages wrapped within a separate protocol on the existing application protocol HTTP ignoring most of the latter protocol's potential. If this sounds over complicated that is probably right. The SOAP technology is frequently associated with the highly indistinct an nebulous service-oriented architecture paradigm (SOA) although they have nothing in common but three letters. Adding to the confusion, the way the Web works has been codified in the REST paradigm (Representational State Transfer) which gave birth to another somewhat vague paradigm of the Resource-oriented Architecture (ROA).
Both architecture paradigms use common terms but with distinctly different and frequently incompatible meanings. The same terms are used in both contexts but with different connotations and from different levels of perspective. In the ROA all resources can also be a service thus in one way making the SOA a subset of the ROA. From another perspective the ROA is an implementation of the broader SOA concept and its largest existing network is the Web.
The difference in the paradigms are that they look at the architecture from a usability point of view whereas what is commonly known as the SOA seems to have evolved from a software perspective. Potentially oversimplifying things again it could be stated that the ROA is the system architect's view that can be implemented by the software architect using Service-oriented architecture software. Luckily in most cases the software in question can be used following either paradigm, in the worst case by putting a facade in front of it that connects to the Web which REST does by itself.
There is a comprehensive list of links pointing to sites with case studies on the OSGeo Wiki web site:
The regular annual OSGeo Journal also contains case studies from all walks of life:
Rhineland Palatinate SDI
As an example for an Open Source based spatial data infrastructure this paper will present and discuss the SDI of the federal state of Rhineland Palatinate, Germany:
The state of Rhineland Palatinate operates a growing number of spatial data services with software from several vendors and needs to provide access to a broad range of users from the general public to specific secure environments. Additional spatial data related to the region is maintained through services operated by the state's municipalities, the German state and private data providers. Access to some data sources has to be secured, controlled and logged on a user, group and functional level.
The need for domain specific web based applications is growing quickly requiring comprehensive tracking and monitoring of services.
From the problem definition the following (shortened) requirements for the implementation of the geoportal were identified. The full list of requirements is available through the Ministry of the Interior.
- Manage any number of OGC WMS and WFS services running on an arbitrary number of different vendor software
- Centralized spatial web service repository
- Service meta data monitoring and custodian notification
- Notification of Web application operator in case of changes of service meta data
- Service availability monitor, notification of custodian / publisher / operator
- Secure access to services that contain privacy related or restrictively licensed data
- Binding 2500+ identities from within public administration authentication system to geoportal authorization and OWS Security Access façade
- Transparent user management for 1000+ users outside of the public administration authentication system and bind to secure OWS access façade
- Provide access management on a group, user, feature and functionality/application basis
- Fully mandate capable user management
- Meta data publishing conforming with INSPIRE
Most requirements defined by the GeoPortal.rlp analysis document could be addressed by standard out of the box software installations. The core software of the geoportal is the Mapbender framework. Some extensions had to be implemented to the meta data management layer, OWS security proxy and WFS-T service repository. All development that could be implemented as generic modules was added to the Mapbender core allowing it to be reused by other clients.
The core software stack of the geoportal installation is completely based on Open Source software:
- PostgreSQL / PostGIS
The geoportal is designed to act as a broker between users and providers of spatial data, information and spatially related services. The portal offers centralized access to spatial data in Rhineland-Palatinate through services organized by meta data in a repository.
GeoPortal.rlp serves federal state agencies, municipal authorities and private data providers as a platform to maintain their meta data and present their data and services. Direct access to the distributed data sources of each spatial product and service provider ensures that information made available by these institutions on a joint platform is up-to-date. The geoportal does not store any data but only its meta data, service meta data, user identity information and application configuration.
The Portal - User Perspective
Within the Spatial Data Infrastructure of Rhineland-Palatinate, the GeoPortal.rlp assumes the role of the service-oriented agent brokering spatial data between users and providers.
The core functionality of the portal provided to anonymous end user follows three steps:
- Search spatial data
- Select result
- Show map
The distributed access paradigm keeps the responsibility for the data, service, actuality and authorization with the body in charge (creator, custodian, reseller) guaranteeing a high level of topicality and actuality. The available panoply of spatial data made enables authenticated and authorized users to create their own individual spatial data product and web application. The consistent and comprehensive application of recognized ISO and OGC standards throughout GeoPortal.rlp allow for seamless integration of maps and data from different services.
The broader Open Source community has over years created a comprehensive stack of software that is used globally to implement spatial data infrastructures. There is a wide selection of components for a wide variety of domains and solutions, implemented in a variety of programming languages and adhering to all major standards. Well defined governance rules as set forth by the Open Source Geospatial Foundation provide for a secure environment for users and developers alike. As a legal entity OSGeo also gives the software projects themselves a legal backing that is also required by businesses who want to provide services for the components of the SDI stack or other software in the Open Source realm. There are no reasons not to use Open Source software and many reasons to do so. But the most convincing arguments can only be found out by actually using them, the entry barrier is very low and the reward high.