SNMP, LDAP and SOAP Together for a distributed Network Management Experience

Goal

To implement and deploy a distributed network management system based on collecting information from agents (SNMP), discovered through a directory service (LDAP), monitoring and steering them using an interoperable messaging (SOAP).

1- Introduction

As networks grow in size and in demand, monitoring the state of the network is becoming critical to avoid congestion, security breaches and various network bottlenecks that might occur in any Local or Wide area networks. The inherent heterogeneity of a local/wide area network require from the network management infrastructure to be as platform independent as possible, thereby imposing a clear definition of the underlying protocol; the extensible markup language (XML), self-describing language allows such a protocol definition to be platform independent and also provides a human readable description of its content; these two points make XML a language of choice in the design of a messaging for a network management tool. With the growth of the internet and of new approaches to computing such as Grid computing, providing seamless access to resources; the issue of resource/peer discovery raises a need for distributed directory services capable of synchronizing their information in an optimal manner. The Lightweight Directory Access Protocol (LDAP) is an effort to provide lightweight directory lookup information accross a heterogeneous network of peers. This paper will present the DIMM (Discovery Messaging Management) infrastructure, enabling SNMP management to hosts discovered through LDAP directories and using an XML-based messaging (SOAP); Section 2 will describe the issues of Discovery, Messaging, Management separately, and the chosen application solutions will be presented, in section 4, background and related work will be presented, section 5 will go through a typical usage scenario. For reference purposes, installation and configuration of the tools used can be found in the appendix.

2- Components of a Network Management system

Any distributed system includes a discovery mechanisnm, an application and a messaging definition.

2.1- Discovery mechanism

In a distributed network, discovering peers/resources is a mechanism that can go from a simple query to a central location known to everyone to a complex distributed location involving advanced querying mechanisms across the nodes of the network.
The requirements of a discovery mechanism in a network management system is to locate a peer or a resource and provide routing to it in an efficient (number of hops, time) manner.
Additionaly, a discovery mechanism should be scalable, allowing any host/node joining the network to be added to the list of active hosts that could be looked up.
The Domain Name Server (DNS) is the most popular and currently most scalable discovery mechanism; it is based on a hierarchichal lookup mechanism in which a server (domain name server) runs the BIND utility, which will convert the name of the requested Uniform Resource Locator name into its equivalent IP address, if the entry does not exist in the database, a recursive lookup to a referring domain name server is performed until the entry is matched, this new entry is then locally cached for later accesses.
Request forwarding mechanisms such as the ones implemented in Chord, Pastry of the Content-Addressable Network are based on a more decentralized approach in which every entry information is stored at a predefined place (given by a specific hashing mechanism) and replicated through some replication policies, the lookup is performed based on hashing the content to be matched and routing through the overlay network towards the destination pointed out by the result of the hash function.

In both of these cases, frequency of updates is an issue that does not have an immediate answer since the scale of the entries might get out of proportion (in the scale of DNS, the internet connected world).
The fundamental question that needs to be resolved is how to synchronize those distributed entries so that querries can be guaranteed to get a valid response and that information stays up to date.

2.2- The network management application

The ability of a network to perform efficiently given its parameters depends on how well it can stay out of congestion.
In today's growing network size, network management tools are becoming essential to control the flow of information across domains. Network administrators need to move from the tedious job of analyzing log files to a more efficient and ubiquitous way of monitoring and controling their network.
This change will require a system to run an information gathering agent and a functionality for manager to query this collected information station. The ISO Network Management Model has identified the different requirements of a management architecture by breaking it up into five areas: {performance, configuration, accounting, fault and security} management.
The goal of Performance Management is to measure the performance of network-related componenets of a system in order to maintain an optimal internetwork functioning.
In a Configuration management system information related to the managed device is maintained in the information base for a quick analysis by the network administrator regarding the device/system configuration; such information can be: {Operating System, Ethernet Interface, TCP/IP software, Netware Software, NFS software, Serial Communication, SNMP ....}
An Accounting management tool measures the network-utilization parameters in order to better understand the usage of the system and therefore tweak the system for optimal resource utilization.
Fault Management's objective is to detect, log, notify users of, and if possible automatically fix network faults/problems to keep the network running effectively.
Security Management is responsible for monitoring and policing network accesses in order to maintain rights of access to the network by enforcing rules and access control.

2.3- The messaging

Any distributed system is based on the capability of nodes to send messages to other nodes in order to perform an operation.
This messaging can be more or less complex depending on the requirements of each endpoint. In a network management system, the need for a heterogeneous messaging, involving a clear description of the underlying protocols is fundamental.
The architecture of such a messaging service requires endpoints to establish a virtual channel on top of the point-to-point connections between the nodes involved in the query/response message.
There has been several approaches to messaging systems so far among which we can point out: Raw network, Remote Procedure Calls, the Electronic mail and message oriented middleware.
In a Raw Network type of messaging, every possible transaction has to be taken care of yielding to high bandwidth (because built right on top of the network protocols) but extensive developement of the endpoint message handlers.
A Remote procedure call infrastructure in which specific subroutines/methods are invoked at a remote end has the advantage of being simple to deploy.
The elctronic mail system, is today one of the most commonly used collaborative tool, despite all the efforts of much more advanced collaborative application providers. One of the reason for this is probably its extensive deployment and the numerous clients that implement the Simple Message Transport Protocol (SMTP). It has the advantage of being asynchronous and simple but on the other hand is unreliable and provides no guarantee of receipt.
A Message Oriented Middleware is somewhat the "combination" of all of the above messaging system, it is based on a middleware set of message brokers that similarly to the ground mail dispatch centers, distribute messages to the different endpoints. It is not an elegant solution for high-bandwidth streamed media; comparatively, sending a letter through ground mail is not going to reach destination in the next second.

3- Selected Solutions to be studied

3.1- OpenLDAP

Directories can be seen as specialized databases providing reading, browsing and searching capabilities. Since directory entries are not meant to be changing very often and that a directory is more often read than written, they do not provide advanced complicated transaction schemes as found in database management systems. Directory updates are typically simple all or nothing changes.

In a network management system, finding a host that matches a profile is of major importance; a system administrator might need a list of all the hosts in the LAN which have an open connection to a specific address, or all hosts which have a specific port open... LDAP securely stores mnemonic strings in a hierachical tree-like structure, referenced through a distinctive name and allows searches to be filtered on a criteria basis. The attributes of the entries are defined in Schema files that describe the structure of the data.
Why LDAP ? :
As mentioned above, LDAP is a directory service protocol as well as a back-end database, it is intended to be used in systems where data is more often read than written (no frequent updates) and provides simple querying features based on simple filters. Given that our system requires registration of each managed device in a directory and that the device won't get to update its entry very often, LDAP seems to be an interesting technology to use.

3.2- Simple Network Management Protocol

The Simple Network Management system allow administrators/managers to query agents monitoring a device (bridges, hubs, routers, network servers etc...). The network-Information is stored in a Management Information Base which can be thought of as an Information Warehouse. A typical agent usually:
  • Implements full SNMP protocol
  • Stores and retrieves management data as defined by the Management Information Base
  • Can asynchronously signal an event to the manager
  • Can be a proxy for some non-SNMP manageable network node.
A typical manager usually:
  • Implements full SNMP protocol
  • Is able to
    • Query agents
    • Get response from agents
    • Set variables in agents
    • Acknowledge synchronous events from agents
In order to allow information to be queried and modified, information is structured following the Structure of Management Information (SMI).
The SMI states that each managed object must have a name, a syntax and an encoding. The name uniquely identifies the object, the syntax defines the data type, the encoding describes how the information is serialized for transmission.
  • SNMP access particular instances of an object
  • All instances of an object in the MIB reside at the leaf of the MIB tree
  • SNMP protocol access objects by formaing an Object identifier of form x.y where x the true OID for the object in the MIB, and y is a suffix specified by the procol that uniquely identifies a particular instance.

Why SNMP ?
Standards in Network Management include the Simple Network Management Protocol and the Common Management Information Protocol, they actually both work in a very similar manner. CMIP is known to be more robust, secure and elaborate than SNMP but it also requires more system resource and is more difficult to program, which might explain its less popular aspect.

3.3- Simple Object Access Protocol

SOAP is a way to invoke methods on classes and objects that exist on remote servers; it is a more advanced version of the earlier XML-RPC remote procedure call mechanism, compared to which it provides fault handling and a more extensive list of data types handling. It is emerging as an Internet-Friendly alternative to other component based languages such as DCOM, CORBA.
SOAP is theoretically transport independent, therefore using it on top of HTTP has made it quite popular, although it currently only defines HTTP as a transport protocol in the specification. SOAP uses the self-describing eXtensible Markup Language for the syntax; by leveraging those two (HTTP+XML) ubiquitous technologies it naturally inherits many advantages among which is interoperability, ubiquity and pervasiveness.
SOAP consists of three parts:
  • The SOAP envelope construct defines an overall framework for expressing what is in a message; who should deal with it, and whether it is optional or mandatory.
  • The SOAP encoding rules defines a serialization mechanism that can be used to exchange instances of application-defined datatypes.
  • The SOAP RPC representation defines a convention that can be used to represent remote procedure calls and responses.

The motivation for such a protocol was to resolve the interoperability issues that currently tie application developpers to specific implementations and limit the interchange of information across systems.
By leveraging the eXtensible Markup Lanaguage, SOAP allows querries and responses to be in an human-readable intermediary format and can be directed to the handler of choice.
Current drawbacks of such a framework is paradoxally closely linked to its advantages in that the intermediary format generates an overhead in the messaging that cuts down performance and therefore limits the usability of such a framework for very systems requiring frequent changes and updates.

Being an advanced form of XML-RPC, offering an inherent interoperable caracteristic, SOAP is a technology that promises to grow exponentially and successfully. Known limitations of SOAP include :
  • The current SOAP spec describes how SOAP payload can be transmitted via HTTP, but does not address any other protocols.
  • SOAP does not address higher-level issues such as object activation or lifetime management.
  • SOAP does not mandate any particular language for interface description, although the XML Schema specification is a reasonable way to describe interfaces and user-defined types.
SOAP was designed with simplicity and extensibility in mind, therefore some features from traditional messaging systems and distributed object systems are not part of the core SOAP specification, such as :
  • Distributed Garbage Collection
  • Boxcarring or batching of messages
  • Objects by reference
  • Activation

4- Description of the experimental setup

Components

SNMP

The simple network management protocol system that was deployed is the freely available net-snmp project (Version 4.2.2) that can be found at http://net-snmp.sourceforge.net.
This work was based on the Carneggie Mellon and University of California at Davis SNMP implementations.
It provides: An extensible agent, An SNMP library, tools to request or set information from SNMP agents, tools to generate and handle SNMP traps.
It runs on almost any flavor of Unix including (Linux OpenBSD Solaris Irix).
The client allows the basic GET, SET, GETNEXT, WALK and TRAP through the MIB trees and more advance features such as TRANSLATE, STATUS, NETSTAT and TEST. SNMP was deployed on a Linux Debian [kernel 2.4.2] running on a Pentium II 350MHz, with the standard mibs: system, interfaces, ip, tcp, udp, icmp, snmp

OpenLDAP

Version 2.0.18. Provides the stand-alone (slapd) and replication engine (slurpd).
Provides client tools : ldapadd, ldapmodify, ldapdelete, ldapsearch.

SOAP

Deployed SOAP::Lite, a perl binding to SOAP back-end as well as the perl SOAP client for querying any soap-enabled back-end.
Also deployed the Apache-SOAP on top of Tomcat servlet+jsp engine.

4-2 Putting them together

The idea is to look for an SNMP managed device in the directory service and use a SOAP client to query the returned device.

4-2.1 The LDAP lookup

Each managed device registers to an LDAP directory using a distinguished name (DN) entry such as: 
	dn: cn=node1, dc=netw-manag, dc=com
	cn: node1
	objectclass: top
	objectclass: device
	objectclass: ipHost
	cn: node1.netw-manag.com
	ipHostNumber: 138.89.1.11
	macAddress: 00:00:92:90:ee:e2
	description: pentium III 1GHz
	l: Piscataway, New Jersey
	c: US
Performing a search on the host at netw-manag.com for objectclasses device would return this entry. The following command-line search shows how this could be done. 
			
ldapsearch -b 'cn=Manager,dc=netw-manag,dc=com'	'objectclass=device'
Once the information is retrieved from the Directory, the manager gets to know of: a short description of the managed device, its IP address and is then ready to establish a direct connection to this node through a SOAP request.

4-2.2 The SOAP query

	#!/usr/bin/perl
	use SOAP::Lite;
	$client=SOAP::Lite->uri('SNMP');
	$myproxy=$client->proxy('http://netw-manag.com/soap/');
	$myresult=$myproxy->get(mib)->result;
	print "SNMP Response: ".$myresult. "\n";
The client initiates a request to the back-end server by invoking the remote method triggering the SNMP response.
Given this mechanism, the client does not need an SNMP client API but requires a SOAP client access to the back-end service.
The back-end handles the call by acting as a bilingual translator between SOAP and SNMP 
	#!/usr/bin/perl -w
	use SOAP::Transport::HTTP;
    SOAP::Transport::HTTP::CGI
    -> dispatch_to('SNMP')
    -> handle;
	
	package SNMP;
     sub get($ARG) {
       $res=`snmpget localhost public $ARG`;
       return $res;
	}

Conclusions

In this work, three growing technologies (SNMP, LDAP, SOAP) were evaluated and combined to provide an interoperable network management system in which managed devices are discovered through a directory service. Despite the efficiency inducing such increasing popularity in each of these technologies, the benefit of combining them does not appear to be optimal given the problem to be solved. The advantages of such a system is to leverage the SOAP remote procedure call and messaging capabilities, providing clients with the ability to query remote LDAP and SNMP servers without the need of their local respective client-side APIs; on the other hand, servers require SOAP to be deployed and provide a strong binding between the back-end applications (LDAP or SNMP) and the remote procedure call.

Appendix A

Apache-Tomcat setup

  • Install Apache
  • Install Tomcat
  • Install mod_webapp (connector for apache)
  • Modifiy Apache/conf/httpd.conf to contain 
	LoadModule webapp_module modules/mod_webapp.so
	AddModuel mod_webapp.c
	...
	WebAppConnection conn warp localhost:8008
	WebAppDeploy     /proj conn
  • Add the new project as a context in Tomcat/conf/server.xml 
    <Context name="/proj" docBase="/where/proj/is" />
    This can also be done by puting the web archive file (.war) directly into Tomcat/webapp directory.
  • Start Tomcat
  • Start Apache
  • Access http://localhost/proj

Appendix B

Apache-SOAP

Apache SOAP is a java-only (for now) SOAP binding.
  • Install soap
  • Deploy soap as a Context in Tomcat or copy the soap.war file directly into Tomcat/webapps
  • When deploying services make sure that SOAP has access to the classes through the CLASSPATH variable in catalina.sh
  • deploy the service
  • Access the service from any SOAP client

Appendix C

SOAP::Lite

SOAP::Lite is a perl to SOAP binding.
  • Install Apache and enable it for cgi's
  • Install soaplite
  • add CGI-based SOAP servers in Apache's cgi-bin directory. Make sure that the script contains the #!/usr/local/bin/perl or wherever path to per you require
  • access the service through any SOAP client

Appendix D

OpenLDAP

Links

Network Management

SNMP-Simple Network Management Protocol
The Net-SNMP project
Network Weather Service
Apache SNMP modules; check http://www.simpleweb.org/ for general information
Network Management at Buffalo
Simple Times:Publications devoted to SNMP
Network Management Using Internet Technologies;Franck Barillaud IBM

SOAP::Simple Object Access Protocol

SOAP Specification 1.1
IBM developerwork SOAP Tutorial
Quick start guide to SOAP using SOAP::Lite
SOAP Chapter 12 of Java & XML, O'Reilly publishing, Brett McLaughlin

LDAP::Lightweight Directory Access Protocol

LDAP Linux HOWTO
LDAP Implementation HOWTO
OpenLDAP 2.0 Administrator's Guide