Netdisco 0.94 - README
Max Baker
Netdisco is an Open Source web-based network management tool.
Designed for moderate to large networks, configuration information and connection data for network devices are retrieved and set by SNMP. With Netdisco you can locate the switch port of an end-user system by IP or MAC address. Data is stored using a SQL database for scalability and speed.
Cisco Discovery Protocol (CDP), Foundry Discovery Protocol (FDP), and SynOptics Network Management Protocol (SONMP) optionally provide automatic discovery of the network topology.
The network is inventoried by both device model and operating system (like IOS). Netdisco uses router ARP tables and L2 switch MAC forwarding tables to locate nodes on physical ports and track them by their IP addresses.
For each node, a time stamped history of the ports it has visited and the IP addresses it has used is maintained. Netdisco gets all its data, including topology information, with SNMP polls and DNS queries. It does not use CLI access and has no need for privilege passwords. Security features include a wire-side Wireless Access Point (AP) locator.
Netdisco was created at the University of California, Santa Cruz (UCSC), Networking and Technology Services (NTS) department. UCSC continues to support the development of Netdisco by providing development servers and financial support. The Netdisco project is hosted by Source Forge.
From the web interface devices connected to switch and router ports are listed by MAC address. A history of which switch ports a MAC address has been seen at is kept. With a click the you can browse a network device connected to an uplink port. With another click you can disable or enable the switch port, logging the reason, user and date.
Only users you specify in Netdisco will have access to switch off a port. Netdisco will also not allow people to switch off uplink ports by accident.
public@101
)
Netdisco supports any Network device that talks SNMP and has basic information available through MIB-II (RFC 1213). Additional vendor-specific information is available for a number of devices, but especially for Cisco, HP, and Bay devices.
Device support is handled through SNMP::Info
-- a Perl module that is
an integral part of Netdisco that handles device-specific code. See the
Device Matrix
at http://snmp-info.sourceforge.net for a list of
devices that have been tested against Netdisco. SNMP::Info can be extended
for new families of devices relatively easily with a little Perl knowledge.
Please use the netdisco-users
mailing list for all problems and comments.
http://lists.sourceforge.net/lists/listinfo/netdisco-users
In case of bugs, please use the Bug interface from SourceForge page at:
http://sourceforge.net/projects/netdisco
Netdisco will attempt to detect uplink ports in case you are missing topology data during macsuck. Check the logs of the macsuck / macwalk for notifications of detected uplink ports, and add that data to your netdisco-topology.txt.
Fun Answer - From Douglas M. McKeown :
``This is where you go to a switch (Layer 2) and find all the MAC (or Ethernet Hardware) addresses which this device is connected to. So you plug your Dell into your HP Switch and that HP Switch is uplinked to your Core switch (not using the word router here. we're talking simple, physical network connections, sort of like electrical wires.) Well your Dell has a MAC address of let's say ''A`` and amazingly, your HP switch has a MAC address of ''B`` and your Core switch has an address of ''1``. Well if you Macsuck your Core switch, it doesn't have your Dell connected to it, but it does have ''B`` which is another switch. So you Macsuck ''B`` and it has MAC addresses for 1, B and A! You don't really Macsuck an end device (your Dell).
So what do we know?
- Core (1) knows about HP Switch "B". - HP Switch "B" knows about Core (1) and Dell "A". - Dell "A" knows about HP Switch "B".
Does ``1'' know about ``A'' ? If it's a router it does. Otherwise it asks who has ``A'' and switch ``B'' says, I know! So 1 goes to B which goes to A.
Got it?``
This process is what lets Netdisco map an Ethernet address to an IP address. Combined with the Macsuck process, Netdisco can ultimately resolve an IP address to a switch port.
If you have a small network that only has layer-2 devices on it, and you use a Linux or BSD box as your router, you will need to install net-snmp on the machine, and then have netdisco discover that machine. Otherwise you will not be able to resolve a MAC address to an IP address.
See Toplogy Information in this file.
From Douglas McKeown :
``CDP is the Cisco Discovery Protocol. Sort of an add-on for when switches talk to switches about who's connected to whom. CDP quickly tells other switches that it has switches connected. Netdisco really likes CDP a lot for mapping out the network and automatically discovering the topology. If your devices don't use CDP, then you need to work with the netdisco-topology.txt file to create a layout of your network.''
Note that FDP (Foundry) and SSOMP (Nortel/Bay) are both supported, and anywhere you see CDP you can assume we mean FDP and SSOMP too.
See the INSTALL document for instructions and requirements to install Netdisco.
Netdisco has three components :
Most of the back-end is controlled by cron jobs.
A background daemon is put resident to run maintenance tasks collected from the front-end. This keeps these sometimes memory intensive tasks and code out of the httpd processes.
Some front-end administration tasks are put in a queue in the database that a daemon running from the back-end picks up and processes.
The number of people using Netdisco can scale with the web server capacity, and will create no extra load on the devices.
carp()
SQL commands. Sets $netdisco::SQLCARP to 1.
Devices listed in arpnip_no
in the config file are excluded. See the
entry below.
expire_*
config file directives (see below).
This routine should be run nightly.
For a full backup run sql/pg_back to backup the whole database.
Devices listed in discover_no
in the config file are excluded. See the
entry below.
Subnet is specified in CIDR format :
192.168.0.0/24
NOTE: You can safely ignore all warnings about size too small for label
.
Make sure you have a relatively new version of GraphViz. You need a newer version of GraphViz if you get an error similar to:
Creating CMAP : /usr/local/netdisco/html/netmap.map warning, language cmap not recognized, use one of: ps hpgl pcl mif...
Devices listed in macsuck_no
in the config file are excluded. See the
entry below.
It is not necessary to do this after every change. This is only a convenience switch.
It's better to use interactive prompts so that the password doesn't get stored in your shell history file and exported to the process table.
netdisco
that runs in the background. From the web Admin Panel
,
jobs are put in a queue in the database. The daemon picks up these jobs and executes them from the
back-end as user netdisco
. The daemon is restarted daily in a cron job, or can be manually started
as root :
su - netdisco -c "/usr/local/netdisco -p restart"
Port Info was designed around data coming out of a Pinnacles database at UCSC, and might
prove to be site-specific. However, see port_info.html
for a good example of how to access other
databases using the netdisco.pm
SQL routines.
Enable this feature by setting port_info
to true in netdisco.conf
To do this the back-end requires a read-write community string for the device in question. The admin daemon must also be enabled. Netdisco keeps a log for each port holding information about why a port was turned on or off.
A reason for turning switch the port is chosen from a list to provide future audits of admin activity.
The user and IP address of the request are stored. To change the default reasons, modify the
%PORT_CONTROL_REASONS
hash in netdisco.pm
Optionally if the portctl_email
setting is set in netdisco.conf
, an e-mail is sent out with a
notification of the switching. Locally at UCSC that e-mail is sent to an administrative mailing list.
To turn this feature off uncheck the Port Control
checkbox from all users in the Admin Panel
.
By default Netdisco will be allowed to shut off
- Switch Ports - IP Phones - Router Ports that are NOT uplinks
By setting certain config file directives you can allow Netdisco to shutoff uplink ports and VLAN interfaces. But this is REALLY NOT RECOMMENDED. See below for the required commands.
The Web console is a reverse proxy that runs on Apache. You must enable it in netdisco_apache.conf
and
netdisco_apache_dir.conf
. The add devices and models to the configuration lines web_console_vendors
and
web_console_models
in netdisco.conf
.
netdisco -d mydevice
As a convenience you can use the topology file to quickly seed Netdisco with devices.
To import all the topology information at once make sure the topology filename is set in netdisco.conf
and use the -T command:
netdisco -T
kill -INT jobpid
There is currently no way to stop a job inside the Admin daemon. Send the daemon an INT signal and it will terminate after its current job has completed.
netdisco -i old-ip-address new-ip-address
Changing the IP address of a device will:
See each item's entry in the Config File Section below for more details.
The expire data routines are called from the -B/Backup routine, which should be running nightly via cron.
This is rather permanent. Make sure you run -Backup before you do this.
node
table,
and the MAC address to IP mapping in the node_ip
table.
To remove nodes from a switch, use the Admin Panel on the web side and choose either
Delete Nodes
or Archive Nodes
. Archiving nodes will set the archive bit so that the
data will be available, but not always showing. You can also delete nodes from the command
line using the -e command with or without the -V flag.
Database Cleanup -K will delete nodes that seem to be extraneous. See -K for more details.
Once you have cleared out nodes from a switch, then run -I to remove unused node to IP mappings.
This is rather permanent. Make sure you run -Backup before you do this.
Add User
form in the Admin Panel. After first installing
Netdisco you need to add an admin user by running -u.
source$ pg_dump -a -d -U netdisco -t users netdisco > user_dump.sql source$ scp user_dump.sql dest:
dest$ cd /usr/local/netdisco/sql dest$ ./pg_run /path/to/user_dump.sql
This turns out to be an INDEX bloat problem on Postgres versions less than 7.4. Recently doing this on a Postgres 7.3 install changed the amount of space that i Netdisco's database was using from 16G to 400M !!!
In order to fix this we do a VACUUM FULL ANALYZE VERBOSE
and REINDEX
from pg_run
.
This command locks each table before it does the VACUUM, and therefor can be more through.
It's a good idea to take netdisco down temporarily while you do this. I do this about
once a month, or when I notice it dragging down. Use Netdisco Statistics
as a good metric
of things slowing down. This may get fixed with changes in VACUUM in Postgres 7.4 and above.
Procedure for doing a vacuum full (as root):
/usr/local/netdisco/bin/netdisco_daemon stop
crontab -u netdisco -r
/usr/local/apache/bin/apachectl graceful
ps killall netdisco
df -h /usr/local/netdisco/sql/pg_run # before comparison : select relname, relpages from pg_class order by relpages desc;
REINDEX TABLE node; REINDEX TABLE node_ip; REINDEX TABLE device; REINDEX TABLE device_port; REINDEX TABLE device_port_log; VACUUM FULL ANALYZE VERBOSE;
# after comparison : select relname, relpages from pg_class order by relpages desc; \q
After:
df -h
/usr/local/etc/rc.d/010.pgsql restart
OR
/etc/rc.d/init.d/pgsql restart
OR
/etc/rc.d/pgsql restart
/usr/local/apache/bin/apachectl graceful
crontab -u netdisco /usr/local/netdisco/netdisco.crontab
/usr/local/netdisco/bin/netdisco_daemon start
cd sql ./pg_run admin.sql
Topology information is crucial to Netdisco's performance. It allows the application to know which ports are uplink ports and which have connected nodes. Ports that are uplink ports that are not marked so in Netdisco will appear to steal MAC address entries from their rightful ports. So it is critical to use the topology file and CDP/FDP/SONMP to maintain a topology.
During macsuck if Netdisco finds the MAC address of a known device or switch port, then that port is marked as an uplink. Nodes will not collect at these switch ports, and a warning message will be printed. Check the logs of your macsuck and macwalk jobs in order to find and correct autodetected uplink ports. Add these ports to your netdisco-topology.txt file.
Netdisco will auto-discover the layer-two topology of a network using CDP. However, many networks have parts of the topology that are not covered by CDP.
Use the manual topology file netdisco-topology.txt
to supply the layout of
the network if your network has devices that don't talk CDP or misreport
information.
The manual topology file only requires one side of the data to be entered. Both directions of a link will be forced to the given data if one side is listed.
File Format
The format of the manual topology consists of four types of lines:
#
They can happen on any line.
link:
or alias:
is assumed to be a the DNS name or
IP address of a network device.
link:
connect two devices together. The format is
link:outgoing port,destination device,Destination port
The outgoing port belongs to the device listed above the link:
line.
The Destination Device and Port tell Netdisco who is on the other end of this link. The device can be a DNS name or an IP Address.
NOTE: The port names must match exactly how Netdisco sees it. Go to the
device and check it out. You might think of it as port 1
but Netdisco might
think of it as RMONPort26onunit1
.
Many network devices like routers have multiple IP addresses assigned to them. If the device cannot or does not supply this information to Netdisco in a standard way, you can add IP addresses used here.
White space in the file (except for line breaks) is ignored. Tabbing over before line:
lines
makes it easier to read, but is not required.
File Uses
Some reasons the manual topology file is used:
[Cisco] ---> [Bay] ---> [HP]
The Cisco and HP devices (CDP speakers) find each other and the Bay device never appears. You would then have to add these lines to the topology file:
ciscoswitch.my.company link:EtherNet0/1,bayswitch.my.company,25 bayswitch.my.company link:26,hpswitch.my.company,J3
This tells Netdisco that port Ethernet0/1
on ciscoswitch
is connected to Port 25
on bayswitch
.
Then in turn Port 26 on bayswitch
is connected to port J3
on hpswitch
.
A note about devices that are CDP Aware and that implement CDP:
CDP Aware devices are devices that probably do not speak CDP (probably for legal reasons) but that are smart enough not to forward CDP packets. Cisco devices that have CDP disabled are usually still CDP Aware and will not forward the packets. Man-in-the-middle situations occur when the device both does not speak CDP and is not CDP Aware.
Netdisco is controlled via cron jobs. Jobs are run as user netdisco
.
Multiple jobs can be run at once.
The default jobs are :
The settings in netdisco.conf
are used both in the back-end and the front-end.
When you make a change in the config file that is used in the web front end, you must reload apache. The config information is shared between processes for speed and memory performance.
su - -c "/usr/local/apache/bin/apachectl restart"
Multiple config files can be used in the back-end by calling Netdisco with the
-C
option:
netdisco -C myotherfile.conf
.ucsc.edu
New in version 0.93 these directives are included to help make Netdisco more self-maintaining.
Setting these will result in permanent data removal.
You can put in a CIDR style subnet or a single IP address :
arpnip_no = 192.168.0.0/24,192.168.0.1,192.168.1.0/24
You can NOT use an abbreviated CIDR subnet like 192/8
.
See arpnip_no for more details.
BOOLEAN. Set to true to ignore aliases that are part of private nets:
10.0.0.0/8 172.16.0.0/16 and 192.168.0.0/24
See arpnip_no for more details.
This option was used to speed up MACsucking on certain Cisco Catalyst family devices where you have to connect to each VLAN with SNMP to get the forwarding tables. Certain default VLANs will not answer to SNMP, and Netdisco has to wait for them to timeout.
VLANs listed here are overrided regardless of macsuck_all_vlans value.
This is a debug option. Set this if you think that the option of not macsucking VLANs that aren't in use on device ports is some how interfering.
Setting this would revert macsuck to the same behavior as 0.93 and before.
Does not override macsuck_no_vlan.
nightly()
(-B) dump an NMIS http://www.sins.com.au/nmis
style Config file. Warning, this file will contain SNMP Community strings.
Optional Override options are :
Network
core
true
true
lan
161
device_root()
for more info.
Default false
The five database settings are db
, db_user
, db_pw
, db_opts
, and db_env
.
You can run multiple database types in Netdisco. See port_info
for an instance of this.
For each of the above settings, the database shortcut name (you choose) is inserted after db
.
Postgres is the required first database, and uses the short name Pg
.
The following lines must be added :
Default : dbi:Pg:dbname=netdisco
Default : PrintError => 1, AutoCommit => 1
Mainly used for Oracle.
Default : not set.
Example :
db_Oracle_env = ORACLE_HOME => /usr/local/oracle7, ORACLE_STUFF=>1
The final solution should be to add sub bulkwalk_no { 1; }
to the device
class that is misbehaving in SNMP::Info. This will turn off bulkwalk for a
class of devices, not all.
Default is on. SNMP::Info 0.10 or higher required.
perldoc SNMP
for more info.
Default is 20. SNMP::Info 0.10 or higher required.
This is only necessary if you turn on the port_control
command.
Port Control
are sent to.
Default False.
EXTREMELY VERY DANGEROUS - Turning off uplinks will take out chunks of your network.
Default False.
EXTREMELY VERY DANGEROUS - Turning off a VLAN could take out most of your network.
Port Info
and Jack Search
features.
Requires web login, password changing and all admin functions to be run in secure space.
netmap.html
component.
epsilon
attribute in GraphViz
used to control the graph solver.
Set to an integer value. This will improve the mapping and visual quality of
them graph. Each integer step can mean an exponential time increase in the solving
of the graph.
neato
and twopi
.
netmap.html
component.
GraphViz
for the overlap=""
feature.
neato
to try and solve the graph. Default 60min.
Variable:Regular Expression:Attribute:Value
Variables that you can use include : label,ip
Attributes can be any node attribute usable in GraphViz, such as fillcolor and color
Examples:
label:cat(?!-g):fillcolor:blue
If the label (dns name) matches cat, but not cat-g, make it blue
ip:^169\.233:color:yellow
If the IP address of the device starts with 169.233, then make the border around the device yellow.
The original ArpNipper was written by Jim Warner at UCSC.
If the device supports the v_name()
call, and has VLANs, then the MacSucker tries to
connect to each VLAN and macksuck()
each VLAN. This is required for some devices like
the Cisco Catalyst 5000, 3500, 1900, 6500 series.
A few speedups are implemented for the devices that require each VLAN to visited:
The original MacSucker was written by Mark Boolootian at UCSC.
Browse the source code or check out netdisco-api for more info.
Netdisco uses PostgreSQL as its database store. Indexing is used heavily to speed up
queries and facilitate large data sets. See the sql/
directory and INSTALL
for more information.
I would like to thank the following people for their contributions to Netdisco :
Mark Boolootian (Through who's ideas Netdisco was born and shaped) (UCSC), Jim Warner (UCSC), Mike Hunter (UCB), Brian Wilson (NCSU), Bradley Baetz (bbaetz), David Temkin (sig.com), Edson Manners (FSU), Dmitry Sergienko (Trifle Co, .ua) Remo Rickli (PSI, Switzerland), Jean-Philippe Luiggi (sagem.com), A.L.M Buxey (Loughborough University, UK), Kevin Cheek (UMICH), John Bigrow (bnl.gov), George Pavel (llnl.gov), Charles Goldsmith (CompUSA), Douglas M. McKeown (saintmarys.edu), Revital Shvarzman (York U, Ontario), Walter Gould (Auburn U), Lindsay Druet and Colin Palmer (U of Waikato, Hamilton NZ), Dusty Hall (Auburn U), Jon Monroe (center pointe), Eric Miller (jeneric).
As well as these people for their donations :
Richard Silver (East Alabama Medical Center) Linda Rosewood (UCSC) "6pax4max!","31337" (???) Dusty Hall (Auburn University) Walter Gould (Auburn University) Frank Chatham (Auburn University)
And probably lots of other people I forgot to put in here. Not to mention the authors and communities of all the other software that Netdisco is built upon.
%text>