Nagios can be a pain in the butt to troubleshoot because finding out what it is passing to it’s plugins isn’t easy to log.
A friend says that this works, haven’t tried it yet, but it seems logical.
strace -s 4096 /usr/local/nagios/bin/nagios -c /usr/local/nagios/etc/nagios.cfg 2>&1 | vim -
syntax on
set filetype=strace
So sometimes I’m on a box and need to test if a port is cleared on a firewall.. i may not yet have apache installed but I want to test if port 80 is open, or I may want to act as a server to grab a request..
simple with Netcat.
nc -l -p 8080
will open a listen on port 8080 and after the first connect, exit and go back to shell… useful for testing
A common monitoring system, Nagios, uses qpage for sending alphanumeric pages via a modem.. yes it sound insane, but think about what happens to your monitoring system when your internet connection drops, unless your using outside monitoring, your not going to get that email to send you a page.
I ran into an issue with the modem not dialing quickly enough and had a few tips for getting it going.
1) Make sure that your calling the correct dialing group.
2) Check the modem in minicom, load it up, and do a few
AT[ENTER]
OK
ATZ[ENTER]
OK
to make sure the modem is reset, and then do a
ATDT 2125551212[ENTER]
{it should pause and dial the number [in this case, 212-555-1212] and either tell you BUSY, CONNECTED, or error out.
Just thought I’d toss out a few reminders.
Having to move large sums of data between boxes is part of an admins life… and sometimes those machines are across the country, and your moving a bunch of sensitive data, and the bosses want to make sure no one can sniff at it… ok, whatever boss 
So here’s what I do, use friendly unix tar, and ssh pipe it to another machine.
Lemme show you some examples:
This shows you how to send files FROM the local machine TO a remote machine.
$ cd directory-with-files
$ tar cf - . | ssh username@remoteserver.com “cd /directory-to-put-files; tar xf -” [ENTER]
Ok, you may be asking me to explain this madness:
Change to a directory with files you want to transfer
tar cf - . - create a tar file, with the name “-”, which means output to screen, in the directory “.” (currently in, or right here)
| ssh username@remoteserver - setup an ssh connection to the remote server
“cd /directory-to-put-files; tar xf -” [ENTER] - once connected, change directorys on the local machine, and then EXTRACT that tar file named “-” or output to the screen.
In this case, the output to the screen will be piped thru ssh to the other computer and boom your set.
$ cd directory-to-put-files
$ ssh username@remoteserver.com “cd /directory-to-get-files; tar cf - .” | tar xf - [ENTER]
Try it out 
I’ve been caught too many times without good backups, so I’ve started to be a bit smarter about things… I try and automate and cron all the backups early on in a servers existance and just have one big backup drive on a server somewhere that keeps everything.
Here’s an example of this is a simple site backup script that I run.
/root/scripts/syncserver
This runs rsync updating any new files on the filesystem to our backup server. Now I just create an ssh key for root to the other machine’s username and copy the entire box over… this isn’t a PRETTY restore if I had to use it, but at least all data, config files, new files, users, etc are saved on the other machine.
Modify paths as necessary.
I deal with alot of troubleshooting in my life, and one of the things I have to do is talk to end users who are having problems hitting the site.
I need them to tell me their IP address… which is a near impossibility for most people… so I’ve been using WhatIP.org to get that information.
I generally tell them to hit whatip.org in a browser and then send me the output of the site… it makes my life easier because then I can search thru error logs or access logs and determine what they’re doing, or do a traceroute and see if the bandwidth is having issues.