UnixSupport.com

Having to move large sums of data between boxes is part of an admins life… and sometimes those machines are across the country, and your moving a bunch of sensitive data, and the bosses want to make sure no one can sniff at it… ok, whatever boss

So here’s what I do, use friendly unix tar, and ssh pipe it to another machine.

Lemme show you some examples:
This shows you how to send files FROM the local machine TO a remote machine.

$ cd directory-with-files
$ tar cf - . | ssh username@remoteserver.com “cd /directory-to-put-files; tar xf -” [ENTER]

Ok, you may be asking me to explain this madness:
Change to a directory with files you want to transfer

tar cf - . - create a tar file, with the name “-”, which means output to screen, in the directory “.” (currently in, or right here)

| ssh username@remoteserver - setup an ssh connection to the remote server
“cd /directory-to-put-files; tar xf -” [ENTER] - once connected, change directorys on the local machine, and then EXTRACT that tar file named “-” or output to the screen.

In this case, the output to the screen will be piped thru ssh to the other computer and boom your set.

---

Now, in order to reverse the logic and transfer FROM a server TO the local machine:

$ cd directory-to-put-files
$ ssh username@remoteserver.com “cd /directory-to-get-files; tar cf - .” | tar xf - [ENTER]

Try it out

Ok, I feel stupid, however I never realized that I could search man pages!

For those that don’t know, unix offers a built in “manual” command called “man”… sometimes it’s helpful, sometimes it’s there just for reference. It’s easy

$ man command

in this case man ls

LS(1) FreeBSD General Commands Manual LS(1)

NAME
ls - list directory contents

SYNOPSIS
ls [-ABCFGHLPRTWabcdfghiklmnopqrstuwx1] [file …]

DESCRIPTION
For each operand that names a file of a type other than directory, ls
displays its name as well as any requested, associated information. For
each operand that names a file of type directory, ls displays the names
of files contained within that directory, as well as any requested, asso-
ciated information.[and on and on]

---

man is a great thing, however sometimes you end up forgetting the command your using, and thats when my new little friend helps out…

$ man -k search-scring[ENTER]

This is awsome, now I can find all the insane instances of locate man -k locate

cfree(3) - free up allocated memory
index(3) - locate character in string
locate(1) - find filenames quickly
locate.updatedb(8) - update locate database
memchr(3) - locate byte in byte string
mmap(2) - allocate memory, or map files or devices into memory
pthread_mutex_destroy(3) - free resources allocated for a mutex
rindex(3) - locate character in string
strchr(3) - locate character in string
strpbrk(3) - locate multiple characters in string
strrchr(3) - locate character in string
strstr(3), strcasestr(3), strnstr(3) - locate a substring in a string
usbhid(3), hid_get_report_desc(3), hid_use_report_desc(3), hid_dispose_report_desc(3), hid_start_par
se(3), hid_end_parse(3), hid_get_item(3), hid_report_size(3), hid_locate(3), hid_usage_page(3), hid_
usage_in_page(3), hid_init(3), hid_get_data(3), hid_set_data(3) - USB HID access routines
whereis(1) - locate programs
which(1) - locate a program file in the user’s path

Give it a whirl!

I’ve been caught too many times without good backups, so I’ve started to be a bit smarter about things… I try and automate and cron all the backups early on in a servers existance and just have one big backup drive on a server somewhere that keeps everything.

Here’s an example of this is a simple site backup script that I run.

/root/scripts/syncserver

---

/usr/local/bin/rsync -av –exclude “*_log” -e “ssh” / username@backup.server.com:/path-to-backups/`/bin/hostname`/

---

This runs rsync updating any new files on the filesystem to our backup server. Now I just create an ssh key for root to the other machine’s username and copy the entire box over… this isn’t a PRETTY restore if I had to use it, but at least all data, config files, new files, users, etc are saved on the other machine.

Modify paths as necessary.

I’ve been called in the middle of the night with “the server is slow” being the emphasis of the call. Sometimes it’s not obvious using standard utilties, or sometimes it only happens during certain times.

I’ve used a simple script to eliminate this issue, and run it from cron every minute or 5 minutes to give me output on whats going on with the system.

This is ment for temporary use only, or at least make sure you clear the files every once and a while, because they get pretty big, thus I wrote a cyclecheck script and even a cyclerebootcheck script to see why a machine rebooted.

---

check script:

#!/bin/sh
echo “——————-” >> /data/`/bin/hostname`.load
echo “——————-” >> /data/`/bin/hostname`.psaux
/bin/date >> /data/`/bin/hostname`.load
/bin/date >> /data/`/bin/hostname`.psaux
/usr/bin/w >> /data/`/bin/hostname`.load
/usr/bin/w >> /data/`/bin/hostname`.psaux
ps -awux >> /data/`/bin/hostname`.psaux

---

cyclecheck script:
mv /data/`/bin/hostname`.psaux /data/`/bin/hostname`.psauwx.old
mv /data/`/bin/hostname`.load /data/`/bin/hostname`.load.old

---

cyclerebootcheck script:

mv /data/`/bin/hostname`.psaux /data/`/bin/hostname`.psauwx.reboot
mv /data/`/bin/hostname`.load /data/`/bin/hostname`.load.reboot
echo `hostname` rebooted `date` | mail pageadmin@site.com

---

install script:

echo “*/1 * * * * root /root/scripts/check” >> /etc/crontab
echo “1 0 * * * root /root/scripts/cyclecheck” >> /etc/crontab
echo “/root/scripts/cyclerebootcheck” /etc/rc.local
killall -HUP crond

---

I place them all in /root/scripts and chmod -R 755 /root/scripts, then run /root/scripts/install to place these things into the crontab.

Then you can view the output in /data/hostname-of-server.load or /data/hostname-of-server.psaux each with a timestamped entry.

——————-
Wed Nov 2 00:02:00 EDT 2005
12:02AM up 123 days, 23:45, 0 users, load averages: 0.15, 0.28, 0.24
(what users are logged on listed here)
USER TTY FROM LOGIN@ IDLE WHAT
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 24498 0.0 0.0 984 224 ?? R 12:02AM 0:00.00 ps -awux

It works at least.. edit for directories you need, this is written for freebsd in this case.