Archive for ‘Sysadmin’ Category
Sysadmin »

Proving the Network is Not the Problem With iperf – Packet Life

datePosted on 15:35, November 24th, 2013 by Many Ayromlou

Proving the Network is Not the Problem With iperf – Packet Life: “When an application fails to perform as expected, the network is often the first thing blamed. I suppose this is because end users typically view the network as the sole limiting factor with regard to throughput, unaware of the intricacies of application, database, and storage performance. For some reason, the burden of proof always seems to fall onto networkers to demonstrate that the network is not at fault before troubleshooting can begin elsewhere. This article demonstrates how to simulate user traffic between two given points on a network and measure the achievable throughput.”


Copying large number of files between two Unix/Linux/OSX Servers

datePosted on 14:38, August 15th, 2012 by Many Ayromlou

Here are some quick tip(s) for copying a ton of files between unixy machines really fast. You’re probably thinking “why not use rsync?”…..well rsync can be miserably slow if your source or destination cpu is underpowered. You can always do a rsync after these commands to make 100% certain that everything checks out, but try using one of these methods for the initial copy:

  • One way of doing it is
    tar -c /path/to/dir | ssh user@remote_server 'tar -xpvf - -C /absolute/path/to/remotedir'

    You’ll be prompted for the remote servers password or you can use the private key of the remote server using the -i switch in the ssh command. This has the side benefit of preserving permissions. An alternate version of this command can also be used to locally move folder structures across mount points while preserving permissions: 

    tar -cf - -C srcdir . | tar -xpf - -C destdir


    cd srcdir ; tar -cf - . | (cd destdir ; tar -xpf -)
  • Another way of doing it with netcat (nc) is
    srv1$ tar -cfv - * | nc -w1 4321

    followed by

    srv2$ nc -l -p 4321 |tar -xpfv - 

    Note that you type the first command on the source machine and the second command on the destination machine.

  • Yet another way of doing it with socat utility is
    host1$ tar -cvf - * | socat stdin tcp4:host2:portnum

    followed by

    host2$ socat tcp4-listen:portnum stdout | tar -xvpf - 

    Note that you type the first command on the source machine and the second command on the destination machine.

Once your favourite process (above) is done you can do a quick rsync to tie up any loose ends.

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir

Rsync will now fly through the filesystem as 99.9% of the time, 99.9% of the files on the destination are good. And as always make sure you understand the commands before you use them…..and keep backups just in case :-).

OSX Server: Could not setup Mach task special port 9

datePosted on 12:57, August 26th, 2010 by Many Ayromlou[1] (0x10f860.cron[43786]): Could not setup Mach task special port 9: (os/kern) no access

If you’re seeing this warning/error in your OSX Server log files, it is more than likely caused by cron running jobs for mailman subsystem. Even if the Mail process is disabled in Server Admin, OSX will try to run these cron jobs. The way around this (only do this if you’re NOT running mail server or mailman mailing list manager on your box) is to comment out all lines in /usr/lib/cron/tabs/_mailman file (insert a # character at the beginning of each line that doesn’t have it). This fixed the problem for me…..hopefully it will also work for you :-).

MySQL Replication howto for Snow Leopard…

datePosted on 16:58, August 23rd, 2010 by Many Ayromlou

We recently upgraded our servers from old XServe G5’s running Tiger to the latest greatest running Snow Leopard. In this small howto I will deal with the procedure I followed to setup mysql master-slave replication between two new servers. I start out with two (master, slave) empty DB’s, setup the replication and then import my data from a third server (my old G5 X-Serve). Of course like anything else I will try to show you how to get out of — what I like to call — Steve Jobs Hell Holes :-). There are other ways of doing this procedure, for more info check out this page.

– First things first….the magic command that stops mysql server from command line, just in case you screw something up and need to restart (This should be used on your master and slave servers prior to them going live…..DO NOT USE THIS ON A LIVE SERVER):

sudo launchctl unload /System/Library/LaunchDaemons/org.mysql.mysqld.plist

This will stop the launch Daemon from continously launching mysql when trouble is brewing. This usually happens when you screw something up badly and the symptom is that the Server Manager Status for MySQL will say “Starting up“, but never changes to “Running“.

– Next I want to blow away my DB files on the master and slave (remember these are NOT production servers yet…..I’m still rebuilding them. You DO NOT want to remove the DB’s on your production servers). You’ll need to find the path in the “Settings” tab of the server manager, under “Database Location“. I removed everything in that directory (rm -rf *) from the command line. AGAIN, I CAN NOT STRESS THIS ENOUGH, BE CAREFUL WHERE YOU ISSUE THESE COMMANDS.

– I did this on both the master and the slave.

– Now on the Master change the following in /etc/my.cnf (You should have this file, if you’ve got a my.cnf.default, copy it my.cnf).


– On the Master still, check the “Allow Network Connections” check box in Server Manager and set the root password (for mysql) by pressing the “Set MySQL Root Password…” button. Then press “Save” followed by “Start MySQL“. This will initialize mysql with the default tables and setup the root password.

– Now on the Slave change the following in /etc/my.cnf (You should have this file, if you’ve got a my.cnf.default, copy it my.cnf).


– Back on the Master, you need to create a user (I call it repl) that has REPLICATION SLAVE privilege. Use the following two commands (make sure you replace and slavepass….LEAVE repl as the userid:

mysql> CREATE USER 'repl'@'' IDENTIFIED BY 'slavepass';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'';

– Next we need to flush the tables and issue a read lock (yes, just because we can):


– Still on the master we find out the current binary log file name and position:

mysql> show master status;
 | File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
 | mysql-bin.000004 |      340 |              |                  |
 1 row in set (0.00 sec)

Note down the filename and the Position number. We will need them later.

– Back on the Slave, go to Server Manager and set the root password (for mysql) by pressing the “Set MySQL Root Password…” button. Then press “Save” followed by “Start MySQL“. Connect to the DB as root and issue the following command:


This will setup the slave to talk to the Master and do it’s thing. While we are here, we might as well verify that the repl user can log into the master from the slave:

mysql -h -u repl -p

should prompt you for password and if you type in the ‘slavepass’ you assigned above, you should be able to get in.

– Back on the master unlock the tables:


– Last but not least on the slave, turn on the slave mode:


Done…..Now you can go to your production server and suck it’s brains (ahemm…..DB’s) out and import it into your master. Your slave should follow and replicate whatever you import into the master Server. I used the following command (you mileage might/will vary):

mysqldump -x -c --add-drop-table --add-drop-database -u root -p --databases dbname1 dbname2 dbname3 >goodies.sql

Transfer “goodies.sql” text file from your production server to the master server (ssh/scp/ftp….whatever).

mysql -u root -p < goodies.sql

MySQL: Transfering users and priviledges to a new server….

datePosted on 15:32, August 19th, 2010 by Many Ayromlou

So this was a great big mystery this morning. How the heck do you transfer the users and their privileges out of a old mysql server and “import” them into a new server. We recently upgraded from OSX 10.4.11 to a couple of spanking new Snow Leopard servers and during the mysql export/import cycle this issue came up. Well the simple answer is…..DON’T USE mysqldump on your mysql DB (you know the default DB that stores all your users and privileges. It’s a bad idea and will probably do more harm than good. Instead use the following procedure:

1) On your old server (the one that has your data/users/tables on it) issue the following command (replace YOUR dbadmin/root username and  password in the 2 appropriate places:

mysql -B -N --user=admin --password=yourpassword -e "SELECT DISTINCT CONCAT('SHOW GRANTS FOR ''', user, '''@''', host, ''';') AS query FROM mysql.user" |mysql --user=admin --password=yourpassword | sed 's/\(GRANT .*\)/\1;/;s/^\(Grants for .*\)/## \1 ##/;/##/{x;p;x;}'

The output of this command is something like this:

## Grants for admin@ ##
## Grants for fabrik@localhost ##
GRANT ALL PRIVILEGES ON `fabrikdb`.* TO 'fabrik'@'localhost' WITH GRANT OPTION;
## Grants for nerdlogger@localhost ##
GRANT ALL PRIVILEGES ON `nerdlogger`.* TO 'nerdlogger'@'localhost' WITH GRANT OPTION;
## Grants for research@localhost ##
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON `research`.* TO 'research'@'localhost';
## Grants for admin@localhost ##
## Grants for root@server.domain ##

2) Now you’re ready to selectively cut and paste the appropriate users and associated grant into a new mysql session (which you have to open) on the new server.


When SysAdmins Ruled The Earth…

datePosted on 11:18, August 6th, 2008 by Many Ayromlou

Just found this on Cory Doctrow’s blog…..When SysAdmins Ruled The Earth….It’s a great read/listen, specially if you’re a current/former SysAdmin.

Here are the different parts read during his podcast over a couple of sessions:

If you like to read it yourself here is the entire story. This is great stuff……..They even kill the CN Tower……Now you can’t ask for more than that people.

Four little Security tools you should install in Ubuntu

datePosted on 14:41, June 12th, 2008 by Many Ayromlou

These should probably also be installed under other linux distros (might already be). But for the sake of completeness here they are:

1) denyhosts: great little package that’s already 98% configured after apt-get install process. It runs as a daemon and monitors /var/log/auth.log file for unsuccessful ssh logins and takes measures to ban the originating IP in /etc/hosts.deny. The cool part is that it does not need access to firewall or anything. Config file is /etc/denyhosts.conf and is pretty self explanatory. Ubuntu package is called “denyhosts” and needs python to work.

2) chkrootkit: another little gem that you install via apt-get install process. Ubuntu package is called “chkrootkit”. After install do “man chkrootkit” for more info, but the gist of it is that when run from command line it uses it’s own utils (located in /usr/lib/chkrootkit) to see if the system is infected.

3) rkhunter: this util is really a giant shell script, but it’s really nice and easy to use. Again use Ubuntu package name “rkhunter” to install it. It’s config file goes into /etc/rkhunter.conf and is pretty nicely setup by default. Next run “rkhunter –update” to update the discription/signature files from their website, then run “rkhunter –propupd” to grab a snapshot of the various files installed on your system. This will be used later, every time you run the command to see if anything has been changed by trojans/rootkits. Finally run “rkhunter –check” to actually run all the tests and see if you’re good to go. At the end if there are warnings check /var/log/rkhunter.log for a list of explanations about those warnings (suspicious filenames, hidden file locations, etc.)

4) ufw: The netfilter (firewall) interface for the rest of us. If you’re like me too dense to remember the iptables lingo, this might be for you. See this page for a good introduction.

Have fun and remember kids Vitamin U(buntu) is good for you.

Changing DNS hostname on OSX Server

datePosted on 13:17, December 15th, 2007 by Many Ayromlou
If you ever need to check the DNS hostname under OSX server here is the command:

$ sudo changeip -checkhostname
If you need to change it (ie: after a DNS table change) use the following command (for OD sites):
changeip /LDAPv3/
changeip also has a lot of other uses (ie: if you change the ip address of your machine you can use changeip to change the setting in the directory as well). Have a look at the man pages for more info.

If you’re having trouble kerberizing your Open Directory under OSX server then you are also most likely having hostname problems. Run the first command to check the hostname information and make sure “Current Hostname” and “DNS Hostname” in it’s output match. If they don’t use the second command to change (fix) it (assuming your DNS is working properly). Then you can go to Server Admin and under Open Directory/Settings press the Kerberize button that Pops up on the lower left.