How to defend against ARP poisoning/spoofing attack in vSphere infrastructure

There are few vSphere Infrastructure enterprise possibilities how to deal with this type of attack.
I know about two ... Vmware vShield  and CISCO Nexus1000v.

However here I would like to share idea how to do it with open source tools integrated into enterprise infrastructure.

Disclaimer: 
Please be aware that this is not out of box enterprise solution and you have to know what you are doing and you have full responsibility for all impacts.

How we can simulate the attack?
Bellow is tutorial inspired by another tutorial from
http://blog.facilelogin.com/2011/01/arp-poisoning-with-dsniff.html
You can simply change installation procedures based on your OS distribution.

ARP poisoning with dsniff
dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.

To install dsniff on CentOS 6.

yum -y install wget

cd /usr/src
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
rpm -ivh epel-release-6-7.noarch.rpm
yum –y install openssl gcc flex bison libpcap-devel libnet

yum install dsniff

Now you need to find out two things,

1. IP address of the target machine - say 192.168.1.4
2. IP address of the Gateway - say 192.168.1.1

Let's start ARP poisoning from the attacker's machine - with arpspoof tool which comes with dsniff.

$ sudo arpspoof -i en1 -t 192.168.1.4 192.168.1.1

This will update target machine's ARP table with attacker's MAC address against the IP address of the gateway.

Now - start a tcpdump on the same interface from your machine - start viewing all the traffic going to and from the target machine.

$ sudo tcpdump -i en1

How we can detect the attack?
We can use aprwatch for example on my favorite OS FreeBSD running in virtual machine,

Installation is simple as
cd /usr/ports/net-mgmt/arpwatch/
make install
...
then you have to add
arpwatch_enable="YES"
in to your /etc/rc.conf
...
 
And last but not least is to enable promiscuous mode on VMware vSwitch portgroup where arpwatch vm is connected to. The best way is to create another portgroup (single port is enough) with the same VLAN ID as protected VLAN and in Security we have to set Promiscuous Mode to Accept. 

... then arpwatch send message to syslog. So you can see something similar in your /var/log/messages
Oct 31 09:08:02 ips arpwatch: flip flop a.b.c.d 0:50:56:8d:2e:bc (54:52:0:fe:47:95)

Arpwatch can also send a e-mail message about incident. The message looks like

hostname:
ip address: 95.80.240.1
ethernet address: 54:52:0:fe:47:95
ethernet vendor:
old ethernet address: 0:50:56:8d:2e:bc
old ethernet vendor: VMWare, Inc.
timestamp: Wednesday, October 31, 2012 8:57:33 +0100
previous timestamp: Wednesday, October 31, 2012 8:57:33 +0100
delta: 0 seconds

How we can protect against the attack?

Well this is another story. It really depends on your environment but in my environment I have vSphere Distributed Switch and all virtual machines have VMtools installed so I trust VMware MAC:IP associations and based on these information (MAC) I can find the attacker port in distributed switch and disable it.

And that's just a small step to do it in automated way by leveraging VMware vCLI (aka VMware Perl SDK).

WARNING!!!

Currently described solution works only on single ESX host and doesn't work among more ESX hosts because of virtual networking principles. VMware virtual switch (or module of distributed switch) is not a switch but port extender. The difference is significant. The ARP poisoning attack is based on permanently unicasting ARP replies to the victim so when Arpwatch server is not on the same ESX as the attacker or the victim then even promiscuous port on distributed switch portgroup will not help us to catch it because arp reply packet is not visible on Arpwatch server.

The solution would be to have arpwatch VM on each ESX host. VMware introduced the concept of ESX agents (aka Agent Virtual Machines) so I believe this is the right use case for ESX agent implementation.
For more information about "Deploying vSphere Solutions, vServices, and ESX Agents" read this document.

I have to test this architecture ... so stay tuned.

Any comments or thoughts are appreciated.

Accessing Microsoft SQL Server from Linux using DBD::Sybase

Citation from: http://www.perlmonks.org/?node_id=392385
Author:  Lindsay Leeds (2004 Sep 20)

Recently, I made yet another attempt to get Perl to access Microsoft SQL Server using DBD.  Usually, when I want to connect to a Microsoft SQL Server, it is from Perl on Windows.  So I take the easy route and use DBD::ODBC and use an ODBC connection.  This time though, I wanted to connect to Microsoft SQL Server 2000 from a Linux box.  Having no ODBC to fall back on, I looked for native DBD driver of some sort.
It took me several hours of struggling to make it work.  I almost gave up several times, so I am writing outline to help anyone else trying to accomplish this same task.
In the end, we will use the DBD::Sybase perl module from CPAN to access the Microsoft SQL Server.  Before we can do that however, we must first compile the freetds library.

Note: From now on I will refer to Microsoft SQL Server as SQL Server.  Please do not confuse this with a generic sql server.  We can all now pause to gripe about the lack of imagination in product naming at Microsoft.
Compiling Freetds
Download and compile freetds from http://www.freetds.org/.

once you unzip and untar it, enter the directory and run:

./configure --prefix=/usr/local/freetds --with-tdsver=7.0
make
make install

Configuring Freetds
Now we have the freetds compiled, but we still have configure it.  This is the part that threw me off and is so different from other DBD drivers.  The DBD::Sybase driver will ultimately be affected by the contents of the /usr/local/freetds/etc/freetds.conf file.  If that file is not configured correctly, your DBD::Sybase connection will fail.
Okay, now that we have established there is a  relationship between the freetds.conf file and the DBD::Sybase module, let's edit the freetds.conf file.
The strategic modifications I made to the freetds.conf file were:
1) uncomment the following lines and modify if necessary:

try server login = yes
try domain login = no

Note: this forces the module to attempt a database login instead of a domain login.  I could not get domain login to work, though I will admit I did not try very hard.
2) uncomment the following line and modify if necessary:

tds version = 7.0

This supposedly sets the default tds version to establish a connection with.  I have only SQL Server 2000 servers, and they won't talk at any lower version.  So I set it to 7.0.  If for some reason you had older SQL Servers, you might leave it at the default 4.2.
3) create a server entry for my server sql1:

[sql1]
        host = sql1
        port = 1433
        tds version = 8.0
    


[download]

Note: My server here is sql1.  Ping sql1 worked, so I am sure I can resolve it using DNS.  You can also specifcy an ip address instead of the host name.  The sql1 in the brackets is just a descriptor.  It could be 'superduperserver' and it would still work as long as my 'host =' is set correctly.  I tried 'tds version 7.0' for my SQL Sever 2000 and it worked.  Version 5.0 though resulted in an error.  You might want to verify your SQL Server is listening on port 1433 with a 'netstat -a -n' run from the command line on the SQL Server.
At this point you can verify your configuration.

/usr/local/freetds/bin/tsql -S sql1 -U sqluser

You will then be prompted for a password and if everything is well, you will see a '1)' waiting for you to enter a command.  If you can't get the 1) using tsql, I doubt your DBD::Sybase perl code is going to work.  Please note that sqluser is not an Active Directory/Windows Domain user, but an SQL Server user.
Compiling DBD::Sybase
Now that we have the freetds library prerequisite for DBD::Sybase installed and configured, we can compile the DBD::Sybase perl module.  Obtain it from www.cpan.org if you haven't already.
once you have untarred it and are in the directory, run:

export SYBASE=/usr/local/freetds
perl Makefile.PL
make
make install

Note: The export line is to let the compilation process know where to find the freetds libraries.

Using DBD::Sybase
You are now ready to test your DBD::Sybase module.

#!/usr/bin/perl

use DBI;

$dsn = 'DBI:Sybase:server=sql1';

my $dbh = DBI->connect($dsn, "sqluser", 'password');
die "unable to connect to server $DBI::errstr" unless $dbh;

$dbh->do("use mydatabase");

$query = "SELECT * FROM MYTABLE";
$sth = $dbh->prepare ($query) or die "prepare failed\n";
$sth->execute( ) or die "unable to execute query $query   error $DBI::errstr";

$rows = $sth->rows ;
print "$row rows returned by query\n";

while ( @first = $sth->fetchrow_array ) {
   foreach $field (@first) {
      print "field: $field\n";
   }
}

Good luck!

Adding ESXi 5.1 to "5.1 vCenter Appliance" fail

Finally I found time to install vSphere 5.1 in my home lab. I have 5.0 environment running so I've bought another old DELL PE 2950 on czech "ebay like" system Aukro (www.aukro.cz) for just 6.500 CZK (approx. 330 USD) to leave my current lab untouched and try 5.1.

So, I upgraded BIOS and DRAC to latest firmwares and installed DELL version of ESXi 5.1 (embedded) on my DELL PE 2950. Then I installed vCenter appliance (OVF) on top of this new ESX.

I was able to add my old ESX5.0 to this new vCenter but not the new one.

WHAT'S WRONG???

Troubleshooting process ...

In vCenter event log I can see message: "A general system error occurred: Timed waiting for vpxa to start".

In /var/log/vpxa.log ...

2012-10-07T08:33:29.941Z [FFFE1B90 error 'SoapAdapter'] Unsupported namespace "urn:vpxa3" in content of SOAP body
-->
--> while parsing SOAP body
--> at line 9, column 0
-->
--> while parsing SOAP envelope
--> at line 2, column 0
-->
--> while parsing HTTP request before method was determined
--> at line 1, column 0

So, what  versions am I running?


ESXi 5.1.0 (799733)
vCenter Appliance 5.0.0 (755629)


Oooops ... I believed I installed vCenter 5.1 because in my local install archive it is stored as
VMware-vCenter-Server-Appliance-5.1.0.5100-799730_OVF10


So the cause is clear ... bad installation image management cost me one hour of troubleshooting :-(

NAKIVO - another virtual infrastructure backup software

NAKIVO (http://nakivo.com) is another virtual infrastructure backup software. It can be installed on Windows or Linux (Ubuntu) server. Linux installation is something which interest me. I have to test it and compare it against Veeam Backup and Replication. 

New Nexus 1000v (2.1) will be also available as free edition

Source

Nexus 1000v version 2.1 will have (2.1 is currently beta) two editions. Essential edition is free of charge so VMware Enterprise Plus customers can leverage CISCO virtual networking. Advanced edition is paid version but with significantly enhanced features. The most interesting thing is that VSG (Virtual Security Gateway) is also included in Nexus 1000v advanced edition.

Enabling Nested ESXi in vSphere 5.1

Nice article how to check physical ESX host capability to virtualize ESX (aka nested ESX).

esxcli for vSphere 5

Excellent introduction into esxcli.

Automating ESXi 5 Kickstart Tips & Tricks

Here is the link to excellent blog post.

iReasoning MIB browser - Free MIB Browser

iReasoning MIB browser is a powerful and easy-to-use tool powered by iReasoning SNMP API . MIB browser is an indispensable tool for engineers to manage SNMP enabled network devices and applications. It allows users to load standard, proprietary MIBs, and even some mal-formed MIBs. It also allows them to issue SNMP requests to retrieve agent's data, or make changes to the agent. A built-in trap receiver can receive SNMP traps and handle trap storm.

Major features:

    Intuitive GUI
    Complete SNMPv1, v2c and v3 (USM and VACM) support
    Complete SNMPv3 USM support, including HMAC-MD5, HMAC-SHA, CBC-DES, CFB128-AES-128, CFB128-AES-192, CFB128-AES-256 (128-bit, 192-bit and 256-bit AES) algorithms
    Robust and powerful SMIv1/SMIv2 MIB parser
    IPv6 support
    Trap Receiver
    Trap Sender
    Log window to display application log and SNMP packets exchanged between browser and agents
    Port view (bandwidth utilization, error percentages) for network interface cards
    Switch port mapper for mapping switch ports
    Table view for MIB tables
    SNMPv3 USM user management (usmUserTable in SNMP-USER-BASED-SM-MIB)
    Device snapshot
    Cisco device snapshot
    Performance graph tool for monitoring of numerical OID values
    Ping and traceroute tools
    SNMP Agents Comparison
    Network discovery tool
    Runs on Windows, Mac OS X, Linux and other UNIX platforms

http://ireasoning.com/mibbrowser.shtml

Note: another free MIB browsers is getif, mibble