Bringing on new clients often flags previously unseen issues; on both the client side, as well as the hosting side of the equation. I’ve mentioned in past blog posts that I work for a large State Government body, and we’re bringing on new departments on a somewhat regular basis.

We are currently in the process of bringing on board a large department of about 500 employees, 50 or so servers, and an infrastructure consisting mostly of Juniper gear. They’re already part of the Telstra GWIP, so we needed Telstra to set up routing between sites, and required adding rules to both ours and their firewalls. This is where a few issues were identified.

From several of our workstations we found that something was trying to hit the new network using a source IP address from the 192.168.0.0/24 subnet (example subnet). While the client network uses 192.168.0.0/24 subnet, we do not–nor have we ever–used this subnet. Naturally our routers would not have known how to route to this network previously and would have simply dropped any return packets; this is no longer the case.

This newly identified problem required a resolution, and I took it as an opportunity to use and learn Cisco’s SPAN (Switch Port Analysis), which I had only read about but never used in the past.

I first identified the switch port that one of the culprit workstations was using with the show mac-address-table command as follows:

NULLSEC-SW1#show mac-address-table | include xxxx.xxxx.xxxx

I then set up a monitoring session on that port to send ingress and egress packets from the source port to the networked port that I was using with Wireshark to capture any packets (my laptop was connected to that port, listening in passive mode):


NULLSEC-SW1#config t
NULLSEC-SW1(config)#monitor session 1 source interface fa3/0/10
NULLSEC-SW1(config)#monitor session 1 destination interface fa4/0/48


Once I started monitoring the Wireshark feed, it became obvious that something was going crazy with HSRP Hello multicasts. So I logged myself another call to resolve this issue (unfortunately as it is the production environment, I couldn’t fix it without proper approval processes), and set a filter to not hsrp so I could actually read the packet flow.

After about 3 hours of capturing packets, Wireshark died with an “Out of Memory” error; I should have seen that coming. I then killed the SPAN session using the following command:

NULLSEC-SW1(config)#no monitor session 1

Because so much data was captured before Wireshark died, I had to split 5GB of data into smaller, manageable files. I wrote the following batch script to achieve this:

echo [+]========================================================[+]
echo [+]--------------| WireShark File Splitter |---------------[+]
echo [+]--------------| Author: Michael Howe    |---------------[+]
echo [+]--------------| Version: 1.0            |---------------[+]
echo [+]========================================================[+]

rem Define working directory
set wireshark="C:\Program Files\Wireshark\"

rem Set the file variable
set /P file="Enter File Name with Full Path: "

rem Obtain packet information for file
echo "Obtaining file information."
%wireshark%\capinfos.exe -c %file%

rem [Set the number of packets in the file
set /P max="Enter Number of Packets in File: "

rem Reset the b variable to 0
set b=0

rem Loop through file splitting into 500000 packets
FOR /L %%a IN (1,500000,%max%) DO (
SET /a b = %%a+499999
%wireshark%\editcap.exe -r %file% %file%_%%a-!b! %%a-!b!
)


Unfortunately monitoring that particular interface didn’t return any results, so the next logical step was to enable monitoring on the entire VLAN. I used the following to enable SPAN on the VLAN:


NULLSEC-SW1#config t
NULLSEC-SW1(config)#monitor session 1 source vlan 2122
NULLSEC-SW1(config)#monitor session 1 destination interface fa4/0/48


This time I thought ahead and had Wireshark split the packet dump into 200MB files. Alas, it didn’t crash again, and the files were much more manageable.

Split Wireshark Dumps into 200MB Files at Time of Capture.

Using the following filter on the 50GB+ of data that I had by the end of the venture, I found that there was no rogue application on the network, and notified the client.

ip.addr == 192.168.0.0/16 or ip.addr == 10.0.0.0/8

It turns out that the spoofed IP packets were coming from an external source.

Update: Upon further investigation, it has been determined that the address spoofing seen was sourced from misconfigured Telstra routers between our GWIP sites. Telstra has confirmed this and rectified the issues.