Host Discovery and Initial Organization

Note: Date is incorrect. I THINK I’ve already converted this into another document (taking notes during a pentest?). Check that.

Discovering hosts on an internal network is one of the first things you’ll do as part of a penetration test, assuming all the initial formalities have been completed. I’m going to go through a few techniques for discovering hosts using Nmap, and crucially — how to organize the data that comes back from these scans.

Here’s a summary of the steps I’m going to take in this article. This will form part of a greater methodology for penetration testing which you can use and modify for your own purposes.

Nmap ping sweep

Ping Sweep

One method of discovering hosts on a network is to use a ping sweep, also known as a ping scan in the Nmap reference guide (man nmap).You can achieve this with the following command:

nmap -sn 192.168.56.0/24

That previous command results in the following, in my case:

Nmap scan report for jx395 (192.168.56.1)
Host is up (0.00061s latency).
Nmap scan report for 192.168.56.106
Host is up (0.00087s latency).
Nmap scan report for 192.168.56.107
Host is up (0.00063s latency).
Nmap scan report for 192.168.56.108
Host is up (0.00065s latency).
Nmap scan report for 192.168.56.109
Host is up (0.0014s latency).
Nmap done: 256 IP addresses (5 hosts up) scanned in 2.38 seconds

I have 4 virtual machines running on my system, and I now know the IP addresses of these hosts. The first line in the previous scan result represents my host machine, which is named jx395.

You should note the difference in results when running Nmap as a regular user, and as the root user. Here’s the same command again but prefaced with sudo:

sudo nmap -sn 192.168.56.0/24

Nmap scan report for 192.168.56.100
Host is up (-0.10s latency).
MAC Address: 08:00:27:4C:03:FE (Oracle VirtualBox virtual NIC)
Nmap scan report for 192.168.56.106
Host is up (0.00031s latency).
MAC Address: 08:00:27:47:3C:5B (Oracle VirtualBox virtual NIC)
Nmap scan report for 192.168.56.107
Host is up (0.00055s latency).
MAC Address: 08:00:27:33:4A:2D (Oracle VirtualBox virtual NIC)
Nmap scan report for 192.168.56.108
Host is up (0.00022s latency).
MAC Address: 08:00:27:52:FE:A7 (Oracle VirtualBox virtual NIC)
Nmap scan report for 192.168.56.109
Host is up (0.00025s latency).
MAC Address: 08:00:27:D8:88:E2 (Oracle VirtualBox virtual NIC)
Nmap scan report for jx395 (192.168.56.1)
Host is up.
Nmap done: 256 IP addresses (6 hosts up) scanned in 3.23 seconds

nmap -sn 192.168.56.0/24

Scanning for Web Servers

Another method for finding hosts is to assume that all hosts on the network are web hosts, and have Nmap run a port scan on TCP port 80 for each IP address:

sudo nmap 192.168.56.0/24 -p 80

It looks like there are a few hosts with port 80 open on my virtual network:

Nmap scan report for 192.168.56.100
Host is up (-0.078s latency).
PORT   STATE    SERVICE
80/tcp filtered http
MAC Address: 08:00:27:4C:03:FE (Oracle VirtualBox virtual NIC)

Nmap scan report for 192.168.56.106
Host is up (0.00040s latency).
PORT   STATE SERVICE
80/tcp open  http
MAC Address: 08:00:27:47:3C:5B (Oracle VirtualBox virtual NIC)

Nmap scan report for 192.168.56.109
Host is up (0.00032s latency).
PORT   STATE SERVICE
80/tcp open  http
MAC Address: 08:00:27:D8:88:E2 (Oracle VirtualBox virtual NIC)

Nmap scan report for jx395 (192.168.56.1)
Host is up (0.000061s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 256 IP addresses (4 hosts up) scanned in 5.57 seconds

Note: If none of these host discovery techniques are chosen, Nmap uses a default which is equivalent to the -PE (ICMP echo) -PS443 (SYN) -PA80 (ACK) -PP (ICMP timestamp) arguments for Windows or privileged (root) Unix users. Attentive readers know that this means an ICMP echo request, a TCP SYN packet, a TCP ACK packet, and an ICMP timestamp request are sent to each machine. An exception to this is that an ARP scan is used for any targets which are on a local ethernet network. from https://nmap.org/book/host-discovery-techniques.html

The Right Combination of Pings

For a typical internal network, the default Nmap discovery option should be sufficient. But, to be sure that you have discovered all hosts on a network, or within a range, it is important to be certain you’ve left no stone unturned.

The -PS, -PA, -PU and -PY options stand for portlist… SYN, ACK, UDP and SCTP respectively.

The official Nmap book (https://nmap.org/book/host-discovery-strategies.html#host-discovery-ideal-probes) recommends the following combination:

-PE -PP -PS21,22,23,25,80,113,443,31339 -PA80,113,443,10042. Adding in --source-port 53 might be worthwhile as well.

Scanning Commonly Used Ports

The --top-ports option, when followed by an integer (a whole number) will conduct a port scan of the top ports as perceived by Nmap…

To truly catch all hosts on a network, you would have do something like scan every single port, of every single potential host on the network. Something like this:

nmap -sT -Pn 192.168.56.0/24 -p-

The previous scan will take a long time to complete and will need to be run in the background, along with your UDP scan(s).

Methodology

Here’s one suggested methodology you could use:

Perform a ping sweep using sudo nmap -sn 192.168.56.0/24.
Perform a second ping sweep using sudo nmap -sn -PE -PP -PS21,22,23,25,80,113,443,31339 -PA80,113,443,10042 192.168.56.024
Perform a basic (default) scan of all hosts found, but with ping disabled (you know they are there at this point): sudo nmap -Pn -iL targets.list
Perform a basic scan of the entire range, most common ports only: `sudo nmap 192.168.56.0/24 –top-ports 10
Do scans with the -sT option also.
Perform the above but using IPv6.
Scan common UDP ports (53 etc)
Scan all UDP ports

Organizing Results

Recording all the results in a meaningful way is as important and problematic as conducting the scans. You will need to review all the information that comes back from these scans in order to start carrying out more targeted probes.

I like to use a simple approach of flat files, organized into folders, with each folder representing a host. Within each host’s folder will be stored sub-folders, such as:

OSINT
DNS
Nmap
Web
Users

You could have a number of markdown documents for different purposes too:

index
my_notes
vulnerabilities.md

Then, a bash script can be used to compile this information into a series of web pages, one for each host, allowing you to easily view the information.

Creating a Folder Structure With Bash

test_root (start date of engagement e.g. 2020-06-03)
- targets.txt
- /IPAddress
  - notes.md
  - basic.gnmap
  - basic.nmap
  - basic.xml

First create a new directory for containing all other files, and host directories. You could base the name of this root folder on the start date of the engagement. Then run an initial Nmap scan, to discover hosts. I like to save the results as Nmap’s grepable format at this point. I’ll use a descriptive name and add .gnmap to the end, so I know this file contains grepable Nmap output.

nmap -sudo sn 192.168.1.0/24 -oG sn.gnmap

Next, I will use grep and awk to filter out a list of IP addresses:

cat sn.gnmap | grep Host: | awk -F " " '{print $2}'

The result of that operation was 6 IP addresses:

192.168.1.65
192.168.1.66
192.168.1.67
192.168.1.69
192.168.1.70
192.168.1.254
192.168.1.139

You can carry out further discovery scans at this point, and save the results to different files. Then cat the files, and re-run the previous grep and awk command:

cat sn.gnmap discovery2.gnmap discovery3.gnmap | grep Host: | awk -F " " '{print $2}' >> targets.list

In the previous command I’ve output the results to a file called targets.list. I’ve used .list to remind me that this is a simple list. You can run further discovery scans, and append them (using >>) to that targets.list file. You will need to sort them, and remove any duplicate entries using the next command, which also puts them in order!

cat sn.gnmap | grep Host: | awk -F " " '{print $2}' | sort --unique -t . -k 3,3n -k 4,4n

To create directories based on this list, use:

for target in `cat targets.list`; do 
    mkdir $target; 
    touch $target/notes.md;
    touch $target/hostnames.list;
done

The previous command also creates a file — notes.md — in each directory, but only if the file does not already exist. Touch will not overwrite that file.

The results of ls will not be in order. If this bothers you then you could set up an alias that sorts the output of ls as shown previously.

Hostnames

A simple way to get the hostname for each host, based on the IP address, is to use host:

for target in `cat targets.list`; do 
    host $target | awk -F " " '{print $5}' >> $target/hostnames.list; 
done

Running Further Scans

Now that you have your directories set up, you can run scans, and save the results to the relevant directory:

for target in `cat targets.list`; do sudo nmap -Pn $target -oA ./$target/basic; done

Creating a Summary of Results

Rather than cd into each directory at this point, run another bash script that pulls out the results you want and presents them to you in a human friendly format. Something like this can be built upon:

#!/bin/bash

for target in `cat targets.list`; do

    echo \#\# $target;
    echo "* Hostname(s):" `cat $target/hostnames.list`;
    echo "* Operating System: " `cat $target/operating_system.txt`;
    echo;
    echo "[Open Local Directory in Web Browser](file://$target/)";
    echo;
    echo "HTTP: [http://$target/](http://$target/)";
    echo;
    echo "HTTPS: [https://$target/](https://$target/)";
    
    cat $target/notes.md;

    echo \#\#\# Basic Nmap Results;
    echo '```';
    cat $target/basic.nmap;
    echo '```';

done;

After creating your results.md file, convert this to an HTML file so that you can view it in your browser. You can do this with pandoc. If you have a stylesheet, you can specify it here.

pandoc --metadata=title:"Penetration Test | 2020-06-03" --standalone --css=style.css -f commonmark -o results.html results.md

Time to Move On

It is important to have some sort of system in place to prevent you getting stuck on a single host for too long. Once you’ve gathered information on a group of hosts, you will want to focus on each one in turn. But, you do not want to spend too long on each host, since it may not be possible to get a shell on that host with the information you currently posses (if at all). This does not mean you cannot make notes on anything useful you see.