When a security analyst performing website penetration testing the initial step should be finding hidden directories of a vulnerable website.

These hidden web directories are essential because they can give useful information i.e. potential attack vectors that would not be visible on the public facing website.

One of the ways to achieve this is by attempting brute-forcing site structure that includes directories and files in websites and for that, you have to choose a powerful tool.

Although there are many tools available used to perform site brute-forcing includes Dirbuster or Dirb but these have their own limitation such as Dirbuster only offers GUI interface that is not feasible all the time and Dirb does not include multithreading feature.

The most popular choice among penetration testers for website brute-forcing is Dirsearch.

Dirsearch, written in python is a command-line website directory scanner. It has a lot of features making it the complete winner in terms of performance:

1. It includes Multithreading, making it faster than any other site scanner tool
2. It performs Recursive brute-forcing
3. It has HTTP proxy support
4. Dirsearch effectively detects invalid web pages
5. It has User agent randomization and Batch processing
6. Supports Request delaying

This tool can be run on any operating system (Windows, Linux, mac) making it more compatible and simpler, yet a powerful tool.

In this setup we will be using Kali Linux as an attacking machine and DVWA on Metasploitable 2 as the target.

Install Dirsearch

You need to install Dirsearch in your Kali Linux attacking machine. First update all the repositories from the command # apt-get update

After that install Dirsearch from Github by using the link: https://github.com/maurosoria/dirsearch

Type command:

# git clone https://github.com/maurosoria/dirsearch

After successfully installing the tool, change the root directory into Dirsearch.

# cd dirsearch/

Type ls to see the content under Dirsearch directoy.

You can see there is a configuration file and a python file named ‘dirsearch.py’ which we are going to execute.

Configuring the Tool

Next step will be configuring the Dirsearch. You can do it in three ways.

As this ‘dirsearch.py’ file needs python 3 to execute correctly, just simply run it with python.

# python3 dirsearch.py

It gives a usage example, stating we need to specify a valid URL.

The second way to run Dirsearch is to run it with Bash. Type ls -la to give us the permissions of everything in this directory.

As we can see from above that this tool is executable, now run it using the dot-slash.

# ./dirsearch.py

The third and the easiest way to run Dirsearch is by creating a symbolic link in the /bin directory as this will permit us to run this tool not only with the directory cloned from GitHub but anywhere in the prompt.

Change into the bin directory and create a symbolic link naming it ‘dirsearch’ by using the ln -s command.

# ln -s ln -s ~/dirsearch/dirsearch.py dirsearch

Site Scanning Using Dirsearch

As we have created the symbolic link, just type dirsearch in any directory to execute.

Use the flag ‘-h’ with the dirsearch to get full help menu that will show a ton of options and potential configuration settings.

So now for searching the site it needs a valid target’s URL and a file extension to run. You can specify a valid URL with the -u flag, and a file extension to search for with the -e flag.

In our case we will give it the URL of our DVWA on Metasploitable2 machine.

# dirsearch -u http://192.168.0.102/dvwa -e php

This showing us a lot of information regarding  the extensions, HTTP methods in use, number of threads, and size of the current wordlist. After that it starts to dig into the directories and returns with its findings that includes the status code, size, and directory name.

If you want to exclude certain HTTP status codes use -x flag.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403

Here we are excluding all the 403 codes by specifying after -x flag. It leads us to get what we required, making it cleaner and simpler.

Also, you can use -w flag to use a wordlist of your choice.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403,301,302 -w /usr/share/wordlists/wfuzz/general/common.txt

You can see it does not find a large pool of results with this mentioned wordlist because the size is smaller.

Now coming to its best feature i.e. performing recursive directory scanning. Use -r flag to run the recursive search.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403,301,302 -r

Now see there is an additional entry of ‘Recursive Level’. On completing the initial scan this will go back and starts scanning each directory it found recursively, in this case it starts config/ then docs/ and so on.

While scanning if you want to exit the scan parse ‘e’ from keyboard. To continue it from the point you have stopped, parse ‘c’. Parse ‘n’ to move to the next directory. These steps will let you have a control over the results as recursive scanning is a time-consuming process.

If you want to set the recursion level to a deeper value, you can use the -R flag and a value of how many levels deep you want to dig.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403,301,302 -r -R 3

So, you can see this starts scanning the /administrator directory which is obviously could not be found on the top level.

Conclusion

This article demonstrates how to use an effective web directory brute-forcing tool Dirsearch to scan and search hidden web directories which may not be visible to a user. This is very a very first task in penetration tasting and Dirsearch do this job much faster than the traditional Dirbuster and Dirb.