Server monitoring with icinga

What is Icinga?
From the website, icinga.org,
 "Icinga is an enterprise grade open source monitoring system which keeps watch over networks and any conceivable network resource, notifies the user of errors and recoveries and generates performance data for reporting"

If you are familiar with nagios, then you will most likely feel comfortable with icinga. Actually, icinga is a fork of nagios and uses its plugins for achieving many of its tasks.This system provides a very nice interface for monitoring systems and services on your network.

Installation and setup:
Installation will more than likely go smoothly if you can follow the instructions from icinga's documentation/quick start guide. I beleive there are packages for icinga in the debian/ubuntu repositories, so you can probably install by doing, apt-get install icinga. However, I installed it from it's sources by following the quickstart guide here.

NOTE: If you decide to create a different user other than icingaadmin from the command:

htpasswd -c /usr/local/icinga/etc/htpasswd.users icingaadmin

 , you may run into minor set backs like i did. When i logged into the web interface with the unique username i created, i was unable to see the default localhost system being monitored. To get things working, i had to modify some files, and basically add my username to certain config directives so i have the relevant permissions to view/retrieve information. Specifically, the cgi.cfg config file in the icinga's etc directory. Below are the directives that i modified. Note that the user icingaadmin is there by default, so i only appended my username.

authorized_for_system_information=icingaadmin,myusername

authorized_for_all_services=icingaadmin,myusername

authorized_for_all_hosts=icingaadmin,myusername

authorized_for_all_service_commands=icingaadmin,myusername

authorized_for_all_host_commands=icingaadmin,myusername

To add remote hosts to be monitored is also simple. I encourage you to read up on this post at howtoforge.com

Basically, if we wanted to add a system to be monitored that will only check for the system being up/alive (ping checks), we can create a config file for this host in the icinga's etc/modules directory. For example, we can name the file dhcp_server.cfg (files must end in .cfg extension).

Here is a sample of the config file contents for a simple ping host check.

define hostgroup{
     hostgroup_name     windows-machines
     alias                         window machines
     members                  dhcp_server
}

define host{
     use                           windows-server
     host_name               dhcp_server
     alias                         dhcp server
     address                     192.168.1.2
     hostgroups                windows-machines
}

define service{
     use                            generic-service
     host_name                dhcp_server
     service_description ping host
     check_command      check_ping!100.0,20%!500.0,60%
}

Save this file the restart the icinga daemon: service icinga restart. If there are no errors all should be well. Log into the web interface and confirm that your new host is being monitored. If the host at 192.168.1.2 is down, then icinga's web interface will indicate so with the red color. The green color indicates that the host is up.

Resources/Good Reading:
www.icinga.org
howtoforge.com icinga article

More password cracking fun, with a twist.

The twist is, we wont be utilizing our CPU or GPU in the traditional sense of cracking a password hash. Let me explain.

In traditional password cracking, you have a hash file with hashes that you may have dumped from a database or operating system (like the sam file in windows or /etc/shadow in linux). You fire up your favorite password cracker like john the ripper, point to it your hash file and a wordlist then john does the rest. What's happening underneath the hood is that each word in the wordlist is ran through an algorithm (say md5 for example) and is thus converted to its hashed form. This hashed form of the word is then compared to the hashes in the hash file that was supplied to john the ripper. If there is a match, then the password was cracked.

Password crackers utilize the CPU or is some cases, the GPU, to do the computational work of converting the words in the wordlist files to their respective hash representations and then comparing them to a hash list. The amouont of words processed per second is dependent on the speed of the CPU or GPU and its not uncommon for password crackers like john and hashcat to process thousands of words per second.

Today, most of this processing work has gone into the cloud. There are many websites that offer password cracking services. You submit a hash, and withing a few seconds, if the hash is found, you will receive its plain text. Most of these services utilize rainbow tables which make for faster hash lookups, thus minimizing the time it takes to receive an answer.

Program: findmyhash.py

While exploring some of the new tools in the recently released bactrack 5 R2, i came across this tool. This tool will accept a hash or a list of hashes in a file and the hash type. It will then utilize various web resources to find you the resulting password in plaintext (this means an internet connection is required).

Usage is simple.
# python findmyhash.py MD5 -h "482c811da5d5b4bc6d497ffa98491e38"

Cracking hash: 482c811da5d5b4bc6d497ffa98491e38

Analyzing with md5hashcracker (http://md5hashcracker.appspot.com)...

***** HASH CRACKED!! *****
The original string is: password123

The following hashes were cracked:
----------------------------------

482c811da5d5b4bc6d497ffa98491e38 -> password123

Another useful utility is the hash identifier python script, which now comes bundled with hashcat in BT5 R2. This script aims to identify the hash type (md5, lm, ntlm, sha-1, etc)of given a hash. Just run the script and enter a hash and it the program will do its very best to identify the hash type. The findmyhash utility takes a hash type as one of its arguments so this hash identifier tool can be used to determine this.

usage: # python hash_id.py

Detecting xss vulnerabilities for cookie harvesting with owasp-zap

In my previous post, i demonstrated how one can use an automated tool like proxystrike to detect sql injection vulnerabilites. In this post, i will use a different tool called owasp-zap (my favorite interception proxy, only cause its free and burpsuite isn't completely free :( ). At its heart, zap (zed attack proxy), is very similar to tools like paros and burp suite. It can intercept http(s) request and responses, giving you the ability to modify the information containted in the requests and responses. It is also a web application scanner, that can look for common vulnerabilites like sql injection and xss. Now that we got that out of the way, onto the demo.

Like the previous post, we will be using the vulnerable web app, DVWA. We will run owasp-zap proxy and then re-configure firefox's proxy settings to run through the proxy (i do this through a firefox plugin called foxyproxy). I then browse the web app, then created a post using the guestbook feature. After the post was create, i instructed zap to perform a scan, using the POST request as the template (note that you can configure a scan policy that dictates what vulnerabilities to look for. From the menu's, Analyze -> Scan policy). The POST request was the one that was sent when we submitted the information we entered in the guestbook.

When zap finds a vulnerability, its is logged in the alert tab. Using the information that I found about the vulnerability in the alert tab, i then go onto inject some script, that would allow us to harvest session cookies. These cookies can then be used to hijack the sessions of other logged on users.

Using proxystrike to deteck sql injection vulnerabilities

I've touched on exploiting sql injection in previous blog posts. I gave some tips on how to manually test inputs for sql injection, but this is a very time consuming process. There are many potential injection points in one web app, and manually identifying each is a task in itself. Surely there must be a better way of identifying these vulnerabilities. So lets answer the follow question; How do security people automate the detection of sql injection vulnerabilities in web apps?

Usually, you will use special purpose scanners (designed for web application testing). There are many web app scanners that do a good job at detecting sql injection. However, no tool is perfect and its is recommended to not rely on any one tool. If you have 3 web app scanners, run them all then compare your results. Also keep in mind that it is possible that your scanner wont find every flaw in the application, so it is absolutely mandatory to have some knowledge of performing these tests yourself manually, to verify scanner results.

One tool that i find useful in detecting sql injection flaws is proxystrike. Proxystrike is a proxy/scanner that looks for sql and xss vulnerabilities. From the proxystrike documentation;

The process is very simple, ProxyStrike runs like a passive proxy listening on port 8008 by default, so you have to browse the desired web site setting your browser to use ProxyStrike as a proxy, and ProxyStrike will analyze all the paremeters in background mode. For the user it is a passive proxy because you won’t see any difference in the behaviour of the application, but in the background it is very active.

In the video below, i demonstrate how proxystrike makes it easy to detect sql injection vulnerabilities. I use DVWA (damn vulnerable web app) in the demo. To verify the vulnerability found by proxystrike, i used sqlmap to test and exploit the flaw. The end result was a dump of all the databases on the database server.

proxystrike from aerokid240 on Vimeo.

Resources / Good Reading:
proxystrike

Local priv escalation - linux kernel 2.6.39 and up

Today, i noticed a new local privilege escalation exploit on exploit-db.com, and decided to check that out. Initially i thought it might be an exploit for very old Linux kernels but that certainly wasn't the case (here is the advisory on securityfocus.com). I figured that i would try this on one of my VMs when i got home from work, but i was a little too impatient. I thought i would remote desktop to my systems and get started. Before i attempted this, i figured why not verify the kernel version of my host machine before initiating a remote desktop session. My laptop is running backtrack 5 R1. So i typed, "uname -sr" and was given the output "Linux 2.6.39.4". Very interesting i thought. Hmm, i figured i'll try it here before any of my VMs (yea i know, bad practice to try someone else's code on a host system before trying it out on your test bench, but i'm not perfect :-P). I went over to exploit-db.com and downloaded the exploit and got to work.

First i made sure i was logged in as an unpriveleged user.
Commands: whoami && id
Output:
noobuser
uid=1001(noobuser) gid=1001(noobuser) groups=1001(noobuser)

I complied the code using: gcc -o local_exploit 18411.c

Then i executed the exploit: ./exploit

I was then greeted with another shell. I then verified who i was logged in as.
Commands: whoami && id
Output:
root
uid=0(root) gid=0(root) groups=1001(noobuser)

There you have it. Even an account named noobuser can pwn systems and become root with lil effort. Unfortunately, noobuser will still continue to be considered anything but elite, atleast in the security community :(.

OpenVPN revisited

I already touch on the basics on setting up openvpn. In that setup, a remote client(or clients) will only be able to connect to the server and access it's services. This time i want to share a little bit of knowledge on setting things up where remote clients can communicate with other devices on the server network.

Network setup:
Internal:
[router:192.168.1.1]
[client1:192.168.1.100] [client2:192.168.1.101]

Server:
[internal:192.168.1.10] - [vpn:10.10.10.1]

VPN clients:
[vpn:remote user:10.10.10.2]


The server is connected to an internal network, 192.168.1.0/24. When the VPN link is establiished, the vpn network of 10.10.10.0/24 will be established and the server will get the ip address of 10.10.10.1, while connecting clients will get different addresses from the vpn's address pool. Now if you've followed my previous openvpn setup, the remote client would only be able to communicate with that server. However many businesses require that remote vpn users have access to the entire subnet's resources. As you will see, upgrading to this setup is quite simple.

Here are the client and server configs:

Server:

dev tun
port 1194
proto udp
daemon
server 10.10.10.0 255.255.255.0
ca ca.crt
cert server.crt
key server.key
dh dh1024.pem
push "route 192.168.1.0 255.255.255.0"
ifconfig-pool-persist ipp.txt
keepalive 10 120
comp-lzo
max-clients 100
user nobody
group nogroup
persist-key
persist-tun

Client:

client
dev tun
proto udp
remote 11.22.33.44 1194
resolve-retry infinite
nobind
ca ca.crt
cert client.crt
key client.key
comp-lzo
user nobody
group nogroup
verb 3

Things to note:

The "daemon" directive tells openvpn to run in the background and send all output and error messages to a syslog file such as /var/log/syslog or /var/log/messages.

Including the "server 10.10.10.0 255.255.255.0" setting defines the vpn address pool. The first address, 10.10.10.1, will be assigned to the vpn server's tun interface.

The push "route 192.168.1.0 255.255.255.0" setting allows the server to advertise this subnet to connecting clients. When a remote client connects to the vpn server, a route will be added for the subnet 192.168.1.0/24, in that client's routing table.

The ifconfig-pool-persist ipp.txt is very interesting. According to the openvpn's manpage in linux, The goal of this option is to provide a long-term association between clients (denoted by their common name) and the virtual IP address assigned to them from the ifconfig-pool.

The "nobind" directive in the client config simply tells the client to not bind to any address and port. This directive is only suitable for clients.

Very Important things to note:

If you were to use the sample configuration files above as is, you will only have proper communication to the server, but not the subnet behind the server. This is what happens if vpn client 10.10.10.2 tried to ping 192.168.1.100; Since the clients will now have a route added for the 192.168.1.0/24 network, the ping packet will get sent over the vpn tunnel. The server already has a route to the internal network so the packet will be routed, ONLY if after setting up forwarding :). In linux you do this by typing the following in a terminal: "echo "1" > /proc/sys/net/ipv4/ip_forward". This turns your machine into a basic router. Without this, your machine will drop all packets that aern't ment for itself. So assuming we have forwarding in place, our inital ping packet will get forwarded to the internal host. Now we run into more problems.

The internal host will recieve the packet but since it doesn't have a route for the vpn client's network (10.10.10.0/24), it will send its response to its default gateway ( routers in most cases), then the router will consult its internal routing table and learn that it has no route for that network then forward the packet to its default gateway, and so on until the packet is dropped. Note that addresses such as 192.168.0.0 and 10.0.0.0 are dropped by routers on the internet as they are flagged as non-route-able addresses.

The solution to this is to add a route to the default gateway. For me, this would be the router, so i would log onto the routers web interface and goto the routing settings and add a maunal route for the 10.10.10.0/24 network to forward packets to the vpn server, 192.168.1.1.

Now, instead of our router forwarding the packets on the internet (where they will eventually be dropped), since we have an entry for the 10.10.10.0/24 network in the gateways routing table, the packets will be forwarded to the vpn server. Again, we run into another problem, NAT. I suggest that you read up on NAT (network address translation), as it can be quite a challenge to define its purpose and what it is. Therefore i will leave that up to you and your googling skills.

However, the solution to the NAT problem takes only one command on the linux terminal:
"iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -o eth0 -j MASQUERADE"

Now everything should be up and working. Remote vpn clients should now be able to communicate successfully with the other hosts on the servers internal network. Try pinging the other hosts to verify connectivity and if all is well, tap yourself on the shoulder.

Various Server services: Best Practice - compilation

This blog post will combine some of the best resources that i could find regarding server setups and best practices. Services will include web server (http), ssh, ftp etc. Of course you can find these on your own but why go through dozens of search engine results just to find that 80% of them aern't that useful, when I've done most of the hard stuff for you :).

SSH server (openssh):

Link 1: cyberciti.biz
Link 2: howtoforge.com
Link 3: teknoteknik.wordpress.com

FTP server(vsftpd):
Link 1: Ubuntu Server documentation
Link 2: brainsware.org vsftpd virtual users
Link 3: Centos.org vsftpd config options
Link 4: SFTP vs FTPS

Web Server (apache):
Link 1: apache.orgs tutorial collection (Best list)
Link 1: yolinux.com

Database server (mysql):
Link 1: ubuntu server documentation
Link 2: greensql.com

dhcp server (dhcpd):
Link 1: Ubuntu server documentation
Link 2: Article at consultingblogs.emc.com (very good read)

As i find more interesting articles, i will update this blog post. If you are interested in implementing any of the above services or just looking for some best practices, hopefully you will find these resources useful as i have.