A Student-Hacker Showdown at the Collegiate Cyber Defense Competition

Date: Mar 31, 2006

Students faced off against experienced hackers at the Mid-Atlantic Regional Collegiate Cyber Defense Competition. The students' goal: lock down unfamiliar systems and secure their networks. The hackers' goal: to own the students' networks and steal important data. Seth Fogie witnessed this real-world competition and reports on its unexpected twists, turns, and even drama.

Imagine if you just graduated with an IS degree and landed a job at a small business as their only IT staffer. You know your way around an operating system and understand some of the protocols and programs that keep data flowing, but for the most part your skills are untested in the real world. Regardless, you are the only thing separating the company's users and data from downtime. Sound like a tough situation? Oh, I forgot to mention there are four of the best hackers in the world trying to get into your digital domain and steal anything of value, including a database of 10,000 credit card numbers. This isn't something seasoned administrators would want to face, much less fresh graduates.

Well, this is exactly what several groups of college students faced recently at the Regional Collegiate Cyber Defense Competition, which was held at several locations around the US. I was able to attend this three day event, and this is my story. Get ready for some fun, shocks, and dohs! as you follow along with me, the Red Team, and the students.

Collegiate Cyber Defense Competition

Before we get into the actual details of the event, it is important to highlight the reason behind the competition. As per their website at http://utsa.edu/cias/CCDC, "Unlike traditional 'hack and defend' or 'capture the flag' contests, this competition tests each team’s ability to operate, secure, manage, and maintain a corporate network. This competition is the first to create, as closely as possible, a realistic corporate administration and security experience — giving the competitors a chance to compare their education and training against their peers and the real world challenges that await them."

In other words, the competition is about creating a practical experience from the classroom knowledge. Students simply take what they have learned and apply it in a simulated environment for educational purposes. However, the competition aspect helps schools know where they stand in relation to other schools that are teaching related content. It is important to note that the program is funded 'in part' by the National Science Foundation (Award ID 0501828 http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0501828) and has paid out a little over $1,000,000 so far to create "problem-based and case-based learning methodologies in order to provide students with activities that simulate real-life work experiences <with a focus on security>." I suggest you check out the above link to see the details of this award.

The contests are first held regionally, and the winner of each regional exercise competes against each other at the national finals in San Antonio, Texas. I attended the Mid-Atlantic Regional Collegiate Cyber Defense Competition, which was hosted in Lancaster, PA by White Wolf Security. The five schools where from the PA/VA/MD area and were a mix of two/four year degrees that range from networking to programming. The participating schools were:

Anne Arundel Community College
Community College of Baltimore County
George Mason University
Millersville University
Towson University

White Wolf Security

The host and operator of the event (White Wolf Security) operates a training facility that serves as a working-training environment for hands on computer lab-based education. Everyone, from local colleges to the US Secret Service, use White Wolf's equipment as a hands-on lab for training events and other related activities. The owner, Tim Rosenberg, is well known in educational/government circles for both his lab (which is mobile) and for his training events.

The Details

There are several different groups involved with the games. Each has a function, and a tag. The white cell is there to keep the games running smoothly. Gold members are the judges and professors who basically monitored the games from afar. The red cell was there to attack, and finally, the law enforcement was there to arrest the hackers.

The Scoring

At the beginning of the game, everyone starts with zero points. If the team can keep all the services open, and a selection of 'target' files available, they keep that zero. However, if a team suffers from a loss of confidentiality, integrity, or availability in any other way, they start collecting points — sometimes quite rapidly. Figure 1 is a shot of the scorebot system, and backend components of the network.

Figure 1: Scorebot system

The Feds

One particular aspect of this game that added significant value was the inclusion of a reporting process to the authorities. On hand was a real US Secret Service agent who deals with computer-related crimes on a regular basis. His job in the game was to show up on scene when a student team detected an attack. If the incident report was filled out correctly (see a real incident report from USSS web site: http://www.secretservice.gov/forms/form_ssf4017.pdf), the team would get points taken away from their growing score. This aspect to the game was actually one of the most valuable as one who has had to deal with the authorities before. Not only will this experience give each of the students something to look back on if they ever have to deal with the government for real, but they also now have someone they can talk to if something does arise.

The Network Layout

The network was separated into seven different subnets. Five were split up between the teams, one was used for scoring systems, and the final was for the Red Team. At the edge of each of the student's networks was a router and firewall, which were off limits until the second day and third days, respectively. Finally, each school had four servers in a DMZ that were connected to the firewall, each with a 'public' IP and a specific purpose. In addition to this, they had two workstations that were in the protected area of the network that were used for syslogging and other functions. The following breaks down the initial system setup — try not to grimace.

Alpha1 – Windows 2003 Server running IIS 6.0 (HTTP/HTTPS), MYSQL and OSCommerce (with PHP support).
Alpha2 – Fedora Core 4.0 with VSFTPD and DNS (BIND)
Alpha3 – Fedora Core 3.0 with SSH
Alpha4 – Windows 2000 Server with IIS5.0, MYSQL, Telnet, SMTP/POP, and DNS (Secondary) running an HR database.

Each system and program was of an unknown patch state/version. In addition, there was a network IP camera thrown in for grins and giggles. Welcome to most small IT shops where money is tight and time is valuable. Figure 2 provides a look at an unmanned pod. Note the four monitors that are connected to the stack via KVM.

Figure 2: Unmanned pod

Business Objectives

To make the games a bit more realistic, each team would receive various business objectives that would have to be completed in due course, or they would lose points. This could be something as simple as add an email account or even install PGP. The details of the objective were up to those running the games.

The Rules

The rules were fairly simple — at least at first glance. Basically, the Red Team could do anything but hurt someone or perform a denial of service attack (network flood). The student teams were a bit restricted, with regard to changing IP addresses and messing with the infrastructure.

Communication was allowed between team members, but only the team leader could talk to the white cell members about problems, etc. The feds could be called over for an investigation and the Red Team was allowed to try to talk to the teams to put a social engineering twist on the games. Finally, all business objectives and administrative requests are sent to the CEO via email.

There are some other rules and regulations that are laid out in some detail (http://student.ccbcmd.edu/~cobrie12/ccdc/docs/
Mid_Atlantic_Regional_Team_Packet.pdf). However, for the most part, the rules were fairly loose to allow some dynamics in the game.

The Competition

The event kicked off around 1PM on a Friday afternoon. All the students were sitting at the 'pod,' waiting for the green light. The Red Team was all set to go in a separate room with their equipment. After some general announcements, Tim Rosenberg introduced the students to their mission. "Your job is to keep the services up, the router routing and keep the store open — as well as everything else". After some brief descriptions as to what and who was involved, he gave the green light. At this point, the students had three hours to figure out what they had just inherited from 'the previous IT person' and fix it. Meanwhile, the Red Team was set loose to discover just who was out there and figure out what they were running. They were not to attack anything until after the three hour limit was up. However, the term 'attack' is very grey and seemed to include rooting routers and firewalls.

When the teams were set loose I positioned myself in the red room to see how the initial information discovery process would go. It was at this point one of the Red Team members stood up, kicked everyone out, and locked the door. Fortunately, I was labeled as trustworthy and was able to stay inside. He next reached inside his bag and pulled out a complete description of the student's setup, including all operating systems, services, web applications, and IP addresses he had obtained from an anonymous source. Everyone in the room immediately got a slightly evil grin on their face as they realized the results of this social engineering reward. Oh yes — things were about to get very bad for the students. Figure 3 gives you a shot of the Red Team in action.

Figure 3: Red team in action

After the disclosure of this damning piece of information, I stepped outside the room to see how the students were managing. Ironically, the students at this point knew less then the Red Team about what was running on their systems. Once again, fortune shined down on me because I happened to know one of the school’s teams leaders. After a short catch-up (I knew him from high school), I started to ask what he knew about the competition and what his students were dealing with. As it turned out, his team was all programmers who jumped into the event at late notice. That said, they seemed to be very busy figuring out what they had to fix and seemed to be fairly astute as to what they needed to do. I saw kernel recompiling, service packs being downloaded and installed, account permissions being locked down, and much more. In fact, as I looked around the room, all of the teams seemed to be in a frantic rush as they tried secure their VERY insecure systems.

I walked around from team to team and quickly realized that no one trusted me, which is a good thing as social engineering was allowed. After an introduction ("I am press") and assurance I was not going to tell the Red Team anything, they allowed me to be near, but still kept one eye on me and the other on what I was looking at. Paranoia had set in. Since the first three hours were critical to their success, I decided to keep my distance from the teams and watched from the sidelines. Considering the feat they were trying to accomplish, they did not need me interfering.

During this time, I was able to talk to several of the team leaders, who were not allowed to interact with their teams. Most of them are college professors (PhD types) who wanted to expose their teams to some real world experience. Since the bill was paid for by the grant, there was little to lose and much to be gained by joining the competition. In fact, I am pretty sure there will be at least one school that will be including a class on router configuration.

Back in the red room, the Red Team was working hard at 'information gathering.' This involved scanning the systems with nmap and popular GUI applications from Windows. After looking at the results, it was pretty obvious that the students had some serious issues to address. However, it was equally as obvious that the much of the content of the Red Team's information packet was going to be learned by the schools in the first half hour — except for the default passwords.

Since the Red Team knew the default passwords for most of the accounts and services of the running servers, they had logged into each of the teams routers and changed the default password to something a bit more hard to guess. They were also logging into the Linux servers via SSH and changing account passwords, plus doing a little system level recon to see what kind of vulnerabilities they could use to raise their newly acquired accounts to root level access. Some might call this active hacking, but the lines were not that clearly drawn, which leaves much to 'interpretation.'

This type of network and system recon continued for roughly three hours, during which time I bounced between the red room and student pods. There were several hiccups in the process, such as an overload on a power circuit that led to a complete loss of power for two teams, but that just made the event that more realistic IMHO. As the teams started to get things under control, they acknowledged my presence and started to talk a bit. I learned that most of the students were expecting to be slaughtered when the Red Team was set loose. I personally agreed. Even if everyone on the team was an experienced veteran, there was no way they could lock down everything in three hours.

Don't Let Your Momma Dress You

When I first entered the White Wolf Security lab, the first thing that caught my eye was a team that was wearing all blue. Everyone else was in typical student attire that mostly consisted of jeans and a t-shirt. Ironically, this attempt at professionalism turned out to be a bad idea because the Red Team also noticed the blue shirts. The result: 'the blue team' became target number 1.

Let the Games Begin — Day One

When the three hour grace period was over, the Red Team slowly worked their way into attack mode. One member started to sort through the information they gleamed from their scans and investigated each possible exploit. Another member fired up a MySQL database client and started to poke around the students databases looking for sensitive data. The two others were adding/changing accounts to routers, firewalls, and systems. However, for the most part, the students were not being pelted with attacks. And this continued for the next several hours.

One interesting event occurred during this first stretch that warrants a mention. As it turns out, a team detected that their router’s default password did not work. They corrected this problem by uploading new configs to the router, which gave them control again. However, a Red Team member realized what happened and decided to find another way into the device. It took a few minutes, but they quickly learned that the router had SNMP enabled and allowed read/write access for public and private. The result was that the attacker used 'private' community access to add a new account to router. Once again, this activity was detected by the students, at which point they attempted to completely secure their router. Unfortunately for them, they messed up this process and inadvertently took themselves out of the game. Since the router is the doorway to their servers, the scoring bot had no way to tell if their servers were running. The point to this is, killing your device might keep an attacker out, but it also keeps valid communications from occurring.

Day Two

Saturday started out slowly, but by the end of the day things would not be looking good for most of the teams. With one team pretty much out of the picture thanks to a router issue, the Red Team really focused on the 'blue shirts'. Their first target was an OSCommerce application that was running on one of their Windows machines. Unfortunately, the blue shirts forgot to change the permissions for the admin directory on the application. As a result, the Red Team had complete access to the configuration manager portion of the application. This not only gave them access to all the order information that included 10,000 credit cards, but also gave them access to a file manager application that allowed them to upload/download/edit files on the system, which they did.

One of the members on the Red Team decided to make the ownering of this application obvious, and renamed the Title of the OSCommerce site to something like 'Welcome to Tim Rosenberg’s School of UDP.' Unfortunately for the Red Team, this was quickly spotted by the students who then started to look at how and why this happened. Meanwhile, the Red Team member had also defaced their home page, which the students again spotted. Access to the admin folder was soon disabled, but the damage was done — integrity was lost, services were denied, and confidentiality was gone. The blue shirts were able to detect and report the web server defacement to the authorities, but they missed the customer information download. Since web defacement is minor on the scale of attacks, the Red Team was only given community service instead of the felony charge they could have been hit with. The end result is that the color blue was a good pick for this team as depression sunk in.

The Red Team did spread the love around a bit after pummeling the blue shirts for a few hours. They discovered an unprotected HR program that was loaded with SQL injection vulnerabilities. This was then used to download/alter employee data, which represented a major loss in confidentiality and integrity. However, it was what the Red Team did after this that was quite clever.

Using some custom code, the Red Team created a SQL injection query that then connected to another team's web server in an attempt to create a denial of service attack. This odd attack forced one of the teams to approach the other team with an apology that went something like this: "Hi. Uh, I am sorry if we are attacking you, but we aren't really doing it." The DoS was stopped soon afterwards upon request of the judges.

At about 2PM the second day, attention was shifted to one team in particular because they had managed to stay out of the limelight. It was soon discovered that this team had failed to change their postmaster email password, which gave the Red Team full control over the emails coming and going to the server. Various methods of abuse were discussed, but it was concluded that the best thing to do was to change the password on the administrator accounts, create a new account, and forward all email from the CEO email account to the Red Team’s account. Once this was set up, the Red Team was told to take a break because the student teams were getting overwhelmed. Thus the attacks stopped, which gave the students time to focus on reporting incidents and securing their systems from the various attacks that had been occurring during the day.

Confusion Techniques

One of the interesting tricks that the Red Team did to keep the students guessing was to run continuous scans from programs like Nessus. They did this for one reason — overload the students. In addition to indirect misinformation, the Red Teams also employed tools like mucus, which have no other purpose but to trigger IDS alerts. They also noted the use of ethereal and injected malicious packets into the network that would crash the sniffer and cause general havoc. It is important to note these techniques because in a real attack, it is not only possible for this to occur, but even probable — especially if the attacker knows you are watching.

The Hacker Mentality

As I sat in with the Red Team, I got to watch how each worked and what tools/methods were used. The four members really represented a wide range of skills and personalities. On the one side was the professional information assurance who really knew the technology and was able to assist with various tasks that led to loss of confidentiality. Another member was very prepared with a rack server that had six CPUs. His system was loaded with programs like Canvas, metasploit, and other automated penetration testing tools. Next was a member who blended the casual professional with rule bending hacker. He proved to be a valuable asset and was the one who coded up the script that performed the SQL denial of service attack. The final member was the pony tail/ear-ring type that really stretched the rules and thought nothing of it. As a career penetration tester, his skills were valuable for the team as well. The point is, each member took a different approach and used different tools to get the job done. They used everything from OS X to Linux and even Windows, not to mention intimidation, ladders, and glow sticks (more on that in the next section).

Day Three

I arrived early on day three with an understanding that there would be one more hour of active Red Team hacking. The rest of the day was set aside for some competition between the students to allow them some Red Team action. However, to my surprise, I arrived to find all the students in a panic. Apparently, someone had messed with the computer systems during the night and no one would fess up as to who had done it or what was done.

While we all waited on the Red Team, it was discovered that during the night the forwarded CEO email account had intercepted two emails from one of the student teams. Unfortunately for them, it contained all the user/passes for every member of the team. As a result, the present Red Team member was able to log into the OSCommerce site and download the customer database and access the accounts on the SSH server, not to mention anything else that required an account. Of interest, the Red Team was not able to use the file manager in OSCommerce to upload/download files because the students had only allowed read/execute access to the admin directory. It was at this time that the Red Team arrived and explained what had happened.

To keep the games interesting, and provide a bit of a educational anomaly, the Red Team had done what any criminal hacker would consider — they broke into the teams’ pods and installed backdoors. Using only the light from a glow stick (the hotel they were staying at didn't have any flashlights), they found a ladder, climbed up the outside of the room (12 foot ceilings), pulled back a drop ceiling tile, and climbed down a wooden rod they collected from nearby. With physical access granted, the Red Team went to town.

Rootkits, backdoors, password changes, system configuration changes and more were fair game with no one around to stop them. One team had locked down the KVM device with a password, but this was quickly bypassed by plugging the monitor into the actual computer. Another team used BIOS password protection, but again, a quick short of the CMOS and the BIOS flash was reset back to default. Windows administrator accounts fell quickly to boot disk based password reset attacks. Root account was gained by 'single user' mode hacks on the Linux machines. From there, log files were deleted, PHP scripts were embedded in programs, backdoors were installed, accounts were created with root level access, and much more. Simply put, the Red Team owned the students through and through. The only way anyone could realistically recover is if they took everything offline and started from scratch, which is exactly what a business would have to do. In fact, in a real case, the feds would probably ask to take the systems as evidence.

The Summary

After a brief break for lunch and some major discussions about the games and the physical break in, the Red Team gave a short talk to the students about what they did. Not including the physical attacks, 90% of the issues were related to default passwords. The remaining problems were related to bad code. They also brought up the blue shirt affect and that avoiding attention is a great technique to staying out of harm’s way. It was also discovered that most of the teams were expecting serious 0-day attacks that they would have to find and stop, when in reality telnet, SSH, and a web browser were the primary weapons. The winning team was actually the one that kept their router running (two teams hosed themselves on this issue), changed most of the default passwords, locked down their permissions, and didn't attract attention to themselves. Oh, and of interest, it was the same team that had only a week to prepare and were all programmers (as seen in figure 4).

Figure 4: The Winners!

The end result was that a group of students got a first-hand experience of just how bad it can be in the real world, and what they would need to do if they ever had to deal with a similar scenario. From setting up a secure shopping cart to understanding how the chain of evidence and how to deal with authorities, the experience was valuable for everyone there, including me. I for one will be back again next year to watch the games!

Winning Team: Millersville University

Todd E Echterling: System administrator for the computer science department
Chad A Billman
Edward J Schwartz
Thomas J Miller
Cory W Adams
Michael A Vicinsky
Mark A Olszewski
Bradley J Chronister

Red Team:

Joe Harwell: Joe is a Security Specialist for Nortel Government Solutions. He currently is responsible for design, integration and testing of many of the "three letter agencies" security systems, and has over 15 years of experience in the field. He was CERT penetration tester for the US Army in a previous life.

Ryan Trost: Ryan is a Senior Security Engineer for Criterion Systems, currently working on a DHS contract. When not overseeing the security architecture of his team, he spends his free time developing a Network Security Snap-on Application that involves IDS Geocoding (patent pending). Ryan will be graduating from George Washington University this May with a Masters in Computer Science.

Adam Meyers, CCE, IAM, IEM: As an information security professional and consultant, Adam Meyers provides clients with complete security expertise, ranging from assessments, forensics, incident response, penetration testing, and security architecture. Additionally he provides physical security assessments and threat analysis. Mr. Meyers is a Certified Computer Examiner (CCE). Prior to joining SRA, he worked with the George Washington University Security Team, as the Network Manager for the 2000 National Democratic Convention, and as a private security consultant, all while pursuing a degree in political science with specific attention to inter-state information warfare.

Tom Parker: Tom is a computer security analyst who, alongside his work providing integral security services for some of the world's largest organizations, is widely known for his vulnerability research on a wide range of platforms and commercial products. Tom regularly presents at closed-door and public security conferences, including the Blackhat briefings, and is often referenced by the world's media on matters relating to computer security.