[ May 2008 ]

1. Troubleshooting technical issues

Many people ( among them technical ) have problems articulating the nature a problem when it occurs. There are a distinct number of steps to follow but the most important is to be logical and follow the natural progression of looking at an issue. Also, it's important that, with more urgent issues, one doesn't lose sight of the core of the problem due to the critical nature of the problem and the surrounding stress.

1. Information gathering: this is the single most important task to perform as it dictates the chances of solving a problem and the speed of doing so. Look at every aspect of the system under review and some out-of-the-box thinking is useful on occasion.
2. Review logs: logs often provide clues to the issue at hand
3. Common areas: look at disk space as running out of or low on this resource can severly hamper or crash many applications
4. Network tests: with network issues, make use of common tools like ping, dig/nslookup and traceroute to determine bascially connectivity and the path of an issue
5. DNS: learn the various types of DNS records available and how to search for them
6. End users: ask questions carefully of users and try it from different angles; take the information they provide and decide whether it's valid in the current context or not
7. Take a break: if you can, this could give you the chance to refocus on the issue and look at it from a dfiferent perspective
8. AV packages: not all of these will find every threat; note though that it's not a good idea to run multiple engines unless your package has specific support for doing so; in addition, make use of a dedicated Anti-Spyware package ( eg. Spybot ) in addition to your AV

2. Applications part 2: reporting packages

Statistics and information reporting is an important part of IT strategic processes and aids in:

1. showing usage by capacity and time

2. providing for more accurate sizing of systems and future growth paths

3. aids cost savings in terms of bandwidth usage

4. showing information regarding misuse and abuse of a system

The 4 categories I'm looking at in this article are firewall, web, proxy and email reporting. I'll be covering the following products:

• Marshall
• eIQ Networks
  
• FortiAnalyser 
• Splunk 
• Pflogsumm
• Lightsquid
• Webalizer
• Analog
• Awstats 

There are many commercial and non-commercial reporting engines in this space, some that provide simple entry-level information and others that go the whole route and cover multiple products and huge amounts of data crunching. First the commercial products.

Marshall

Focus: web, email, firewall, proxy 

... has as part of it's reporting suite a product called Webtrends Firewall Suite. Of course, Webtrends itself has always had support for multiple log types including and not limited to firewall, web and email. However it commands a fair price premium being a commercial product. I've not used this in anger so won't pass any judgement except to say it's available.

eIQNetworks Firewall Analyzer

Focus: firewall 

This is the original product that Fortinet's FortiReporter software product was based on, so I'm not going to cover too much here ( there'll be a lot of info in the Fortinet section ) except to say that this product more or less covers 90% of the comemrcial device market int erms of web, firewall and proxy support. Fantastic product at an even more fantastic ( high ) price point.

Fortinet FortiAnalyser

Focus: firewall, web, email 

Fortinet's FA started life out as a rebranded version of the FA product above but has now morphed into a distinct Fortinet-focussed hardware product. The units come in 4 flavours:

100b

800b

2000a

4000a 

The following features are available:

• Reporting, logging, alerting and content archiving
  
• Over 300 customizable reports, scheduled or on-demand
• Advanced features such as Event Correlation, Forensic Analysis, and Vulnerability Scanning
• Secure data aggregation from multiple FortiGate and FortiMail security appliances
• Network capacity and utilization data reporting
  

Reports cover areas such as:

• viruses
• attacks
• firewall events
• mail and web usage 
• protocols
• and anything else based on raw data generated by FortiGate devices 

 This is beyond doubt one of the most generous reporting tools available albeit tied to Fortinet's platform. For anyone with many FortiGate devices, this is a must.

Now onto the OSS products:

Splunk

Focus: everything 

Is one of those unsual products that comes around now and again. It's not specifically used for any of these categories of reporting but is more of an aggregator and data mining tool for logs from just about anything you can imagine. The offical line is:

With a variety of flexible input methods you can index logs, configurations, traps and alerts, messages, scripts, and code and performance data from all your applications, servers and network devices. Fast, free form search on anything, not just a few predetermined fields. Boolean, nested, quoted string and wildcard searches. No knowledge of specific data formats required. Combine time and term searches. Find errors across every tier of your infrastructure and configuration changes in the seconds before a system failure occurred. Fields are identified from your results as you search -- providing much more flexibility than a rigid set of field mapping rules imposed ahead of time.

So that's quite a mouthfull but for those with a heterogeneous network full of different devices, apps and logs, this is the ultimate aggregator of logging data.

Pflogsumm

Focus: Postfix email reporting  

Is basically a perl script that was devised to provide in-depth reporting for Postfix MTAs.

Features:

• Total number of:
  • Messages received, delivered, forwarded, deferred, bounced and rejected
  • Bytes in messages received and delivered
  • Sending and Recipient Hosts/Domains
  • Senders and Recipients
  • Optional SMTPD totals for number of connections, number of hosts/domains connecting, average connect time and total connect time
• Per-Day Traffic Summary (for multi-day logs)
• Per-Hour Traffic (daily average for multi-day logs)
• Optional Per-Hour and Per-Day SMTPD connection summaries
• Sorted in descending order:
  • Recipient Hosts/Domains by message count, including:
    • Number of messages sent to recipient host/domain
    • Number of bytes in messages
    • Number of defers
    • Average delivery delay
    • Maximum delivery delay
  • Sending Hosts/Domains by message and byte count
  • Optional Hosts/Domains SMTPD connection summary
  • Senders by message count
  • Recipients by message count
  • Senders by message size
  • Recipients by message size
  with an option to limit these reports to the top nn.
• A Semi-Detailed Summary of:
  • Messages deferred
  • Messages bounced
  • Messages rejected
• Summaries of warnings, fatal errors, and panics
• Summary of master daemon messages
• Optional detail of messages received, sorted by domain, then sender-in-domain, with a list of recipients-per-message.
• Optional output of "mailq" run

Seeing as Postfix is becoming one of the most popular MTA's used worldwide, this is a must have as it touches every area of an MTA's usage. Reports are delivered in email format by the way.

Lightsquid

Focus: Squid logs

Lightsquid provides a web-based interface to Squid logs.

Features:

• fast and simple install
• fast log parser generatesmall per user data file
• perl based cgi script for dynamic generated report pages
• html template for design
• no database required
• no additional perl module
• various reports
• user groups support
• graphics report (v 1.6+)
• real name (v 1.6+)
• multilangual interface

Areas reported on include:

• user, group and total usage by day, week, month and year
• user and group listings with complete cross-drill available
• ip to name conversion
• authenticated user support
• top sites, users, etc.
• graphing for month usage
• per user time reports
• bug files threshold report

Webalizer

Focus: web

The Webalizer is a fast, free web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for viewing with a standard web browser. Unlimited log file sizes and partial logs are supported, allowing logs to be rotated as often as needed, and eliminating the need to keep huge monthly files on the system. Supports standard Common Logfile Format server logs. In addition, several variations of the Combined Logfile Format are supported, allowing statistics to be generated for referring sites and browser types as well. Now also has native support for wu-ftpd xferlog FTP and squid log formats as well. Supports virtual web hosts via separate config files. Overall a fairly decent log analaysis package but there's better.

Analog

Focus: web

• Ultra-fast
• Scalable
• Highly configurable
• Reports in 32 languages
• Works on any operating system
• Free software

Not as full featured and pretty as other systems but you can spruce it up a bit with ReportMagic.

Awstats

Focus: web, proxy, email

The swiss knife of log analysis, Awstats does everything from web, ftp, email to proxy. AWStats is a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from all major server tools like Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar, IIS (W3C log format) and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers.

* Number of visits, and number of unique visitors,
* Visits duration and last visits,
* Authenticated users, and last authenticated visits,
* Days of week and rush hours (pages, hits, KB for each hour and day of week),
* Domains/countries of hosts visitors (pages, hits, KB, 269 domains/countries detected, GeoIp detection),
* Hosts list, last visits and unresolved IP addresses list,
* Most viewed, entry and exit pages,
* Files type,
* Web compression statistics (for mod_gzip or mod_deflate),
* OS used (pages, hits, KB for each OS, 35 OS detected),
* Browsers used (pages, hits, KB for each browser, each version (Web, Wap, Media browsers: 97 browsers, more than 450 if using browsers_phone.pm library file),
* Visits of robots (319 robots detected),
* Worms attacks (5 worm's families),
* Search engines, keyphrases and keywords used to find your site (The 115 most famous search engines are detected like yahoo, google, altavista, etc...),
* HTTP errors (Page Not Found with last referrer, ...),
* Other personalized reports based on url, url parameters, referer field for miscellanous/marketing purpose,
* Number of times your site is "added to favourites bookmarks".
* Screen size (need to add some HTML tags in index page).
* Ratio of Browsers with support of: Java, Flash, RealG2 reader, Quicktime reader, WMA reader, PDF reader (need to add some HTML tags in index page).
* Cluster report for load balanced servers ratio.

As a conclusion to this article, one should take note of syslog-ng with phpSysLog which provides for a SQL backend to log collection and a searchable web frontend for viewing of the data. Not quite as fancy as Splunk but nonetheless a very viable option.

3. Open Standards news

 ISO puts standard for Microsoft's OOXML document formats on hold

After member states filed four complaints against the standardisation of Microsoft's Office Open XML (OOXML) document format, the International Standards Organisation (ISO) and the International Electrotechnical Commission (IEC) in Geneva have responded by postponing publication of the revised specification. As the ISO announced, the planned ISO/IEC DIS 29500 cannot be published until these complaints have been heard. Procedure requires that they be dealt with by the end of June, when the ISO and IEC have to hand over their comments on the complaints to two management committees for a final decision.

Brazil, India, South Africa, and Venezuela have officially filed complaints against the controversial certification of OOXML in expedited proceedings in Geneva. These emerging nations are concerned that no consensus was reached about which changes need to be made to the specification, which is more than 6000 pages long, during consultation on the numerous comments submitted at the end of February, after the first attempt to adopt OOXML as a standard failed in 2007. Specifically, they complained that concrete technical objections were not individually discussed .

 4. Email Clients

There are a variety of email cients available, some simple and others advanced, some Windows only and other cross platform. I'll be covering some of the main players in this field and make some personal comments, which you're welcome to take with seriously or a pinch of salt .

Outlook

This is Microsoft's premier email client and has been around for many years only as part of the MS Office suite. As a result it's got very good integration with MS' office and server platforms ( Exchange ) and features strong organisational, calendaring, publishing and sharing capabilities. Some of these features are only available when working against Exchange though. Archiving ( a necessary evil with Outlook ) is fairly robust and the resulting files can be included in the standard folder/account view. One of Outlook's biggest drawcards are it's universal support for mobile phone synchronisation by the manufacturers.

On the bad side, Outlook continues to use the PST format for it's storage mechanism ( especially when in standalone mode ) and suffers from severe slowdown when the size of the PST approaches or exceeds 2GB. Recent versions ( in MSO 2003 or later ) are better but still have performance constraints when used with large email stores. Junk/spam controls are reasonable rather than stellar and there are no bayesian controls or automated learning. Handling of email certificates is very poor and not intuitive at all. The address book uses a proprietary mechanism but you can export as csv if required.

Overall, a reasonable client with strong calendaring and collaboration for Windows users although there are stronger clients from a pure email handling point of view.

Thunderbird ( Netscape Mail, Mozilla Mail ) 

Thunderbird is a simple straight up email cient and has gained a fair amount of users in the last 2 years as a result of the tie up with the Firefox browser and its increasing popularity. Very strong from a pure email point of view with capacity for massive email store handling ( I'm running a 13GB store ). Move/copy operations are handled very quickly and surprisingly even when using IMAP email stores. IMAP support is strong by the way although it can get its knickers in a twist sometimes but fixing store issues is fairly straightforward. Part of the performance value of T/Bird is its used of mutliple flat files mirroring your folder layout. This means that a typical email a/c with a few folders can hold 10's of thousands of email without a problem. The files are also indexed for high speed access to email operations.

The address book uses the industry standard vcard format and can export/import from and to a variety of formats. There are a variety of views available for your inbox with the grouped view ( by today, yesterday, last week, older email ) being quite popular. Threaded is brilliant for news groups and rssfeeds. That reminds me, T/Bird has support for pop, imap, rss, ldap, ical and nntp content types. Some other standard features including colour coded mail tagging, saved realtime updated search folders, a strong antispam/junk filter with learning, multiple user support, strong authentication and encryption support, and a powerful filtering/rule system. 

T/Bird has no calendaring by default but this leads me to the next big advantage of Thunderbird - it's plugin system. This allows 3rd party authors to add additional functionality to the application and allows the user to decide exactly what advanced functions they need. These include:

• Lightning ( the Mozilla calendar )
• an enhanced certificate viewer
• address book synchroniser using http, ftp, imap
• displaying the sender email client
• world clocks
• mail redirect/bounce
• quote colours ( great for following an email thread )
• intelligent signature switcher
• builtin web browser
• post-it-notes
• many more

Evolution

This client has been around for a few years now and has gathered strong support based on it's outstanding feature set. This includes:

• intelligent junk mail control ( based on the T/Bird system )
• search folders 
• integrated SMIME and GPG support 
• strong filters and searching
• strong support for web calendars through iCal support
• collaboration support for Exchange and Groupwise
  
• assisted multiple emali account creation
• cross platform support ( Gnome libs and Windows )
  

Outlook Express

Available as standard in Windows versions up until, but not including Vista, this is a straight forward no-frills email client. Not much to say except that it does email. No collaboration tools, email handling is ok and is uses the Windows wab-based address book.

There are also:

• KMail - part of the KOffice KDE package
• Eudora - an old favourite but long in the tooth
• Groupwise client - good collaboration with Groupwise server
• Lotus Notes - the grandaddy of collaborative clients
• Opera Mail
• Sylpheed - strong Gnome-based client

 Wikipedia has a good comparison here

5. This week's security

A tough week - probably worse than it appears. Substantial numbers of critical vulnerabilities were reported for users of widely deployed software - Microsoft Bluetooth, Internet Explorer, and DirectX, Apple QuickTime and Cisco and other vendors' SNMP. But also in the less visible world of web applications where a massive wave of attacks against web apps became more visible in this week's data - nearly 80 new vulnerabilities in commercial web apps this week alone -- and hundreds of thousands of sites compromised because of flaws in their custom-developed web applications.

• CRITICAL: Microsoft Bluetooth Remote Code Execution
• CRITICAL: Microsoft Internet Explorer Multiple Vulnerabilities
• CRITICAL: Microsoft DirectX Multiple Vulnerabilities
• CRITICAL: Apple QuickTime Multiple Vulnerabilities
• CRITICAL: Multiple SNMP Implementations Authentication Bypass VulnerabilityHIGH: OpenOffice.org Remote Code Execution
• HIGH: Novell GroupWise Messaging Client Buffer Overflow
• LOW: Microsoft Windows Pragmatic General Multicast Denial-of-Service

Viruses this week 

Downloader-UA.h, a Trojan that spreads by masquerading as an MP3 music or MPEG video file available from popular file-sharing services such as Limewire and eDonkey. The malicious files are named differently in multiple languages and vary in size to make them appear like legitimate music or video files. Attempting to play one of the malicious files will trigger a program called PLAY_MP3.exe; to download, launch, and force advertisements to appear on the infected computer.
More information:

MalDoc-Fam, a Trojan that is spreading through infected Word documents attached to emails posing as news about the Chinese earthquake disaster. The malware-tainted emails typically appear with body text suggesting they contain news from China's official press agency, Xinhua. Opening the attached Word document triggers an exploit that downloads malware onto vulnerable PC's.
More information: http://www.theregister.co.uk/2008/05/22/china_earthquake_trojan/

 6. And some more security

As part of the range of services I provide, Network Mapping, Vulnerability Scanning and Penetration Testing are sometimes the most interesting for me because it provides a chance to see how well someone has secured their systems ( or how badly ). Quite unexpectedly is the fact that there are still basic mistakes being made in respect of having systems publicy accessible on the internet. Some of these include:

• ports open that are either not in use, required or provide known vulnerable services
• web, mail and other servers without firewall/ids protection
• servers running old versions of software
• service banners enabled
• MS Exchange being used without a relay
• administrative areas on web servers not secured
• bad application coding
• poor network design
  

By far the most serious issue as of late is web servers running applications that are prone to SQL injection attacks. "SQL Injection" is subset of an unverified/unsanitized user input vulnerability ("buffer overflows" are a different subset), and the idea is to convince the application to run SQL code that was not intended. If the application is creating SQL strings naively on the fly and then running them, it's straightforward to create some real surprises. The latest figures indicate that around 2 million web servers worldwide are current suffering some form of SQL Injection attack.

Regarding MS SQL Server, some are not aware of xp_cmdshell

Microsoft's SQL Server supports a stored procedure xp_cmdshell that permits what amounts to arbitrary command execution, and if this is permitted to the web user, complete compromise of the webserver is inevitable. If we can run commands, the webserver itself cannot help but be compromised. Access to xp_cmdshell is usually limited to administrative accounts, but it's possible to grant it to lesser users.

And how to get around this?

• Sanitize the input
• Escape/Quotesafe the input Limit database permissions and segregate users
• Use bound parameters (the PREPARE statement)
• Use stored procedures for database access
• Isolate the webserver
• Configure error reporting

The upshot of this is that programmers and software architects need to take cognisance of this issue when designing and coding.