Monday, October 30, 2006

Sending email through Google SMTP from Perl

The do no evil people from Google kindly provide a publicly accessible SMTP server to which you can send email. The catch(s)?

By requiring authentication Google hopes to quickly identify and shut down spammers if they spam from a gmail account.

Since I use Perl to administer my servers I would like to be able to send alerts from Perl scripts to my gmail account when something happens. (And from my gmail account a copy can be sent to my other email accounts, my cell phone, etc.).

To support SASL you need to install the following modules on top of the standard Perl distribution:

Unfortunately the Net_SSLeay and IO-Socket-SSL contain C code and need to be compiled. Fortunately you might be able to find a Perl repository that has already built these packages for your platform e.g. http://archive.apache.org/dist/perl/win32-bin/ppms/ for Perl 5.8 and the Win32 platform. So using the Perl Package Manager (PPM) you can install the packages by doing something like:


rep add "Apache Perl 5.8.x" http://archive.apache.org/dist/perl/win32-bin/ppms/
install Net_SSLeay.pm
install IO-Socket-SSL
install Authen-SASL

With Net-SMTP-SSL you can simply extract the SSL.pm file from the distrbution and place it in the <PERL_INSTALL_DIR>/lib/Net/SMTP directory.

Now that you have all of the packages you need installed, a simple Perl script to send email via the Google SMTP server looks something like:


#!/usr/bin/perl -w

use Net::SMTP::SSL;

sub send_mail {
my $to = $_[0];
my $subject = $_[1];
my $body = $_[2];

my $from = 'johnny@gmail.com';
my $password = 'MySecretGmailPassword';

my $smtp;

if (not $smtp = Net::SMTP::SSL->new('smtp.gmail.com',
Port => 465,
Debug => 1)) {
die "Could not connect to server\n";
}

$smtp->auth($from, $password)
|| die "Authentication failed!\n";

$smtp->mail($from . "\n");
my @recepients = split(/,/, $to);
foreach my $recp (@recepients) {
$smtp->to($recp . "\n");
}
$smtp->data();
$smtp->datasend("From: " . $from . "\n");
$smtp->datasend("To: " . $to . "\n");
$smtp->datasend("Subject: " . $subject . "\n");
$smtp->datasend("\n");
$smtp->datasend($body . "\n");
$smtp->dataend();
$smtp->quit;
}

# Send away!
&send_mail('johnny@mywork.com', 'Server just blew up', 'Some more detail');


Sending email with attachments is a little trickier since we have to construct multi-part messages. The Net::SMTP::Multipart module provides a wrapper around Net::SMTP (but not Net::SMTP::SSL) to support attachments, but I don't like the syntax it requires and lack of MIME types guessing so I extracted the core logic from that module into the example below:

#!/usr/bin/perl -w

use Net::SMTP::SSL;
use MIME::Base64;
use File::Spec;
use LWP::MediaTypes;

sub send_mail_with_attachments {
my $to = shift(@_);
my $subject = shift(@_);
my $body = shift(@_);
my @attachments = @_;

my $from = 'johnny@gmail.com';
my $password = 'MySecretGmailPassword';

my $smtp;

if (not $smtp = Net::SMTP::SSL->new('smtp.gmail.com',
Port => 465,
Debug => 1)) {
die "Could not connect to server\n";
}

# Authenticate
$smtp->auth($from, $password)
|| die "Authentication failed!\n";

# Create arbitrary boundary text used to seperate
# different parts of the message
my ($bi, $bn, @bchrs);
my $boundry = "";
foreach $bn (48..57,65..90,97..122) {
$bchrs[$bi++] = chr($bn);
}
foreach $bn (0..20) {
$boundry .= $bchrs[rand($bi)];
}

# Send the header
$smtp->mail($from . "\n");
my @recepients = split(/,/, $to);
foreach my $recp (@recepients) {
$smtp->to($recp . "\n");
}
$smtp->data();
$smtp->datasend("From: " . $from . "\n");
$smtp->datasend("To: " . $to . "\n");
$smtp->datasend("Subject: " . $subject . "\n");
$smtp->datasend("MIME-Version: 1.0\n");
$smtp->datasend("Content-Type: multipart/mixed; BOUNDARY=\"$boundry\"\n");

# Send the body
$smtp->datasend("\n--$boundry\n");
$smtp->datasend("Content-Type: text/plain\n");
$smtp->datasend($body . "\n\n");

# Send attachments
foreach my $file (@attachments) {
unless (-f $file) {
die "Unable to find attachment file $file\n";
next;
}
my($bytesread, $buffer, $data, $total);
open(FH, "$file") || die "Failed to open $file\n";
binmode(FH);
while (($bytesread = sysread(FH, $buffer, 1024)) == 1024) {
$total += $bytesread;
$data .= $buffer;
}
if ($bytesread) {
$data .= $buffer;
$total += $bytesread;
}
close FH;

# Get the file name without its directory
my ($volume, $dir, $fileName) = File::Spec->splitpath($file);

# Try and guess the MIME type from the file extension so
# that the email client doesn't have to
my $contentType = guess_media_type($file);

if ($data) {
$smtp->datasend("--$boundry\n");
$smtp->datasend("Content-Type: $contentType; name=\"$fileName\"\n");
$smtp->datasend("Content-Transfer-Encoding: base64\n");
$smtp->datasend("Content-Disposition: attachment; =filename=\"$fileName\"\n\n");
$smtp->datasend(encode_base64($data));
$smtp->datasend("--$boundry\n");
}
}

# Quit
$smtp->datasend("\n--$boundry--\n"); # send boundary end message
$smtp->datasend("\n");
$smtp->dataend();
$smtp->quit;
}

# Send away!
&send_mail_with_attachments('johnny@mywork.com', 'Server just blew up', 'Some more detail', 'C:\logs\server.log', 'C:\logs\server-screenshot.jpg');


A couple of interesting things to note:
  • The Google SMTP server is available on port 465, which is the standard port for secure SMTP.
  • Google SMTP always uses your fully qualified gmail account name as the "from" (i.e. the account name you used to authenticate with the SMTP server), even if you specify a different from when constructing the message headers and body.

Saturday, October 28, 2006

Using Spring AOP to log swallowed exceptions

One example of bad programming style is to catch exceptions and then silently do nothing with - them, aka "swallowing" exceptions - which looks something like this:



...
preWork();

try {
doWork();
} catch (Exception e) {
// FIXME
// Dunno what to do with this exception...
}

postWork();
...


If this is your own code then you should fix it (and then severly punish yourself!). If it is someone elses code - for example a third party lib - and you have to use it then it can be very frustrating to have your code die silently; it makes code so much more difficult to debug.

(As a side note, exception swallowing is probably more common in Java because of a feature of the language - typed exceptions. But I've seen swallowed exception code in plently of other C++-like languages.)

I recently had such a problem when I was enhancing a Spring MVC application that used Tiles to lay out pages. Tiles often swallows exceptions. gggrrr.

What to do? It would take a fair amount of effort to fix exception handling in the Tiles code and also a fair amount of effort to step through the application code with a debugger.

Fortunately, Spring - the ultimate software hammer - provided the base infrastructure for a simple solution:
  • Use ThrowsAdvice AOP advice to intercept an exception thrown from the application code, log the exception at ERROR level, and let the exception flow continue.
Step 1: Create a ThrowsAdvice to that will intercept and log an exception


package robertmaldon.util;

import java.lang.reflect.Method;

import java.io.PrintWriter;
import java.io.StringWriter;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import org.springframework.aop.ThrowsAdvice;

public final class LogThrowsAdvice implements ThrowsAdvice {
private Log logger = LogFactory.getLog(this.getClass());

public void afterThrowing(Method method, Object[] args,
Object target, Exception ex) throws Throwable {
logger.error("Intercepted exception of type ["
+ ex.getClass().getName()
+ "] thrown by target class ["
+ target.getClass().getName()
+ "] and method [" + method.toString() + "]");

logger.error(getStackTrace(ex));

// Rethrow the exception in case something else downstream can
// handle it.
throw ex;
}

private String getStackTrace(Exception ex) {
try {
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw);
ex.printStackTrace(pw);

return sw.toString();
} catch (Exception e) {
return e.toString();
}
}
}


Step 2: Create the bean definition for the ThrowsAdvice and configure an autoproxy to wire the advice to your application code.


<!-- Bean definitions for your application code -->
<bean id="petStoreController1" .../>
<bean id="petStoreController2" .../>

<!-- Define the ThrowsAdvice bean -->
<bean id="logThrowsAdvice" class="robertmaldon.util.LogThrowsAdvice" />

<!-- Wire the advice to the application code -->
<bean class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
<property name="beanNames"><value>petStoreController*</value></property>
<property name="interceptorNames">
<list>
<value>logThrowsAdvice</value>
</list>
</property>
</bean>


Step 3: There is no step 3!

Very simple, very easy.

This approach is good for development, but may introduce too much overhead for production.

Tuesday, October 24, 2006

CSS changes in IE7

A list of CSS changes in IE7. The UI expert on our team was extremely happy to hear IE finally supports transparent PNGs!

You can test IE7 without installing it (i.e. without blowing away your current IE installation).

Thursday, October 19, 2006

Who says Wikipedia is inaccurate?

Wikipedia Celebrates 750 Years Of American Independence

Wikipedia, the online, reader-edited encyclopedia, honored the 750th anniversary of American independence on July 25 with a special featured section on its main page Tuesday...

The special anniversary tribute refutes many myths about the period and American history. According to the entry, the American Revolution was in fact instigated by Chuck Norris, who incinerated the Stamp Act by looking at it, then roundhouse-kicked the entire British army into the Atlantic Ocean. A group of Massachusetts Minutemaids then unleashed the zombie-generating T-Virus on London, crippling the British economy and severely limiting its naval capabilities...

Tuesday, October 17, 2006

ASP.NET providers now pluggable

A couple years ago I wanted to learn C#, so a good way I thought was to write a medium sized project in C#. For the project I decided to do a Tomcat/Jetty-like "servlet" engine, but using ASP.NET instead of JSPs.

The Microsoft company line at the time was "if you want to do ASP.NET, you have to buy/install/run on IIS". Well, after I did a bit of research that turned out to be only half true. The actual ASP.NET engine was part of the .NET core runtime. Although it wasn't documented how to pass HTTP requests through to the ASP.NET engine Microsoft kindly provided just such an example with the Cassini sample web server.

So far so good. However, when I looked into slightly more complex things like security and session replication I hit a dead end - all of these things were very closely tied to SQL Server, all of the interfaces that would have been useful to extend and make generic were marked "sealed". Bummer. I gave up that pet project because I didn't want to be tied to just SQL Server.

I was therefore plesently surprised to see an infoq article titled "Create Your Own ASP.NET Providers", which tells you how to implement user and session providers in ASP.NET 2.0. I'm very tempted to resurrect that old project and implement some of the new Jetty 6 features like continuations and large scale NIO in C#. (Jetty.net anyone?)

Friday, October 13, 2006

John Davies podcast on the current Investment Banking Technology Stack

In this podcast, taken from a keynote presentation given at TSSJS-Europe, C24 co-founder and CTO John Davies discusses the technology stack that is currently used by most of the investment banks. (Those of us who work in the industry have seen these trends, but it is good to get this type of info out in the open :))

Some of the highlights (in no particular order):

A typical stack
  • Excel (in his view is the only part of the Microsoft stack that is 'necessary' because it is the tool of choice by traders. Better not tell that to all of the people building rich .NET clients :))
  • Tibco/RV (as well as being high performance, it integrates well with Microsoft technologies)
  • MQSeries (aka MQ or WebSphereMQ), and often using JMS to interact with MQ rather than the native MQ API
  • A J2EE or more "light-weight" Java application server stack
Application Servers

WebLogic/WebSphere/JBoss are still heavily used but are on the way out. New projects must justify the use of an application server. If they can't justify an application server then they need to use a more light-weight stack such as Tomcat/Spring/Hibernate.

No more EJBs!

Entity Beans in particular have been "banned for years".

Open Source

It's not just the low cost that banks find attractive, its more that the source code is always available of needed - no worries about the company that makes the product going broke or stopping development/innovation/support of the product.

AMQ - Messaging will become a commodity

AMQ, a wire-level protocol which a number of vendors are cooperating on, is set to commoditize messaging - and break the back of Tibco and IBM who charge lots of money for RV and MQ licenses! JP are already running two implementations of AMQ internally. RedHat will ship AMQ implementations with its operating system packages.

Databases are dead - Long live the database!

Your traditional relational database - Oracle, Sybase, etc - are on the way out. Memory is so cheap that finanacial systems are now loading a database into memory at the start of the day and dumping the modified data back out at the end of the day. Mr Davies was very careful to point out that these were in-memory databases and not caches, but didn't go into details about how integrity was preserved if the server power failed.

Wednesday, October 11, 2006

How many toolbars can you add to IE7?

Funny idea. Find out why someone would want to do this.

Web 2.0 Attacks

A good article on the ways to attack a Web 2.0 application, thanks to the growing number of technologies used in "rich internet applications" (html, flash, applets, javascript, etc):

  1. Cross-site scripting in AJAX. A pre-AJAX browser exploit, but AJAX makes it easier to hide such an attack.
  2. XML poisoning. A denial-of-service attack that tries to exploit weaknesses in server-side XML parsers. Do something nasty like apply a recursive payload to an XML block.
  3. Malicious AJAX execution. Some javascript that sits in the background, captures info from a server (e.g. the user has signed into their on-line bank account) and sends that info to a identity theft server. All done silently.
  4. RSS / Atom feed injection. RSS is becoming a popular distribution mechanism. Since most people view feeds in their browser it becomes very easy to slip some malicious javascript into a feed.
  5. WSDL scanning and enumeration. Looking at the exposed method interfaces of a web service a malicious person may be able to guess at ways to atack the interface. e.g. if you see a "debug" or "override" parameter you might be tempted to see what it does.
  6. Client side validation in AJAX. For those people who define validation on the client side but are too lazy to do the equivalent validation on the server side, don't be surprised if someone breaks your application by making a malicious call directly to the server.
  7. Web services routing issues. WS-Routing allows SOAP messages to travel in specific sequence from various different nodes on the Internet. Often encrypted messages traverse these nodes. A compromise of any of the intermediate nodes results in possible access to the SOAP messages traveling between two end points.
  8. Parameter manipulation with SOAP. Try manipulating the values of the SOAP messages to do things like SQL-injection attacks.
  9. XPATH injection in SOAP messages. ditto to the point above.
  10. RIA thick client binary manipulation. An attacker can reverse engineer the binary file (e.g. flash .swf files) and decompile the code.
Update: DLG asked what are some guidelines to protecting against such attacks? Some of my thoughts are:

  • AJAX calls only work cross-site if using iframes. As with javascript the browsers should pop us a security alert asking if cross-site AJAX calls should be allowed on the page.
  • XML Poisoning / Parameter manipulation / XPATH injection. You should unit test a number of such attacks to see what happens. You may need to work with your vendor to fix any issues in the XML stack they provide.
  • WSDL scanning. A code review should find any methods that should not be exposed to clients. What could make this tricker, however, is that many of the SOAP standards now use a document-centric approach, where the WSDL is a thin wrapper around an XML "blob" payload.

Tuesday, October 10, 2006

Chin up Rover!

It will probably never be a big selling item, but I liked the cute Doggy Car Window Head/Chin Rest I saw in the Newport stores while doing the tourist thing on the weekend. It's a ruff life. Woof!

Wednesday, October 04, 2006

Clippy gets ported to Linux




Clippy the Microsoft Office Assistant has inspired a Linux clone called Vigor. Need help with the evil six editor (VI)? No problem!

Fixing those AJAX Javascript memory leaks

One thing I've noticed with a number of the recent wave of AJAX applications is that they leak memory. For example, editing googlepages grinds my 512MB computer to a halt in 20 minutes of so.

Jack Slocum has a few tips on avoiding javascript leaks, including using the LeakMonitor Firefox extension, and points out leaks in common AJAX libraries like prototype.

Tuesday, October 03, 2006

Sarbanes-Oxley squashes IPOs?

Interesting story I saw on Power Lunch while working out at the gym. IPOs are down in Q3 in the US, but steady or up in overseas markets like Europe. The reason? Many business people are finding Sarbanes-Oxley requirements too onerous for listing on the Nasdaq and are preferring to take a company private or list on a UK exchange instead.

The future of Web 3.0: Thick clients?

The New York Times has launched Times Reader, a .NET thick client that provides a more faithful rendering of Times stories than can be achieved with a browser (so they claim), such as:

  • the same font and column structure you see in the printed paper; and
  • no scrolling necessary -- just use the arrow keys on your keyboard to turn the page
Can you do this with a browser? Depends on the browser :) The standard bodies have done there job with the CSS 3 Multi-Column Layout spec. Some clever people have created javascript hacks that display multi-coulmns for pure text. And the Mozilla dudes have preliminary support for multi-column content in the current production version of Firefox 1.5.

So which browser doesn't support multi-column layouts in a meaningful way? You guesses it, Internet Explorer. And font support tends to be operating system related. So why don't the NY Times push Microsoft to make Internet Explorer standards compliant instead of building thick clients that will only run on Windows?

(Lack of CSS support, including the upcoming IE 7, is still the biggest gripe I hear when talking to web design professionals.)

On a related note, what will our interface to the internet look like post-Web 2.0? Will it be thick clients instead of a browser? I guess I'm used to a browser, but the thought of having to learn a different UI to navigate each "web site" I access is frightening. Some of the Finetix UI experts I've discussed this with can only see AJAX + CSS in the near and medium term.