Boisseau.co.uk Blog

I’ve seen some people recently struggling with the ancient application, ZTerm, on Mac OS X. Generally people only need it to use with a USB/Serial adapter, when configuring devices via their console port (for example Cisco routers, managed switches, or the ‘hard-line’ into a server).

Not many people know that most of the ZTerm functionality is available to you as part of the Unix underpinnings of Mac OS X. Some time ago, I wrote a ‘.command’ file (just a single liner) to completely replace my need for ZTerm. I thought I would share the love, and publish this little helper tool:

In my case, the contents of the ‘.command’ file is:

cu -l /dev/tty.usbserial -s 9600

This uses the ‘cu’ command, which stands for ‘call up’ - allows you to literally call up (as-in phone) another system. This was used in the old-days for dialing into other machines to get access to the command line remotely. Obviously this involves using a ‘device’ to connect to the other machine. In most cases, this would have been a modem device that is presented as a ‘tty’ in the ‘/dev’ directory. In our case, we’re just talking to a serial device.

Because I am invariably talking to a Cisco or Netgear device, I set my baud rate to 9600 and don’t need any other setting. However, if you lookup the man page on ‘cu’ you can see all the various options for devices that need specific connection requirements such as parity, XON/OFF etc.

If you need to find out what your device is presented as in ‘/dev’, run the following command to see all the available tty devices:

ls -l /dev | grep 'tty\.'

To create a ‘.command’ file, just copy/paste the command into a text editor and save it as ’something.command’. This should make it a double-click-able icon. If it doesn’t want to open in Terminal on it’s own, remember you need to set the ‘execute’ permissions for the file:

chmod +x /path/to/something.command

I hope that’s a time-saver for others as it has been for me.

The built in Server Admin tools often aren’t enough to setup some email behaviour, that OS X Server is capable of (as part of Postfix). Postfix has a huge amount of configuration, however there’s a few things that I seem to be doing on every OS X Server installation, so I figured I’d write them down for my benefit as well as yours.

Making a no-reply blackhole address

Using your favourite text editor (in my case nano), open the following file:

sudo nano /etc/aliases

Scroll down to where it says:

# Put your local aliases here.

Be careful not to write into the bit below, as Server Admin will overwrite this to create internal mail-groups. Add the following line to create a noreply@ address on your default domain name.

noreply: /dev/null

This basically sends any mail that arrives at noreply@yourdomain.com to the /dev/null space, which is basically a pointer to the sacred unix black-hole. To apply this change, exit from the text editor, and run the command:

sudo newaliases

Piping mail to a script

If you’re a scripting/programming nut, as I am, you will appreciate the ability to directly pipe email coming into your server straight into stdin of a script. It is possible to pipe it into a old-school shell/perl script (or anything that accepts data from STDIN), however I normally like to pipe it straight into my comfort zone of PHP.

Let’s say you want to make an email address of autoreply@yourdomain.com which is handled by a PHP script. As above, use your text editor to open up /etc/aliases. Under where it says:

# Put your local aliases here.

Add the line:

autoreply: "|/path/to/scripts/pipe.php"

This pipes the RAW source of the mail to STDIN of the script specified. For a quick tip on how to receive the data from STDIN in your PHP script, here’s some code:

#!/usr/local/php5/bin/php -q
// Above line tells the shell where PHP lives on your server.
// Make sure the path is correct.  Type 'whereis php' into Terminal.

// Grab the data from STDIN
$data = file_get_contents('php://stdin');

// Now do something with the $data variable

If you want to extract the different parts of the email from the raw source, I suggest you download this great MIME Parser PHP Class.

Handling ‘aliases’ on virtual mail domains

If you want to setup any of the aliases you have created above on one of your ‘virtual hosting’ domain names, you need to do one or two extra things:

Using your text editor, you need to edit the file:

sudo nano /etc/postfix/virtual

By default the file just contains a commented-out copy of the manpage. So you can type this anywhere, either above or below the existing content:

noreply@virtualdomain.com noreply

This basically makes the existing noreply@ address (as above) available under the virtual domain. I have kept the names the same to keep things simple, however there’s nothing to stop you forwarding a different name on the virtual domain to the address you have setup on your default domain.

Once you have made the relevant changes in the virtual file, you need to tell postfix’s main config where to look for it:

sudo nano /etc/postfix/main.cf

Add the following line somewhere in the file. I put it at the bottom under the rest of the config Server Admin added:

virtual_alias_maps = hash:/etc/postfix/virtual

Finally, to apply the changes run the following commands in Terminal:

sudo postfix reload
sudo postmap /etc/postfix/virtual

You can now try to send emails to these new addresses. If you get a bounce message, you’ve done something wrong!!

As you may know from previous posts one of our larger clients is a School here in Edinburgh. One of the most important hurdles for maintaining a network for a school is to ensure the kids have safe access to the internet. I’m a true believer that schools should offer the internet to all pupils without massive amounts of supervision or over zealous filtering. I have known some schools to enforce rules whereby everyone must be supervised on the internet, while other schools use the approach of white-listing only certain educational sites that staff approve.

While both of these methods work in protecting the children from the evils of the internet, they also completely prohibit kids from understanding the true ‘open and free’ ethos of the world wide web. With the net playing a bigger part in everyone’s lives, and more people embracing the Web 2.0 ideals of the likes of blogs, Facebook and Twitter - I believe this idea of free speech and free browsing is more important than ever.

The other aspect that a lot of people forget when talking about child protection on the net is the maintenance headache and cost implication. When schools decide to spend money on new IT infrastructure, web-content filtering is often an afterthought, and not properly budgeted for. In our case, we did have the budget for our implementation, however I didn’t want to spend thousands a year on a commercial solution that you become tied to, when that money could be much better spend on enriching the IT education, rather than protecting it.

So what’s the answer? The first step to doing real protection, on a budget, is having a proxy server that does on-the-fly content filtering. Traditionally proxy servers were just used to speed up web browsing by caching pages, back in the days when the fastest connection institutions could afford was that of ISDN or slow leased lines. These days, with ADSL or Metro Ethernet services being affordable and accessible by most schools (in the UK anyway), proxying isn’t really necessary - in fact it can be a nuisance. These days it’s not unreasonable to expect sites to update on a minute-by-minute basis, whereby 10 years ago, most sites wouldn’t update any more than once a day or even once a week.

Having said that, proxy servers had developed a second purpose; web-filtering. There are some great tools out there that were built to run on old-school proxy servers to do filtering of all sorts. Even if it’s just to block certain sites in an office where employees are wasting too much company time on Facebook, Bebo any Myspace, proxy servers are cropping up all over the place once again.

Hacking a Proxy

There’s a fundamental problem with proxy servers though. Generally, you can’t just plug-and-play to get a bullet-proof system. Usually, you have to setup your workstations to point to the proxy. While this isn’t a problem in the first instance, normally there’s simple ways round this setting. Remember, the people using your network do not want a proxy, and I’ve seen people go to extensive measures to bypass the proxy. The reason I know all the tricks is because I used to try them all - I had tried and tested methods for both the high-schools I went to! Now the shoe is on the other foot, I hope to be doing a better job than my schools did with me!! So here we go; the hacker’s tools of the trade exposed (from basic, up to hardcore)…

Your users could just try turning off the proxy setting, if your computers aren’t setup to stop them doing this. Assuming you’ve got your user privileges setup to prevent this, users could just go download another web-browser such as Firefox, Camino or Chrome which don’t rely on the system proxy settings. Assuming you got around this problem by blocking those sites with your proxy, they could download a browser at home and bring it in on USB flash drive or CD (or even email it in). So lets just say you’re really smart and have an outbound firewall on your network. You decide to block port 80 and 443 so people can’t get out on the web at all without the proxy server. Your users then decide to use an unrestricted external proxy server (possibly one they’ve setup themselves) that runs on a port you haven’t blocked. You then figure this out, so you block the port they are using, so they use a different one. You eventually decide to block all ports except the ones you need; your users then realise that you can’t do without SMTP or some other external port and start using their external proxy on this very port. You then find out that the best thing to do is completely restrict your users LAN from the outside world, so they can’t connect directly onto the internet without your proxy. Your users then decide to embrace your proxy; and start browsing through one of the many ‘tunnel’ websites. You block these on your proxy.

While you spend your life playing cat and mouse with your users, you have wasted days in man-hours and you have ended up with a very restrictive network. I know - this is a very unlikely case and may only happen on VERY large campuses or corporate networks, but the risks are still there.

Traditional Proxy Setup

Unless you are prepared to spend a lot of time on these issues, and end up with a network that is more restrictive than the Labour government, you will probably want to do everything you can to ease the path from day one. If you’re maintaining a large number of computers, especially if they’re all differing OSs and specs, the biggest headache comes with the first step; configuring all your computers to use the proxy in the first place. While you can use a PAC file or login scripting to automate this task, it’s always a nightmare. For example when you get foreign computers on-site, you must ask your guests to configure thier proxy setting manually, or when you have mobile devices such as PDAs and WiFi enabled phones, you must setup complicated location profiles to enable the proxy on-site and disable the proxy when off-site.

So, here’s the topology most people will be used to.

So each client access-point is setup manually to use your proxy server, instead of send internet traffic straight out over the router. As you can see, normally one would only bother doing this for fixed computers. Often iPhones, PDAs and even servers get left out from the setup, leaving security holes for you to fill in other ways. This is a bit of a hassle to keep maintained, especially on large diverse networks.

Transparent Proxy Setup

The easiest way to get round this problem of client setup is to make your proxy server ‘transparent’ on the network.

Note: Some people don’t like the term ‘transparent’, because in fact the proxy isn’t completely transparent. The web-servers at the remote end see the request with some tell-tale signs that it has gone through a proxy. First of all, the proxy frequently adds an HTTP Header that would say something like: Via: proxy.yourcompany.com. While you can turn this off, the request may also be cached at the proxy or if your proxy server appears to the outside world as a different IP that the PC would have normally these would be other signs your request has been routed through a proxy. But who cares, right? You could go to further lengths with your router to do IP spoofing and all kinds of nasty deep-packet monkey business, but I have yet to find a good reason to do so.

So lets have a look at the typical topolgy of a network with a transparent proxy:

As you can see here, all devices (no matter how big or small) access the internet as usual, via the router. The router then does some magic to re-route the packet to the proxy instead of straight out onto the net. The proxy then makes the request on behalf of the browser, checks it’s content and spits out a response (either the page, rendered as usual, or an error message saying the request was blocked for whatever reason). The response is sent back to the browser without them knowing it was proxied at all (unless they got an error from the proxy). The beauty of this system is that there’s no client configuration at all. Any old computer or mobile device can plug into the network (via any means; ethernet or wifi) and browse the web straight away. You don’t need to worry that foreign machines are vulnerable, or unable to browse because of firewall restrictions.

So, we’re in agreement - this is a better solution, right? Good. I’ll go on and tell you how to achieve it!!

Setting it up

Before you start, you’ll need to get a few things:

• A capable Linux/Mac OS X machine to act as proxy server

You could probably do this with a Windows server, but I wouldn’t like to try. My setup when writing this guide is a Dell PowerEdge SC1430 with Fedora 8 installed. My 2GB of RAM is plenty and the 2GHz Xeon will blast out much more power than I need - but this is a cheap server, and we’re not in the business of running these sorts of applications on old P2 boxes like some people will tell you is possible. While I have said Mac OS X is possible, I’m not going into it here. I think a nice OS X box is a waste of hardware.

• A Cisco Router

One of the things we are using here (which is not tricky - so don’t get scared off) is a Cisco protocol called WCCP (Web Cache Communication Protocol), so we need to have a Cisco router to use it. I have attempted to do this same thing with other routers using static protocol-specific routing rules. I have even tried the same thing on high-end Cisco kit. Having tried it all, I can tell you now that nothing works as well as WCCP - and it’s not difficult. Also, Cisco routers are fantastic bits of kit. These days, any installation where we’re doing anything more than a handful of computers accessing the net, I will always insist the Client uses Cisco kit. They’re bullet proof and they “just work”. I’ve not done a great deal with their switches or enterprise routers, but the Cisco 800/1700/1800 series are absolutely fantastic. Note: you will need to check that the IOS version you have on your router supports WCCP. If your router doesn’t already have it, the chances are you can get hold of an IOS image that does. I think only the bottom end 800 series don’t support WCCP.

• Squid, Dansguardian and Webmin

These are the nice free open-source tools you will need. Once you have your linux Distro installed and working, go and install these.

This is just a configuration guide (not an install guide). So the things I’m missing out of this (because there are many other guides online - reacquaint yourself with your good frind Mr Google) are as follows:

• How to install Linux on your PC
• How to compile and install the software on your Linux box
• How to setup a Cisco router

Before I send you off to install all this, I will give you a bit of an intro to the software side of things to give you an understanding of how it works together.

Squid

Squid is the old-school proxy software that most people use. By default it takes connections on port 3128, makes the request to the intended destination, caches the result in memory, and spits it out to the original requester. Each cached page will have a set time-to-live, so while it’s still alive, any subsequent requests (by anyone else using the proxy) will return the cached page instead of going out onto the web to get it.

Squid is configured mainly with the squid.conf file, and monitored by various log files.

Dansguardian

Dansguardian is a great piece of content filtering software written by Daniel Barron. It is designed to sit on-top of a web-cache such as Squid, and act as a content filter. Many people use this for the same purpose as we are, however others use it simply to filter advert urls, so their web-browsing is ad-free! Dansguardian is configured with a number of files, one main configuration file and a bunch of files that contain black-lists, white-lists and weighted phrase lists. The weighted phrase lists make DG a very powerful filter, because they let you filter sites based on the severity of it’s content.

For example; words such as those describing genitalia may have a negative weight of 20, while other words such as education and the more medical phrases to describe these things may have a positive weight of 50 each. So even if a page mentioned these things very frequently, so long as they are counter-balanced with words and phrases that would indicate it is a sex-education context, the page would be delivered. If the page was full of ‘rude’ words, however had no redeeming content, the page would be blocked.

Webmin

Webmin is a great (self sufficient) tool that allows you to configure all manor of services running on Linux and Mac OS X machines. Often it is known as ‘the missing GUI’. In our case it is very useful as it offers a web-based GUI for both Squid and Dansguardian, as well as helping out with a bunch of general Linux admin tasks.

Your Network

I’m going to assume you have a basic NAT network, with DHCP and your Proxy is already configured on a static IP. You should have something that looks like the ‘Transparent’ topology diagram above. Let’s make some assumptions for you to use as a reference:

LAN Subnet: 192.168.30.0/24
Cisco Router: 192.168.30.1
Linux Proxy: 192.168.30.5

I’ll assume you’re on an ADSL WIC on your Cisco, so your WAN interface is called Dialer0. I don’t care what you’re public IP is, or whether you have NAT mappings going elsewhere. If you have a firewall that’s fine, so long as there’s no rules that could conflict with our setup.. If in doubt - turn it off for testing, and build up the rules from zero if you have problems when you turn it back on.

On your Linux box, we’ll assume you’re using the main ethernet interface is eth0, which has the primary IP as above (192.168.30.5).

Configuring Your Proxy

Assuming you have setup your Linux box, installed Squid and Dansguardian; you should now be able to manually point your browser’s proxy setting towards your proxy. If you haven’t done this before, for Internet Explorer you’ll find it under Windows “Internet Options” in the Tools menu or in Control Panels in XP. For Safari on Mac you’ll find it in System Preferences in the Network pref-pane under ‘Proxies’ on the network interface you’re using to connect to the LAN. For other browsers, the setting is found in the program’s own options/settings window. If you’ve done it correctly, your Squid install will be running on the default port, 3128, and DG will be configured on something like port 8080. It’s useful to know that you can point your proxy settings at either of these ports. Obviously if you use 3128, you won’t be filtered by DG.

So, with your browser configured to use your proxy IP on port 8080, you can now try browsing to some sites. You can check the DG and Squid logs to see your requests traversing your proxy. Here’s a useful command to keep running in a terminal window on the Linux box, or via SSH on your own machine:


tail -f /path/to/your/logfile.log


This will keep the file open and print new log entries as they happen.

Obviously, the acid-test for making sure your proxy is working correctly is to go to Google and type in the worst obscenities you can think of! It’s a really fun game!!

If you have got to this point without any problems, you will now want to make your working proxy transparent to the browser. The first thing you need to do is change a couple of settings in squid.conf. Use your favourite text editor (using sudo) to change/add the following lines:


wccp_router 192.168.30.1
http_port 3128 transparent


You might be able to get away with doing most of this in Webmin, however you will need to do the http_port line in the terminal, because Webmin doesn’t support the transparent keyword.

Once you have made these changes, you need to change the port that Dansguardian runs on, to port 80. We do this so we don’t have to re-route packets to different port numbers. If you have a dedicated box, this shouldn’t be a problem. This can be done from Webmin under Servers -> Dansguardian. Once you have made these changes, remember to tell Webmin to restart Squid and DG.

Configuring Linux

Now you’re running your proxy on port 80, and you’ve told Squid you are running transparent, as well as using WCCP to route packets from your router IP.

What we need to do now is setup Linux to talk to your Cisco router. WCCP uses a GRE (Generic Routing Encapsulation) tunnel to encapsulate packets and keep them intact in transport in the redirection process. In most Linux Distro’s this is done via the ip_gre module. You can check it is installed by running:


modprobe ip_gre


As far as I know, no result from this command signifies that the module is installed. If it isn’t installed, I suggest you do a google search on how to compile and install ip_gre. It should be pretty easy, but on some distros a kernel recompile may be needed. Certainly my out of the box Fedora 8 install had it already.

Once you’ve ascertained that ip_gre is installed, you need to create your GRE tunnel. We will tell Linux that we want to tunnel from our LAN IP (192.168.30.5) to our Cisco (192.158.30.1). This will be created as a standard POSIX network interface called ‘wccp0′:


ip tunnel add wccp0 mode gre remote 192.168.30.1 local 192.168.30.5 dev eth0
ip addr add 192.168.30.5/32 dev wccp0
ip link set wccp0 up


Once you have set this up, a call to ifconfig should show your new wccp0 interface amongst all your usual ones (eth0, lo0 etc.).

The final part of configuration on the proxy before we setup the Cisco, is to direct the traffic coming over the GRE tunnel into port 80, where our proxy is waiting. To do this, we will use the iptables command:


iptables -t nat -A PREROUTING -i wccp0 -j REDIRECT -p tcp --to-port 80


Configuring the Router

At this point your proxy should be completely ready to go. The final step is to get the router to redirect packets destined for the www onto your proxy. I will assume at this point that the router is functional, and you have a working NAT setup. You should telnet/console onto your router and type the following:


conf t
ip wccp version 1
ip wccp web-cache


What we have done here is enabled wccp v1 (v2 does exist, but isn’t compatible with your average copy of Squid) and we are giving the instance of wccp a nice name, ‘web-cache’.

You then need to tell the WAN interface (we’re assuming it’s ‘Dialer0′) to redirect traffic:


int Dialer0
ip wccp web-cache redirect out


You may notice that we haven’t specified any IP address to redirect onto. This is because the proxy is establishing the GRE tunnel from it’s end. So the Cisco should already have an outlet, without us having to configure it here. There is a hidden benefit to this: we can have multiple proxy servers calling into the router and offering their GRE tunnel. This would give you a load-balancing platform, as well as some redundancy. Because the router isn’t manually configured to forward to certain places, it will only pass traffic to the servers that are available.

Once you have made the changes to your router (remember to come out of ‘conf t’ mode, and issue a ‘write’ command to save the config), you should have everything in place. You can now remove all proxy configuration from your client machines, and try browsing. Once some traffic has passed the interface, the Cisco should start redirecting it shortly after. You can check this by issuing the following command to the Cisco command line:


sh ip wccp


Once some traffic has passed the interface, you should get an output something like:


Global WCCP information:
Router information:
Router Identifier: 192.168.30.1
Protocol Version: 1.0

Service Identifier: web-cache
Number of Service Group Clients: 1
Number of Service Group Routers: 1
Total Packets s/w Redirected: 3941
Process: 0
Fast: 0
CEF: 3941
Redirect access-list: -none-
Total Packets Denied Redirect: 0
Total Packets Unassigned: 0
Group access-list: -none-
Total Messages Denied to Group: 0
Total Authentication failures: 0

Finishing up

So there you have it. A transparent proxy on a budget. It should save you days in client configuration, and make life a lot easier with your workstation deployment. There are one or two things to watch out for:

1. My box has been known to loose the GRE tunnel on restart. I don’t know why this was happening, however if it does the ‘ip tunnel add’ commands and the ‘iptables’ commands could be configured in a shell script to run at startup.

2. This does not prevent people from bypasssing your proxy. You will still need to run an outbound firewall (which the Cisco will do nicely, lookup Cisco Extended Firewall), and load in a list of all the ‘mirror’ and ‘tunnel’ sites available. You will also need to update of this on a regular basis, as new ones crop up frequently. There are a bunch of sites around that simply mirror a particular site such as facebook or bebo, you need to watch out for these also.

3. The default ‘block’ page that DG throws is horrible looking. Having said that, you can easily edit the template html files for your particular language. Any images you want to show must be sitting on an external server, as DG will only serve a single request - it’s not a web server.

4. Make sure your proxy is not accessible from the WAN. Many bots and viruses out there will trawl the web for open proxy servers. If a publicly accessible proxy is found, your internet connection will be raped by other viruses, perverts and criminals attempting to browse the web anonymously. It’s also worth noting that if you have a web-server on your network that IS publicly accessible, be careful with the interfaces you enable WCCP on your router. If you mistakenly enable WCCP on inbound connections to your webserver, it is the same as opening your proxy port to the WAN. Your router doesn’t know not to re-route inbound traffic to your web-server via the proxy. If it does, the proxy can still be exploited.

Hopefully this will be of use to some people. I looked for days for good resources on this subject, and couldn’t really find any. The one page I did find very helpful with the Linux bits was the Squid Wiki page on the subject. Thanks to whoever wrote that one!!

Recently, I’ve been doing quite a lot in PHP with FileMaker - building web front-ends to our various systems in the office. When FileMaker 9 came out, I had been playing with the new built-in PHP classes. They’re really nice OO classes, however with all things OO, the processing overhead was adding to an already sluggish back-end.

Having said that, it wasn’t the reason I reverted to FX.php. Some months ago, building my first big system in the new PHP classes, I had a problem with related data. One thing that was really annoying me was the fact that related data always needed another call to the DB. Even trivial related fields that I would have previously been able to pull into the relevant web layout in FM didn’t seem to be showing up. I put this down to the developers of the PHP classes being to purist about the OO layout of the calls.

These shortfalls didn’t sit right with me, because I know that the PHP classes were basically built on the same XML grammar that FX uses, so it doesn’t have control of the data that is pulled down from the database. I asked myself, why would the developers restrict the data delivered to the API deliberately, for the sake of a cleaner OO structure? It seemed odd that they would purposely make the API less efficient.

Anyway, recently I was working on the same project. We had gone to the effort of re-writing the DB calls using the FX.php API instead. This time round I went to do the same old relationship I was having difficulty with before, and was shocked to be having the same issue as before. It looked like the related fields I had pulled into my web layout weren’t showing up in the XML output. I looked around a couple of other apps, which use the same FX version and are served by the same server. Strangely enough the other apps, which had the same kind of external files in the relationships were working fine.

So this one was turning out to be a real head-scratcher. I did some more rummaging around in the database, checking the FMXML extended privilege set was setup the same as my working DB, checking the file permissions on the server were correct, generally checking everything was the same.

Finally, I found a discrepancy. I looked into the “Manage External Data Sources” window and immediately saw something odd. On my working example, all the related files were refereced as just file:Database, whereas on my problem-database, the related DB I was trying to access was referenced as fmnet://fmp.mycompany.com/Database. I figured this might be a problem (although it shouldn’t be), so I changed it to just use file: and hey presto.. The relationships now worked!!

Strangely, when this was fixed I realised something else got fixed as a result. Any fields that had calculated or looked up auto-enter values that referenced data from this external file were being entered when new records were created via web publishing (as you would expect), however they weren’t working before. I had put it down to just another strange web-publishing shortcoming, however it turned out to be problem with external data sources.

So, the moral of the story; when using external sources in FileMaker solutions with web-publishing, remember to check the files are all referenced with just file:YourDatabase. Anything else is likely to cause months of hassle!!

Hopefully posting this will help other with the same issue.

I finally got round to dusting off the old blog!! I decided to ditch the old PHP site, in favour of running the whole thing from WordPress. Apologies if there’s something that was here before that now isn’t.

From now on, this will mostly be filled with technical tidbits. I’m hoping to use this as a place to write down things I might otherwise forget, in the hope that other people might benefit from it also.

So, if you’re into programming, networking, macs, server administration (including dastardly Windows!!) - bookmark this place as it will soon be filling up with tricks-of-the trade!